Translating Motion to Notation: Hand Labanotation for Intuitive and Comprehensive Hand Movement Documentation

Authors: ling  li
wenrui  yang
junliang  xing
xinchun  yu
xiao-ping  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-26 14:05:15

Abstract: Symbols play a pivotal role in the documentation and dissemination of art. For instance, we use musical scores and dance notation to document musical compositions and choreographic movements. Existing hand representations do not fit well with hand movement documentation since (1) data-oriented representations, e.g., coordinates of hand keypoints, are not intuitive and vulnerable to noise, and (2) the sign language, another widely adopted representation for hand movements, focuses solely on semantic interaction rather than action encoding. To balance intuitiveness and precision, we propose a novel notation system, named Hand Labanotation (HL), for hand movement documentation. We first introduce a new HL dataset comprising $4$M annotated images. Thereon, we propose a novel multi-view transformer architecture for automatically translating hand movements to HL. Extensive experiments demonstrate the promising capacity of our method for representing hand movements. This makes our method a general tool for hand movement documentation, driving various downstream applications like using HL to control robotic hands.

Main paper: 
 introduction: 
 In music, people symbolize frequencies into various musical notes, thereby creating the musical staff [4] for the documentation and dissemination of music. In the domain of dance, dance movements are symbolized, resulting in the Human Labanotation for recording dance movements [9, 16, 31 ]. An intuitive, accurate, and comprehensive method should document the content practically. As shown in Figure 1 , symbols replace data-based methods for documenting music and dance movements.Hand movements are a crucial component of daily human communication and human-computer interaction [23] , making their documentation and dissemination equally important. However, existing methods do not adequately fulfill the requirements for documenting hand movements. Data-based recording methods such as Skeleton [8, 28, 34] , Mesh [18, 19] , and MANO [2, 26] , though capable of capturing hand movements with relative precision, are deficient in intuitiveness. Sign Language (SL), as a set of symbols for gestural communication, is likewise unsuitable for recording hand movements. SL as a language focuses more on conveying semantic information than the movements themselves, limiting its ability to cover the full spectrum of hand movements. Additionally, the execution of SL sometimes necessitates the involvement of upper limb movements and facial expressions. Therefore, the sign language notation is not suitable for documenting hand movements.Inspired by human Labanotation, this paper introduces a novel methodology for documenting hand movements, termed Hand Labanotation. It employs simple symbols to represent the motion state of each part of the hand, thereby documenting hand movements. Hand movements revolve around the joints, and based on the hand's physiological structure, we have defined 20 regional vectors based on the hand skeleton model to represent the spatial state of each part of the hand. As illustrated in Figure 2 , our framework designs the spatial division method by the activity characteristics of the fingers, resulting in a total of 26 basic Hand Labanotation symbols. To automate the conversion of hand movements into Hand Labanotation scores, we construct a large-scale multiview Hand Labanotation dataset (HLD) with over 4 million annotated hand movement images. We also propose a learning model featuring a novel multiview transformer (MHLFormer) architecture for the automated translation of images into Labanotation scores. To evaluate MHLFormer's performance on HLD, we introduced three evaluation metrics. Furthermore, we propose the Labanotation Hierarchical (LH) Loss Function to facilitate precise model translation. Extensive experimental results validate the accuracy of our method for documenting hand movements. We can also use it as an intermediary representation to control robotic hands.In summary, our contributions are as follows:• We introduce Hand Labanotation, a novel symbol system for hand movements, facilitating symbolic documentation and analysis, streamlining hand movement analysis, and applying it to robotic hand control. • Our creation of the HLD dataset, derived from annotated Interhand2.6M [21] and Freihand [36] data, represents the first extensive multi-view dataset for Hand Labanotation, marking a leap in hand movement documentation. • We propose and train the MHLFormer model using the innovative LH loss function, achieving outstanding performance in translating hand movements into HL scores. 
related work: 
 Existing Representation Methods for Hand Movement Documentation. We can broadly categorize feasible methods into three classes.(1) Using video to record movements. While video recording offers the advantage of being intuitive, it fails to address the issue of occlusions effectively [8, 24] . Additionally, the requirement for substantial storage space is another drawback of video recording [25, 27] . (2) Utilizing data-based hand representation methods for documentation. In computer vision research, 3D hand movement documentation relies on methods such as Skeleton [34, 35] , Mesh [18, 19] , and MANO [2, 26] . Although these methods can record hand movements with relative accuracy, they severely lack intuitiveness. It is challenging for observers to replicate hand movements merely based on coordinates data. Moreover, capturing accurate hand pose data is also difficult. (3) Employing symbolic methods for documentation. Symbolic recording methods also offer the advantage of being intuitive and, thus, easier to replicate. Existing Sign Language methods [17] have developed their unique system of symbols. However, since Sign Language focuses on semantic information, we cannot document many non-semantic hand movements using Sign Language symbols. Therefore, it is essential to design an intuitive and reliable symbolic system. Labanotation for Human Body. Human Labanotation has a long history. Developed in the early 20th, Labanotation is a symbolic method for recording body movement [9] . With the continuous development of Labanotation, it eventually evolved into a system composed of four basic elements: body, space, time, and dynamics [12] . The body represents the body's moving parts, such as the 'left leg', 'right hand', etc., typically comprising 11 body parts. Space represents the methodology of characterizing movements through direction, level, distance, and degree. Time indicates the duration of each movement, while dynamics denote the emotional components contained within the movement. In the Labanotation score, we describe movements using two dimensions: body and time [15, 31] . The vertical axis represents the temporal progression of the action sequence, with time flowing from the bottom to the top. Each column on the horizontal axis represents a part of the body. The shape and color of symbols represent the spatial position of each body part [9] . With the development of Human Labanotation, the differentiation of human body regions by various Labanotation symbols has become increasingly refined [16, 31] . This refinement provides a viable method for documenting hand movements.Motion-to-Labanotation Translation. The goal of Labanotation translation is to be rapid and accurate. Initially, experts manually recorded Labanotation while observing dance [7, 9] . To streamline this process, computer-assisted tools like Genlaban [6] , Laban Writer [3, 30] , and Led&Lintel [11, 13] were developed, simplifying symbol creation and speeding up notation. While aiding in Labanotation symbol editing and scoring, these tools demand substantial Labanotation expertise, posing barriers for many users [15, 31] . Recent advances in computer science have led to automated Labanotation translation from 3D motion data, increasing efficiency [31] . This method relies on the accuracy of 3D pose estimation. However, during the pose estimation process, the accuracy of the annotations may be affected due to frequent occlusions. The subsequent manual proofreading workload remains significant. Researchers have developed end-to-end methods and datasets such as Laban16 [15] and Laban48 [31] in response. However, these datasets only cover basic upper limb movements. Lower limb and hand movements tend to be more complex. There is still a lack of existing datasets. Hand Modeling. The skeletal model of each human hand consists of 21 keypoints based on the hand's skeleton, which we refer to as nodes. As shown in Figure 3 (a), the red node on each hand is located at the wrist, representing the origin of the hand joints. 
method 3.1 hand labanotation score: 
 Coordinate Setting. Based on the skeleton model, we can define a coordinate system of the hand. We set the root node as the coordinate origin, let 𝑌 -axis be in the direction from the root node towards the base of the middle finger, 𝑍 -axis be in the direction perpendicular to the 𝑌 -axis and the left on the plane of the palm, and let 𝑋 be the cross product of 𝑌 and 𝑍 . The hierarchical relationship among the 21 nodes depends on the edge between the nodes and the distance from the joint root and the hand model. For each pair of nodes at the ends of an edge, the node closer to the joint root is regarded as the father of the other.Regional Vectors. From the coordinate information of the nodes, we get 20 vectors ì 𝑣 𝑖 , 𝑖 ∈ [1, 20], which we call regional vectors. Each vector corresponds to a non-root node state, utilizes its father node as the reference, and stands for the node's relative position concerning its father node. These vectors represent the state of the particular section of the hand. Thus, the hand movement is represented by the set 𝑉 = ( ì 𝑣 1 , . . . , ì 𝑣 20 ), as shown in Figure 3 (a). Finally, each regional vector ì 𝑣 𝑖 can be converted to a symbol in the Hand Labanotation score system based on their spatial positions. In this way, the current hand movement of a single hand can be represented by 20 Hand Labanotation symbols, 40 for both hands. Since the entire palm is close to a rigid structure, we use a region vector parallel to the 𝑦 axis to represent the whole palm section.Basic Hand Labanotation Symbol. On the surface of a cuboid, we select 8 vertices, the centers of the 6 faces, and the midpoints of the 12 edges. Thus, we obtain 26 points, each corresponding to one of the Hand Labanotation symbols. As shown in Figure 2 , we select the center of the cuboid as the coordinate origin, the horizontal plane as the 𝑋𝑌 plane, and the vertical direction as the 𝑍 -axis direction. For a given spatial region vector ì 𝑣 𝑖 , 𝑖 ∈ [1, 20], we determine the shape of the symbol based on the projection of ì 𝑣 𝑖 on the 𝑋𝑌 plane and the texture of the symbol based on the angle between ì 𝑣 𝑖 and the 𝑍 -axis. This way, we can map any spatial region vector to a Hand Labanotation symbol. As shown in Figure 3 (b), the range of motion for the thumb is predominantly along the 𝑍 -axis. Here CVA is for Cross View Attention, MHSA is for Multi-Head Self-Attention, and MLP is for multilayer perception. For more details, refer to [22] .We divide the 𝑍 -axis direction into 3 intervals. Within the 𝑋𝑌 plane, the four fingers other than the thumb primarily rotate around the 𝑍 -axis, so we make a more detailed division of the regions within the 𝑋𝑌 plane, dividing it into 9 intervals.Due to the reason that the manual drafting of Hand Labanotation scores is both costly and cumbersome, it is necessary to develop a dataset-trained, end-to-end model capable of autonomously translating Hand Labanotation scores for arbitrary hand movements. We formulate this task as follows. The given input data is firstly formulated in the form ofV ∈ R 𝑇 ×𝑁 ×𝐻 ×𝑊 ×𝐶 . V = {V 1 , V 2 , . . . V T },where 𝑇 represents the number of frames of the data. For ∀𝑖 ∈ [1,𝑇 ], V i ∈ R 𝑁 ×𝐻 ×𝑊 ×𝐶 , which is collected from 𝑁 cameras located in the same experimental site.V i = {V 1 i , V 2 i , . . . , V N i }, for ∀𝑗 ∈ [1, 𝑁 ], V j i ∈ R 𝐻 ×𝑊 ×𝐶represents the hand movement information at the 𝑖-th moment from the 𝑗-th view in the data. Then, we use the function T to represent the system's transformation from the image data to the translated Hand Labanotation sequence.EQUATIONwhere L = {𝑝 1 𝑟 , 𝑝To facilitate the automatic translation of Labanotation scores, we construct a dataset named Hand Labanotation Dataset (HLD) based on InterHand2.6M dataset [21] and Freihand dataset [36] . To our knowledge, it is the first comprehensive dataset of multiview and single-view hand movements. Our dataset encompasses more than 4 million hand movement images, covering a diverse range of hand movements. Specifically, it includes more than 200 in distinct types of hand movements and involves more than 26 in different male and female participants. Each entry is detailed with high-resolution spatial and temporal annotations, capturing nuances in hand movements. The dataset is developed using algorithmic and manual methods to annotate Hand Labanotation. Here, we briefly introduce the HLD annotation process. First, we use rules demonstrated in Sec. 3.1 to calculate all the regional vectors ( ì 𝑣 1 , . . . , ì 𝑣 20 ) of each hand. Each of the regional vector ì 𝑣 = (𝑥, 𝑦, 𝑧) is then transformed from a Cartesian coordinate system to a spherical coordinate system by the following ruleEQUATIONIdentifying the region where the rotation angles (𝜃, Φ) are determined in the corresponding Hand Labanotation symbols. In our dataset, annotation is conducted through a hybrid approach. 19.85% of the joint frames have incomplete information. We manually annotated all joints for these joint frames. For these joints, when the manually annotated joints with existing information match the algorithmic calculations, we consider the annotation of the missing joint information to be correct. After annotating these frames, we use an algorithm to obtain the results for the remaining frames. The final annotations are preserved as texts. The resultant dataset is categorized into 3 subsets: training, validation, and testing. Figure 6 features a six-frame sequence of a "fist-clenching" gesture captured from multiple views and the corresponding HL symbols.3.4.1 Cross Fusion Mechanism. The Cross Fusion mechanism to fuse information from different views is depicted in Figure 5 . The overall fusion strategy is designed to draw inspiration from the linear fusion approach in MVT [32] . This entails jointly executing self-attention on a single view. We sequentially blend the information between every two adjacent view pairs to efficiently fuse information while conserving computational resources. Since the number of tokens is the same for each view after encoding, there are no issues related to size matching. The cross-fusion processing between two adjacent views is illustrated as followsEQUATIONwhere CVA represents the cross-view attention mechanism, which the following formula can express.EQUATIONConsider that W 𝑄 , W 𝐾 , and W 𝑉 represent the query, key, and value projection matrices, respectively, utilized within the attention mechanism [29] . As depicted in Figure 5 (b), W proj is the weight matrix after a linear layer during the fusion of adjacent views. Within the Cross-View Attention (CVA) module, the dashed line represents the features of the previous view after passing through the Encoder Block. Furthermore, we incorporate a residual connection surrounding the cross-view attention mechanism.Encoder. The Encoder Block modules from different views consist of the Multiview Encoder, illustrated in Figure 5 (b). For single frame hand movement, the input data is denoted as V ∈ R 𝑁 ×𝐻 ×𝑊 ×𝐶 , where 𝑁 denotes the number of views of the input image. We employ ResNet-50 [10] to obtain feature maps. Figure 5 (a) shows that the Encoder Block obtains token sequences by extracting feature maps from images from different views. After passing through the Multiview Encoder block, these token sequences yield feature and confidence maps from different views. They have the same size and hence can be denoted as∀𝑖 ∈ [1, 𝑁 ], z i ∈ R 𝑑 ×𝑐𝑙𝑠 .Here, 𝑑 denotes the total count of regional vectors associated with a hand movement, while 𝑐𝑙𝑠 signifies the number of fundamental categories within the Hand Labanotation symbol taxonomy. In the Global Encoder module, we concatenate them and then process them through Multi-Head Self-Attention (MHSA) [29] . Finally, we map the encoded classification to the output by a linear classifier.Since the structure of the hand can be viewed as a tree-like structure emanating from the root node, there is a dependency relationship between different keypoints. We can perform hierarchical supervision based on the structural differences of the keypoints. Accordingly, we introduce the Labanotation Hierarchical (LH) loss to train our model, comprised of HXE loss [1] and penalty loss. We calculate the correlation matrix based on the orientation ì 𝑑 of different HL symbols in space. Since each symbol corresponds to a portion of the spatial region, we use the direction from the coordinate origin to the center of the spatial region as the direction of the symbol. The calculation process for correlation is as followsEQUATION)Based on the degree of correlation between different categories in Figure 4 , we propose a penalty matrix to generate penalty loss, which assigns weights to various erroneous translations. The penalty matrix is defined as followsEQUATION)where 𝐶𝑜𝑟 is the correlation matrix between different categories, and 𝛼 is a positive scaling factor to control the rate at which the penalty coefficient varies with the correlation coefficient. The expression of the penalty loss isEQUATIONIn the above expression, 𝑦 is the ground truth label, ŷ is the raw translation value, and 𝑐𝑙𝑠 represents the number of basic Hand Labanotation symbol categories. 𝑚 𝑝 is the weight value of the penalty matrix 𝑀 𝑝 between ground truth and predicted labanotation symbols. Finally, we define the LH loss function of the model asEQUATION)where 𝛽 is a hyperparameter ranging between 0 and 1. 
experiments 4.1 dataset: 
 The HLD is constructed based on the Interhand2.6M [21] and Freihand [36] The results are illustrated in Figure 7 . We find that the distribution of Hand Labanotation in space is relatively uniform. Figure 6 offers a visualization of the dataset, which displays photographs of hand movement sequences and their corresponding HL scores.To assess the efficiency of our network's automatic translation result of Hand Labanotation, it is advantageous to adopt a distance metric to quantify the disparity between the translated Hand Labanotation score and the ground truth Hand Labanotation. To provide a comprehensive evaluation, three metrics are proposed in this work: 𝐿𝑎𝑏𝑎𝑛𝑎𝑐𝑐 @1 ↑ 𝐻𝑜𝑟𝑎𝑐𝑐 @1 ↑ 𝑉 𝑒𝑟𝑎𝑐𝑐 @1 ↑ Para. (MB) FLOPs (Giga)OpenPose 𝐻𝑜𝑟 𝑎𝑐𝑐 , 𝑉 𝑒𝑟 𝑎𝑐𝑐 , and 𝐿𝑎𝑏𝑎𝑛 𝑎𝑐𝑐 . Each metric evaluates performance across the horizontal, vertical, and overall dimensions. Labanotation symbols translation result can be represented as a matrix L, which is illustrated in Sec. 3.2. The columns correspond to different regional vectors of two hands, and the rows are associated with individual frames. Each element of the matrix indicates the direction of a regional vector. These directions are divided into 26 distinct categories, each corresponding to a specific Hand Labanotation symbol. For 𝑡 frames data, the translation result from the network of Hand Labanotation is denoted as [1, 20] , 𝑝{𝑡, 𝑟 𝑖 } and 𝑝{𝑡, 𝑙 𝑖 } represents the transformation result of the Hand Labanotation symbols corresponding to the 𝑖-th regional vectors of the right and left hands in the hand movement at frame 𝑡. In our dataset, each sample comprises three distinct components: the left hand, the right hand, and their interaction. Applying a mask matrix, denoted as M, is imperative during the testing phase. The matrix can filter out non-essential elements by using mask weights represented by binary numbers. To evaluate horizontal and vertical Hand Labanotation translation accuracies, we introduced two specific mapping matrices: the horizontal mapping matrix H and the vertical mapping matrix V. These matrices facilitate the transformation of each Labanotation into forms that exclusively contain horizontal or vertical elements. Building on the ingredients we discussed above, the overall accuracy rate of Labanotation translation is formulated as follows:L =          𝑝 {𝑡,𝑟 1 } . . . 𝑝 {𝑡,𝑟 20 } 𝑝 {𝑡,𝑙 1 } . . . 𝑝 {𝑡,𝑙 20 } . . . . . . . . . . . . . . . . . . 𝑝 {2,𝑟 1 } . . . 𝑝 {2,𝑟 20 } 𝑝 {2,𝑙 1 } . . . 𝑝 {2,𝑙 20 } 𝑝 {1,𝑟 1 } . . . 𝑝 {1,𝑟 20 } 𝑝 {1,𝑙 1 } . . . 𝑝 {1,𝑙 20 }          , for ∀𝑖 ∈𝐿𝑎𝑏𝑎𝑛 𝑎𝑐𝑐 = 1 𝐾 𝑇 𝑖=1 𝑀 𝑗=1 𝛿 (L 𝑖 𝑗 , L ′𝑖 𝑗 ) • M 𝑖 𝑗 .Similarly, the accuracy rate of horizontal Labanotation translation is calculated by𝐻𝑜𝑟 𝑎𝑐𝑐 = 1 𝐾 𝑇 𝑖=1 𝑀 𝑗=1 𝛿 ((L • H) 𝑖 𝑗 𝑝𝑟𝑒𝑑 , (L ′ • H) 𝑖 𝑗 𝑔𝑡 ) • M 𝑖 𝑗 . Lastly, the accuracy rate of vertical Labanotation translation is determined by 𝑉 𝑒𝑟 𝑎𝑐𝑐 = 1 𝐾 𝑇 𝑖=1 𝑀 𝑗=1 𝛿 ((L • V) 𝑖 𝑗 𝑝𝑟𝑒𝑑 , (L ′ • V) 𝑖 𝑗 𝑔𝑡 ) • M 𝑖 𝑗 .In these formulas, 𝐾 = 𝑇 𝑖=1 𝑀 𝑗=1 M 𝑖 𝑗 , 𝑇 represents the total number of sample frames, 𝑀 = 40 denotes all hand parts within a single frame, L and L ′ represent the predicted and ground truth Labanotation symbols on each regional vector, respectively, and 𝛿 (𝑥, 𝑦) is an Backbone Method Loss 𝐿𝑎𝑏𝑎𝑛 𝑎𝑐𝑐 @2 ↑ 𝐻𝑜𝑟 𝑎𝑐𝑐 @2 ↑ 𝑉 𝑒𝑟 𝑎𝑐𝑐 @2 ↑ 𝐿𝑎𝑏𝑎𝑛 𝑎𝑐𝑐 @4 ↑ 𝐻𝑜𝑟 𝑎𝑐𝑐 @4 ↑ 𝑉 𝑒𝑟 𝑎𝑐𝑐 @4 ↑ indicator functionEQUATION)Hand movement documentation is a novel task, similar to 3D hand pose estimation. We can apply methods from 3D hand pose estimation to our task and compare them with our method. Since our dataset is constructed based on the Interhand2.6M [21] and Freihand [36] datasets, we select models trained on the Interhand dataset and fine-tune them on the Freihand dataset. After the model training is complete, we can obtain HL results by calculating based on the 3D coordinates derived from the 3D pose estimation method. We compare our method with methods OpenPose [5] , InterNet [33] ,and IntagHand [14] . For fairness, we standardize the number of input image views to one. Thus, we can compare our method with existing 3D hand pose estimation methods. As seen from Table 1 , the performance of our model is significantly superior to methods that generate HL based on existing hand pose estimation models. The size and computational demand of our model are also very competitive.In our experiments, we compared various loss functions. Table 2 shows that our proposed loss function significantly outperforms the traditional Cross-Entropy (CE) and HXE losses, indicating its superior efficacy with different backbones. Additionally, Table 2 highlights the effectiveness of the cross-fusion module in enhancing model performance, which suggests that it facilitates better information integration across views. Notably, @2 in the table represents the use of 2 views in the experiment. The Interhand2.6M [21] dataset provides many images captured from multiple views, from which we select various viewpoints. We select 6 multiview schemes in the experiment with 1 to 6 views around the hand. The details can be found in Figure 9 (a). The accuracy of Hand Labanotation translation 𝐿𝑎𝑏𝑎𝑛 𝑎𝑐𝑐 for Hand Labanotation at each iteration is calculated for each scheme.  
discussion and applications: 
 Hand Labanotation can be an intermediate representation for downstream tasks, yielding satisfactory results. Herein, we will present some potential applications of HL.Hand Labanotation, as a symbolic system, effectively applies to documenting hand movements and controlling robotic hands. Our proposed Hand Labanotation symbol quantifies the rotation angles of individual hand vectors in polar coordinates, which shares similarities with the current approach of using dense rotation angle parameters to control robotic hands. Hand Labanotation can directly serve as an intermediary representation for non-high-precision robotic hand movement imitation, efficiently realizing an integrated process from documenting to simulating hand movements. Although HL loses some precision regarding angles, it also acts as a natural filter for angles, which can prevent severe jitter. Experimental results indicate that HL can replicate human hand movements with relative accuracy as a language for controlling robotic hand actions. Our method is faster and more efficient than capturing images with optical cameras and then using a 3D hand pose estimation model to estimate hand coordinates for controlling a robotic hand.Sahand [20] is a dataset comprising 64, 000 images of American Sign Language (ASL), covering 10 numbers and 26 English alphabet. The essence of this task is a gesture classification problem with 36 categories. Initially, after pre-training the IntagHand model [14] on the Interhand2.6M dataset [21] , we predict the 3D gesture representation from input images, including Skeleton, MANO, and Mesh. Subsequently, we directly connect the results to the final output through the Linear layer. Similarly, we employ the MHLFormer model, pre-trained on the HL dataset, to predict HL representations.Based on HL, we also use the Linear layer for gesture classification.To further explore the rationality of Hand Labanotation, we converted the predicted Labanotation back to the hand skeleton. After initializing the root node coordinates, the rotational information of the hand is restored based on the Hand Labanotation symbols for each segment. We represent the spatial orientation with the central axis of the space corresponding to HL symbols. By assigning phalange lengths proportional to those of an adult hand, we obtain the skeleton results as shown in Figure 8 . A comparison with the original input images reveals that while not completely precise, the skeleton structure reconstructed from Hand Labanotation still accurately records the hand movements relatively well. For example, the two actions shown in Figure 8 are both hand-opening movements, and the results after reconstruction still allow for an intuitive understanding of the corresponding actions.Based on the aforementioned capabilities and the distinct differences in shape and texture of HL symbols, HL can be used as Braille to encode gestures, further expanding its application scenarios. 
conclusion: 
 This paper introduces Hand Labanotation, a symbolic system designed to document hand movement sequences intuitively. The expansive Hand Labanotation dataset (HLD), a comprehensive repository comprising over 4 million frames capturing single and multiview hand movements, accompanies this novel notation system. Furthermore, we propose the MHLFormer method, a pioneering network specifically designed for the autonomous translation of hand movements into HL symbols. Experimental results demonstrate that our method achieves commendable performance. We achieve relatively good results using the predicted HL for hand movement reproduction. HL can also directly serve as an intermediary representation for controlling robotic hands, thereby unifying documentation and dissemination.Limitations: Due to the loss of some information in the process of symbolization, there will be a small amount of unavoidable errors when we store hand movements with HL and then restore them. 
 
 Appendix:  


Decision: This paper received four borderline/weak acceptances from the reviewers. Overall, all reviewers acknowledged the potential of the proposed symbolic representation, the new dataset for hand movement research, and the MHLFormer model. The reviewers raised concerns regarding the clarity of the descriptions and figures, which should be straightforward for the authors to address.

Reviews: review: paper_summary: paper_summary: In this paper, a hand labanotation (HL), a new notation system for hand  movement documentation is proposed. The authors construct  a new HL dataset with 4M annotated images. A multi-view transformer architecture is proposed. The comparison results to the state-of-the-art methods make this paper convincing.
strength_weakness: strength_weakness: The hand labanotation score issue proposed in 3.1 and Fig. 3 is well defined. The system architecture in Fig. 5 is clear and readable. The hand labanotation dataset proposed in Sec. 3.3 and the cross fusion mechanism proposed in Sec. 3.4 are reasonable. The comparison results shown in Table 1 demonstrate that the proposed method achieved the best result in most aspects. Teh FLOPs of the proposed method achieved the best result. The results in Table 2 and Table 3 make this paper more convincing. The multiview results in Fig. 9 are convincing for the possible readers.
limitations: limitations: The proposed method not achieving the smallest para. in Table1. The results in Fig. 7 has higher values in the case 13, more discussions are experiments are needed.
paper_summary: paper_summary: The paper proposes the use of Labanotation, a symbolic representation of body movements, for hand annotation. To do so, it describes the proposed hand notation system and transformation between the orientation of the edges among hand joints (called regional vectors) and the symbols, and proposes a novel large-scale, multiview dataset of hand movement videos annotated with the proposed Labanotation symbols, which is based on existing hand datasets Interhand2.6M and Freihand. The paper also proposes a methodological baseline based on the existing multi-view transformer (MVT) to convert from images to symbols directly. It also proposes to use a combination of the existing hierarchical cross-entropy loss along with a cross-entropy loss informed with the correlation of different symbols. 

Evaluation includes: a comparison with some 3D pose estimation methods (e.g. OpenPose), from which the labanotation symbols can be extracted based on the proposed transformation process, showcasing the improved performance of the proposed method; a comparison among different hand pose representation methods (e.g. MANO) with respect to accuracy and storage, showing competitive performance. The paper also performs a series of ablations, including different CNN backbones and the impact of the number of views used on performance. Finally, the proposed representation is preliminary validated in three applications: a control signal for a robotic hand, a representation for ASL, and symbol-to-gesture recovery, demonstrating that the symbolic representation is resource-efficient despite losing some information detail in the process.
strength_weakness: strength_weakness: - The paper deals with an interesting problem and, to my knowledge, this is the first time that Labanotation is proposed for the representation of hand poses.
- The problem is properly motivated and contextualized. 
- The hand labanotation score is explained clearly and detailedly.  
- The created dataset greatly contributes to fostering this new research topic.
- The images throughout the paper are very informative.
- The ablations are mostly exhaustive and provide rich insights beyond reporting accuracy.
- Despite not achieving the highest accuracy among other hand representation methods, it achieves the goal of efficient storage. 
- Having preliminary results for three downstream applications is also greatly appreciated, making the paper complete.
limitations: limitations: The main limitation I’ve encountered is the lack of clarity in the methodological part (sections 3.3. to 3.5), and part of the evaluation (Sec 4.2) hindering its understanding:
- L387-L389 and L433-L437: it is unclear to me how the dataset is annotated from these explanations. Is it manually, semi-automatically, completely automated? And how are the annotations achieved for each case? 
- Sec. 3.4. starts directly with the cross-fusion mechanism, without an introduction to the overall pipeline. It feels as if such introduction is just included in the caption of Figure 5 to save space, but that impacts the paper negatively. 
- In Sec 3.4.2, there are some parts that are different from Figure 5. For instance, it is said the Encoder block extracts features, when that is supposedly done with ResNet/CNN blocks. Also, it is said that the confidence maps are given after passing through the Multiview encoder block, but in the figure they appear to come from some CNNs.
- Similarly, the pipeline is unclear. In Figure 5, each view is apparently processed by two different CNNs (with different colors), but that is not explained in the text. The output of one goes to the encoder block while the other’s output apparently directly creates a confidence map. How is this confidence map created? Also, according to the text, the CNNs are used to extract features, but Figure 5 depicts the features as ‘multiple’ images. Is this a visualization issue or is there something else?
- Sec 3.4.2 mentions that the last layer is a linear classifier (L488), but the representation is d (20 regional vectors) x cls (26 possible symbols), and the loss in Eq. 7 is a binary cross-entropy-like loss. It is unclear to me how these three items are compatible with each other. What is the format of the final network predictions, and how is the loss computed?
- In Eq. 7, it is also unclear to me how the penalty term is applied and its effect.
- L618 mentions M for the first time, being ‘imperative’ because ‘The matrix can filter out non-essential elements by using mask weights represented by binary numbers.’ However, the text does not describe which type of elements have to be filtered and why.

In addition, the text claims that the paper proposes a novel multi-view transformer, but the only difference I can see compared to the original MVT is the use of a confidence map, which is not justified either. 

With respect to the comparison to other methods, I wonder why it was not possible to compare to existing methods for Labanotation, like [15] and [31]. Even if they are originally considered for body notation, they could be extended to hand notation to have a more fairer comparison.

Other comments:
- Using the same math symbols for different elements throughout the text can be confusing, e.g. ‘i’ takes different meanings in L274, and Eq.3. (Also, in Eq. 3 ‘i’ is not defined).
- The current text lists figures unorderedly. For instance, Figure 4 is located on page 3, but not mentioned until page 5. 
- For completeness, the HXE loss should be defined.
- In Figure 9 (b), It is surprising that using 5 views the performance saturates, being worse than using 3 or 4. And using 6 works better than 5. This suggests that the issue is not the number of views specifically, but their location.

For these reasons, I believe the paper is not ready to be published in its current form, hence my rating of ‘borderline reject’. But as mentioned, most concerns are due to a lack of understanding so those may be addressed in the rebuttal.
paper_summary: paper_summary: This work introduces Hand Labanotation (HL), a novel system for documenting hand movements. The authors contributed the HLD dataset and proposed a multi-view transformer architecture to automatically translate hand movements into HL. The paper also provides potential applications and evaluations on downstream tasks.
strength_weakness: strength_weakness: This work has several strengths:

(1) It introduces a novel notation system for documenting hand movements, which is an intriguing task.

(2) It proposes the MHLFormer model, which uses the LH loss function to achieve great performance in translating hand movements into HL scores.

(3) Additionally, this work contributes a multi-view dataset for Hand Labanotation.
limitations: limitations: (1) In the related work, it says, “Sign Language cannot capture the spatial details of a wide range of hand movements…” near line 169. Please provide further support for this claim.

In American Sign Language (ASL), there are five parameters to convey meaning: handshape, palm orientation, movement, location, and expression/non-manual signals. The sign language book provides detailed text descriptions on how to correctly make sign language in spatial space. For ASL analysis methods like 3D landmarks or mesh rotation matrices, numbers are used to indicate the spatial details. In contrast, employing discrete symbolic Hand Labanotation (HL), which clusters some spatial spaces into one symbol, affects the accuracy of representing spatial details. 

(2) One of the novel aspects mentioned in the MHLFormer model architecture is the Multiview Encoder. However, at line 694, it mentions standardizing the number of input image views to one. Please provide more details about this process: do you select one specific view branch or simply duplicate it multiple times?

(3) Alphabet recognition is the simplest task in ASL, yet in Table 3, this work still does not achieve better accuracy than current methods. Although the storage is efficient, accuracy is the first priority in conveying meaning. More experiments on word-level or sentence-level ASL are needed to support your claim near line 800.

This method might be helpful in encoding handshapes but is questionable in ensuring the correct semantic recognition or translation in ASL. For example, in ASL, the same handshape against the chin means "mother," while against the forehead it means "father." This work likely cannot cover such nuances well.
paper_summary: paper_summary: This paper introduces Hand Labanotation (HL), an innovative notation system for hand movement documentation. To help convert hand movements into HL, the authors have constructed a large-scale multiview Hand Labanotation dataset (HLD) with over 4 million annotated hand movement images. With this dataset, they further propose a novel multiview transformer (MHLFormer) architecture for the automated translation of images into Labanotation scores. Some novel metrics and loss functions are applied to facilitate the training and evaluation of the framework. This is a meaningful exploration and also helps promote the development of various downstream tasks.
strength_weakness: strength_weakness: 1. The proposed Hand Labanation is interesting, which utilizes simple symbols to represent the movements of the human hand.
2. A dataset consisting of Interhand2.6M and Freihand datasets was constructed, with over 4 million images labeled with corresponding symbols.
3. Furthermore, the authors also propose a novel multiview transformer (MHLFormer) architecture to translate the input images into Labanotation scores. Meanwhile, both evaluation metrics and Labanotation Hierarchical (LH) Loss Function are proposed to facilitate the translation.
limitations: limitations: 1. I suggest the author adjust Figure 1 so that it can visually display the proposed tasks.
2. For section4.4, please supplement more analysis and explanation, for example, analyzing why the proposed loss is more effective.
3. I suggest the author place the section4.5 at the beginning of Chapter 4, which may be more convenient for readers to follow this paper.
4. Both the InterNet and IntagHand in Table 1 are mainly designed for modeling interacting hands. For those single hand frames, I think comparing them with the SOTA methods of single hand modeling is more convincing.
5. Although the constructed dataset reach up to 4 million, as mentioned in main paper, it comes from existing data and the authors re-labeling them to obtain  HLD. So using it as a key contribution may be relatively fragile.