Making Body Movement in Sign Language Corpus Accessible for Linguists and Machines with Three-Dimensional Normalization of MediaPipe
Authors: Victor Skobov
Mayumi Bono
Venue: EMNLP 2023 Findings (2023)
Keywords: sign language
pose processing
wlasl
movement processing
TL;DR: The paper presents a new 2D to 3D computation method for estimated pose data that enhance sign language processing.

Abstract: Linguists can access movement in the sign language video corpus through manual annotation or computational methods. The first relies on a predefinition of features, and the second requires technical knowledge. Methods like MediaPipe and OpenPose are now more often used in sign language processing. MediaPipe detects a two-dimensional (2D) body pose in a single image with a limited approximation of the depth coordinate. Such 2D projection of a three-dimensional (3D) body pose limits the potential application of the resulting models outside the capturing camera settings and position. 2D pose data does not provide linguists with direct and human-readable access to the collected movement data. We propose our four main contributions: A novel 3D normalization method for MediaPipe's 2D pose, a novel human-readable way of representing the 3D normalized pose data, an analysis of Japanese Sign Language (JSL) sociolinguistic features using the proposed techniques, where we show how an individual signer can be identified based on unique personal movement patterns suggesting a potential threat to anonymity. Our method outperforms the common 2D normalization on a small, diverse JSL dataset. We demonstrate its benefit for deep learning approaches by significantly outperforming the pose-based state-of-the-art models on the open sign language recognition benchmark.

Making Body Movement in Sign Language Corpus Accessible for Linguists and Machines with Three-Dimensional Normalization of MediaPipe
Victor Skobov! and Mayumi Bono?
"The Graduate University for Advanced Studies (SOKENDAT), Kanagawa, Japan Nattional Institute of Informatics, Tokyo, Japan ('vskobvv, {bono)@nii.ac..
Abstract
guists must annotate the sign language features on the video to process them. And annotate a feature, it must first be clearly defined and thor- oughly explained to the annotators. That process is not only time-consuming, but it also limits ac- cess to the collected data and the processing poten- tial. Pose estimation methods like OpenPose (Cao et al., 2017, 2021) and MediaPipe (Lugaresi al., 2019) are more often included in the sign language processing pipeline. Recently published sign lan- guage datasets often include pose estimation data. How2Sign',, a large multimodal dataset Amer- ican Sign Language (ASL) presented in Duarte et al. (2021), and the Word-Level American Sign Language? 2 (WLASL) video dataset presented in Li al. (2020a), both provide estimated pose data. However, the detection accuracy of pose estimation techniques still requires improvement. Moryossef et (2021) reported the negative influence of in- accurate or missing estimations on model perfor- mance and applicability beyond training datasets. Generally, sign language researchers indepen- dently develop their own ways of processing pose data for specific body joints and features. Recent approaches still rely on raw pixel data (Sadeghzadeh and Islam, 2022) or a combination of pixel and pose data (Shi et al., 2021). Morcover, sign language datasets vary in terms of the position of the camera relative to the signer, resulting in dis- similarity in 2D pose estimation output for similar movements. Such inconsistencies prevent model generalization, thereby limiting movement feature extraction and inference outside the dataset.
Linguists can access movement in the sign lan- guage video corpus through manual annota- tion or computational methods. The first relies on predefinition of features, and the second requires technical knowledge. Methods like MediaPipe and OpenPose are now more often used in sign language processing. MediaPipe detects two-dimensional (2D) body pose in a single image with a limited approximation of the depth coordinate. Such 2D projection of a three-dimensional (3D) body pose limits the potential application of the resulting mod- els outside the capturing camera settings and position. 2D pose data does not provide lin- guists with direct and human-readable access to the collected movement data. We propose our four main contributions: A novel 3D normaliza- tion method for MediaPipe's 2D pose. novel human-rcadable way of representing the 3D normalized pose data, an analysis Japanese Sign Language (JSL) sociolinguistic features using the proposed techniques, where we show how an individual signer can be identified based unique personal movement patterns suggest- ing potential threat to anonymity. Our method outperforms the common 2D normalization a small, diverse JSL dataset. We demonstrate benefit for deep-learning approaches by sig- nificantly outperforming the pose-based state- of-the-art models on the open sign language recognition benchmark.
1 Introduction
Our research aims to find movement representa- tion that allows processing of sign language move- ment directly, without relying on annotations or systems of predefined movement features, such as the Hamburg Notation System (Prillwitz al., 1989). And help to overcome the camera settings constraints of the available datasets.
The commonly used standard normalization pro- cess proposed in Celebi et al. (2013), recently adopted in Schneider et al. (2019) and Fragkiadakis et al. (2020), involves coordinates axis origin trans- lation "neck key point" and scaling of all co- ordinates so that the distance between shoulder
1.1 Problem
Due to its visual nature, sign language data are stored and distributed a video format. Lin-
Available at: https://how2si:: hwtsssiiinggtththh github. io/ 2 Available at: https://dxlin github. io/NLASL/

---

key points equals one. This work will refer this normalization method the "basic." This method successfully eliminates the inftuence body size differences. However, features such as body rota- tion (toward the camera), posture, and body pro- portions in the dataset can still influence feature extraction.
An analysis of JSL sociolinguistic features present in movement patterns
Substantially outperforming state-of-the-art results on the WLASL-100 public benchmark for pose-based deep-learning methods
Our solution and reproduction code are available as a mp2signal python package and as a GitHub repository".
1.2 Related Work
Overcoming the camera setting boundaries and pose estimation are being actively researched. Complete 3D body pose estimation using a sin- gle camera is of the main goals (Ji et al., 2020). Activity classification results obtained using 2D and 3D pose data were compared, with no signifi- cant difference emerging; the features 2D data were sufficient (Marshall ct al., 2019). pipeline that includes learning 3D pose coordinates from 2D pose data collected from sign language videos was proposed and used recognizing tasks and synthe- sizing avatar animations (Brock et al., 2020). The Skeletor (Jiang et al., 2021), deep-learning ap- proach that refines the pose's estimated coordinates three dimensions. However, it relies on contex- tual information and therefore requires a sequence frames.
Methodology
To achieve our goal of directly processing sign language, we set the following requirements for the desired movement representation: the adherence the triangle inequality, the capability of movement synthesis, being intuitive and understandable humans, and being low-dimensional.
The adherence triangle inequality is essential for automated data processing techniques like cluster- ing, machine learning, and indexing.
Movement representation data must be dis- tributed across space compatible with the notion of distance and similarity. Sampling from such space should return the corresponding pose, and moving through should produce movement to meet the movement synthesis requirement.
The space should not have any latent features, and its dimensions must be perceivable by human. To promote readability and facilitate processing, the space must be low-dimensional possible to climinate unnecessary information from repre- sentations.
1.3
Our Proposal
Here, we propose chree-dimensional normaliza- tion and encoding method for MediaPipe pose data that is entirely algorithmic and normalizes pose data based on fixed human proportions. Data is presented series of angular joint changes. It could be applied to various sign language process- ing tasks and datasets and allows the search and extraction of movement features. Both methods are detailed in Sections 2.1 and 2.3.
Normalization must transform pose data into the desired space, and encoding must represent it suit- ably for human perception.
To determine the degree of adherence to these requirements and compare our method to the basic normalization method, mentioned in Section 1.1, conducted experiments two types of sign language data using standard machine learning and deep-learning techniques.
We tested our method against the basic normal- ization on continuous sign language data using standard machine learning techniques in Section 3.1 and isolated sign language data using the deep- learning in Section 3.2. We show how the JSL sociolinguistic features present in the signing movement and how they can be explored using our methods.
2.1 3D Normalization
The MediaPipe's 's holistic model along with two and y provides a limited estimation of the z coor- dinate for each body key point on the given frame.
The main contributions are:
We propose a procedure to improve the depth z coordinate estimated by MediaPipe.
Novel
three-dimensional
normalization
method for MediaPipe
Available at: mp2signal
https://github..co//skkooot/ con/vskobov/
Novel movement representation method for a direct sign language processing
'Avillable at: https://google git.giitttbb.... io/mediapipe/ colutions/holistiee html

---

Joints do not move in isolation; they naturally interact within the skeleton. For example, the move- ment of the arm changes the position of the hand and its fingers. Therefore, we propose processing pose skeleton data as a "joint tree structure" that respects actual human proportions, with the root node in the neck key point. We aimed to use all available information to simplify pose data pro- cessing. We selected 137 joints from the holistic model: 22 for each hand, nine for the upper body, and 84 for facial expression. We created human skeleton, which showed rigidness, proportions, and connectivity line with the human body.
value from MediaPipe's original z estimation
length; propj * (cyyddislance/0.23 (I)
The joint tree structure allows us to control the order of calculation with the traversal and process only the desired part of the tree if needed. To obtain the coordinates for joint with the origin set at the neck point, sum the coordinates for all its parent joints in the tree. A detailed example of the 3D estimation step is shown in the middle part of Figure
To improve the depth coordinates estimated by MediaPipe, we use the proportions of the body. In Herman (2016), an overview of standard human body proportions is provided. For simplification, we assume that the body proportions for all data processed with MediaPipe are constant when using our method.
Fuluskude. 11-12. Inter
Side view
Semi-front view
We captured short video of body standing upright and used MediaPipe to calculate the ra- tio of distances between key points relative to the distance between the shoulders. The maximum distances across frames were used to calculate the proportions. The joint free model of the human body aligned with MediaPipe key points stores the proportional values for each joint. From the length just one joint in real space, we can compute the lengths of all joints, which requires some reliable MediaPipe estimation as a basis.
Standard 2D Normalization
The holistic MediaPipe model includes a face model for estimating key facial points; its depth coordinates are the most accurate. The distance be- tween the eyes in the MediaPipe model is selected basis for calculating the lengths of body joints. We trust its estimation the most. Eyes positions calculated based on the average positions of key points 159, 143, 157, and 149 for the left eyc, and key points 384, 386, 379, and 372 for the right eye. Relative to the distance between the shoulders, the distance between the eyes was calculated as 0.237 from the previously captured short video.
Figure 1: Normalization pose captured from the side (top-left) and "semi-front" (top-right), basic nor- malization (middle), our 3D normalization (down).
2.2 Scaling and Rotation
With this ratio, we calculate the lengths of all body joints in 3D using the Formula 1, where eyedistance gives the distance between the eyes according MediaPipe and prop, gives the "cap- tured proportion" of the joint. We calculate the z coordinate using the length, and relative x and y coordinate with the origin in the "parent in the joint tree structure. Lastly, we apply the sign
After 3D coordinates refinement, the pose data are represented a joint tree structure with coordi- nates in 3D space. To address the variation in cam- era angle and relative position, we rotate and scale the coordinates the final part of the normaliza- tion process. The resulting pose consistently rotated toward the camera and is fixed in size. The

---

root node (neck point) is the origin (0.0,0), and the left shoulder is at (0.5,0,0).
and z-axes ranging from -t π in radians are encoded in the red, green, and blue channels, re- spectively, as 8-bit integers ranging from 255. Figure 3 shows an example image of a representa- tive movement.
Both scaling and rotation are performed through a linear transformation in 3D space. To generate the transformation matrix, scaling factor and rotation angles are required, which we compute for each frame. We apply the transformation matrix to all joint coordinates, joint tree structure traversal to obtain rotated and scaled coordinates for each joint. A detailed example of a scaled and rotated pose is shown in Figure (bottom panel).
For handshapes, it might serve as an additional visual clue to add one-dimensional (1D) encoding of the absolute 3D angle (0--180%; blue 0", red 90%, green 180%) between the hand joints and their parents. Figure 2d provides an example.
Facial expressions are essential for processing sign language. Therefore, we perform an additional separate transformation only for face points. We scale and rotate the key face points so that the nose points toward the camera along the z-axis, while the point between the eyes is on the y-axis. Addi- tional normalized facial key point data are shown in Figure (bottom panel, upper right corners) and 2e-h.
5 (128, 191, 191)
J4 R
7 (185, 206, 143)
G
(184, 160, 161)
B
5
r+1tt2
frames
7 72
2.3 Representation
Our normalization process returns pose data as 137 "tree-structured joints" with 3D coordinates, which is helpful for decomposing movement. We use relative coordinates for each joint, with the ori- gin set at the parent joint, to represent the joint's movement in space independently. Since the pro- portions are fixed and known, independent move- ment may be estimated with arccosine of direction cosines values, the angles between joint and axes, which range from - to ı in radians. The resulting body movement appears as a series of 411 (3+ 137) isolated signals. Each signal shows a value for the angle between the corresponding joint and the corresponding axis every frame. The result- ing decomposition allows the quick determination of when, where, and what movement of the body is captured, providing direct access to it.
joints
d
g
Figure 2: Proposed encoding scheme: a) calculating joint's direction angles with arccosine; b) translating the angle values into RGB color space; displaying the RGB-en encoded data as an image: d) adding additional 3D angles between the joints with color- color-coding: e) nor- malizing facial expressions with relaxed baseline face; and f-g) example color-coding of various facial expres- sions deviating from the relaxed baseline face.
Initially, for each key body point, we obtained three values from the MediaPipe. After normaliza- tion, the key body points became joints with three direction angles values, striped from the variation in size, rotation, and body proportions. The di- mensionality of the information remained the same while the representation space changed, adhering to the requirements in Section 2.
Key face points are encoded differently. We captured a relaxed facial expression as the baseline (Figure 2e) and encoded deviation angles for each key point in three axes. The angle changes are usually tiny,
multiplied the difference by factor of four (determined by trial and error) boost visual interpretation. Figure 2f-g shows an example of encoded facial expressions.
We use image RGB color space visualize series of direction angles for joints to simplify the interpretation of the movement. The process is shown in Figure 2a-b: direction angles with x-, y-,
For computer processing methods, channels must be separated; thus, data will be encoded as an

---

8-bit integer with only one channel per pixel. This method allows for movement manipulation via im- age manipulation. Image processing a highly ad- vanced area with various methods and approaches applicable to movement processing. Movement detection and recognition can borrow approaches from object detection (image processing).
Figure shows an encoded representation of 13 simultaneous closings and openings of rotating fists. Where and when the movement occurs easily detectable with the naked eye, and might also be easily detectable via modern pattern-reoognition methods.
Figure 4: Hands movement is clearly visible the encoding plot of processed video sample with 13 clos- ings openings rotating fists.
this Section, compare the performance of our method to the basic normalization method, men- tioned in Section 1.1, on a dataset composed of continuous samples - the JSL Dataset and a pub- lic benchmark dataset collected isolated samples the WLASL-100 dataset. The JSL dataset has a variation in the camera angle and includes coding for various sociolinguistic features. However, it is small and very diverse dataset; therefore, it will be used for feature exploration and camera settings boundary testing using standard machine learning algorithms. The isolated WLASL-100 is more suitable for deep-learning testing since it is an established public sign language recognition benchmark.
The proposed encoding scheme is straightfor- ward, and the image can easily be decoded back to direction angles and coordinates. Movement pat- terns are fully explainable and can produce skeleton animations, aiding visual comprehension and thus satisfying the requirements in Section 2.
We hope to encourage researchers to explore the capabilities of encoded movement data, aug- menting their sign language knowledge explore movement features. Section 3 discusses how the proposed methods compare to the standard normal- ization process used for linguistic and sociolinguis- tic features extracted from sign language datasets.
frames
3.1 Continious Signs JSL
additional dimensional 3d angle (0-180) encoding for hands and lower arms
We created a simple dataset from the Japanese Sign Language (JSL) Corpus presented by Bono et al. (2014, 2020). The JSL Corpus is a continuous and dialog-based corpus that includes utterance- and word-level annotations. It consists of conversations between two participants freely discussing various topics. The signers vary in age, geographic area, gender, and the deaf school they attended. Conver- sations were captured from both the semi-front and side positions; a sample from the dataset shown in Figure 1.
right am and hand
left arm and hand
411(137*3)-
dimensional
mouth lips
space
eyes and brows
3.1.1 JSL Dataset Satisties
Using the word-level annotations, we have selected lexical signs from the JSL Corpus with more than 25 samples. For each lexical sign, we extracted 25 video examples for each camera view (a total of 50 samples). Some samples had insufficient capturing quality for pose estimation, our final dataset comprised 674 semi-front view and 608 side view videos. The resulting number classes and samples per class for each feature is shown Table 4.
Figure 3: Structure our movement encoding method.
3 Experimental Setup
The proposed normalization explicitly devel- oped to process data with high variance, which is typical of data captured in real life. The decomposi- tion property of our approach allows for comparing pure movement data on a jointt joint-by-join basis. In

---

For comparison, we created second dataset from the same samples by normalizing the Me- diaPipe pose data using the basic normalization method. The resulting samples vary in duration from four to 120 frames, and we had to resize them using linear interpolation to fit the longest sam- ple in the dataset. The JSL Corpus includes the signer ID, prefecture of residence, deaf school, age group, and gender. This information was added to the dataset since we were interested in examining whether these features affected signing movements.
3.2.2 WLASL Preprocessing
The WLASL dataset is distributed in video for- mat, requiring preprocessing before training. Our preprocessing flow starts with the mediapipe pose data extraction and normalization using the pro- posed methods and basic normalization to create two datasets for comparison. The next preprocess- ing steps are visualized in Figure 8 and include finding and cutting to the part where both hands are visible, resizing using the linear interpolation to a fixed 100 frames, and removing the facial and relative joint information rows from the samples reduce the dimensionality from 455 and 411 in ba- sic normalization case to 159 values per 100 frames. We want to point out that start and end frames for basic normalization samples were determined using corresponding sample of the proposed normaliza- tion dataset to guarantee consistency between the two datasets.
3.1.2 Classification
First, we used the "Neighhourhood Components Analysis" (NCA) approach presented in Gold- berger et al. (2004) to visualize the embedding space for each sociolinguistic feature in the dataset. We tested various classification techniques using the scikit-lear package S "(Pedregosa et al., 201 1). including the linear support vector classifier (SVC) (Fan et al., 2008), nearest neighbor classifier (Cover and Hart, 1967), naive Bayes (Zhang, 2004), and decision tree (Breiman et al., 1984), check for the presence of fcatures in the data and assess the potential applicability of our normalization method to classification tasks.
3.2.3 Model
We chose the Conformer model presented by Gulati et al. (2020) as the core unit since is aimed at two- dimensional signal representations. Figure 5 shows the overview of our model, where use the adap- tive average pool layer to reduce each sample 15 frames and add one fully connected layer before and one after the conformer. Both do not have bias nor an activation but have an L.2 normalization and a dropout layer after. The resulting model ends up as simple as possible. We train it using the Adam optimizer (Kingma and Ba, 2015) and the log loss for 200 epochs with the mini-batch size 32, Before training, all samples are standard-scaled" on the training set, and during training, a 50% uniform noise added the samples.
We designed experiment in which model trained data captured from the front per- spective and tested using data captured from the side perspective. We did this address the camera angle boundary and generalization issue mentioned in Section 1, i.c., to determine the applicability to other datasets and capture conditions.
3.2 Isolated Signs WLASL
We used the popular public deep-Icarning bench- mark, the Word Level American Sign Language Dataset (Li et al., 2020a), to demonstrate the utility of our normalization and representation methods in deep-learning pose-based approaches.
Results and Analysis
4.1 Continious Signs
Figure 6 shows the well-distinguished clusters for Signer ID, Prefecture, Deaf School, and Age Group, with the only exception for the Gender feature. Table 4 shows all classification results and samples per class distribution, whereas Table 2 shows only the best results summary.
3.2.1 WLASL-100 Dataset Satisties
We selected the WLASL-100, a WLASL subset subset of the top one hundred samples per sign. The split statisties are shown in Table
Classes
Train
Validation
Test 258
For the Lexical Sign feature, our method outper- forms the basic normalization method. Signer ID was the best-performing feature on front view data (accuracy = 78.57%) when using naive Bayes, for
100
1442
338
Table I: WLLAS-1100 data subset satistics.
"Implementation used: https://gist..ee github. com/ com/ farahmand- m/ 1al244463332277aaa 49f92ce4708bebbb
'Aaillall at: https://scikit-learn..

---

FC 1024
NO Bias
NO Activation
p-2
Dropout 0.275
Conformer
StandardScaler(hnung
norm
output
FC 256
p-2
+ +Noize
Adaptive avg poolid
NO Bias
norm
Dropout
NO Activation
0.275 Output
Conformer
FC 100
emb __dddiimi 1024
ffn ffn_dim 4096
layers dropout 0.375 group_ norm=0
32, 100
32, 100, 159
32, 159
32, 256
32. 15, 1024
32, 15, 1024
Figure 5: An adaptive average pooling layer, with single conformer layer (Gulati et al., 2020) trained with cross-entropy loss.
which the baseline was 8.33% and 43/12 samples per number classes ratio. Also, Lexical Sign with 49.78% accuracy on front plus side view data from naive Bayes compared to 3.7% chance of guess- ing with only 43/27 samples per number classes ratio. Each signer has unique movement pattern (i.e., signing style). Likewise, stylistic characteris- ties uniquely vary depending on the prefecture of residence, deaf school, and age group; only gender had no influence signing movements.
outperforms the state-of-the-art pose-based results on both datasets. Moreover, the proposed normal- ization pose-only dataset exceeds the models with combined modalities. As for comparing the malization of datasets, the results suggest a great performance improvement using the proposed normalization over the basic normalization, going from 75.85% to 84.26% using the same pose es- timated data from MediaPipe. In Figure 7, the accuracy curve of the test data set during training is shown, indicating clear improvement in learning with the proposed normalization.
The results, shown in Table 2, indicate that the only feature that did not significantly improve pre- diction performance from the chance level was Gen- der (accuracy 75.53%), even though had the most samples per class among all features and binary in data. This is consistent with the NCA em- bedding visualization. We cannot predict the JSL signer's gender based on their signing movements. Model performances other features were consis- tently above the baseline, except for the learning transfer tests (training on front view images and testing on side view images), thus confirming the presence of the movement patterns attributed to them. JSL experts validated the results, confirming our findings based on their experience.
Discussion and Conclusions
The proposed methods allow linguists and engi- neers to directly access the movement captured in the sign language corpus. Before, they had to use human annotation or recognition methods, which both relied on a predefinition of the features and were effectively limited by it.
Sign language movement can now be repre- sented and stored in human-readable form with the proposed encoding method, allowing researchers to observe and comprehend it visually. Normal- ized pose data are distributed over a joint-based, low-dimensional feature space with distinct and fully explainable dimensions. Machine learning methods can also process it directly since it com- plies with the distance notion and the triangular incquality.
The last two columns in Table 2 indicate that our method retains the extracted features better than the basic normalization method for all features, overcoming the camera angle setting boundary.
4.2 Isolated Signs
The embedding results showed the presence of stylistic movement features that correspond known sociolinguistic features JSL., similar predictions of the speaker's 'ss country of origin based on their accent. Linguists and sign language ex-
In Table 3, report the average accuracy across ten runs for each dataset with the top 1, top and top 10 prediction scores as established in the WLASL benchmark reporting practice. Our model

---

Train Dataset
Front + Side
Side View
Front View
Front View
Test Dataset
Front + Side
Side View
Front View
Side View
Feature
Baseline
Ours
Basic
Ours
Basic
Ours
Basic
Ours
Basic
Lexical Sign
3.7
49.78
25.32
37.27
16.36
39.02
21.14
32.73
8.18
Signer
8.3
72.64
66.04
83.33
78.57
62.07
40
25
Prefecture
25
50
57.35
56.25
45.45
56.76
66.67
34.38
30.3
Deaf School
16.67
48.05
46.75
54.05
45,95
55
43,9
35.13
29.73
Age Group
16.67
49.12
42.1
44.44
51.72
45.16
39.29
33.33
24.14
Gender
50
76.88
70.52
71.61
76.83
76.34
78.02
75.31
57.32
Table 2: Comparison of the normalization methods in terms of JSL feature classification performance.
Pose- based
Frame+ Backbone
Model
Top
Top 5
Top 10
Li et al. (2020a)
65.89
84.11
89.92
Li et al. (2020b)
77.55
91.42
Hosain al. (2021)
75.67
86.42
90.16
Maruyama al. (2021)
81.38
94.13
96.05
Tunga et al. (2021)
60.15
83.98
88.67
Boháček and Hrúz (2022)
63.18
Maruyama et al. (2021)
71.07
90.13
92.42
Naz et al. (2023)
72.09
88.76
92.64
Ours (basic norm) Agv, Std
75.85(+1.11)
92.75(+:.6)
95.27(++0.18)
Ours (basic norm) Max
77.51
93.8
96.
Ours (3D norm) Avg, Std
84.26(+0.82)
95.66(+00442)
96.86(+0.277)
Ours (3D norm) Max
85.27
96.12
97.28
Table The comparison of accuracy scores on the WLASL--100 test data. We report the performance of our model with the proposed and basic normalization method.
perts can apply their knowledge of language prop- erties and the proposed method to uncover novel features. Nevertheless, our results raise a concern about signer privacy protection since stylistic fea- tures of individual signers can be predicted based solely signing movement.
Limitations
The proposed representation method can be used for any direeddimnssonal pose estimation. How- ever, the proposed normalization method relies en- tirely on the initial data recording quality and esti- mation accuracy of MediaPipe and is incompatible with two-dimensional pose estimation methods like OpenPose. Our normalization method recalculates the value of coordinate but relies on McdiaPipe's s depth estimations to determine the order of the final coordinates. Even under ideal conditions, ac- counting for body proportions is difficult since the normalization method assumes all humans have the same body proportions. It may lead to instances where the hands detect an important sign language feature.
The deep-leaming WLASL-100 benchmark re- sults are consistent with the JSL dataset tests. Our method significantly outperforms other pose-based methods and successfully competes with multi- modal approaches. Sign language is naturally con- veyed through body movement; extracting it from the collected video data improves performance and robustness.
Our method performs consistently well across all data sets. We satisfied the initial requirements outlined in Section and addressed the generaliza- tion issue discussed in Section 1. The proposed methods are suitable for any sign language, and multiple sign languages can be encoded into one signing space, thus facilitating cross- laosuge stud- ies in future research.
ideally touching, failing
Processing some facial expressions, mouth ges- tures, and mouthing is limited and requires addi- tional modalities (e.g., pixel data). Still, the de- tected facial key points can provide aid in pixel extraction.

---

Ethics Statement
with weighted dynamic time warping. In Proceed- ings of the International Conference on Computer Vision Theory and Applications Volume I: VIS- APP, (VISIGRAPP 2013), pages 620-625. INSTICC, SciTePress.
In our work, we used anonymized pose skeleton data extracted from the published Japanese Sign Language Corpus (Bono et al., 2014, 2020) in ac- cordance with the ACL Ethies Policy. However, one of the work's conclusions proposes that a per- son can be identified, to a limited degree, using their signing style. We encourage the research com- munity to pay closer attention to the possible de- velopment of future authentification systems using movement habits.
T. Cover and P. Hart. 1967. Nearest neighbor pattern classification. JEEE Transactions on Information Theory, 13(1):21-27.
Amanda Duarte. Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. 2021. How2sign: A large-scale multimodal dataset for continuous ameri- can sign language. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 2735-2744.
References
Rong-En En Fan, Kai-Wei Chang, Cho-Jui Hsich, Xiang- Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A library for large linear classification. the Journal of machine Learning research, 9:1871-1874.
Matyáš Boháček and Marek Hrúz. 2022, Sign pose- based transformer for word-level sign language recog- nition. In 2022 IEEF/CVP Winter Conference on Ap- plications of Computer Vision Workshops (WACVW), pages 182-191.
Manolis Fragkiadakis. Victoria Nyst, and Peter van der Putten. 2020. Signing as input for dictionary query: Matching signs hased on joint positions of the dom- inant hand. In Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pages 69- 74, Marseille, France. European Language Resources Association (ELRA).
Mayumi Bono, Kouhei Kikuchi, Paul Cibulka, and Yu- taka Osugi. 2014. A colloquial corpus of Japanese Sign Language: Linguistic resources for observ- ing sign language conversations. In Proceedings of the Ninth International Conference on Language Revources and Evaluation (IRREC'14), pages 1898- 1904, Reykjavik, Iceland. European Language Re- sources Association (ELRA).
Mayumi Bono, Rui Sakaida, Tomohiro Okada, and Yusuke Miyao. 2020. Utterance-unit annotation for the jsl dialogue corpus: Toward a multimodal ap- proach to corpus linguistics. In Proceedings of the 9th Workshop on the Representation and P Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pages 13- 20, Marseille, France. European Language Resources Association (ELRA).
Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. 2004. Neighbourhood com- ponents analysis. In Advances in Neural Information Processing Systems, volume 17, MIT Press.
Anmol Gulati, James Qin, Chung- -hhnng Chiu, Niki Parmar, Yu Zhang. Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmeett former for Speech Recognition. pages 5036-5040.
Trans-
Irving P. Herman. 2016. Terminology, the Standard Hu- man, and Scaling, pages 1-38. Springer International Publishing, Cham.
L. Breiman. Jerome H. Friedman. Richard A. Olshen, and C. J. Stone. 1984. Classification and regression trees. Biometrics, 40:874.
Al Amin Hosain, Panneer Selvam Santhalingam, Parth Pathak, Huzefa Rangwala, and Jana Košecká. 2021. Hand pose guided 3d pooling for word-level sign lan- guage recognition. In 202/ IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1428-3438.
Heike Brock, Felix Law, Kazuhiro Nakadai, and Yuji Nagashima. 2020. Learning three-dimensional skele- ton data from sign language video. ACM Trans. Intell. Syst. Technol. 11(3).
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation us- ing part affinity fields. CVPR.
Xiaopeng Ji, Qi Fang. Junting Dong, Qing Shuai, Wen Jiang, and Xiaowei Zhou. 2020. A survey on monoc- ular 3d human pose estimation. Virtual Reality & Intelligent Hardware, 2(6):77
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2021. Openpose:
Realtime multi-person 2d pose estimation using part affinity fields. volume 43, pages 172-186.
Tao Jiang, Necati Cihan Camgöz, and Richard Bow- den. 2021. Skeletor: Skeletal transformers for robust body-pose estimation. In 2021 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3389-3397.
Sait Celebi. Talha T. Temiz, Ali S. Aydin, and Tarik Arici. 2013. Gesture recognition using skeleton data

---

Diederik P. Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. In 3rd Inter- national Conference ICLR 201S, San Diego CA. USA, May 7-9, 2015, Conference Track Proceedings.
Arezoo Sadeghzadch and Md Baharul Islam. 2022. Triplet besse convolutional neural network for static sign language recognition. 2022 Innovations in Intelligent Systems and Applications Conference (ASYU), pages 1-6.
Learning Representations,
Dongxu Li, Cristian Rodriguez-Opaza, Xin Yu, and Hongdong Li. 2020a. Word-level deep sign language recognition from video: new large-scale dataset and methods comparison. 2020 IEEE Winter Con- ference on Applications of Computer Vision (WACV), pages 1448-1458.
Pascal Schneider, Raphael Memmesheimer, Ivanna Kramer, and Dietrich Paulus. 2019. Gesture recogni- tion in rgb videos using human body keypoints and dynamic time warping. In RoboCup 2019: Robot World Cup XXIII, pages 281-293, Cham. Springer International Publishing.
Dongxu Li, Xin Yu, Chenchen Xu, Lars Petersson, and Hongdong Li. 2020b. Transferring cross-domain knowledge for video sign language recognition. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 6213-6213.
Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. 2021. Fingerspelling detection in american sign language. 2021 TEEÉ/CVF Confer- ence on Computer Vision and Puttern Recognition (CVPR), pages 4164-4173.
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong. Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. 2019. Medi- apipe: A framework for building perception pipelines. ArXiv,
Anirudh Tunga, Sai Vidyaranya Nuthalapati. and Juan Wachs. 2021. Pose- based-baeddd sign language recognition using gen and bert. 2021 IEEE Winter Confer- ence on Applications of Computer Vision Workshops (WACVW), pages 31-40.
Harry Zhang. 2004. The optimality of naïve bayes. In Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2004).
Fiona Marshall, Shuai Zhang, and Bryan Scotney. 2019. Comparison activity recognition using 2d and 3d skeletal joint data. In Irish Machine Vision and Image Processing Conference proceedings, page 13. Irish Pattern Recognition and Classification Society.
A Appendix
Mizuki Maruyama, S. Ghose, Katsufumi Inoue, Partha Pratim Roy, M. Iwamura, and Michifumi Yoshioka. 2021. Word-level sign language recog- nition with multi-stream neural networks focusing on local regions. ArXiv, abs/2106.1599 15989.
Amit Moryossef, Toannis Tsochantaridis, Joe Dinn, Necati Cihan Camgoz, Richard Bowden, Tao Jiang, Annette Rios, Mathias Muller, and Sarah Ebling. 2021. Evaluating
immediate applicability of pose estimation for sign language recognition. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition (CVPR) Workshops, pages 3434-3440.
Neelma Naz, Hasan Sajid, Sara Ali, Osman Hasan, and Muhammad Khurram Ehsan. 2023. Signgraph: An efficient and accurate pose-base based graph convolution approach toward sign language recognition. IEEE Access, pages 19135-19147.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 2830.
Siegmund Prillwitz, Regina Leven, Heiko Zienert, Thomas Hanke, and Jan Henning. 1989. Ham- NoSys version 2.0. Hamburg notation system for sign languages- an introductory guide. Seanum-Verlgg

---

Lexical Sign
"to make"
Signer ID
"swoet"
Ab
33
BB
3.75
V
21
BO
PR
"now"
2b
17
"middle"
S
Deaf School
Prefecture
Group
Gender
15
Figure 6: The NCA embedding visualization of various Japanese sign language corpus features,
top-1
top-5
top-10
80
70
Accurscy
Accursicy
20
Normalization
Normalization
Normalization
10
Dasic
Dasic
Dasic
30
40
80
120
160
200
40
80
120
260
200
40
120
160
200
Epochs
Eppcch
Epochs
Figure 7: Accuracy (%) on top (left), top (middle), and top 10 (right) on the test dataset of WLASL-100 dataset during training.
Preprocessed sample:
Propused
Proposed
Proposed
Basic
Basic
Basic
(22 9 2 hands joints
+ 9 body joints)
3 dimensions
159 values over 100 frames
Proposed
Basic
Normalized input samples Start and end frames cuts Resized to 100 frames
Facial and relative rows removed
Figure 8: WLASL preprocessing: to simplify the data, we focus only on the parts of the sample where hands are visible. Then, we resize the data 100 frames eliminate facial and relative joint information to reduce the complexity.

---

Train Dataset
Front + Side
Side View
Front View
Front View
Test Dataset
Front Side
Side View
Front View
Side View
Ours Basic
Ours
Basic
Ours
Basic
Ours
Basic
Lexical Sign
Baseline:
3.704
Number of Classes:
Samples per Class:
Nearest Neighbors
30.9
12.02
27.27
15.46
28.45
12.2
27.27
5,46
Linear SVM
49.78
25.32
37.27
14.54
39.02
21.14
32.73
3.64
Decision Tree
19.74
17.17
25.45
16.36
23.58
20.32
18.18
8.18
Naive Bayes
24.03
11.16
19.09
12.73
29.27
12.2
16.36
4.54
Best
49.78
25.32
37.27
16.36
39.02
21.14
32.73
8.18
Signer ID
Baseline:
8.333
Number of Classes:
12
Samples per Class:
43
Nearest Neighbors
41.51
45.28
50
39.58
37.5
36.21
40
8.33
Linear SVM
72.64
66.04
70
56.25
64.29
50
34
14.58
Decision Tree
50.94
56.6
62
54.17
50
62.07
20
25
Naive Bayes
66.98
39.62
72
83.33
78.57
51.72
14
16.67
Best
72.64
66.04
72
83.33
78.57
62.07
40
25
Prefecture
Baseline:
25
Number of Classes:
Samples per Class: 84
Nearest Neighbors
32.35
57.35
45.45
37.84
41.67
21.88
21.21
Linear SVM
50
39.71
50
27.27
51.35
66.67
21.88
30.3
Decision Tree
30.88
39.71
56.25
33.33
32.43
55.56
15.62
15.15
Naive Bayes
38.23
33.82
50
33.33
56.76
63.89
34.38
21.21
Best
50
57.35
56.25
45.45
56.76
66.67
34.38
30.3
Deaf School
Baseline:
16.667
Number of Classes:
Samples per Class: 65
Nearest Neighbors
33.77
46.75
40.54
43.24
41,46
27.03
18.92
Linear SVM
42.86
46.75
48.65
45.95
42.5
43.9
29.73
29.73
Decision Tree
28.57
32.47
40.54
32.43
37.5
31.71
35.13
21.62
Naive Bayes
48.05
36.36
54.05
40.54
55
29.27
21.62
18.92
Best
48.05
46.75
54.05
45.95
55
43.9
35.13
29.73
Age Group
Baseline:
16.667
Number of Classes:
Samples per Class:
47
Nearest Neighbors
31.58
28.07
18.52
37.93
32.26
39.29
33.33
17.24
Linear SVM
45.61
42.1
29.63
37.93
38.71
28.57
22.22
24.14
Decision Tree
31.58
35.09
33.33
27.59
35.48
32.14
25.93
10.35
Naive Bayes
49.12
26.32
44.44
51.72
45.16
39.29
14.81
20.69
Best
49.12
42.1
44.44
51.72
45.16
39.29
33.33
24.14
Gender
Baseline:
50
Number of Classes:
Samples per Class:
423
Nearest Neighbors
76.88
70.52
71.61
62.2
70.97
60.44
61.73
50
Linear SVM
69.36
68.79
71.61
76.83
68.82
78.02
75.31
57.32
Decision Tree
65.32
67.05
70.37
67.07
75.27
64.83
45.68
39.02
Naive Bayes
65.32
61.85
54.32
71.95
76.34
58.24
61.73
48.78
Best
76.88
70.52
71.61
76.83
76.34
78.02
75.31
57.32
Table 4: Comparison of the performance of the various standard classification methods for JSL feature prediction.


Reviews:
[{"review_id": "aTUYdAxGfF", "reviewer": "Reviewer_ynFA", "summary": "", "questions": "Was there a reason to choose only one sign language to highlight the risks of movement habits present in pose features? I believe the claim is Sign language agnostic and would be better to validate on another sign language. I understand due to dataset availability and other resource constraints; this might be out of the scope of this work. However, I was curious if such cases exist across multiple sign languages. It would be great if you could provide some insights along with a reason to choose JSL specifically for this analysis.\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comments on Reasons To Reject:\n\n    Was there a reason to use only the WLASL-100 set to report the improvements? It would be better to report the performances obtained over all the settings in WLASL, including WLASL2000. The results over multiple would make the results more reliable and the findings more transparent and valuable for the research field.\n\nComment: \n\nFrankly, this paper was submitted before without deep learning testing. Our point is that the proposed methods are, first and foremost, for humans, hence the article's name. The WLASL-100 is the subset testing that was included after we faced the reality of today's state of research: it has to be beneficial for deep learning algorithms. There are many reasons why benchmarks do not represent the real task, but here are some that are important for us. The isolated nature of the sign recording is not informative for linguists because it rarely (almost never) occurs in the real sign language conversation. The mislabeling in the WLASL dataset itself (source: Dafnis, Konstantinos & Chroni, Evgenia & Neidle, Carol & Metaxas, Dimitris. (2022). Isolated Sign Recognition using ASL Datasets with Consistent Text-based Gloss Labeling and Curriculum Learning. Section 1.1 http://www.lrec-conf.org/proceedings/lrec2022/workshops/sltat/pdf/2022.sltat-1.3). The main reason is that we do not intend to get on the competition's leaderboard but rather address the issue of the movement's inaccessibility for linguists. We emphasize linguists because data scientists and computer vision engineers can always come up with a way of extracting the needed features for the required task (I3D, VGG16, and others) that only they can understand and use. \n\n    The Japanese Sign Language dataset done for less number of samples may need a re-verification for other datasets, making the movement habits claim to be applicable for less number of samples. A sufficiently large dataset may marginalize such noise, leading to generalized feature representations obtained from poses.\n\nComment: \n\nYou have stated is your self, below: \" I understand due to dataset availability and other resource constraints; this might be out of the scope of this work.\"\n\nWe only found that pattern interesting and reported on it; the lack of gender-related movement patterns has to be further studied. This is out of the scope of this work.\n\nQuestions For The Authors:\n\n    Was there a reason to choose only one sign language to highlight the risks of movement habits present in pose features? I believe the claim is Sign language agnostic and would be better to validate on another sign language. I understand due to dataset availability and other resource constraints; this might be out of the scope of this work. However, I was curious if such cases exist across multiple sign languages. It would be great if you could provide some insights along with a reason to choose JSL specifically for this analysis.\n\nJSL Corpus was chosen because it has the following properties: Unlike other sign languages, JSL is the sign language that does not have a unified and standardized grammar or the way it is taught. Each deaf school has its own understanding of what JSL is. The JSL Corpus is collected in different prefectures and is very diverse in the JSL style. The JSL Corpus is continued and dialog-based, where the conversations are unrestricted. The main reason we had access to the corpus and linguists who are working with it. We hope that other laboratories will use our methods to report their results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper makes a novel contribution by highlighting the effectiveness of posed-based sign language recognition methods and devices, a new normalization specifically for sign language processing. With all the details provided in detail, the method becomes more reliable and would be useful for the sign language processing research field.\n\n* The paper reports significant performance improvement over the entire training trajectory, highlighting the performance boost with the proposed normalization technique.\n\n* The analysis of Japanese Sign Language is detailed and would be helpful for the research community.\n\n\n", "reasons_to_reject": "* Was there a reason to use only the WLASL-100 set to report the improvements? It would be better to report the performances obtained over all the settings in WLASL, including WLASL2000. The results over multiple would make the results more reliable and the findings more transparent and valuable for the research field.\n\n* The Japanese Sign Language dataset done for less number of samples may need a re-verification for other datasets, making the movement habits claim to be applicable for less number of samples. A sufficiently large dataset may marginalize such noise, leading to generalized feature representations obtained from poses. \n\n", "questions_to_authors": "Was there a reason to choose only one sign language to highlight the risks of movement habits present in pose features? I believe the claim is Sign language agnostic and would be better to validate on another sign language. I understand due to dataset availability and other resource constraints; this might be out of the scope of this work. However, I was curious if such cases exist across multiple sign languages. It would be great if you could provide some insights along with a reason to choose JSL specifically for this analysis.\n\n", "paper_topic_and_main_contributions": "The paper proposes a new Normalization technique designed specifically for sign language processing. The primary contribution of the proposed method is in improving the quality of the z dimension (depth) and capturing movement via normalized baseline. The paper further validates the quality of the proposed normalization technique via detailed analysis and performance improvement on the WLASL dataset in a 100-class setting. In addition, the paper also provides an analysis of sociolinguistic features present in movement patterns present in Japanese Sign Language.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ci1yJQrelM", "reviewer": "Reviewer_2kwG", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comments on Reasons To Reject: \n\n    It remained unclear how robust the proposed normalization approach worked for the dataset where singers can have different physical conditions. The testbed here (collected by the authors) did not have that many variations.\n\nComment: \n  It remained unclear what was meant under \"different physical conditions.\" How does that invalidate the method or the results?\n\n    Figure 5 did not help much with the model explanation.\n\nComment: \n  The mentioned Figure 5 includes all model components with all the parameters. The explanation of the modules is out of the scope of the work; the used Conformer model is explained in the referenced paper: Gulati, Anmol, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu and Ruoming Pang. “Conformer: Convolution-augmented Transformer for Speech Recognition.” ArXiv abs/2005.08100 (2020)\n\n    Several experimental details are missed, especially the approaches related to the normalization of joints.\n\nComment: \n  Sections 2.1 \"3D Normalization\" and 2.2 \"Scaling and Rotation\" explain the approaches related to the normalization of joints in detail.\n\nQuestion to the reviewer: \n\nObviously, you are lowering the score and deliberately coming up with reasons for it. Do you act in the interest of the research community by doing this? Do you act in your own interest?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-The normalization approaches are carefully adjusted based on observations and are helpful by sharing the details.\n- The work aims to address a critical problem in the sign language processing field.\n- Experiments demonstrated that the proposed pose processings help the model to learn about the features and assist in the downstream tasks.", "reasons_to_reject": "- It remained unclear how robust the proposed normalization approach worked for the dataset where singers can have different physical conditions. The testbed here (collected by the authors) did not have that many variations.\n- Figure 5 did not help much with the model explanation. \n- Several experimental details are missed, especially the approaches related to the normalization of joints.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes a novel three-dimensional normalization method for MediaPipe, as well as the novel movement representation method to model the movement of sign languages better. This is important as the 2D images can cause information loss and suppress the 3D movement understanding. The authors further demonstrate the effectiveness of the proposed approach on a small JSL dataset with care analysis, as well as the open-sign language recognition task.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DX5QvlB5bs", "reviewer": "Reviewer_iboj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comments on Reasons To Reject:\n\n    the use of a single \"standard\" body measurement for reference and normalization seems like a weakness and also raises ethical concerns. Learners working on this representation may suffer from poorer accuracy on body types not matching the \"chosen standard\". This is analogous to for example how facial recognition systems trained on faces of specific types misidentify faces of others leading to significant harms. I strongly urge the authors to consider moving away from a single standard for normalization. This is not discussed as a concern by the authors in their ethics section.\n\nComment: \n\nThank you. We have been considering addressing it. However, we did not have a proper scientific background to do so correctly and effectively since this is not our field. The following work may be used as guidance on the topic: Herman, I.P. (2016). Terminology, the Standard Human, and Scaling. In: Physics of the Human Body. Biological and Medical Physics, Biomedical Engineering. Springer, Cham. https://doi.org/10.1007/978-3-319-23932-3_1, https://link.springer.com/chapter/10.1007/978-3-319-23932-3_1 \n\nThe body proportions can be recalculated, of course. If you look at the code, we use it as a dictionary, which can be adjusted. \n\nThis is a thoughtful comment; however, is it a reason to reject the work entirely? \n\n    some steps are not adequately described, such as how scaling and rotation are done.\n\nComment: \n\nWe have a dedicated Section 2.2 \"Scaling and Rotation\" where we describe in detail how the scaling and rotation are done. We use linear transformation (a basic, widely known and used technique in any 3D operations, some course, not ours: https://cseweb.ucsd.edu/classes/wi18/cse167-a/lec3.pdf) \n\n    the figures are not helpful to illustrate the operations. Only the end result is shown not what is done to get there.\n\n    it was not clear to me how z coordinates were being calculated. \n\nComment: \n\nline 215: we calculate the lengths of all body joints in 3D using the Formula 1, where... \n\nLenght - is the length in 3D space, that we approximate using the body proportions (Formula 1 , line 224).  \n\nline 219: We calculate the z coordinate using the length, and relative x and y coordinate with the origin in the \"parent joint\"... \n\nUsing the ''Pythagorean theorem\" : \nlength^2 = x^2 + y^2 + z^2 -> z = square root(length^2 - x^2 + y^2)\n\nline 222: Lastly, we apply the sign (plus or minus) value from MediaPipe’s original z estimation to it.  -- Because it is a square root, z could be minus or plus, and we trust the MediaPipe's estimation on it (plus or minus). \n\nThe explanation did not mention the application of the Pythagorean theorem, which is true. You can find these operations it the code, along with rotation and scaling. \n\nGeneral comment: \n\nThe body proportions can be changed, but I just had to start somewhere and not argue to use it as a standard. We will expand the section where we introduce the body proportions and include the concerns you have brought up in the ethics section as well; we highly appreciate your comment. \n\nThe whole process is two simple formulas, that simple. The rotation and scaling are performed with simple, straightforward matrix multiplication. Those are basic middle school and high school math operations, and we can see how it can be unexpected. If it is so simple, why is there a paper about it in 2023? The missing ingredients are the length calculation in 3D, where we propose to use proportions for it, and the initial z estimations of MediaPipe, which makes it possible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ addressing a very important problem\n+ generally very clearly written description\n+ very detailed normalization scheme taking into account 3D body shapes and a variety of measurements\n+ good preliminary experimental data suggesting the scheme is beneficial", "reasons_to_reject": "- the use of a single \"standard\" body measurement for reference and normalization seems like a weakness and also raises ethical concerns. Learners working on this representation may suffer from poorer accuracy on body types not matching the \"chosen standard\". This is analogous to for example how facial recognition systems trained on faces of specific types misidentify faces of others leading to significant harms. I strongly  urge the authors to consider moving away from a single standard for normalization. This is not discussed as a concern by the authors in their ethics section.\n- some steps are not adequately described, such as how scaling and rotation are done.\n- the figures are not helpful to illustrate the operations. Only the end result is shown not what is done to get there.\n- it was not clear to me how z coordinates were being calculated.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Making Body Movement in Sign Language Accessible\n\nIn this paper the authors describe a detailed normalization pipeline to convert annotated sign language images into a standard format. They show that learning algorithms working on their format are able to do better on sign language tasks than alternative normalization schemes.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main strength of the paper is that its contribution is important to processing related specifically to sign language, where there has been less work relative to spoken or written language. The reviewers also found the paper clear and well written. Room for improvement include adding the needed details for the method to be reproducible. Some reviewers also thought that the method should be tested on a larger available dataset.
Decision: Accept-Findings