Linguistically Motivated Sign Language Segmentation
Authors: Amit Moryossef
Zifan Jiang
Mathias Müller
Sarah Ebling
Yoav Goldberg
Venue: EMNLP 2023 Findings (2023)
Keywords: sign language
sign language segmentation
TL;DR: -

Abstract: Sign language segmentation is a crucial task in sign language processing systems. It enables downstream tasks such as sign recognition, transcription, and machine translation.
In this work, we consider two kinds of segmentation: segmentation into individual signs and segmentation into \textit{phrases}, larger units comprising several signs. We propose a novel approach to jointly model these two tasks.

Our method is motivated by linguistic cues observed in sign language corpora. We replace the predominant IO tagging scheme with BIO tagging to account for continuous signing. Given that prosody plays a significant role in phrase boundaries, we explore the use of optical flow features. We also provide an extensive analysis of hand shapes and 3D hand normalization.

We find that introducing BIO tagging is necessary to model sign boundaries. 
Explicitly encoding prosody by optical flow improves segmentation in shallow models, but its contribution is negligible in deeper models.
Careful tuning of the decoding algorithm atop the models further improves the segmentation quality.

We demonstrate that our final models generalize to out-of-domain video content in a different signed language, even under a zero-shot setting.
We observe that including optical flow and 3D hand normalization enhances the robustness of the model in this context.

Linguistically Motivated Sign Language Segmentation
Amit Moryossef 1.2 Zifan Jiang" Mathias Müller?, Sarah Ebling?, Yoav Goldberg' Blar-Ilan University, 'University of Zurich ami toooooseee@@@aaaaii com, jiangêcl. uzh. ch
Abstract
B
B
B
0
0
sign
sign 2
sign 3
Sign language segmentation crucial task in sign language processing systems. It en- ables downstream tasks such as sign recogni- tion, transcription, and machine translation. In this work, we consider two kinds of segmen- tation: segmentation into individual signs and segmentation into phrases, larger units compris- ing several signs. We propose novel approach to jointly model these two tasks.
phrase
0 0
Figure I: Per-frame classification of sign language utterance following BIO tagging scheme. Each box represents a single frame a video. propose a joint model segment signs (top) and phrases (bottom) the same time. B=beginning, I=inside, O=outside. The figure illustrates continuous signing where signs often follow each other without an O frame between them.
Our method motivated by linguistic cues ob- served in sign language corpora. We replace the predominant TO tagging scheme with BIO tag- ging to account for continuous signing. Given that prosody plays significant role in phrase boundaries, we explore the use of optical flow features. also provide an extensive analysis of hand shapes and 3D hand normalization.
Sign language transcription and translation sys- tems rely on the accurate temporal segmentation of sign language videos into meaningful units such as signs (Santemiz et al., 2009; Renz et al., 2021a) signing sequences corresponding subtitle units! (Bull al., 2020b). However, sign language seg- mentation remains a challenging task due the difficulties in defining meaningful units in signed languages (De Sisto et al., 2021). Our approach is the first to consider two kinds of units one model. We simultaneously segment single signs and phrases (larger units) in a unified framework. Previous work typically approached segmenta- tion as binary classification task (including seg- mentation tasks in audio signal processing and com- puter vision), where each frame/pixel predicted to either part segment or not. However, this approach neglects the intricate nuances of contin- uous signing, where segment boundaries are not strictly binary and often blend in reality. One sign or phrase can immediately follow another, transi- tioning smoothly, without frame between them being distinctly outside (Figure and $3.1).
We find that introducing BIO tagging nec- essary to model sign boundaries. Explicitly encoding prosody by optical flow improves seg- mentation in shallow models, but its contribu- tion is negligible in deeper models. Careful tuning of the decoding algorithm atop the mod- els further improves the segmentation quality.
We demonstrate that our final models gener- alize to out-of-domain video content in dif- ferent signed language, even under zero-shot setting. We observe that including optical flow and 3D hand normalization enhances the ro- bustness model this context.
1 Introduction
Signed languages are natural languages used by deaf and hard-of-hearing individuals to commu- nicate through a combination of manual and non- manual elements (Sandler and Lillo-Martin 2006). Like spoken languages, signed languages have their own distinctive grammar, and vocabulary, that have evolved through natural processes of language de- velopment (Sandler, 2010).
We propose incorporating linguistically moti- vated cues to address these challenges and improve sign language segmentation. To cope with contin-
Subtitles may not always correspond directly to sentences. They frequently split within a sentence and could be tempo- rally offset from the corresponding signing segments.
"Equal contribution authors.

---

The annotation of the first phrase in video from the test set (dgskorpus _oe___ge__2_______ in yellow, signing: "Why do you smoke?" through the use three signs: WHY (+mouthed), TO-SMOKE, and gesture (+mouthed) towards the other signer. At the top, phrase segmentation model predicts single phrase that initiates with B tag (in green) above the B-threshold (green dashed line), followed by an (in light blue), and continues until falling below a certain threshold. At the bottom, our sign segmentation model accurately segments the three signs.
Figure 2:
uous signing, we adopt a BIO-tagging approach (Ramshaw and Marcus, 1995), where in addition to predicting a frame to be in or out of segment, also classify the beginning of the segment as shown in Figure 2. Since phrase segmentation primarily marked with prosodic cues (i.e., pauses, extended sign duration, facial expressions) (Sandler, 2010; Ormel and Crasborn, 2012), we explore using op- tical flow to explicitly model them ($3.2). Since signs employ a limited number of hand shapes, we additionally perform 3D hand normalization ($3.3). Evaluating on the Public DGS Corpus (Prillwitz al., 2008; Hanke et al., 2020) (DGS stands for German Sign Language), we report enhancements in model performance following specific modifica- tions. We compare our final models after hyperpa- rameter optimization, including parameters for the decoding algorithm, and find that our best architec- ture using only the poses is comparable the one that uses optical flow and hand normalization. Reassuringly, we find that our model generalizes when evaluated on additional data from different signed languages in a zero-shot approach. We ob- tain segmentation scores that are competitive with previous work and observe that incorporating opti- cal flow and hand normalization makes the model robust for nut-of-domall data.
Related Work
2.1 Sign Language Detection
Sign language detection (Borg and Camilleri, 2019; Moryossef et al., 2020; Pal al., 2023) is the task of determining whether signing activity is present in a given video frame. A similar task in spoken languages is voice activity detection (VAD) (Sohn et al., 1999; Ramurez et al., 2004), the detection of when human voice is used in an audio signal. As VAD methods often rely on speech-specific rep- resentations such as spectrograms, they are not necessarily applicable to videos.
Borg and Camilleri (2019) introduced the classi- fication of frames taken from YouTube videos as ei- ther signing or not signing. They took spatial and temporal approach based on VGG-16 16 (Simonyan and Zisserman, 2015) CNN to encode each frame and used Gated Recurrent Unit (GRU) (Cho al., 2014) encode the sequence of frames in win- dow of 20 frames at 5fps. In addition the raw frame, they either encoded optical-flow history, ag- gregated motion history, or frame difference.
Moryossef et al. (2020) improved upon their method by performing sign language detection in real time. They identified that sign language involves movement of the body and, such, de- signed a model that works on top of estimated hu- man poses rather than directly on the video signal, They calculated the optical flow norm of every joint detected on the body and applied a shallow yet ef- fective contextualized model to predict for every frame whether the person is signing or not.
Lastly, we conduct an extensive analysis of pose- based hand manipulations for signed languages (Appendix C). Despite not improving our segmen- tation model due to noise from current 3D pose esti- mation models, we emphasize its potential value for future work involving skeletal hand poses. Based this analysis, directions for improving 3D pose estimation.
propose several measurable
While these recent detection models achieve high performance, we need well-annotated data including interference and non-signing distractions for proper real-world evaluation. Pal et al. (2023) conducted a detailed analysis of the impact
Our code and models are available at https: //github. transcription.

---

signer overlap between the training and test sets on two sign detection benchmark datasets (Sign- ing the Wild (Borg and Camilleri, 2019) and the DGS Corpus (Hanke et al., 2020)) used by Borg and Camilleri (2019) and Moryossef et al. (2020). By comparing the accuracy with and without over- lap. they noticed a relative decrease in performance for signers not present during training. As a result, they suggested new dataset partitions that eliminate overlap between train and test sets and facilitate a more accurate evaluation of performance.
aries between signs in continuous sign language videos by employing 3D convolutional neural net- work representations with iterative temporal seg- ment refinement to resolve ambiguities between sign boundary cues. Renz et al. (2021b) fur- ther proposed the Changepoint Moduulted Pscudo- Labelling (CMPL) algorithm to solve the problem of source-free domain adaptation.
Bull al. (2021) presented a Transformer-basee approach to segment sign language videos and align them with subtitles simultaneously. encod- ing subtitles by BERT (Devlin et al., 2019) and vidcos by CNN video representations.
2.2 Sign Language Segmentation
Segmentation consists of detecting the frame boundaries for signs or phrases videos to di- vide them into meaningful units. While the most canonical way of dividing spoken language text is into linear sequence of words, due to the si- multaneity of sign language, the notion of a sign language "word" is ill-defined, and sign language cannot be fully linearly modeled.
Motivating Observations
To motivate our proposed approach, we make a series of observations regarding the intrinsic na- ture of sign language expressions. Specifically, we highlight the unique challenges posed by the contin- uous flow of sign language expressions ($33.1,, the role of prosody in determining phrase boundaries ($3.2), and the influence of hand shape changes in indicating sign boundaries ($3.3).
Current methods resort to segmenting units loosely mapped to signed language units (Santemiz et al., 2009; Farag and Brock, 2019; Bull et al., 2020b; Renz et al., 2021a,b; Bull et al., 2021) and do not explicitly leverage reliable linguistic pre- dictors of sentence boundaries such as prosody in signed languages (i.e.. pauses, extended sign du- ration, facial expressions) (Sandler, 2010; Ormel and Crasborn, 2012). De Sisto et al. (2021) call for a better understanding of sign language struc- ture, which they believe is the necessary ground for the design and development of sign language recognition and segmentation methodologies.
3.1 Boundary Modeling
When examining the nature of sign language ex- pressions, we note that the utterances are typically signed in a continuous flow, with minimal to no pauses between individual signs. This continuity is particularly evident when dealing with lower frame rates. This continuous nature presents a signifi- cant difference from text where specific punctua- tion marks serve as indicators of phrase boundaries, and semi-closed set of tokens represent the words.
Santemiz et al. (2009) automatically extracted isolated signs from continuous signing by align- ing the sequences obtained via speech recognition, modeled by Dynamic Time Warping (DTW) and Hidden Markov Models (HMMs) approaches. Farag and Brock (2019) used a random forest classifier to distinguish frames containing signs in Japanese Sign Language based on the composition of spatio-temporal angular and distance features between domain- speciifc pairs of joint segments.
Given these characteristics, directly applying conventional segmentation or sign language de- tection models--hatt is, utilizing IO tagging in a manner similar to image or audio segmentation models-may not yield the optimal solution, par- ticularly at the sign level. Such models often fail to precisely identify the boundaries between signs. A promising alternative is the Beginning-Inside- Outside (BIO) tagging (Ramshaw and Marcus, 1995). BIO tagging was originally used for named entity recognition, but its application extends to any sequence chunking task beyond the text modality. In the context of sign language. BIO tagging pro- vides more refined model for discerning bound- aries between signs and phrases, thus significantly improving segmentation performance (Figure 1). To test the viability of the BIO tagging approach
Bull et al. (2020b) segmented French Sign Lan- guage into segments corresponding to subtitle units by relying on the alignment between subtitles and sign language videos, leveraging spatio-temporal graph convolutional network (STGCN; Yu et al. (2017)) with BiL.TMM on 2D skeleton data.
Renz et al. (2021a) located temporal bound-

---

in comparison with the 10 tagging, we conducted an experiment on the Public DGS Corpus. The en- tire corpus was transformed to various frame rates and the sign segments were converted to frames using either BIO or IO tagging, then decoded back into sign segments. Figure 4 illustrates the results of this comparison. Note that the 10 tagging was unable to reproduce the same number of segments the BIO tagging the gold data. This under- scores the importance of BIO tagging in identifying sign and phrase boundaries.
utilizing maximum of two hand shapes. For ex- ample, linguistically annotated datasets, such as ASL-LEX (Sehyr et al., 2021) and ASLLVD (Nei- dle et al., 2012), only record one initial hand shape per hand and one final hand shape. Mandel (1981, p. 87) argued that there can only be one set of se- lected fingers per sign, constraining the number of handshapes in signs. This limitation is referred to as the Selected Fingers Constraint. And indeed, Sandler al. (2008) find that the optimal form of sign is monosyllabie, and that handshape change is organized by the syllable unit.
100%
To illustrate this constraint empirically, show histogram of hand shapes per sign in SignBank for 705, 151 signs in Figure 5.
98%
96%
94%
BIO
60%
92%
10
40%
10
20
30
40
Figure 4: Reproduced sign segments in the Public DGS
20%
Corpus by BIO and IO tagging at various frame rates. 99.7% of segments reproduced 25fps by BIO tagging.
0%
0
Figure 5: Number of hand shapes per sign in SignBank.
3.2 Phrase Boundaries
Linguistic research has shown that prosody is re- liable predictor of phrase boundaries in signed lan- guages (Sandler, 2010; Ormel and Crasborn, 2012). We observe that this is also the in the Public DGS Corpus dataset used in our experiments. To illustrate this, we model pauses and movement us- ing optical flow directly on the poses as proposed by Moryossef et al. (2020). Figure demonstrates that change in motion signifies the presence of a pause, which, in turn, indicates phrase boundary,
Additionally, we found that a change in the dom- inant hand shape often signals the presence of a sign boundary. Specifically, out of 27, 658 sen- tences, comprising 354, 955 pairs of consecutive signs, only 17.38% of consecutive signs share the same base hand shape". Based on these observa- tions, we propose using 3D hand normalization as an indicative cue for hand shapes to assist detecting sign boundaries. We hypothesize that per- forming 3D hand normalization makes it easier for
3.3 Sign Boundaries
Ptttp:://sii /sigsbnkk org/si gtges/innpuddee..co ' is important to note that this percentage is inflated, as it may encompass overlaps across the dominant and non- dominant hands, which were not separated for this analysis.
We observe that signs generally utilize a limited number of hand shapes, with the majority signs
body
face
left
right
body
Figure 3: Optical flow for conversation between two signers (signer top, signer 2 bottom). The x-axis is the progression across 30 seconds. The yellow marks the annotated phrase spans. (Source: Moryossef et al. (2020))

---

the model to extract the hand shape. We expand on this process and show examples in Appendix C.
4.2 Methodology
Our proposed approach for sign language segmen- tation is based on the following steps:
4 Experimental Setup
1. Pose Estimation Given video, we first ad- just it 25 fps and estimate body poses using the MediaPipe Holistic pose estimation sys- tem. We do not use OpenPose because lacks a Z-xxis, which prevents 3D rotation used for hand normalization. The shape of pose is represented as (frames x keypoints x axes).
In this section, we describe the experimental setup used to evaluate our linguistically motivated ap- proach for sign language segmentation. This in- cludes description of the Public DGS Corpus dataset used in the study, the methodology em- ployed to perform sign and phrase segmentation. and the evaluation metrics used to measure the per-
formance of the proposed approach.
2. Pose Normalization To generalize over video resolution and distance from the camera, we normalize each of these poses such that the mean distance between the shoulders of each person equals 1, and the mid-point is at (0, 0) (Celebi et al., 2013). We also remove the legs since they are less relevant to signing.
4.1 Dataset
The Public DGS Corpus (Prillwitz et al., 2008; Hanke et al., 2020) is a distinctive sign language dataset that includes both accurate sign-level anno- tation from continuous signing, and well-aligned phrase-level translation in spoken language.
3. Optical Flow We follow the equation in Moryossef et al. (2020, Equation 1).
The corpus comprises 404 documents 714 videos4 with an average duration of 7.55 minutes, featuring either one signer or two signers, at 50 fps. Most of these videos feature gloss transcriptions and spoken language translations (German and En- glish), except for the ones in the "Joke" category, which are not annotated and thus excluded from our m The translations are comprised of full spoken language paragraphs, sentences, or phrases (i.e., independent/main clauses).
4. 3D Hand Normalization We rotate and scale each hand to ensure that the same hand shape is represented in consistent manner across different frames. We rotate the 21 XYZ key- points of the hand so that the back of the hand lies on the XY plane, we then rotate the hand so that the metacarpal bone of the middle fin- ger lies on the -axis, and finally, we scale the hand such that the bone is of constant length. Visualizations are presented in Appendix C.
Each gloss span considered a gold sign seg- ment, following a tight annotation scheme (Hanke et al., 2012). Phrase segments are identified by examining every translation, with the segment as- sumed to span from the start of its first sign to the end of its last sign, correcting imprecise annotation.
5. Sequence Encoder For every frame, the pose is first flattened and projected into standard dimension (256), then fed through LSTM encoder (Hochreiter and Schmidhuber, 1997).
This corpus is enriched with full-body pose esti- mations from OpenPose (Cao et al., 2019; Schul- der and Hanke, 2019) and Mediapipe Holistic (Gr- ishchenko and Bazarevsky, 2020). We use the 3. d.O-uen- split from Zhang et al. (2023). After filtering the unannotated data, we are left with 296 documents/ 583 videos for training, 6/ 12 for validation, and 9/ 17 for testing. The mean number of signs and phrases in a video from the training set is 613 and 1 111, respectively.
BIO Tagging On top of the encoder, we place two BIO classification heads for sign and phrase independently. B denotes the begin- ning of a sign or phrase, denotes the middle of sign or phrase, and O denotes being out- side sign or phrase. Our cross-entropy loss is proportionally weighted in favor of B as it is a rare label compared to and O.
Greedy Segment Decoding To decode the frame-level BTO predictions into sign/phrase segments, we define segment to start with the first frame possessing a B probability
'The number of videos is nearly double the number documents because each document typically includes two signers, each of whom produces one video for segmentation. "Wo also exclude documents with missing annotations. id E (1289910, 1245887, 1289868, 1246064, 1584617)
"B:1:0 is about 1:5:18 18 for signs and 1:58:77 for phrases.

---

greater than a predetermined threshold (de- faulted 0.5). The segment concludes with the first frame among the subsequent frames, having either a or O probability exceeding the threshold. We provide the pseudocode of the decoding algorithm in Appendix B.
6. E5: Autoregressive Encoder We replace the encoder with the one proposed by Jiang et al. (2023) for the detection and classification of great ape calls from raw audio signals. Specif- ically, we add autoregressive connections be- tween time steps to encourage consistent out- put labels. The logits at time step 1 are con- catenated to the input of the next time step, t 1. This modification is implemented bidi- rectionally by stacking two autoregressive en- coders and adding their output up before the Softmax operation. However, this approach is inherently slow, as we have to fully wait for the previous time step predictions before we can feed them to the next time step.
4.3 Experiments
Our approach evaluated through series of six sets of experiments. Each set repeated three times with varying random seeds. Preliminary experiments were conducted to inform the selec- tion of hyperparameters and features, the details of which can be found in Table 3 in Appendix A. Model selection is based validation metrics.
4.4 Evaluation Metrics
1. E0: 10 Tagger We re-implemented and re- produced" the sign language detection model proposed by Moryossef et al. (2020), in Py- Torch (Paszke et al., 2019) as naive baseline. This model processes optical flow as input and outputs (is signing) and (not signing) tags.
We evaluate the performance of our proposed ap- proach for sign and phrase segmentation using the following metrics:
Frame-level F1 Score For each frame, we apply the argmax operation to make local prediction of the BIO class and calculate the macro-averaged per-class FI score against the ground truth label. We use this frame-level metric during validation the primary metric for model selection and carly stopping, due to its independence from a potentially variable segment decoding algorithm ($5.2).
2. E1: Bidirectional BIO Tagger We replace the IO tagging heads in EO with BIO heads to form our baseline. Our preliminary experi- ments indicate that inputting only the hand and body keypoints and making the LSTM layer bidirectional yields optimal results.
3. E2: Adding Reduced Face Keypoints Al- though the 75 hand and body keypoints serve as an efficient minimal set for sign language detectionn/semmenttiioo models, we investi- gate the impact of other nonmanual sign lan- guage articulators, namely, the face. We intro- duce reduced set of 128 face keypoints that signify the signer's face 8
Intersection over Union (IoU) We compute the IoU between the ground truth segments and the predicted segments to measure the de- gree of overlap. Note that we do not perform a one-to-one mapping between the two using techniques like DTW. Instead, we calculate the total IoU based all segments in video.
Percentage of Segments (%) To comple- ment IoU, we introduce the percentage of segments to compare the number of seg- ments predicted by the model with the ground truth annotations. It is computed as follows: The optimal value is 1.
4. E3: Adding Optical Flow At every time step we append the optical flow between é and t t- 1 to the current pose frame an additional dimension after the XY Z axes.
# prodicted segments
5. E4: Adding 3D Hand Normalization At ev- ery time step, we normalize the hand poses and concatenate them to the current pose.
#ground truth segments'
Efficiency We measure the efficiency of each model by the number of parameters and the training time of the model a Tesla V100- SXM2-32GB GPU for 100 epochs
"The initial implementation uses OpenPose (Cao al., 2019). at fps. Preliminary experiments reveal that these differences do not significantly influence the results.
"We reduce the dense FACE, LANDMARKS in Mediapipe Holistic to the contour keypoints according to the variahle mediapipe.son sol.ipopesouutisssonoooooss holistic. BAcitiiFFAAAMM FACEMESH CONTOURS.
"Exceptionally the autoregressive models in E5 were trained on an NVIDIA A A100-SXM4-80GB GPUA100 which doubles the training speed of V100, still the training is slow.

---

Sign
Phrase
Efficiency
Experiment
F1
IoU
%
F1
loU
%
#Params
Time
EO
Moryossef et al. (2020)
0.46
1.09
0.70
1.00
102K
0:50:17
E1
Baseline
0.56
0.66
0.91
0.59
0.80
2.50
454K
1:01:50
E2
E1 Face
0.53
0.58
0.64
0.57
0.76
1.87
552K
1:50:31
E3
Optical Flow
0.58
0.62
1.12
0.60
0.82
3.19
473K
1:20:17
E4
E3 + Hand Norm
0.56
0.61
1.07
0.60
0.80
3.24
516K
1:30:59
E1s
E1 + Depth=4
0.63
0.69
1.11
0.65
0.82
1.63
1.6M
4:08:48
E2s
E2 Depth=4
0.62
0.69
1.07
0.63
0.84
2.68
1.7M
3:14:03
E3s
E3 Depth=4
0.60
0.63
1.13
0.64
0.80
1.53
1.7M
4:08:30
E4s
E4 Depth=4
0.59
0.63
1.13
0.62
0.79
1.43
1.7M
4:35:29
E1s#
Els + Tuned Decoding
0.69
1.03
0.85
1.02
E4s*
E4s + Tuned Decoding
0.63
1.06
0.79
1.12
E5
E4s + Autoregressive
0.45
0.47
0.88
0.52
0.63
2.72
1.3M
-3 days
Table I: Mean test evaluation metrics for our experiments. best score of each column is in bold and a star (*) denotes further optimization of the decoding algorithm without changing the model (only affects IoU and %). Table 4 in Appendix A contains a complete report including validation metrics and standard deviation of all experiments.
5 Results and Discussion
In E2s, reaffirmed that adding face keypoints does not yield beneficial results, so we exclude face in future experiments. Although the face an es- sential component to understanding sign language expressions and does play some role in sign and phrase level segmentation, we believe that the 128 face contour points are too dense for the model to learn useful information compared to the 75 body points, and may instead confuse the model.
We report the mean test evaluation metrics for our experiments in Table 1. We do not report F1 Score for EO since it has a different number of classes and is thus incomparable. Comparing El to EO, we note that the model's bidirectionality, the use of poses, and BIO tagging indeed help outperform the model from previous work where only optical flow and 10 tagging are used. While El predicts an excessive number of phrase segments, the loUs for signs and phrases are both higher.
In addition, the benefits of explicitly including optical flow (E3s) fade away with the increased model depth and we speculate that now the model might be able to learn the optical flow features by itself. Surprisingly, while adding hand normaliza- tion (E4s) still slightly worsens the overall results, it has the best phrase percentage.
Adding face keypoints (E2) makes the model worse, while including optical flow (E3) improves the F1 scores. For phrase segmentation, includ- ing optical flow increases IoU, but over-segments phrases by more than 300%, which further exagger- ates the issue in E1. Including hand normalization (E4) on top of E3 slightly deteriorates the quality of both sign and phrase segmentation.
From E4s proceeded with the training of E5, an autoregressive model. Unexpectedly, counter our intuition and previous work, E5 underachieves our baseline across all evaluation metrics
From the non-exhaustive hyperparameter search in the preliminary experiments (Table 3), we exam- ined different hidden state sizes (64, 128, 256, 512, 1024) and range of 1 to 8 LSTM layers, and con- clude that a hidden size of 256 and 4 layers with le 3 learning rate are optimal for EI, which lead Els. We repeat the setup of E2, E3, and E4 with these refined them outperform their counterparts, notably in that they ease the phrase seemenegmenttiion problem.
5.1 Challenges with 3D Hand Normalization
While the of 3D hand normalization is well- justified in $3.3, believe it does not help the model due to poor depth estimation quality,
10g5 should have more parameters than E4s, but because of an implementation bug. each LSTM layer has half the parameters. Based on the current results, we assume that autoregressive connections (even with more parameters) will not improve our models.
and show that all of

---

further corroborated by recent research from De Coster et al. (2023). Therefore, we consider it a negative result, showing the deficiencies in the pose estimation system. The evaluation metrics we propose in Appendix C could help identify better pose estimation models for this use case.
0.7
Els segments
0.6
Els segments
Probability density
0.5
Gold segments
0.4
0.3
5.2 Tuning the Segment Decoding Algorithm
0.2
We selected Els and E4s to further explore the segment decoding algorithm. As detailed in $4.2 and Appendix B, the decoding algorithm has two tunable parameters, Uhresholdy and thresholdg We conducted grid search with these parameters, using values from 10 to 90 increments of 10. We additionally experimented with a variation of the algorithm that conditions on the most likely class by argmax instead of fixed threshold values, which turned out similar to the default version.
0.1
0.0
10
15
Length in seconds
Figure 6: Probability density of phrase segment lengths.
sign to the end of its last sign) is tighter than the subtitle units used in their paper and that use different training datasets of different languages and domains. Nevertheless, we implemented some of their frame-level metrics and show the results in Table 2 on both the Public DGS Corpus and the MEDIAPI-SKEL dataset (Bull et al., 2020a) in French Sign Language (LSF). We report both zero-shot out-of-domain results" and the results of models trained specifically on their dataset without the spatio-temporal graph convolutional network (ST-GCN) (Yan et al., 2018) used in their work for pose encoding.
We only measured the results using loU and the percentage of segments at validation time since the Fl scores remain consistent in this case. For sign segmentation, we found using threshold, = 60 and threshold, = 40/50/ yields slightly better results than the default setting (50 for both). For phrase segmentation, we identified that higher threshold values (threshold, - 90, threshold, = 90 for EIs and thresholds 80, threshold, 80/90 for E4s) improve the default significantly, especially on the percentage metric. We report the test results under # and #4s respectively.
Despite formulating single model, we under- line separate sign/phrase model selection process to archive the best segmentation results. Figure illustrates how higher threshold values reduce the number of predicted segments and skew the distri- bution of predicted phrase segments towards longer ones in s As Bull et al. (2020b) suggest, advanced priors could also be integrated into the decoding algorithm.
Data
Model
ROC-AUC
FI-M
full (theirs)
0.87
body (theirs)
0.87
LSF
E1s (ours, zero-shot)
0.71
0.41
E4s (ours, zero-shot)
0.76
0.44
E1s (ours, trained)
0.87
0.49
E4s (ours, trained)
0.87
0.51
Els (ours)
0.91
0.65
DGS
E4s (ours)
0.90
0.62
5.3 Comparison to Previous Work
We re-implemented and re-purposed the sign lan- guage detection model introduced in Moryossef et al. (2020) for our segmentation task as a baseline since their work is the state-of ofute-of-the-ar and the only comparable model designed for the Public DGS Corpus dataset. As result, show the necessity of replacing 10 tagging with BIO tagging tackle the subtle differences between the tasks.
Table 2: Evaluation metrics used in Bull et al. (2020b). ROC-AUC is applied exclusively on the O-tag. For comparison FI-M denotes the macro-averaged per-class F1 used in this work across all tags. The first two rows the best results taken from Table in their paper. The next four rows represent how our models perform on their data in a zero-shot setting, and in supervised setting, and the last perform on our data.
represent how models
For phrase segmentation, we compare to Bull et al. (2020b). We note that our definition of sign language phrases (spanning from the start of its first
"Thhe zcro-shot results are not directly comparahle to theirs due to different datasets and labeling approaches.

---

For sign segmentation, we do not compare to Renz et al. (2021a,b) due to different datasets and the difficulty in reproducing their segment-level evaluation metrics. The latter depends on the de- coding algorithm and a way to match the gold and predicted segments, both of which are variable.
featuring white participants.
Encoding of Long Sequences
In this study, we encode sequences of frames that are significantly longer than the typical 512 frames often seen in models employing Transformers (Vaswani et al., 2017). Numerous techniques, rang- ing from basic temporal pooling/downsampling to more advanced methods such as a video/pose en- coder that aggregates local frames into higher-level 'tokens' (Renz et al., 2021a), a)), graph convolutional networks (Bull al., 2020b), and self-supervised representations (Baevski et al., 2020), can alleviate length constraints, facilitate the use of Transform- ers, and potentially improve the outcomes. More- over, a hierarchical method like the Swin Trans- former (Liu et al., 2021) could be applicable.
6 Conclusions
This work focuses on the automatic segmentation of signed languages. We are the first to formulate the segmentation of individual signs and larger sign phrases as a joint problem.
We propose a series of improvements over previ- ous work, linguistically motivated by careful anal- yses of sign language corpora. Recognizing that sign language utterances are typically continuous with minimal pauses, we opted for a BIO tagging scheme over IO tagging. Furthermore, leverag- ing the fact that phrase boundaries are marked by prosodic cues, we introduce optical flow features as a proxy for prosodic processes. Finally, since signs typically employ a limited number of hand shapes, to make it easier for the model to understand hand- shapes, attempt 3D hand normalization.
Limitations of Autoregressive LSTMs
In this paper, replicated the autoregressive LSTM implementation originally proposed by Jiang al. (2023). Our experiments revealed that this implementation exhibits significant slowness, which prevented us from performing further exper- imentation. In contrast, other LSTM implemen- tations employed in this project have undergone extensive optimization (Appleyard, 201 60, includ- ing techniques like combining general matrix mul- tiplication operations (GEMMs), parallelizing in- dependent operations, fusing kernels, rearranging matrices, and implementing various optimizations for models with multiple layers (which are not nec- essarily applicable here).
Our experiments conducted on the Public DGS Corpus confirmed the efficacy of these modifica- tions for segmentation quality. By comparing to previous work in a zero-shot setting, we demon- strate that our models generalize across signed lan- guages and domains and that including linguisti- cally motivated cues leads to a more robust model in this context.
Finally, we envision that the proposed model has applications in real-world data collection for signed languages. Furthermore, a similar segmentation approach could leveraged in various other fields such as co-speech gesture recognition (Moryossef, 2023) and action segmentation (Tang et al., 2019).
A comparison of CPU-based performance demonstrates that our implementation is x6.4 times slower. Theoretically, the number of operations performed by the autoregressive LSTM is equiv- alent that of regular LSTM. However, while the normal LSTM benefits from concurrency based on the number of layers, we do not have that lux- ury. The optimization of recurrent neural networks (RNNs) (Que et al., 2020, 2021, 2022) remains an ongoing area of research. If proven effective in other domains, we strongly advocate for efforts to optimize the performance of this type of network.
Limitations
Pose Estimation
In this work, we employ the MediaPipe Holis- tic pose estimation system (Grishchenko and Bazarevsky, 2020). There is possibility that this system exhibits bias towards certain protected classes (such as gender race), underperforming in instances with specific skin tones or lower video quality. Thus, we cannot attest to how our system would perform under real-world conditions, given that the videos utilized in our research are gener- ated in a controlled studio environment, primarily
Interference Between Sign and Phrase Models
In our model, share the encoder for both the sign and phrase segmentation models, with a shallow linear layer for the BIO tag prediction associated with each task. It remains uncertain whether these two tasks interfere with or enhance each other. An

---

ablation study (not presented in this work) involv- ing separate modeling is necessary obtain greater insight into this matter.
data from automatic speech recognition / spoken language translation (Tsiamas al., 2022).
However, for signed languages, there is nei- ther standardized and widely used written form nor a reliable transcription procedure into some potential writing systems like SignWriting (Sut- ton, 1990), HamNoSys (Prillwitz and Zienert, 1990), and glosses (Johnston, 2008). Transcrip- tion/recognition and segmentation tasks need to be solved simultaneously, so we envision that a multi- task setting helps. Sign spotting, the localization of a specific sign in continuous signing, is simplifi- cation of the segmentation and recognition problem in closed-vocabulary setting (Wong et al., 2022; Varol et al., 2022). It can be used to find candidate boundaries for some signs, but not all.
Noisy Training Objective
Although the annotations utilized in this study are of expert level, the determination of precise sign (Hanke al., 2012) and phrase boundaries remains a challenging task, even for experts. Training the model on these annotated boundaries might intro- duce excessive noise. A similar issue was observed in classification-bart
pose estimation (Cao et al.. 2019). The task of annotating the exact anatomical centers of joints proves to be nearly impossible, leading to a high degree of noise when predicting joint position as 1-hot classification task. The solution proposed in this previous work was to dis- tribute a Gaussian around the annotated location each joint. This approach allows the joint's cen- ter to overlap with some probability mass, thereby reducing the noise for the model. A similar solu- tion could be applied in our context. Instead of predicting strict 0 or 1 class probability, could distribute& Gaussian around the boundary.
A Acknowledgements
This work was funded by the EU Horizon 2020 project EASIER (grant agreement no. 101016982). the Swiss Innovation Agency (Innosuisse) flag- ship IICT (PFFS-21-47) and the EU Horizon 2020 project iEXTRACT (grant agreement no. 802774).
We also thank Rico Sennrich and Chantal Amrhein for their suggestions.
Naive Segment Decoding
We recognize that the frame-level greedy decod- ing strategy implemented in our study may not be optimal. Previous research in audio segmen- tation (Venkatesh et al., 2022) employed a You Only Look Once (YOLO; Redmon et al. (2015)) de- coding scheme to predict segment boundaries and classes. We propose using a similar prediction atop a given representation, such as the LSTM output or classification logits of an already trained network. Differing from traditional object detection tasks, this process is simplified due to the absence of a Y axis and non-overlapping segments. In this sce- nario, the network predicts the segment boundaries using regression, thereby avoiding the class imbal- ance issue of the BIO tagging. We anticipate this to yield more accurate sign language segmentation.
References
Jeremy Appleyard. 2016. Optimizing recurrent neural networks in cuDNN 5. Accessed: 2023-06-09.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances neural information processing systems, 33:12449 12460.
Mark Borg and Kenneth Camilleri. 2019. Sign lan- guage detection "in the wild" with recurrent neural networks. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1637-1641. IEEE
Hannah Bull, Triantafyllos Afouras, Gül Varol, Samuel Albanie, Liliane Momeni, and Andrew Zisserman. 2021. Aligning subtitles in sign language videos. In Proceedings of the IEEE/CVF International Confer- ence Computer Vision pages 11552-11566
Lack of Transcription
Speech segmentation is close task to our sign language segmentation task on vidcos. In addi- tion to relying on prosodic cues from audio, the former could benefit from automatic speech tran- scription systems, either in terms of surrogating the task to text-level segmentation and punctuation (Cho et al., 2015), or gaining additional training
Hannah Bull, Annelies Braffort, and Michèle Gouif- fes. 2020a. MEDIAPI-SKEL- a 2D-skeleton video database of French Sign Language with aligned French subtitles. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 6063-6068. Marseille, France. European Language Resources Association.

---

Hannah Bull, Michèle Gouiffès, and Annelies Braffort. 2020b. Automatic segmentation of sign language into subtitle-units. In European Conference on Com- puter Vision, pages 186-198. Springer.
Ivan Grishchenko and Valentin Bazarevsky, 2020. Me- diapipe holistic.
Thomas Hanke, Silke Matthes, Anja Regen, and Satu Worseck. 2012. Where does sign start and end? segmentation of continuous signing. In Language Resources and Evaluation Conference.
Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. 2019. OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intel- ligence.
Thomas Hanke, Mare Schulder, Reiner Konrad, and Elena Jahn. 2020, Extending the Public DGS Corpus in size and depth. In Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community. Technological Challenges and Application Perspectives, pages 75- 82, Marseille, France. European Language Resources Association (ELRA).
Sait Celebi, Ali Selman Aydin, Talha Tarik Temiz, and Tarik Arici. 2013. Gesture recognition using skeleton data with weighted dynamic time warping. In Inter- national Conference on Computer Vision Theory and Applications.
Eunah Cho, Jan Niehues, Kevin Kilgour, and Alex Waibel. 2015. Punctuation insertion for real-time spoken language translation. In Proceedings of the 12th International Workshop on Spoken Language Translation:
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neurul computation, 978):1735- 1780.
Papers, pages 173-179.
Kyunghyun Cho, Bart van Merriënboer; Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Laguugge Processing (EMNLP), pages 1724- 1734, Doha, Qatar. Association for Computational Linguisties.
Zifan Jiang, Adrian Soldati, Isaac Schamberg, Adri- ano R Lameira, and Steven Moran. 2023. Auto- matic sound event detection and classification of great ape calls using neural networks. arXiv preprint M.iiv:2301.02214
Trevor Alexander Johnston. 2008 From archive to corpus: transcription and annotation in the creation of signed language corpora. In Pacific Asia Conference on Language, Information and Computation.
Mathieu De Coster, Ellen Rushe, Ruth Holmes, An- thony Ventresque, and Joni Dambre. 2023. Towards the extraction of robust sign embeddings for low re- source sign language recognition. arXiv preprint arXiv:2306. 17558.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF international conference on computer vi- sion. pages 10012-10022.
Mirella De Sisto, Dimitar Shterionov, Trene Murtagh, Myriam Vermeerbergen, and Lorraine Leeson. 2021. Defining meaningful units. challenges in sign seg- mentation and segment-meane meaning mapping (short pa- per). In Proceedings of the 1st International Work- shop on Automatic Translation for Signed and Spoken Languages (AT4SSI.), pages 98-103, Virtual. Associ- ation for Machine Translation in the Americas.
Mark Alan Mandel. 1981. Phonotactics and mor- phophonology in American Sign Language. Uni- versity of California, Berkeley.
Amit Moryossef. 2023. Addressing the blind spots in spoken language processing. arXiv preprint 20Xi::2309.0657..
Jacob Devlin, Ming-Wei. Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Amit Moryossef, Toannis Tsochantaridis, Roce Aharoni, Sarah Ebling, and Srini Narayanan. 2020. Real-time sign-language detection using human pose estimation. In Computer gow, UK, August 23-28, 2020, Proceedings, Part II 16, SLRTP 2020: The Sign Language Recogni- rion, Translation and Production Workshop, pages 237-248.
2020 Workshops: Glas-
Iva Farag and Heike Brock. 2019. Learning motion dis- fluencies for automatic sign language segmentation. In ICASSP 2019-2019 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 7360-7364. IEEE
Carol Neidle, Ashwin Thangali, and Stan Selaroff. 2012. Challenges in development of the American Sign Language lexicon video dataset (ASSLVD) corpus. In Sth workshop on the representation and processing of sign languages: interactions between corpus and Lexicon, LREC. Citeseer.
Adam Frost and Valerie Sutton. 2022. Sign Hand Symbols. SignWriting Press.

---

Ellen Ormel and Onno Crasborn. 2012. Prosodic corre- lates of sentences in signed languages: A literature review and suggestions for new types of studies. Sign Language Studies, 12(2):27--315.
voice activity detection algorithms using long-term speech information. Speech communication, 42(3- 4):271-287.
Lance Ramshaw and Mitch Marcus. 1995. Text chunk- ing using transformation-batt -baeed learning. In Third Workshop on Very Large Corpora.
Abhilash Pal, Stephan Huber, Cyrine Chaabani, Alessan- dro Manzotti, and Oscar Koller. 2023, On the impor- tance of signer overlap for sign language detection. arXiv preprint arXiv:2303. 10782.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Gir- shick, and Ali Farhadi. 2015. You only look once: Unified, real-time object detection. 2016 IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 779-788.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, pigh-performance deep learning library. In Advances in Neural Information Process- ing Systems 32, pages 8024-8035. Curran Associates, Inc.
Katrin Renz, Nicolaj C Stache, Samuel Albanie, and Gül Varol, 202 Sign language segmentation with temporal convolutional networks. In ICASSP 2021- I- 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 2135- 2139. IEEE
Katrin Renz, Nicolaj C Stache, Neil Fox, Gul Varol, and Samuel Albanie. 1b. Sign segmentation with changepoint-moduie modulated pseudo-labe labelling. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 3403-3412.
Siegmund Prillwitz, Thomas Hanke, Susanne König, Reiner Konrad, Gabriele Langer, and Arvid Schwarz. 2008. Dgs corpus project-developmeet of a cor- pus based electronic dictionary german sign lan- guage/german. In sign-lang at LREC 2008, pages 159-164. European Language Resources Association (ELRA).
Wendy Sandler. 2010. Prosody and syntax in sign lan- guages. Transactions of the philological society. 10883))298-328.
Siegmund Prillwitz and Heiko Zienert. 1990. Hamburg notation system for sign language: Development of sign writing with computer application. In Cur- rent trends in European Sign Language Research. Proceedings of the 3rd European Congress on Sign Language Research, pages 355-379.
Wendy Sandler and Diane Lillo-Martin. 2006. Sign language and linguistic universals. Cambridge Uni- versity Press.
Wendy Sandler, Peter Macneilage, Barbara Davis, Kris- tine Zajdo, New York, and Taylor Francis. 2008. The syllable in sign language: Considering the other nat- ural language modality.
Zhiqiang Que, Hiroki Nakahara, Hongxiang Fan, He Li, Jiuxi Meng, Kuen Hung Tsoi, Xinyu Niu, Eriko Nurvitadhi, and Wayne W. C. Luk. 2022. Remarn: A reconfigurable multi-threader multi-core accelerator for recurrent neural networks. ACM Transactions on Reconfigurable Technology and Systems, 16:1 - 26.
Pinar Santemiz, Oya Aran, Murat Saraclar, and Lale Akarun. 2009. Automatic sign segmentation from continuous signing via multiple sequence alignment. In 2009 IEEE 12th International Conference on Com- puter Vision Workshops, ICCV Workshops, pages 2001-2008. IEFF
Zhiqiang Que Hiroki Nakahara, Eriko Nurvitadhi, Hongxiang Fan, Chenglong Zeng, Jiuxi Meng. Xinyu Niu, and Wayne W. C. Luk. 2020. Optimizing reconfigurable recurrent neural networks. 2020 IEEE 28th Annual International Symposium on Fredd-Progrummable Custom Compuling Machines (FCCM), pages 10-18.
Mare Schulder and Thomas Hanke. 2019. OpenPose in the Public DGS Corpus. Project Note AP06-2019. 01, DGS-Korpus project, IDGS, Hamburg University, Hamburg, Germany.
Zed Seveikova Sehyr, Naomi Caselli, Ariel M Cohen- Goldberg, and Karen Emmorey. 2021. The ASI.- LEX 2.0 project: A database of lexical and phono- logical properties for 2,723 signs in american sign language. The Journal of Deuf Studies and Deaf Education, 26(2):263-277.
Zhiqiang Que, Erwei Wang, Umar Marikar, Eric Moreno, Jennifer Ngadiuba, Hamza Javed, Bartiomiej Borzyszkowski, Thea Klaeboe Aarrestad, Vladimir Loncar, Sioni Paris Summers, Maurizio Picrini, Peter Y. K. Cheung, and Wayne W. C. Luk. 2021. Accelerating recurrent neural networks for gravitational wave experiments. 2021 IEEF 32nd International Conference on applicationsppeitne Systems. Architectures and Processors (ASAP), pages 117-124.
Karen Simonyan and Andrew Zisserman. 2015, Very deep convolutional networks for large-scale image recognition. CoRR.
Jongseo Sohn, Nam Soo Kim, and Wonyong Sung. 1999. A statistical model-based voice activity detection. IEEE signal processing letters, 6(1):1-3.
Javier Ramirez, José Segura, Carmen Bemitez, Angel De La Torre, and Antonio Rubio. 2004. Efficient

---

William C Stokoe 1960. Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf. The Journal of Deaf Studies and Deaf Education, 10(1):3337.
Valerie Sutton. 1990. Lessons in sign writing. Sign- Writing.
Yansong Tang, Dajun Ding. Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. COIN: A large-scale dataset for comprehen- sive instructional video analysis. In IEEE Confer- ence Computer Vision and Pattern Recognition (CVPR).
Ioannis Tsiamas, Gerard I. Gállego, José A. Fonol- losa, and Marta R. Costa-jussà. 2022. SHAS: Approaching optimal segmentation for end-to-end speech translation. In Proc, Interspeech 2022, pages 106-110.
Gül Varol, Liliane Momeni, Samuel Albanie, Triantafyl- los Afouras, and Andrew Zisserman. 2022. Scaling up sign spotting through sign language dictionaries. International Journal of Computer Vision, 130:1416 1439.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Satvik Venkatesh, David Moffat, and Eduardo Reck Miranda. 2022. You only hear once: YOLO-like algorithm for audio segmentation and sound event delection. Applied Sciences, 12(7):3293.
Ryan Wong. Necati Cihan Camgoz, and R. Bowden. 2022. Hierarchical 13D for sign spotting. In ECCV Workshops.
Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial temporal graph convolutional networks for skeleton- based action recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Ting Yu. Haoteng Yin, and Zhanxing Zhu. 2017. Spatio- temporal graph convolutional networks: A deep learn- ing framework for traffic forecasting. In Interna- tional Joint Conference on Artificial Intelligence.
Biao Zhang, Mathias Müller, and Rico Seunrich. 2023. SLTUNET: A simple unified model for sign language translation. In The Eleventh International Confer- ence on Learning Representations, Kigali, Rwanda.

---

A Extended Experimental Results
We conducted some preliminary experiments (starting with PO) on training a sign language segmentation model to gain insights into hyperparameters and feature choices. The results are shown in Table 312, We found in P1.3.2 the optimal hyperparameters and repeated them with different feature choices.
Sign
Phrase
Experiment
FI
loU
%
IoU
PO
Moryossef et al. (2020)
test 0.4
1.45
0.65
0.82
dev
0.35
1.36
0.6
0.77
PO.1
PO Holistic 25fps
test
0.39
0.86
0.64
0.5
dev
0.32
0.81
0.58
0.52
PI
Pl baseline
test 0.55
0.49
0.83
0.6
0.67
2.63
dev
0.56
0.43
0.75
0.58
0.62
2.61
PI.1
P1 encoder eecoder_biddreee bidirectional
test 0.48
0.45
0.68
0.5
0.64
2.68
dev
0.46
0.41
0.64
0.51
0.61
2.56
P1.2.1
PI fiidennddim-te
test 0.47
0.42
0.44
0.52
0.63
1.7
dev 0.46
0.4
0.43
0.52
0.61
1.69
P1.2.2
P1 hidden 41m=1044
test 0.48
0.45
0.42
0.58
0.65
1.53
dev 0.46
0.41
0.36
0.53
0.61
1.49
P1.3.1
P1 encoder_deptlle
test 0.55
0.48
0.76
0.58
0.67
2.56
dev 0.56
0.43
0.69
0.58
0.62
2.52
P1.3.2
P1 encoder 4
test
0.63
0.51
0.91
0,66
0.67
1.41
dev 0.61
0.47
0.84
0.64
1.39
P1.4.1
Pl hidden_dim==12 encoder_deptler
0.58
0.8
0.67
2.0
dev
0.55
0.43
0.75
0.54
0.62
2.03
P1.4.2
Pl middeen_dm==155 encoder_depthes
0.62
0.91
0.68
2.43
0.6
0.47
0.83
0.62
2.57
P1.4.3
P1 1idd encoder_depth-i
0.59
0.91
3.04
0.6
0.47
0.84
0.62
3.02
P1.5.1
P1 hidden dim=64 + encoder depth=4
test
0.57
0.8
0.68
2.11
0.58
0.75
0.59
0.62
2.39
P1.5.2
Pl hidden dim=64 + encoder depth=8
test
0.62
0.51
0.85
0.64
0.68
2.53
dev
0.6
0.46
0.79
0.6
0.62
2.53
P2
Pl optical flow
test 0.58
0.5
0.95
0.63
0.68
3.17
0.59
0.45
0.84
0.59
0.61
3.08
P2.1
P1.3.2 + optical flow
0.63
0.92
0.67
1.51
0.62
0.46
0.81
1.53
P3
P1 hand normalization
test 0.55
0.48
0.77
0.58
0.67
2.79
dev
0.55
0.42
0.71
0.57
0.62
2.73
P3.1
P1.32 hand normalization
test
0.63
0.51
0.91
0.66
0.66
1.43
dev
0.61
0.46
0.82
0.64
0.61
1.46
P4
P2.1 + P3.1
test 0.56
0.51
0.92
0.61
0.66
1.45
dev
0.61
0.46
0.81
0.63
0.6
1.41
P4.1
P4 encoder_depther
test
0.6
0.51
0.95
0.62
0.67
1.08
dev
0.61
0.47
0.86
0.62
0.6
1.12
P5
P1.3.2 + reduced_
test 0.63
0.51
0.94
0.64
0.66
1.16
dev
0.61
0.47
0.86
0.64
0.58
1.14
P5.1
P1.322 full_face
test
0.54
0.49
0.8
0.6
0.68
2.29
dev
0.57
0.45
0.7
0.59
0.62
2.29
Table 3: Results of the preliminary experiments.
NNot that due to an implementation issue on edge cases (which we fixed later), the JoU and % values in Table are lower than the ones in Table and Table thus not comparable across tables. The comparison inside of Table 3 between different experiments remains meaningful, In addition, the results in Table are based on only one run instead of three random runs.

---

We selected some promising models from our preliminary experiments and reran them three times using different random seeds make the final conclusion reliable and robust. Table includes the standard deviation and the validation results (where we performed the model selection) for readers to scrutinize.
Sign
Phrase
Efficieney
Experiment
*Params
Moryossef et (2020)
test
0.46
1.09
0.41
0.70 0.01
1.00
0.06
102K
0:50:17
dev
0.42 2: 0.05
1.21
0.59
0.61 1 0.06
2.17 - 11.85
102K
0:50:17
Baseline
test
0.56
0.03
0.66 0.01
0.91
0.05
0.59 0.02
0.50 1: 0.03
2.50 4: 0.13
454K
1:01:50
1:01:50
E2
El + Face
test
0.53 a: 0.05
0.58 à: 0.07
0.84 à: 0.30
0.57 1: 0.02
0.76 à: 0.03
1.87 1: 0.83
552K
1:50:31
0.50
0.07
0.53 0.11
0,90
0.19
0.53 = 0.05
0.71 0:07
2.13
1.02
552K
1:50,31
Optical Flow
test
0.58
0.01
0.82
1.12
0.05
0.00
0.82 0.03
3.19
473K
1:20:
dev
0.58 1 0.00
0.02 1. 0.00
1.50 1 0.19
0.59 1 0.01
0.79 0.00
3.94 1. 0.14
473K
1:20: 17
Norm
test
0.56
0.02
0.61 0.00
1.07
0.00
0.60 0.00
0.80 0.00
3.21
U.17
516K
1:30,59
dev
0.57
0.01
0.81 0.01
1.50
0.07
0.58 1 0.00
0.00 0.79
4.04
1 0.51
516K
1:30:59
EIs
El + Depth=4
test
0.63 L 0.01
0.69 1 0.00
1.11 1 0.01
0.65 L 0.02
0.82 1 0.04
1.63
1.63.10.11 0.10
1.6M
4:08:48
0.00
0.63 +
1:27
0:01
0.63+
0.77 0.01
2.17+ 0.18
4:08:48
F2s
Depth-4
0.62 0.02
0.89
1.07 0.03
0.63
0.84 0:03
2.68
1.7M
3:14:03
dev 0.60
0.01
0.63 1 0.01
1.20 1 0.12
0.59 1 0.02
0.76 0.05
3,30
1 n.62
1.7M
3:14:03
Dopthm4
0.60 0.01
0.63
1.13 0.01
0.64
0.80 0.03
1.53 0.18
4:08:30
dev 0.62 1 0.00
0,63 0.00
1.63 0.05
0.63 1
0.76 0.00
2.14
1.7M
4:08:30
Fds
+ Depth=4
test 0.59
0.00
0.63 + 0.00
1.13 + 0.03
0.62 + 0.00
0.79 + 0.00
1.43 + 0.10
1.7M
4:35/29
0.00
0.63
0.04
0.77 0.01
E4ha
Eds Autoregressive
test 0.45 0.03
0.47
0.05
0.56
0.08
0.52 +
0.63 0.10
2.72 1.33
1.3M
days, 21:28:42
dev
0.10 0.10=2000 0.01
0. 10 = 0.01
2.02 0.73
0.17 = 0.00
0.57 = 0.04
4.26 1.26
1.3M
days, 21:28-42
Table 4: Mean evaluation metrics our main experiments. A complete version of Table

---

B Greedy Decoding Algorithm
We provide our exact decoding algorithm in Algorithm 1. We opt to employ adjustable thresholds rather than argmax prediction, as our empirical findings demonstrate superior performance with this approach ($5.2).
Algorithm 1 Probabilities to Segments Conversion.
Require: probs, list of probabilities from 0 to 100
1: threshold, 50.0
2: Uhreshold, <- 50.0
3:
4: start <- None
5: flid_pass_slat +- False
6:
7: for -- 0 len(probs) do
8: b,i,0 probs[i]
10:
if start None then
11:
b thresholdy then
12: start +
13: end if
14: else
15:
if did fid_ppas____ pass_start then
16:
if > thresholds or > threshold, then yield (start, 1)) start + None did fld_pass_steee startsstttt False
19:
20:
end if else
21:
22:
if b < thresholds then hid_pass_stare t- True
23:
24:
end if
25:
end if
end if
27: end for
28:
29: if start + None then
30: yield (start, (en(pro))))
31: end if

---

C Pose Based Hand Shape Analysis
C.1 Introduction to Hand Shapes in Sign Language
The most prominent feature of signed languages their use of the hands. In fact, the hands play an integral role in the phonetics of signs, and a slight variation hand shape can convey differences meaning (Stokoe Jr, 1960). In sign languages such as American Sign Language (ASL) and British Sign Language (BSL), different hand shapes contribute to the vocabulary of the language, similar to how different sounds contribute to the vocabulary of spoken languages. ASL is estimated to use between 30 to 80 hand shapes!3, while BSL is limited to approximately 40 hand shapes SignWriting (Sutton, 1990), system of notation used for sign languages, specifies superset of 261 distinct hand shapes (Frost and Sutton, 2022). Each sign language uses subset of these hand shapes.
Despite the fundamental role of hand shapes in sign languages, accurately recognizing and classifying them challenging task. In this section, we explore rule-based hand shape analysis in sign languages using 3D hand normalization. By performing 3D hand normalization, we can transform any given hand shape to fixed orientation, making it casier for a model to extract the hand shape, and hence improving the recognition and classification of hand shapes in sign languages.
C.2 Characteristics of the Human Hand
The human hand consists 27 bones and can divided into three main sections: the wrist (carpals), the palm (metacarpals), and the fingers (phalanges). Each finger consists of three bones, except for the thumb, which has two. The bones are connected by joints, which allow for the complex movements and shapes that the hand can form.
MIDDLE JOINTS
Intershdaangall
Prosimee Interphslargeed
Medazarpophhlangeel
Carpamatacurpuu
RING JOINTS
Interphalangeal
INDEX JOINTS
Prosimal Interpfastagged
Metsstrpophhsangeel
Proximal Interpholanggal
Carpometacurpal
Methaunpoophkkaggol
Carponetacarpal
Distta Interpfuubergued
Inteeppaateeggee
Metsscoroophakanges
Carpormetacarpal
THUMB JOINTS
Interphalangeal Metocospophalnnget
Carpometaaarpul
Figure 7: Anatomy of a human hand. @American Society for Surgery of the Hand
"htttps//assfont wttp/asslfotttttth.... gith fttrssooooo--ottt-o---Asss/l/// html "https://bbe //bsl surrey, ac. http://nncpplleii-nnnhh-hhhapeep

---

Understanding the different characteristics of hands and their implications in signed languages is crucial for the extraction and classification of hand shapes. These characteristics are based on the Sign definitions of the five major axes of hand variation: handedness, plane, rotation, view, and shape.
Handedness is the distinction between the right and left hands. Signed languages make a distinction between the dominant hand and the non non-dominant hand. For right-handed individuals, the right hand is considered dominant, and vice-versa. The dominant hand is used for fingerspelling and all one-handed signs, while the non-dominant hand is used for support and two-handed signs. Using 3D pose estimation, the handedness analysis is trivial, the pose estimation platform predicts which hand is which.
Plane refers to whether the hand is parallel to the wall or the floor. The variation in the plane can, but does not have to, create a distinction between two signs. For example, in ASL the signs for "date" and "dessert" exhibit the same hand shape, view, rotation, and movement, but differ by plane. The plane of hand can be estimated by comparing the positions of the wrist and middle finger metacarpal bone (M_ _MMP..
Algorithm Hand Plane Estimation
1: < M My W RIST.yl 1.5 // add bias to 2: - M_MMCP.z - WRIST.2
3:
return y > x 'wall' 'floor'
Rotation refers to the angle of the hand in relation to the body. Sign SignWriting groups the hand rotation into eight equal categories, each spanning 45 degrees. The rotation of hand can be calculated by finding the angle of the line created by the wrist and the middle finger metacarpal bone.
View refers to the side of the hand as observed by the signer, and is grouped into four categories: front, back, sideways, and other-sideways. The view hand can be estimated by analyzing the normal the plane created by the palm of the hand (between the wrist, index finger metacarpal bone, and pinky metacarpal bone).
Algorithm Hand View Estimation
1: normal f math.normal (RRIST, 1 MCP, P_MCP)
2: plane < get_plane(WRISS) M_MCP)
3: if plane 'wall' then
4: angle t (normal.zz, normal..))
5: return angle > 210 ? 'front' (angle > 150 'sideways" 'back')
6: else
7: angle + [(normal.y y, normal.x))
8: return angle > 0 ? 'front' (angle > -60 ? 'sideways' 'back')
end if
Shape refers to the configuration of the fingers and thumb. This characteristic of the hand is the most complex to analyze due to the vast array of possible shapes the human hand can form. The shape of a hand is determined by the state of each finger and thumb, specifically whether they are straight, curved, bent, and their position relative each other. Shape analysis can be accomplished by examining the bend and rotation of each finger joint. More advanced models may also take into consideration the spread between the fingers and other nuanced characteristics. 3D pose estimation can be used to extract these features for a machine learning model, which can then classify the hand shape.
C.3 3D Hand Normalization
3D hand normalization is an attempt at standardizing the orientation and position of the hand, thereby enabling models to effectively classify various hand shapes. The normalization process involves several

---

key steps, as illustrated below:
1. Pose Estimation Initially, the 3D pose of the hand is estimated from the hand image crop (Figure 8).
Figure 8: Pictures of six hands all performing the same hand shape (v-shape) taken from six different orientations. Mediapipe fails at estimating the pose of the bottom-middle image.
2. 3D Rotation The pose is then rotated in 3D space such that the normal of the back of the hand aligns with the Z-axis. As a result, the palm plane now resides within the XY plane (Figure 9).
Figure 9: Hand poses after 3D rotation. The scale difference between the hands demonstrates a limitation of the 3D pose estimation system used.

---

3. 2D Orientation Subsequently, the pose is rotated in 2D such that the metacarpal bone of the middle finger aligns with the Y-xiis (Figure 10).
Figure 10: Hand poses after being rotated.
4. Scale The hand is scaled such that the metacarpal bone of the middle finger attains constant length (which we typically set to 200, Figure 11).
Figure 11: Hand poses after being scaled.
5. Translation Lastly, the wrist joint is translated to the origin of the coordinate system (0, 0, 0). Figure 12 demonstrates how when overlayed, we can see all hands producing the same shape, except for one outlier.

---

Figure 12: Normalized hand poses overlayed after being translated to the same position. The positions of the wrist and the metacarpal bone of the middle finger are fixed.
By conducting these normalization steps, hand model can standardized, reducing the complexity of subsequent steps such as feature extraction and hand shape classification. This standardization simplifies the recognition process and can contribute to improving the overall accuracy of the system.
C.4
3D Hand Pose Evaluation
In order to assess the performance of our 3D hand pose estimation and normalization, we introduce two metrics that gauge the consistency of the pose estimation across orientations and crops. Our dataset is extracted from the SignWriting Hand Symbols Manual (Frost and Sutton. 2022), and includes images of 261 different hand shapes, from 6 different angles. All images are of the same hand, of an adult white man.
Multi Angle Consistency Error (MACE) evaluates the consistency of the pose estimation system across the different orientations. We perform 3D hand normalization, and overlay the hands. The MACE score is the average standard deviation of all pose landmarks, between all views. A high MACE score indicates a problem in the pose estimation system's 's ability to maintain consistency across different orientations. This could adversely affect the model's performance when analyzing hand shapes in sign languages, signs significantly vary with hand rotation.
Figure 13: Visualizations of 10 hand shapes, each with 6 orientations 3D normalized and overlayed.
Figure 13 shows that our 3D normalization does work to some extent using Mediapipe. We can identify differences across hand shapes, but still note high variance within each hand shape.

---

Crop Consistency Error (CCE) gauges the pose estimation system's consistency across different crop sizes. We do not perform 3D normalization, but still overlay all the estimated hands, shifting the wrist point each estimated hand the origin (0, O, 0). The CCE score the calculated average standard deviation all pose landmarks across crops. A high CCE score indicates that the pose estimation system is sensitive to the size of the input crop, which a significant drawback as the system should be invariant to the size of the input image.
Figure 14: Visualizations of 10 hand shapes, each with 48 crops overlayed.
Figure 14 shows that for some poses, Mediapipe is very resilient to crop size differences (c.g. the first and last hand shapes). However, it is concerning that for some hand shapes, exhibits very high variance, and possibly even wrong predictions.
C.5 Conclusion
Our normalization process appears to work reasonably well when applied to different views within the crop size. It succeeds in simplifying the hand shape, which in turn, can aid in improving the accuracy of hand shape classification systems.
However, it is crucial consistency and reliability in a dynamic context, such as a video, may be quite different. In a video, the crop size can change between frames, introducing additional complexity and variance. This dynamic nature coupled with the inherently noisy nature of the estimation process can pose challenges for a model that aims to consistently estimate hand shapes.
that while this method may seem to perform well on static image, its
In light of these findings, it is clear that there is a need for the developers of 3D pose estimation systems to consider these evaluation methods and strive to make their systems more robust to changes in hand crops. The Multi Angle Consistency Error (MACE) and the Crop Consistency Error (CCE) can be valuable tools in this regard.
MACE could potentially be incorporated as loss function for 3D pose estimation, thereby driving the model to maintain consistency across different orientations. Alternatively, MACE could be used as an indicator to identify hand shapes that require more training data. is apparent from our study that the performance varies greatly across hand shapes and orientations, and this approach could help in prioritizing the allocation of training resources.
Ultimately, the goal of improving 3D hand pose estimation is to enhance the ability to encode signed languages accurately. The insights gathered from this study can guide future research and development efforts in this direction, paving the way for more robust and reliable sign language technology.
The benchmark, metrics, and visualizations are available at https: //gi com/ wawpinngddd--hhn----eeeee hands-benchharkk


Reviews:
[{"review_id": "3s8FvPGfaZ", "reviewer": "Reviewer_i9D9", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Rebuttal \n\n\n#### Insufficient Novelty\n**Reviewer’s Comment:**  \nReplacing IO tagging with BIO tagging is not novel, given that it was proposed in 1995 for similar NLP tasks.\n\n**Response:**  \n\nOur innovation is the application of BIO tagging to sign language segmentation—a context where it has not been used before. This adaptation addresses unique challenges and extends BIO tagging from text to the continuous modality of video, similar to how Vision Transformers extended Transformers to images.\n\n---\n\n#### Experimental Results\n**Reviewer’s Comment:**  \nThe experimental results are not convincing enough and questions whether the baseline model is a state-of-the-art model.\n\n**Response:**  \nThank you for highlighting this concern. It is indeed the state-of-the-art and the only comparable model designed for this dataset. We will make it clearer in our paper.\nWe additionally highlight two other works on other datasets in different languages that are not immediately and directly comparable, and in section 5.2 we show our trained models still achieve reasonable zero-shot performance on one of them.\n\n\n---\n\n#### Optical Flow and 3D Hand Normalization\n**Reviewer’s Comment:**  \nThe improvements due to optical flow and 3D hand normalization appear to be trivial. \n\n**Response:**  \nThe added features, like optical flow, do improve the performance in shallower models. In deeper models, they offer more than just trivial in-domain improvements; they crucially enhance the model's adaptability to new contexts and languages (Table 2). \nWhile the use of 3D hand normalization is well-justified in the paper, it does not help the model, due to poor depth estimation quality, as further corroborated by recent research [De Coster et al. 2023](https://arxiv.org/pdf/2306.17558.pdf). Therefore, we consider it a negative result, showing the deficiencies in the 3D pose estimation system. This analysis will be further clarified in the camera ready.\n\n\n---\n\n#### Results in Table 2\n**Reviewer’s Comment:**  \nYou question the utility of the results in Table 2 and ask whether it's possible to train the model with the MEDIAPI-SKEL dataset.\n\n**Response:**  \nWe posit our results in Table 2 specifically as a zero-shot, out-of-domain generalization. While the datasets are annotated differently, we can still see the improvement from our feature engineering, most importantly in ROC-AUC, which leads us to believe the features we added are indeed important for out-of-domain generalization.\nHowever, we agree that including results trained on the MEDIAPI-SKEL dataset could offer additional insights. We will incorporate this into the revised paper.\n\n---\n\n#### Reproducibility\nWe believe our work is highly reproducible. The code is available on GitHub (the links are broken due to the anonymity requirements), including all of the settings to train each model and variation presented in this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a new scheme of sign language segmentation, which include two aspects: replacing the predominant IO tagging scheme with BIO tagging to account for continuous signing and segmenting phrases as well as segmenting signs. By using the new scheme, better segmentation performance may be achieved.", "reasons_to_reject": "1. Novelty seems insufficient. Although replacing the IO tagging scheme with BIO tagging may benefit sign language segmentation, the idea is not completely novel (as the authors mentioned, it is proposed in 1995 for similar NLP tasks. \n2. Experimental results seems not convincing enough. Only one baseline is compared to. Is that the SOTA model? Comparison to more methods or models may be needed.\n3. The improvement brought by optical flow and 3D hand norm seems trivial. And by using them, the IoU and percentage of segments even drops. Is there any analysis or explanation about this?\n4. Results in Table 2 cannot demonstrate the superiority of the proposed method, since the conditions are different. Is it possible to train the proposed method with the MEDIAPI-SKEL dataset?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new scheme of sign language segmentation, which include two aspects: replacing the predominant IO tagging scheme with BIO tagging to account for continuous signing and segmenting phrases as well as segmenting signs. The authors also propose a method using optical flow and 3D hand normalization. Experimental results show the proposed method outperforms a baseline system.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PcmmpndFNI", "reviewer": "Reviewer_ha7w", "summary": "", "questions": "In table 2, you describe how E1s and E4s perform on the LSF and DGS datasets. are those modes have the best results on those dataset? I would like to see the performance of other experiments just to see how well they generalize.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Rebuttal \n\n#### Predictability of Using BIO Tagging\n**Reviewer’s Comment:**  \nThe use of BIO tagging instead of IO tagging is very predictable.\n\n**Response:**  \nWhile we acknowledge that BIO tagging itself isn't new, its application to sign language segmentation addresses specific challenges unaddressed thus far by the current literature.\n\n---\n\n#### Applicability Beyond Sign Language\n**Reviewer’s Comment:**  \nThe study is specific to sign language and does not see its applicability in other tasks.\n\n**Response:**  \nWorking on sign language should be a sufficient motivation. However, the segmentation approach could be leveraged in various other fields:\n\n1. **Gesture Segmentation in Spoken Language**: Spoken language often incorporates gestures and facial expressions, integral to human communication, enriching the semantic layer and aiding in emotional connection. Our segmentation is directly applicable here, allowing for a better analysis of this blend of verbal and non-verbal cues.\n2. **Action Segmentation**: In computer vision, breaking down a video into a sequence of actions is crucial for several applications. Our approach could be adapted to tackle this issue effectively. [Tang et al. 2019](https://arxiv.org/abs/1903.02874)\n3. **Speech Segmentation**: In scenarios where segmentation is performed without the accompanying transcription step. Our methods could be applied in these contexts.\n\n---\n\n#### Questions for the Authors\n**Reviewer’s Question:**  \nYou are interested in knowing how different experiments performed on the LSF and DGS datasets, questioning if E1s and E4s have the best results on those datasets.\n\n**Response:**  \nSince E1s and E4s are the most different (E4s includes all our engineered features), we find it interesting to apply them on another unseen dataset and compare them.\nWe posit our results in Table 2 specifically as a zero-shot, out-of-domain generalization. While the datasets are annotated differently, we can still see the improvement from our feature engineering, most importantly in F1 score, which leads us to believe the features we added are indeed important for out-of-domain generalization.\nHowever, we agree that including results trained on the MEDIAPI-SKEL dataset could offer additional insights. We will incorporate this into the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental setup and the results are solid. The \"efficiency\" metric is very useful.\nThe idea of using 3D hand gestures is original.\nOverall speaking, this paper is well-written and easy to follow.\nI had some questions, but all of them were answered in the limitations section.", "reasons_to_reject": "The idea of using BIO instead of IO is very predictable.\nThe study is very specific to sign language, I don't see how to use it in other tasks.\nother then that, I do not see a clear reason to reject it.", "questions_to_authors": "In table 2, you describe how E1s and E4s perform on the LSF and DGS datasets. are those modes have the best results on those dataset? I would like to see the performance of other experiments just to see how well they generalize.", "paper_topic_and_main_contributions": "The work suggest a method for automatic segmentation of sign language. The novelty in the paper is the use BIO tagging scheme, using prosodic cues for marking phrase boundaries and using 3D hand normalization. To test their methods the authors conducted several experiments and obtained strong results. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ViRgilu0wR", "reviewer": "Reviewer_7cdW", "summary": "", "questions": "A.Why not use additional publicly available corpora to increase training data and generalization? Aside from DGS, there are other European corpora (Swedish, Greek, etc.)\nB.The seq2seq and transformer models use attention and/or positional encoding, making the BIO tagging unnecessary. How would you defend your work against this statement?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Rebuttal\n\n#### Limited Improvement Over Baseline and Inconsistent Standard Deviations\n**Reviewer’s Comment:**  \nThe experimental results do not show significant improvements over the baseline and questions the standard deviations in Table 4.\n\n**Response:**  \nThe results given by E2s, E3s, and E4s indeed are not an improvement over E1s in the in-domain setting. \nThe added features, like optical flow, do improve the performance in shallower models. In deeper models, they offer more than just trivial in-domain improvements; they crucially enhance the model's adaptability to new contexts and languages (Table 2). \n\n---\n\n#### Analysis of Facial Expression Data and Hand Normalization\n**Reviewer’s Comment:**  \nWhy the inclusion of facial expression data and hand normalization did not lead to performance benefits.\n\n**Response:**  \nFor the face points, although the face is an essential component to understanding sign language experessions, and does play some role of it in sign and phrase level segmentation, we believe that the 128 face points are too dense for the model to learn useful information compared to the 75 body points, and may on the opppsite, confuse the model. We will include more speculations on this line in our paper.\nWhile the use of 3D hand normalization is well-justified in the paper, it does not help the model, due to poor depth estimation quality, as further corroborated by recent research [De Coster et al. 2023](https://arxiv.org/pdf/2306.17558.pdf). Therefore, we consider it a negative result, showing the deficiencies in the 3D pose estimation system. This analysis will be further clarified in the camera ready.\n\n---\n\n#### Length of Appendix and Justification\n**Reviewer’s Comment:**  \nYou comment on the extra-long appendix and its lack of obvious justification.\n\n**Response:**  \nYou have a very valid point. The intention was to provide comprehensive information, including quality estimation of our methodology, but we realize this may have led to unnecessary complexity. We will consider publishing the appendix separately as part of the technical documentation of the benchmark we are introducing, and focus on providing only essential information in the revised version of the manuscript.\n\n---\n\n#### Questions for the Authors\n**Reviewer’s Questions:**  \n1. Why not use additional publicly available corpora for training?\n\n**Response:**  \nDespite there being other sign language datasets publicly available, the training of sign-level segmentation requires accurate sign-level gloss annotation from continuous signing, and the training of phrase-level segmentation requires perfect alignment between sign language video and spoken language sentences, which is rarely the case in most datasets except the Public DGS Corpus dataset we use. \nWe do highlight two previous works and the dataset each use fulfills one of the requirements above, so we also compare our models to one of them on phrase-level segmentation. In the end, the models we trained show surprisingly good zero-shot performance even without seeing additional data in a different signed language.\n\n**Reviewer’s Questions:**  \n2. How to defend the necessity of BIO tagging against seq2seq and transformer models?\n\n**Response:**  \nWe are not entirely sure about the question, but we will try to clarify that the use of BIO tagging is about labelling the video data with the sign/phrase start and end time, so it is irrelavant to what model archtecture we use.\nIf the question is more about why we did not use a seq2seq and transformer model, the short answer is because the sequence of video frames is very long, and we have more detailed eleboration of this point in our limitation section “Encoding of Long Sequences”.\n\n---\n\n#### Reproducibility\nWe believe our work is highly reproducible. The code is available on GitHub (the links are broken due to the anonymity requirements), including all of the settings to train each model and variation presented in this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed model can be used in the outside domain for further data collection in the wild.", "reasons_to_reject": "Experimental results do not show an improvement over the suggested baseline E1, arguing against one of the contributions of the paper (Hand Norm)\nReported Stds in Table 4 indicate that reported performance differences might be insignificant.\nThe analysis of the results needs to be more comprehensive. Why does adding facial expression data lowers the performance, and why does not the proposed hand norm benefit it as well?\nThe paper has an extra six pages long second paper in Appendix section C, with no obvious justification. Author could just mention that they perform a linear transformation of 3D coordinates for normalization, that would be sufficient.", "questions_to_authors": "A.Why not use additional publicly available corpora to increase training data and generalization? Aside from DGS, there are other European corpora (Swedish, Greek, etc.)\nB.The seq2seq and transformer models use attention and/or positional encoding, making the BIO tagging unnecessary. How would you defend your work against this statement?", "paper_topic_and_main_contributions": "The paper proposes a BIO frame tagging method for sequences and single signs within those sequences. Authors argue that it is beneficial for segmentation tasks.\n\nThe paper proposes a 3D normalization method for handshapes using MediaPipe's hand model.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In the context of SL, more suitable terms are:\nword -> sign, gloss, or annotated unit/token\nphrase -> utterance  (line 216) \n\nReproducibility, Ethics Review, Anonymity Requirement and Overall Recommendation\nhttps://github.com/anon/anonymous\nhttps://github.com/anon/anonymous2\n\nBoth links did not work - returned 404; is that a placeholder?"}}]

Author rebuttal:

Meta-review: The paper introduces a novel approach to sign language segmentation by replacing the conventional IO tagging with BIO tagging. The authors also propose a method using optical flow and 3D hand normalization, resulting in improved segmentation quality and robustness across different signed languages and domains.
The proposed BIO tagging scheme and 3D hand normalization are discussed as potential improvements for sign language segmentation, but there are concerns about the experimental results and the significance of these contributions. The paper is well-written and easy to follow. While the use of BIO tagging is not considered highly original, the introduction of 3D hand normalization is seen as more novel. However, there are questions about the effectiveness of these contributions. While there is potential for the proposed method to be used in sign language segmentation, concerns are raised about the experimental results, the lack of a clear superiority over the baseline, and the applicability of the method to other tasks.

Pros:
- The paper is well-written and easy to follow.
- The use of 3D hand normalization is seen as a potentially original and valuable contribution.
- The paper addresses potential questions and limitations effectively.
- The proposed model may have applications in data collection for sign language in real-world settings.

Cons:
- Concerns are raised about the experimental results, including the lack of a clear superiority over the baseline and the drop in performance with the introduction of certain features, which the authors have addressed.
- The applicability of the proposed method to tasks outside of sign language segmentation is questioned but the authors have addressed.
Decision: Accept-Findings