Handshape-Aware Sign Language Recognition: Extended Datasets and Exploration of Handshape-Inclusive Methods
Authors: Xuan Zhang
Kevin Duh
Venue: EMNLP 2023 Findings (2023)
Keywords: Sign language recognition
handshape
TL;DR: This paper extends the existing PHOENIX14T dataset with handshape labels and proposes novel handshape-aware sign language recognition methods, which constantly outperform the baseline systems.

Abstract: The majority of existing work on sign language recognition encodes signed videos without explicitly acknowledging the phonological attributes of signs. Given that handshape is a vital parameter in sign languages, we explore the potential of handshape-aware sign language recognition. We augment the PHOENIX14T dataset with gloss-level handshape labels, resulting in the new PHOENIX14T-HS dataset. Two unique methods are proposed for handshape-inclusive sign language recognition: a single-encoder network and a dual-encoder network, complemented by a training strategy that simultaneously optimizes both the CTC loss and frame-level cross-entropy loss. The proposed methodology consistently outperforms the baseline performance. The dataset and code can be accessed at: www.anonymous.com.

Handshape-Aware Sign Language Recognition:
Extended Datasets and Exploration of Handshape Saaddsapp-Inclusive Inclusive Method
Xuan Zhang
Kevin Duh Johns Hopkins University
Johns Hopkins University xuanzhang@jhu.cm edu
kevinduh@cs. jhu. edu
Abstract
The majority of existing work sign lan- guage recognition encodes signed videos with- out explicitly acknowledging the phonological attributes of signs. Given that handshape is vital parameter in sign languages, we explore the potential of handshape-awar sign language recognition. We augment the PHOENIX 14T dataset with gloss-level handshape labels, re- sulting in the new PHOENIX14T-BB dataset. Two unique methods are proposed for handshape-inclusive sign language recogni- tion: single-encoder network and dual- encoder network, complemented by a train- ing strategy that simultancously optimizes both the CTC loss and frame-level cross- entropy loss. The proposed methodology consistently outperforms the baseline perfor- mance. The dataset and code can be accessed at https://itthbb.. http://ggituu..ooEEsseeeeee com/Este lee/sr handshape.git.
WHITE
LIKE
Figure 1: An example of handshape minimal pair in Both signs start from identical handshape but end with a distinct one. In practical scenarios, when a signer signs rapidly, the terminal handshape of LIKE may closely resemble that of WHITE, leading to poten- difficulties in differentiation.
ten representations of signs typically denoted by spoken language words. Among the array of SLR products, sign gloves, and wearable devices using sensors track hand movements, widespread. However, these devices have faced criticism from the Deaf community, primarily due to the social stigma associated with wearing them. This feed- back has motivated us to explore video-based SLR, an alternative approach that utilizes cameras to record signs and feed them into the system video inputs. By doing inclusive society, preserving the valuable sign lan- guages serving as the heart of the Deaf culture, and thereby facilitating improved communications between the Deaf and hearing communitics.
1 Introduction
Sign languages are primarily the languages of Deaf people. They are the center of the Deaf culture and the daily lives of the Deaf community. In the U.S., estimates suggest that between 500,000 to million people communicate using American Sign Language (ASL), making it the fifth most- used minority language in the country after Span- ish, Italian, German, and French (Lane et al., 1996). Natural sign languages, which develop indepen- dently and possess unique grammatical structures distinct from surrounding spoken languages, just as crucial include in the field of natural lan- guage processing (NLP) any other language, as Yin al. (2021) advocates.
aspire to foster more
Signs can be defined by five parameters: hand- shape, orientation, location, movement, and non- manual markers such facial expressions. Signs that differ in only one of these parameters form minimal pairs. An example of a handshape min- imal pair in ASL is illustrated in Figure As reported by Fahey and Hilger (2022), among all parameters, handshape minimal pairs are identified with the lowest accuracy only 20%, compared to palm orientation (40%), location (47%), and movement (87%). This indicates the complexity in-
One direction in sign language processing (SLP) is sign language recognition (SLR), a task of recog- nizing and translating signs into glosses, the writ-
DDeaf sociolinguist Barbara Kannapell: "It is our language in every sense of the word. We create it, we keep alive, and it keeps us and our traditions alive." And further, "To reject ASL is to reject the Deaf person."
Images clipped from https//hahysignlanguug..com

---

Classifier
Linear
Softrax
CTC
Model II.
Predictor
Decoder
Sign Video
S3D
Head
Predictor
Glosses
Model I.
Joint Head
Predictor
Glosses
Head
Predictor
Glosses
Sign
Sign Video
S3D
Head
Predictor
+ Left HS
S3D
Head
Predictor
Leeft
Video
Head
Predictor
Right HS
Head
Predictor
Right HS
Figure 2: We propose two handshape-inclusiee SLR network variants. Model employs a single video encoder, while Model implements both a gloss encoder and a handshape (HS) encoder, applying joint head to the concatenated representations produced by the two encoders.
volved in distinguishing handshapes, underscoring their importance in correctly interpreting signs.
our approach, underscoring the adaptability our methods.
The majority of existing research on SLR does not incorporate phonological features such as hand- shapes into their system designs, with only a few ex-
We set benchmark on the PHOENIX14THH dataset with the proposed methods. Our models outperform previous state-of-the-art (SOTA) single- modality SLR networks, which utilize only RGB videos as input and were trained on PHOENIX14T.
ceptions (Koller al., 2016; Cihan Camgoz et al.,
2017; Koller et al., 2019). Typically, signs are inter- preted cohesive whole, meaning that SLR model is expected correctly recognize all five parameters simultaneously to accurately identify a sign. This constitutes a major distinction between spoken and sign languages the former is linear, while the latter incorporates both linearity and si- multaneity (Hill et al., 2018). This uniqueness introduces considerable challenges to SLR tasks.
Related Work
2.1 SLR
In recent developments of CSLR, a predominant methodology has emerged that employs hybrid model. The model usually composed of three essential components: a visual encoder, which ex- tracts the spatial features from each frame of the sign video; a sequence encoder, responsible for learning the temporal information; and an align- ment module which monotonically aligns frames to glosses. The visual encoder component could be built with various architectures, including 2D- CNNs (Koller al., 2019; Cheng al., 2020; Min et al., 2021), 3D-CNNs (Chen et al., 2022a,b), or 2D-CNNs followed by ID-CNNs (Papastratis et al., 2020; Pu et al., 2020; Zhou et al.. 2021a,b), The sequence encoder can be implemented using LSTMs (Cui et al., 2019; Pu et al., 2020; Zhou et al., 202 Transformer encoders (Niu and Mak, 2020; Camgoz et al., 2020; Zuo and Mak, 2022: Chen et al., 2022b), or ID-CNNs (Cheng et al., 2020). terms the alignment module, research attention has been redirected from HMM (Koller et al., 2017, 2019) to connectionist temporal classi- fication (CTC) (Hao et al., 2021; Zhou et al., 202 Zuo and Mak, 2022: Chen al., 2022b).
The limited interest in integrating handshapes into SLR systems can be attributed largely to the absence of handshape annotations in existing Con- tinuous SLR (CSLR*) datasets. response to this, we have extended one the most widely used SLR datasets, PHOENIX14T, with handshape annota- tions, sourced from online dictionaries and man- ual labeling, thus creating the PHOENIX14T-HS dataset. Our hope is that this will facilitate more research into handshape-aware SLR.
Moreover, we introduce two handshape- inclusive" SLR networks (Figure 2), designed with either single or dual-encoder architectures. These proposed models extend the basic SLR network, which doesn't t include handshape information in gloss prediction. Thus, any cxisting SLR can adopt
*CSLR refers to recognition of sign language at the sentence level, as opposed Isolated SLR. which operates word level. Our work
on CSLR R due broader practical application and higher level of complexity.
"We use handshape-awarn to denote SLR that incorpo- rates handshape information during training, while handshape- inclusive pertains to the deliberate inclusion of handshape predictions within SLR.
Various approaches have been proposed to im- prove SLR system performance.

---

Multi-stream network The multi-stream net- works use multiple parallel encoders extract features from distinct input streams. In addition to the RGB stream, Cui et al. (2019) incorporate an optical flow stream, while Zhou et al. (202 lb);
tem, where two separate streams are built to predict handshapes and glosses, respectively. These two streams are then merged and trained for gloss recog- nition. The aforementioned studies are all carried out on the PHOENIX 14 dataset, made possible by the efforts of Forster et al. (2014), which extended the dataset with handshape labels. Our work in- stead focuses the PHOENIX14T dataset.
Chen et al. (2022b) use key points. Koller et al. (2019) and Papadimitriou and Potamianos (2020) introduce two extract encoders for hand and mouth encoding, directing the system's 's focus towards crit- ical image areas.
Datasets
Cross-entropy loss Training objectives beyond CTC loss can also be employed. Cheng et al. (2020); Hao et al. (2021) train their models to also minimize the frame-level cross-entropy loss, with frame-level labels derived from the CTC decoder's most probable alignment.
We have enriched the SLR dataset PHOENIX14T by incorporating handshape labels derived from the Sign Writing dictionary and manual labeling. In the subsequent sections, we initially present the original 14T dataset (3.1) and the Sign- Writing dictionary (3.2), followed by detailed description of the updated PHOENIX14T dataset (PHOENIX14T-HS)) now featuring handshape la- bels (3.3).
2.2 Handshape-inclusive Datasets
Currently, datasets frequently employed for the continuous SLR task, such as RWTH-PHOONIX- Weather 2014T (Camgoz et al., 2018) and CSL Daily (Zhou et al., 2021a) generally lack hand- shape annotations, except for RWTH-PHOENIX- Weather 2014 (Koller et al., 2015), which is ex- tended by Forster al. (2014) with handshape and orientation labels. The annotating process in- volved initially labeling the orientations frame-by- by- frame, followed by clustering within cach orien- tation, and then manually assigning a handshape label to each cluster. Additionally, a subset of 2k signs is annotated using the SignWriting (Sutton and DAC, 2000) annotation system. To facilitate handshape recognition, Koller et al. (2016) intro- duced the AMillion-Hande dataset, comprising 1 million cropped hand images from sign videos, each labeled with a handshape. The dataset con- sists of two vocabulary-Lovel datasets in Danish and New Zealand sign language, where handshapes are provided in the lexicon, and continuous SLR dataset, PHOENIXI4, 14, annotated with SignWriting It also includes 3k manually labeled handshape test images.
3.1 PHOENIX14T
PHOENIX14T (Camgoz et al., 2018) is one of the few predominantly utilized datasets for SLR tasks nowadays. This dataset consists of German sign language (DGS) aired by the German public TV station PHOENIX in the context of weather fore- casts. The corpus comprises DGS videos from 9 different signers, glosses annotated by deaf experts, and translations into spoken German language. Key statistics of the dataset are detailed in Table. PHOENIX14 14T (Camgoz et al., 2018), an exten- sion of PHOENIX14 (Koller et al., 2015), features redefined sentence segmentations and a slightly reduced vocabulary compared to its predecessor. Despite Forster et al. (2014) having expanded PHOENIX14 with handshape labels, their extended dataset not publicly accessible and only includes labels for the right hand. contrast, our anno- tated data will be released publicly, encompassing handshapes for both hands.
3.2 SignWriting
The SignWriting dictionary (Sutton and DAC, 2000; Koller et al., 2013) publicly accessible, user- edited sign language dataset, encompassing more than 80 distinct sign languages. Adhering to the International SignWriting Alphabet, which pre- scribes a standard of icon bases, users represent signs via abstract illustrations of handshapes, facial expressions, orientations, and movements. These depictions can be encoded into XML format and
2.3 Handshape-aware SLR
Research on leveraging handshape labels to sup- port SLR has been relatively scarce. Koller al. (2016) applied the statistical modelling from Koller al. (2015) and incorporated a stacked fusion with features from the 1-Million-Hande model and full frames. While Cihan Camgoz et al. (2017) and Koller et al. (2019) utilized a multi-stream sys-

---

Sign
Gloss
MORGEN
TEMPERATUR
ACHT
BIS
DREIZEHN (THIRTEEN)
MAXIMAL
DREIZEHN (THIRTEEN)
(English)
(TOMORROW) (TEMPERATURE)
(EIGHT)
(TO)
(MAXIMUM)
Right Handshape
1
3
index
3
b_ b_thumb
Left Handshape
5
b thtmub
Figure 3: One sample from the PHOENIXI4T-HS where handshape labels have been appended for both hands atop the PHOENIX14T dataset. When loaded into a Python environment, this sample appears as a Python dictionary: [name: train/01 AAril ppril_2010____ Thursday Soansday_heut-6703. signer: Signer04, gloss: MORGEN TEMPERATUR ACHT BIS DREIZEINN MAXIMAI. DREIZEHN, hundshape-righr: [[7], [f], [3], [index], [3], [b (b_tumbb), [3]], handshape-left: (D) (f). (5). D. (I. (b thumb), [I]).
handshape
3.3 Handshape-extended PHOENIX14T (PHOENIXI4T-HS)
30000
There are 17,947 entries for DGS in the SignWrit- ing dictionary. However, 314 signs/glosses the PHOENIX14T dataset are either not included or lack handshape annotations in the dictionary (Ta- ble. This implies that 4,366 of the 7,096 samples in the train set contain signs devoid of handshape labels. We thus manually labeled these 314 signs. This results in the following annotation steps:
5000
thumb
4000
index_fex
ItaLopen
2000
Vndes
handishae--ee
1. Look up the SignWriting dictionary.
1300
2. Manually label handshapes for signs not present in Sign
6000
thumb
Index
$900
4300
The author, who has competent understand- ing of ASL and Sign Language Linguistics, yet lacks formal training DGS, annotated by simul- tancously watching the corresponding sign vidco to ensure alignment. The task proves particularly demanding when consecutive gaps-signs missing handshapes-emerge To delineate the boundaries of these signs, the author resorted online DGS dictionaries (not SignWriting). The entire manual annotation process took around 30 hours.
noihhntt
2000
somonnt
Figure 4: Top 10 most frequent handshapes for each hand in PHOENIXI4T-HS..
Our method contrasts with that of Koller et al. (2016), which applied frame-level handshape anno- tations. We have instead adopted gloss-level hand- shape annotations. While the frame-level approach is more detailed, Koller et al. (2016) reported a sig- nificant number of blurred frames, making the task of frame-by-frame labeling both challenging and time-intensive. Moreover, given that sign language recognition essentially a gloss-level recognition task, our aim is maintain consistency in granu- larity when integrating handshape recognition as an additional task within the framework.
converted into textual descriptions. We utilized the SignWriting parser provided by Koller et al. (2013) to extract handshapes for both hands from the original Sign Writing dictionary.
"http::/github.com. hhttss//gtthulccommmeeiinniiiin com//herlimm/siiiwiiinng parser "Note that one-handod signs do not have handshape labels for the left hand, which is the non non-dominant hand for this
signer.

---

PHOENIX14T
Train
Dev
Test
Propout
Sign Video
S3D
mma
PE
#samples
7,096
519
642
(Tx224x224:)
T/4x832
vocab.
1,066
393
411
avg. gloss len.
7.80
7.23
6.65
Gloss Probabiities
Softmax
Closifier
Linear
Inperform
Drepout
Dent
Ganrid
vocab. not in SW
299
70
1/4x512
T/KKG
vocab. missing HS in SW
15
9
CTC
#samples w/ missing HS
4,366
304
362
Gloss Predictions
Decoder
(U)
Table 1: Statistics of PHOENIX14T, In the train set, 299 signs are absent from the SignWriting (SW), and 15 signs lack handshape (HS) annotations in SW, which indicates that 4,366 samples include signs without hand- shape annotations. We thus manually annotated the combined total of 314 15) signs.
Figure 5: Architecture the vanilla SLR network. The head network is composed of orange and green blocks. T is the number of frames in the sign video. G is the size of the gloss vocabulary. L the length the predicted gloss sequence. RGB frames are resized to 224x224 during preprocessing.
We adopt the Danish taxonomy for handshape labels as in (Koller al., 2016), which includes 60 unique handshapes. The application results in the PHOENIX14T-HS dataset, from where an example is shown in Figure 3. Given that all 9 signers in PHOENIX14T are right-hand dominant, it is ap- propriate to employ the default annotations from SignWriting without necessitating side-switching. Figure 4 presents the frequency distribution of the top 10 most prevalent handshapes for each hand in the PHOENIX14T dataset. It is important to note that single sign may comprise multiple hand- shapes. In fact, 13.5% of the signs in the dataset incorporate more than hand, whereas the left hand employs more than one handshape in 5% of the signs.
we did not account for co-articulation, the transi- tion phase between two consecutive signs, in our handshape labeling.
Methods
The task of SLR can be defined as follows. Given an imput sign video V (U1,..., uT) with T frames, the goal is to learn network predict sequence of glosses g (01. .... gL) with words, mono- tonically aligned to T frames, where T L.
In this section, we start by describing the vanilla SLR network, where handshapes are not provided or learned during training in Section 4.1. We then introduce the two wuudshapp-inclusive network chitectures employed in this study in Section 4.2. Specifically, these networks designed to predict sign glosses and handshapes concurrently. Finally, we elaborate on our chosen training and pretraining strategy in Section 4.3 and 4.4.
handshape for the right
We would like to note certain limi- tations in the proposed PHOENIX14T-HE dataset. While approximately one-third of the signs were manually labeled with handshapes, the remaining two-thirds were labeled using the user-generated SignWriting dictionary. As a result, these hand- shape labels may contain noise and should not be seen as curated. When dealing with sign variants, i.e. multiple entries for a single sign, our selection process random and thus may not necessarily correspond with the sign video.
Limitations
4.1 Vanilla SLR networks
The architecture of the vanilla SLR network is il- lustrated in Figure 5. Similar to Chen et al. (2022a) and Chen et al. (2022b), we use an S3D (Xie et al.. 2018) as the video encoder, followed by head network, where only the first four blocks of S3D are included to extract dense temporal representa- tions. Then, a gloss classification layer and a CTC decoder are attached to generate sequences of gloss predictions.
Moreover, individual signers possess unique signing preferences, leading them to opt for differ- ent sign variants. Furthermore, the signers might deviate from the dictionary-form signs, resulting in discrepancies between the real-world usage and the standardization form. In terms of our labeling process, we omitted handshape labels during the initial and final moments of each video, when the signers' hands are a resting position. Finally,
4.2 Handshape-inelusive SLR networks
Figure 2 depicts our proposal of two handshape- inclusive SLR network variants, which are expan- sions upon the vanilla network. Both variants ex- plicitly utilize handshape information by training

---

the model to predict glosses and handshapes con- currently. The key distinction between the two variants lies the employment of either a single encoder or dual encoders.
by our models. First, we extract the best path for glosses from a CTC decoder and fill in the blanks with neighboring glosses. After this, a particular gloss has only one associated handshape, we assign that handshape to all frames within the extent of that gloss. If there is more than one handshape, we gather the handshape probabilities produced by the handshape classifiers within that segment and feed them into CTC decoder to determine the optimal handshape labels for the frames within that gloss's range". Finally, we calculate the cross- entropy loss between the pscudo-labels and the handshape probabilities. This enables more fine- grained frame-level supervision.
Model I. In comparison to the vanilla network, this model forwards the S3D feature to two addi- tional heads, each tasked with predicting the hand- shapes for the left and right hand respectively. The loss for this model computed as follows:
Madel, = CETC + X Eêre + X Crro (1)
where fere represents the CTC loss of the gloss predictor. f'Te and fETO denote the CTC
losses for the left and right handshape predictors. weighted by A and R
The loss function then becomes:
CSLR = Modelyy + NCEECE AB AE (3)
Model II. This model employs dual encoders, each dedicated to encoding the representations for glosses and handshapes independently. While both encoders receive the same input (sign videos) and share the same architecture, they are trained with different target labels (gloss vs. handshape). We also incorporate a joint head, which combines the visual representation learned by both encoders to generate gloss predictions. The architecture of this joint head mirrors that of the gloss head and the handshape head. Therefore, the loss for this model is computed as follows:
where L'D and L& are cross-entropy loss for left and right hand weighted by N and X&E respec- tively.
4.4 Pretraining
Given that our target dataset is relatively sparse, it's crucial to pretrain the model to ensure solid ini- tialization. We first pretrain the S3D encoder on the action recognition dataset, Kinetics-400 (Kay et al., 2017), consisting of million video clips. Follow- ing this, we further pretrain on a word-level ASL dataset, WLASL (Li et al., 2020), which includes 21 thousand videos.
Modely ECro CCCrotCeTot + CoTO + X C
(2) where fOTe denotes the CTC loss of the joint gloss predictor.
Experiments
For this model, we also adopt a late ensemble strategy. This involves averaging the gloss prob- abilities predicted by both the gloss head and the joint head. The averaged probabilities are then fed into CTC decoder, producing the gloss sequence.
In this section, we present the performance of our top-performing model (Section 5.1) and further conduct ablation study (Section 5.2) to analyze the crucial components of our implementations.
4.3
Training strategy
5.1 Best model
The CTC loss is computed by taking the negative logarithm of the probability of the correct path, which corresponds to the true transcription. It is a relatively coarse-gras poarse-grained metric because it operates at the gloss level, not requiring temporal boundaries of glosses. Given that handshape prediction could potentially operate on a frame level, it stands to reason for us to compute the loss at this level as well. However, as the PHOENIX14T-HS dataset does not provide temporal segmentations, opt to estimate these with gloss probabilities?
Our tighest-performing system utilizes the dual- encoder architecture of Model II. After initial pre- training on Kinetics-400 and WLASL datasets, we freeze the parameters the first three blocks of the S3D. For the hyperparameters, we set A and R to 1, while A and XBE are set to 0.05, The initial learning rate is 0.001. Adam is used as the optimizer.
A comparison of our premier system (HS-SLR) with leading SLR methods on the PHOENIXI4T
generated
*For the left hand, when the sign does not have correspond- ing handshape, we label it with the special token <pad>.
For Model II, they are the averaged prohabilitics.

---

Method
Dev
Test
Model
Hands
Dev
Test
CNN-LSTM (Koller et al., 2019)*
22.1
24.1
Vanilla
23.54
23.69
SFL (Niu and Mak. 2020)
25.1
26.1
Model I
right
23.22
22.94
FCN (Cheng et al., 2020)
23.3
25.1
Model
right + left
22.52
23.43
Joint-SLRT (Camgoz et al., 2020)
24.6
24.5
Model II
right
21.06
22.56
CMA (Papastratis et al., 2020)*
23.9
24.0
Model II
right + left
21.03
22.07
SignBT (Zhou et al., 2021a)
22.7
23.9
Model II+*
right
21.51
22.12
MMTLB (Chen et al., 2022a)
21.9
22.5
Model II+
right left
21.86
22.40
SMKD (Hao et al., 2021)
20.8
22.4
(Ssr)))(ours)
20.3
21.8
Table 3: Performance comparison of model variants.
Models marked with are variants that employ only the right handshape head. Please note that the experimental setup differs from the HS-SLR model presented in Table Here, we unfreeze parameters in S3D and exclude the optimization of the cross-entropy loss.
STMC-R (Zhou et al., 2
19.6
21.0
CLR (Zuo and Mak, 2022)*
20.5
20.4
TwoStream (Chen et al., 2022b)*
17.7
19.3
Table 2: Comparison with previous work on SLR on PHOENIX147 14T evaluated by WER. The previous best results are underlined. Methods marked with denote approaches that utilize multiple modalities besides RGB videos, such as human body key points and optical flow. Notably, our best model (Model III achieves the lowest WER among single-modality models.
5.2.2 Pretraining for gloss encoder
We delve into optimal pretraining strategies for the S3D encoder that's coupled with gloss head. We conduct experiments using Model I, as shown in Table 4. We contrast the efficacy of four distinct pretraining methodologies: (1) pretraining solely on Kinetics-400; (2) sequential pretraining, first on Kinctics-400, followed by WLASL; (3) triple- tiered pretraining on Kinetics-400, then WLASL, and finally on handshape prediction by attaching two handshape heads while deactivating the gloss head; and (4) similar threc-stage stace process, but focusing gloss prediction in the final step.
dataset shown in Table 2. While we do not sur- pass the existing records, our system ranks as the top performer among all single-modality models. It is worth noting that the extension of multi-modality models into mundshape-inclusii models, such as introducing handshape heads additional hand- shape encoder the original networks, could po- tentially enhance the SOTA performance further."
Pretrained tasks
Dev
Test
5.2
Ablation study
Kinetics
23.65
24.77
Kinetics + WLASL
22.52
23.43
5.2.1
Model variants
Kinetics + WLASL HS
22.42
23.71
In our amalysis, we contrast our suggested model variants, Model I and Model II (discussed in Sec- tion 4.2), with the Vanilla SLR network (described in Section 4.1). Additionally, we compare models that feature solely a right handshape head against those equipped with two heads, one for each hand. An extended variant, Model II+, which adds two handshape heads to the gloss encoder, is also con- sidered in our experimentation.
Kineties + WLASL Gloss
22.31
23.71
Table 4: Performance of Model I with different pertain- ing strategies. HS pretrains the model on predicting the handshapes only (the gloss head deactivated). Gloss pretrains the model predicting the glosses only (the handshape heads are deactivated).
5.2.3 Pretraining for handshape encoder
As demonstrated in Table 3, Model II outper- forms Model and Model II+. The performance differs marginally between models with handshape heads for both heads versus those with single rinh head.
Table outlines various pretraining strategies adopted for the handshape encoder in Model II. The results pertain to right handshape predictions the PHOENIXI4T-HS dataset. Both Kinetics and WLASL employed for gloss predictions, as they lack handshape annotations. We also test the 1-Million-Hands dataset (Koller et al., 2016) for pretraining purposes. This dataset comprises a million images cropped from sign videos, each
"Due to computational resource constraints, we are cur- rently unable to fit such models into our GPU devices. Future work may explore the expansion of multi mulii-modality models to include handshape-inclusive models.

---

Pretrained tasks
Input
Dev
Test
Pseudo-labels
weight
Dev
Test
None
full
52.81
50.75
Ensemble
0.01
20.50
21.95
None
hand
73.38
72.01
Ensemble
0.05
20.26
21.79
Kinetics full
26.92
27.81
Ensemble
0.1
21.22
21.77
Kinetics
hand
40.56
40.18
Ensemble
0.5
20.31
22.07
Minetics+WAALL
full 25.76
25.74
Ensemble
20.63
21.86
Kinetics+LLASL
hand
43,45
40.05
HS Ensemble, HS
1
21.38
22.05
1-Million-Hands
full 47.59
46.82
1,
20.87
21.86
1-Million-Hands hand
59.04
57.68
Table 7: Performance of Models II with varying weights and methods acquiring pseudo-labels used in cross- entropy loss calculation.
Table 5: Performance of right handshape predictions on PHOENIX14 14T-HS with various pretraining strategies for the S3D encoder.
search by offering bandshape-cnriched dataset, PHOENIX14T-HS,, and proposing two distinctive handshape-inclusive SLR methods. Out method- ologies maintain orthogonality with existing SLR architectures, delivering top performance among SLR models. Our goal is to draw increased attention from the research community toward the integration of sign language's phono- logical features within SLR systems. Furthermore, invite researchers and practitioners in NLP to contribute to the relatively nascent and challenging research area of SLP, thus fostering a richer un- derstanding from linguistic and language modeling perspectives.
labeled with handshapes. To adapt these images for S3D, we duplicate each image 16 times, creat- ing a 'static' video. Furthermore, we experiment with two input formats: the full frame and the right- hand clip. As indicated in Table 5, both pretraining and full-frame input significantly outperform their counterparts.
5.2.4 Frozen parameters
We evaluate the impact of freezing varying num- bers of blocks within the pretrained S3D encoders in Model II. The results are presented in Table 6.
Frozen blocks
Dev
Test
20.55
22.26
In future work, we would like to explore three primary avenues that show promise for further ex- ploration: (1) Extension of multimodal SLR mod- els. This involves expanding multi-moddlity SLR models, which use inputs various modalities like RGB, human body key points, and optical flow, to become handshape-aware. This approach holds potential as different streams capture distinct as- pects of sign videos, supplying the system with richer information set. (2) Contrastive learning. Rather than using handshape labels supervision, they can be employed generate negative exam- ples for contrastive learning. This can be achieved by acquiring the gloss segmentation from the CTC decoder and replacing the sign in the positive exam- ples with its counterpart in the handshape minimal pair. The resulting negative examples would be par- ticularly challenging for the model to distinguish, thereby aiding in the development of better repre- sentations. (3) Data augmentation. Alternatively, create negative examples for contrastive learn- ing, the data volume could be increased using the same method that generates negative examples for contrastive learning.
1,
20.55
22.26
1, 2, 3
20.34
22.21
1, 3,
20.66
21.74
Table 6: Performance Model II with various frozen blocks in both S3D encoders.
5.2.5 Cross-entropy loss
In Table 7, we investigate the computation of cross-entropy loss handshape predictions uti- lizing pseudo-labels obtained via two methods: En- semble and HS. The former pertains to pseudo- labels gathered as outlined in Section 4.3, while the latter relies on CTC decoder-applied handshape probabilities from the handshape head to produce pseudo-labels. We also examine hybrid approach (Ensemble, HS), which sums the losses from both methods. In addition, we tune the weights > and Xlt in Equation 3, setting them to the same value.
6 Conclusions
this work, we introduce the concept of handshape-aware SLR, enriching this area of re-

---

Linitations
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, and Richard Bowden. 2017. Subunets: End-to-end hand shape and continuous sign language recognition. In Proceedings of the IEEE international conference on computer vision, pages 3056-3065.
Noisy labels: As highlighted in Section 3, the handshape labels we create might be noisy, since two-thirds of them are from a user-edited online dictionary. Additionally, these labels may not cor- respond perfectly to the sign videos due to the vari- ations among signers and specific signs.
Runpeng Cui, Hu Liu, and Changshui Zhang. 2019. A deep neural framework for continuous sign language recognition by iterative training. IEEF Transactions on Multimedia, (17):1880-1891.
Single annotator: Finding DGS experts to serve as our annotators proved challenging. Also, ob- taining multiple annotators and achieving inter- annotator agreements proved to be difficult.
Taylor Fahey and Allison Hilger. 2022. Impact of man- ual american sign language parameters on intelligi- bility.
The dataset used in our study does not account for other sign language param- eters including orientation, location, movement, and facial expressions. Morcover, these parameters are not explicitly incorporated as subsidiary tasks within our SLR methodologies.
Single parameter:
Jens Forster, Christoph Schmidt, Oscar Koller, Martin Bellgardt, and Hermann Ney. 2014. Extensions of the sign language recognition and translation corpus twth-phoenix,
In LREC, pages 1911 1916..
Aiming Hao, Yuecong Min, and Xilin Chen. 2021. Self- mutual distillation learning for continuous sign lan- guage recognition, In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 11303-1133 11312.
Single dataset: We only extend single dataset with handshape labels. It remains to be seen whether the methods we propose will prove equally effective on other datasets, featuring different sigm languages, domains, or sizes.
Joseph C Hill, Diane C Lillo-Martin, and Sandra K Wood. 2018. Sign languages: Structures and con- texts. Routledge.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang. Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The kinetics human action video dataset. CoRR, abs/1705 10550006006
References
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neu- ral sign language translation. In Proceedings of the IEEE conference compuler vision and patlern recognition, pages 7784-7793.
Oscar Koller, Necati Cihan Camgoz, Hermann Ney, and Richard Bowden. 2019. Weakly supervised learn- ing with multi-stream enn-Istm-hmms sequential parallelism in sign language videos. IEEE transactions on pattern analysis and machine intelli- gence, 42(9):2306- 2320.
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign language trans- formers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF con- ference computer vision and pullern recognition, pages 10023-10033.
discover
Oscar Koller, Jens Forster, and Hermann Ney. 2015. Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers. Computer Vision and Image Under- standing. 141:108-125.
Yutong Chen, Fangyun Wei, Xiao Sun. Zhirong Wu. and Stephen Lin. 2022a. A simple transfer learning baseline for sign language transla- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5120-5130.
Oscar Koller, Hermann Ney, and Richard Bowden. 2013. May the force be with you: force-aligned signwriting for automatic subunit annotation of corpora. In 2013 10th IEFE International Conference and Workshops on Automatic Face and Gesture Recognition ( (FG), pages 1-6. IEEE.
Yutong Chen, Ronglai 7 Fangyun Wei, Yu Wu, Shu- jie Liu, and Brian Mak. 2022b. Two-stream network for sign language recognition and translation. arXiv preprint arXiv:2211.01367.
Ka Leong Cheng. Zhaoyang Yang, Qifeng Chen, and Yu-W wing Tai. 2020. Fully convolutional networks for continuous sign language recognition. In Com- puter BssvvnEECVV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28. 2020, Proceed- ings, Part XXIV 16, pages 697-714. Springer.
Oscar Koller, Hermann Ney, and Richard Bowden. 2016. Deep hand: How to train a cnn on million hand images when your data is continuous and weakly labelled. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3793-3802.

---

Oscar Koller, Sepehr Zargaran, and Hermann Ney. 2017. Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent cnn-hmms. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4297-4305.
Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. 2021b. Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transac- rions on Multimedia, 24:768-779.
Ronglai Zuo and Brian Mak. 2022. C2str: Consistency- enhanced continuous sign language recognition. In Proceedings of the IEEF/CVE Conference on Com- puter Vision and Pattern Recognition, pages 5131- 5140.
Harlan Lane, Robert Hoffieister, and Benjamin Bahan. 1996. A journey into the deaf-world. Dawn Sign Press.
Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. 2020. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 1459-1469)
Yuecong Min, Aiming Hao, Xiujuan Chai, and Xilin Chen, 2021. Visual alignment constraint continu- ous sign language recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1152-1155!!
Zhe Niu and Brian Mak. 2020. Stochastic une-grained labeling multi-state sign glosses for continuous sign language recognition. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 172-186. Springer.
Katerina Papadimitriou and Gerasimos Potamianos. 2020. Multimodal sign language recognition via tem- poral deformable convolutional sequence learning. Interspeech, pages 2752-2756.
Maas Papastratis, Kosmas Dimitropoulos, Dimitrios Kon- stantinidis, and Petros Daras. 2020. Continuous sign language recognition through cross-modal alignment of video and text embeddings in a joint-latent space. IEFE Access, 8:91 170-91 91000
Junfu Pu, Wengang Zhou, Hezhen Hu, and Houqiang Li. 2020. Boosting continuous sign language recog- nition via cross modality augmentation. In Proceed- ings of the 28th ACM International Conference on Multimedia, pages 1497-1505.
Valerie Sutton and DAC. 2000. Sign writing. Deaf Action Committee (DAC). Technical Report.
Saining Xie, Chen Sun, Jonatham Huang, Zhuowen Tu, and Kevin Murphy. 2018. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European con- ference on computer vision (ECCV), pages 305-321.
Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. 2021. Including signed languages in natural language processing. arXiv preprint ar..v22105.05222.
Hao Zhou, Wengang Zhou, Weizhen Qi. Junfu Pu, and Houqiang Li. 2021a. Improving sign language trans- lation with monolingual data by sign back-translation. In Proceedings of the IEEF/CVF Conference Com- puter Vision and Pattern Recognition, pages 1316 1325.


Reviews:
[{"review_id": "iX325UsVrP", "reviewer": "Reviewer_ZC2W", "summary": "", "questions": "- Why Model II+ performs worse than Model II?\n- Between the proposed single and two encoder models, it was not clear why two encoder model performs better. ", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n# In response to your questions:\n> *Why Model II+ performs worse than Model II?* \n\nModel II is designed with two encoders: a gloss encoder and a handshape encoder. Model II+ is an extension of Model II that includes two additional handshape heads in the gloss encoder, in addition to the gloss head present in Model II. Although this addition would ostensibly improve the model, the Word Error Rate (WER) for Model II+ was 22.40, slightly higher than the 22.07 WER of Model II. \nWe believe this unexpected result stems from the learning focus of the gloss encoder. The primary function of the gloss encoder is to predict glosses, however, the addition of handshape heads in Model II+ may have inadvertently shifted the focus towards handshape predictions instead. As the evaluation of the systems is based on gloss predictions, this shift in focus could potentially have a detrimental effect on performance.\n>*Between the proposed single and two encoder models, it was not clear why two encoder model performs better.*\n\nInitially, it was unclear whether the single-encoder or the two-encoder model would yield superior results, which is why we included both versions in our study and compared their performances. Our intention was to provide a comprehensive reference for others in the field of sign language recognition (SLR)  and to suggest that it might be worthwhile to experiment with both models in practical applications.\nMoreover, we hypothesize that the observed performance difference could be related to the learning focus of the encoders, as mentioned in response to your first question. The two-encoder model allows each encoder to specialize in gloss or handshape prediction, potentially leading to better results overall.\n\nWe hope this explanation addresses your questions and provides a clearer understanding of our findings.\n\n# In response to the critiques:\n> *Authors employ single annotators to build the proposed dataset and it might have noisy labels.*\n\nWe acknowledge the concern regarding the use of single annotators to build the proposed dataset and the potential for noisy labels as a result. We concur that our labels are noisy and that the quality of the dataset could have been enhanced by employing multiple annotators. Nonetheless, we firmly believe that this new dataset will be a valuable resource for future research in sign language recognition.\nFirstly, the currently available PHOENIX14T dataset is relatively small, containing only 7,096 sentences, which limits its utility in real-world applications. Therefore, there is a pressing need to create larger, more comprehensive datasets. Our work, which successfully demonstrates handshape-aware SLR on the PHOENIX14T dataset, underscores the importance of including handshape annotations when developing new datasets.\n\nSecondly, handshape labeling is both costly and inherently noisy due to the challenges associated with determining handshapes during coarticulations (transitions between two signs). However, our study demonstrates that even with handshape labels obtained from a user-edited, publicly available dictionary and a single annotator, it is possible to improve the performance of SLR systems. This is a promising finding, as it suggests that handshape labels are beneficial, even when they are somewhat noisy. \nIn conclusion, despite the limitations associated with using a single annotator and potentially noisy labels, our work highlights the value of handshape annotations and provides a foundation for the development of more advanced and practical SLR systems.\n>*The proposed model utilizes extra label information (handshape labels) to outperform the baselines, which may not be fair comparisons.*\n\nThe primary objective of our study is to investigate whether the inclusion of additional handshape label information can lead to the development of more effective SLR systems. As such, our intention is not to create an unfair advantage over baseline models but rather to explore ways to enhance the performance of SLR systems in a practical context.\n\nWe hope our responses have addressed your concerns adequately. With these considerations in mind, we respectfully request that you reconsider your evaluation of our paper. We are committed to addressing any further questions or concerns you may have and are always open to ongoing dialogue. Thank you for your time!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Authors build a new augment dataset for SLR with handshape annotations, which may be useful for subsequent research.\n- This paper proposed the models for SLR and the best model outperforms the baselines of single modality (i.e., video).\n", "reasons_to_reject": "- Authors employ single annotators to build the proposed dataset set and it might have noisy labels.\n- The proposed model utilizes extra label information (handshape labels) to outperform the baselines, which may not be fair comparisons.\n", "questions_to_authors": "- Why Model II+ performs worse than Model II?\n- Between the proposed single and two encoder models, it was not clear why two encoder model performs better. ", "paper_topic_and_main_contributions": "Authors tackles the problem of Sign Language Recognition (SLR), and augment the existing the PHOENIX14T dataset with handshape labels, called PHOENIX14T-HS dataset. Then they propose two model variations with single encoder and two encoders. In experimental results, they show that the proposed approach outperforms baseline models of single modality (i.e., video).", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Obj2SAKEzE", "reviewer": "Reviewer_uuuR", "summary": "", "questions": "Question A:\nGiven the significance of handshapes in SLR, how do you envision the applicability and adaptability of your proposed methods to sign language translation tasks?\n\nQuestion B:\nThe paper mentions the potential noise in handshape labels, primarily sourced from a user-edited online dictionary. Could you provide more insights into the validation process of these labels? How do you ensure the reliability of such labels, especially when considering the variations among signers and specific signs?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n# In response to your questions:\n> *Question A: Given the significance of handshapes in SLR, how do you envision the applicability and adaptability of your proposed methods to sign language translation tasks?*\n\nThank you for raising this important question. The applicability and adaptability of our proposed methods to sign language translation tasks are indeed promising. Our SLR system can be readily adapted to a sign language translation system, and this adaptability is not unique to our model; it is a feature of most SLR systems. There are two primary approaches to this adaptation: the end-to-end system and the cascaded system, both of which have parallels in speech translation.\n\nFor the end-to-end system, we would retain both the gloss and handshape encoders and attach a sequence-to-sequence decoder, such as a transformer decoder, to the system. This would enable videos to be mapped directly to translations in spoken languages. On the other hand, for the cascaded system, we would first obtain the output glosses from the SLR system and then use them as input to train a gloss-to-text machine translation system, which could be a transformer model.\n\nBoth approaches have their merits and can be chosen based on specific requirements and constraints of the task.\n\n> *Question B: The paper mentions the potential noise in handshape labels, primarily sourced from a user-edited online dictionary. Could you provide more insights into the validation process of these labels? How do you ensure the reliability of such labels, especially when considering the variations among signers and specific signs?*\n\nWe acknowledge that obtaining clean and trustworthy handshape labels is ideal but also inherently challenging due to the application of a dictionary to a sentence-level dataset, and the variations among signers and specific signs. This means that it is not guaranteed that the handshape label from the dictionary will be accurate for signs in the sentence-level dataset. Verifying the accuracy of handshape labels for each example in the dataset is a time-consuming process. Given our limited resources, we simplified the validation process by randomly sampling approximately 100 sentences from the labeled dataset and correcting handshape labels where mistakes were identified.\n\nDespite these challenges, we believe that our approach of applying a dictionary to sentence-level datasets has practical value. The field of sign language recognition is still emerging, and there are limited datasets available. Building larger, real-world reflective datasets is a crucial challenge to address for the community. Our paper underscores the importance of including handshape labels in datasets. However, annotating handshapes and ensuring the accuracy of every handshape in a specific dataset entails a significant additional workload, which would need to be repeated for each new dataset created in the future. A more practical approach is to build a sign dictionary or leverage an existing online dictionary, which could substantially save effort and resources. Although this method comes with unavoidable noise, our research demonstrates that it still leads to improvements in the SLR system. \n\nIn conclusion, the inevitability of noise in the labels mirrors real-world scenarios, and our study presents a promising and successful example of leveraging such noisy labels to enhance SLR systems.\n\n# In response to the critiques:\n> 1. *Insufficient details and comparative analysis: while other methods utilize multiple modalities, the proposed method in this paper leverages additional labels and is pretrained.*\n\nWe appreciate your feedback regarding the details and comparative analysis of our method. Our study indeed focuses on leveraging additional labels and pertaining.\n\n- Multiple modalities: Our study utilizes a single modality (RGB), consistent with several previous works such as SFL (Niu and Mak, 2020), FCN (Cheng et al., 2020), and others mentioned. As shown in Table 2, our model outperforms these previous works, all of which adopted a single modality. Additionally, our method is adaptable to different modalities; it can be easily extended into a multi-modal system by adding extra encoders, such as keypoint and optical flow encoders, which holds promise for further performance improvement.\n\n- Additional labels: The main focus of our study is to determine whether adding additional labels improves the performance of SLR systems. CNN-LSTM (Koller et al., 2019) is a comparable system that also adopted extra labels, yet our system outperforms theirs (Table 2, 21.8 vs. 24.1 WER).\n\n- Pretraining: Several studies, including SignBT (Zhou et al., 2021a), MMTLB (Chen et al., 2022a), and SMKD (Hao et al., 2021), have adopted pretraining strategies. We not only compare our results with these studies in Table 2 but also provide a thorough examination of various pretraining strategies in section 5.2.2 (Pretraining for gloss encoder) and section 5.2.3 (Pretraining for handshape encoder).\n\nWe strive to make our experiments and ablation studies as comprehensive as possible. If there are any specific experiments or details you believe are necessary to better justify our claims, we would be grateful for your suggestions. Our primary claim is that handshape-aware SLR is necessary, and we believe our work provides substantial evidence to support the assertion.\n\n>2.*Experimental validation and generalizability: The results are validated on a single dataset. The potential impact of noisy labels compounds these concerns.*\n\n- Single dataset: We acknowledge the limitation of validating our results on a single dataset and agree that testing on additional datasets would provide a more robust justification for our claims. However, we would like to point out that the SLR community currently has only two readily available sentence-level datasets: the PHOENIX dataset and a Chinese sign language dataset, CSL-Daily. We initially planned to conduct experiments with CSL-Daily as well, but the SignWriting dictionary has extremely limited coverage of Chinese sign language. This means that to test on CSL-Daily, we would need to label the dataset ourselves, a task that, given our limited resources, is nearly impossible. We have done our best to conduct meaningful research within these constraints.\n\n- Noisy labels: We openly acknowledge that our labels are noisy. As mentioned in our response to your second question, we believe that noisy labels are a realistic reflection of real-world scenarios. Moreover, even with noisy labels, our handshape-aware SLR still achieves SOTA results. This underscores the success of our approach, as it suggests that clean labels could potentially lead to even better performance.\n\n# In response to the concerns of the reproducibility:\n\nEnsuring reproducibility is of utmost importance to us, and we have taken several steps to facilitate this. As stated in the abstract, we have committed to making our dataset and code publicly available. Additionally, we have created a concerted effort to provide detailed information about the system configuration and experimental setups in the paper.\nWe understand that reproducibility encompasses various aspects, and we are open to making any additional modifications that may be necessary. If there are specific elements that you feel are missing or could be further elaborated upon to enhance reproducibility, we would greatly appreciate your guidance. We are fully committed to addressing any gaps and providing all the necessary information to ensure that our work can be accurately replicated by others.\n\nWe sincerely appreciate the time and effort you have invested in reviewing our paper and considering our detailed response. It is our firm belief that this paper will have a meaningful impact on the sign language processing community, and hopefully, we have done our utmost to communicate this belief clearly through both our response and the original paper. We kindly ask you to take all of this into account and reconsider the value of the work. Thank you very much for your consideration!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Innovation: The paper introduces the concept of handshape-aware SLR, marking a relatively novel direction in the field.\n2. Dataset Contribution: The provision of the handshape-enriched dataset, PHOENIX14T-HS, offers a valuable resource for researchers.\n3. Methodology: Two unique handshape-inclusive SLR methodologies are proposed, maintaining orthogonality with existing SLR architectures, and suggesting practical applicability.\n4. Future Directions: The paper outlines several promising avenues for future research, offering insights for further exploration by researchers.", "reasons_to_reject": "1. Insufficient Details and Comparative Analysis:\n\nThe paper introduces two distinct SLR methods but could benefit from a more detailed exposition of experimental procedures and validations to firmly back its primary claims.\nA deeper comparative analysis with existing works is essential. While other methods utilize multiple modalities beyond just RGB videos, such as human body key points and optical flow, the proposed method in this paper leverages additional labels and is pretrained. This distinction necessitates a more thorough comparison to truly ascertain the advantages of the introduced methods over existing ones.\n\n2. Experimental Validation and Generalizability:\nThe experimental section might need further elaboration and validation to solidify the paper's primary arguments.\nThe results are validated on a single dataset, raising concerns about the generalizability of the proposed methods. The potential impact of noisy labels, especially when sourced from a user-edited online dictionary, and the challenges tied to relying on a single annotator further compound these concerns.", "questions_to_authors": "Question A:\nGiven the significance of handshapes in SLR, how do you envision the applicability and adaptability of your proposed methods to sign language translation tasks?\n\nQuestion B:\nThe paper mentions the potential noise in handshape labels, primarily sourced from a user-edited online dictionary. Could you provide more insights into the validation process of these labels? How do you ensure the reliability of such labels, especially when considering the variations among signers and specific signs?", "paper_topic_and_main_contributions": "The paper primarily delves into the concept of handshape-awareness in Sign Language Recognition (SLR). The authors have augmented the PHOENIX14T dataset by introducing handshape labels, leading to the creation of the new PHOENIX14T-HS dataset. Two distinct handshape-inclusive SLR methodologies are proposed: a single-encoder network and a dual-encoder network. A training strategy that concurrently optimizes both the CTC loss and frame-level cross-entropy loss is also introduced. The proposed methodologies consistently surpass the baseline performance.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AMVy7nIEsg", "reviewer": "Reviewer_TyCu", "summary": "", "questions": "1. Considering the challenges in finding DGS experts as annotators, how did you address potential biases or inconsistencies in the annotations? Did you perform any inter-annotator agreement analysis to assess the reliability of the handshape labels?\n\n2. While the PHOENIX14T-HS dataset focuses on handshape labels, sign language recognition involves multiple parameters like orientation, location, movement, and facial expressions. How do you plan to address the exclusion of these other important parameters in future work, and how might it affect the performance of the proposed methodologies in more comprehensive sign language recognition tasks?\n\n3. The paper discusses the potential extension of multimodal SLR models to become handshape-aware. Can you elaborate on the advantages and challenges of integrating other modalities beyond handshapes, and how this integration could impact the overall performance and robustness of the proposed approaches?\n\n4. In the evaluation, the paper primarily focuses on accuracy improvements in SLR. Have you considered conducting experiments to evaluate the real-world impact and practical usability of the proposed methods in sign language recognition systems? If not, do you have any plans for such evaluations in future work?\n\n5. The paper emphasizes the performance on the PHOENIX14T-HS dataset. How well do the proposed methodologies generalize to other sign language datasets with different sign languages, domains, and sizes? Have you considered conducting multi-dataset evaluations to assess the generalizability of the methods?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed reviews! \n# In response to the questions:\n> 1. *Considering the challenges in finding DGS experts as annotators, how did you address potential biases or inconsistencies in the annotations? Did you perform any inter-annotator agreement analysis to assess the reliability of the handshape labels?*\n\nWe recognize the importance of conducting inter-annotator agreement analysis to validate and enhance the quality of handshape labels. However, due to the limited availability of a single annotator, we were unable to conduct such an analysis, which likely resulted in noisy labels. Nonetheless, we would like to emphasize that these noisy handshape labels actually mirror real-world scenarios: handshape labels are inherently noisy due to the ambiguity in determining handshape during coarticulations (the transition between two signs) and the application of an existing sign dictionary. Therefore, while we unintentionally created noisy labels, we have demonstrated that handshape-aware SLR shows promise even in the presence of such noise.\n\n> 2. *While the PHOENIX14T-HS dataset focuses on handshape labels, sign language recognition involves multiple parameters like orientation, location, movement, and facial expressions. How do you plan to address the exclusion of these other important parameters in future work, and how might it affect the performance of the proposed methodologies in more comprehensive sign language recognition tasks?*\n\nOur approach is designed to be easily extendable to incorporate other sign parameters. For example, one could add an extra head to predict, for example, orientation (as in Model I in the paper), or add an extra encoder for orientation (as in Model II in the paper) on top of our existing system. Since this would involve providing more information to the model, we anticipate that it would result in improved performance. However, it should be noted that the weights of the loss from different parameters might be an important hyperparameter to tune carefully.\n\n>3. *The paper discusses the potential extension of multimodal SLR models to become handshape-aware. Can you elaborate on the advantages and challenges of integrating other modalities beyond handshapes, and how this integration could impact the overall performance and robustness of the proposed approaches?*\n\nOur approach is single-modal in that the system inputs are in RGB. However, the system is designed to be readily extendable to a multimodal approach, which would involve adding extra encoders for keypoints and/or optical flow. Keypoints and optical flow capture different aspects of the sign video compared to RGB features. Specifically, the signers’ appearance and background are omitted in the former, while some important details might also be omitted due to the application of an external keypoint or optical flow extractor. By making the handshape-aware system multimodal, we anticipate that the performance and robustness of the system would be further improved. However, it also presents challenges such as the need for more computational resources.\n\n>4. *In the evaluation, the paper primarily focuses on accuracy improvements in SLR. Have you considered conducting experiments to evaluate the real-world impact and practical usability of the proposed methods in sign language recognition systems? If not, do you have any plans for such evaluations in future work?*\n\nThank you for raising this important question. The primary focus of this paper is to serve as a reference and guide for future work in real-world SLR applications. The PHOENIX dataset, while valuable, has only 7,096 training examples and is concentrated in a specific domain (weather forecast), limiting its applicability to broader real-world scenarios. To facilitate real-world applications and evaluations, there is a pressing need for much larger annotated datasets, representing one of the most urgent challenges in the field of SLR. However, we believe our paper holds significant implications for future developments in SLR. It underscores the importance of annotating handshapes when creating new datasets and demonstrates the potential of building handshape-aware SLR systems. We are eager to test our approach on a comprehensive real-world sign language dataset as soon as one becomes available.\n\n>5. *The paper emphasizes the performance on the PHOENIX14T-HS dataset. How well do the proposed methodologies generalize to other sign language datasets with different sign languages, domains, and sizes? Have you considered conducting multi-dataset evaluations to assess the generalizability of the methods?*\n\nWe appreciate your concern regarding the generalizability of our methodologies across different sign languages, domains, and dataset sizes. We fully acknowledged the limitation imposed by validating our results on a single dataset, and concur that testing on additional datasets would afford a more robust justification of our claims. However, it is important to note that the SLR community currently has access to only two readily available sentence-level datasets: the PHOENIX dataset and a Chinese sign language dataset, CSL-Daily. We had initially planned to conduct experiments with CSL-Daily as well, but unfortunately, the SignWriting dictionary has extremely limited coverage of Chinese sign language. This implies that to test on CSL-Daily, we would be required to label the dataset ourselves, a task that, given our limited resources, is nearly unfeasible. We have striven to conduct meaningful research within these constraints.\n\nWe are truly grateful for the valuable time you have invested in reviewing this paper and the insightful questions you have posed. We sincerely hope our responses have addressed your questions thoroughly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Handshape-Enriched Dataset: The paper introduces a new dataset, PHOENIX14T-HS, which includes handshape labels for sign language videos. This dataset represents a valuable resource for researchers and practitioners working on Sign Language Recognition (SLR) and related tasks. It enables the development and evaluation of handshape-aware SLR models, which is crucial for advancing the understanding of sign languages.\n\n2. Advancement in SLR: The proposed handshape-inclusive SLR methods, Model I and Model II, represent significant advancements in the field of SLR. Model II, in particular, achieves top performance among single-modality SLR models. These methods open up new possibilities for exploring multimodal SLR models and further improving SLR accuracy.\n\n3. Pretraining Strategies: The paper thoroughly explores various pretraining strategies for the S3D encoder in SLR models. This analysis provides valuable insights into effective pretraining methodologies for sign language-related datasets, which are often low-resource and challenging to handle. The findings can guide researchers in efficiently utilizing pretraining to boost performance on SLR tasks.\n\n4. Impact on SLR Model Design: By incorporating handshape information into the SLR models, the paper addresses a crucial aspect of sign languages that has been largely underexplored in previous work. This not only improves the accuracy of SLR models but also aligns with the linguistic properties of sign languages, contributing to more linguistically meaningful SLR systems.\n\n5. Pseudo-Label Weighting Analysis: The investigation into pseudo-label weighting for handshape predictions provides useful guidance on how to handle noisy labels and improve the learning process. The results shed light on the effective utilization of pseudo-labels for fine-grained frame-level supervision in SLR.\n\n6. Potential for Further Research: The paper opens up promising avenues for future research in SLR, such as exploring multimodal SLR models, contrastive learning for negative examples, and data augmentation techniques. These areas hold potential for further advancing the performance and understanding of SLR systems.", "reasons_to_reject": "1. Noisy Handshape Labels: One of the main weaknesses of the paper is that the handshape labels in the PHOENIX14T-HS dataset are potentially noisy. Two-thirds of the handshape labels are obtained from a user-edited online dictionary, which may not always accurately correspond to the sign videos. This could introduce noise and uncertainty in the dataset, affecting the reliability of the results.\n\n2. Single Annotator: The paper mentions that finding experts to serve as annotators for the handshape labels was challenging, and obtaining multiple annotators for inter-annotator agreement was difficult. This limitation raises concerns about the consistency and reliability of the annotations, as inter-annotator agreement is crucial for ensuring high-quality labeled data.\n\n3. Limited Sign Language Parameters: The dataset used in the study only considers handshape labels and does not include other sign language parameters such as orientation, location, movement, and facial expressions. While the focus on handshape is a specific contribution, the exclusion of other important parameters may limit the broader applicability of the proposed methodologies.\n\n4. Single Dataset Evaluation: The paper primarily evaluates the proposed methods on the PHOENIX14T-HS dataset. While this dataset is enriched with handshape labels, it would be beneficial to see how the methods generalize to other sign language datasets, representing different sign languages, domains, and sizes. The lack of multi-dataset evaluation raises questions about the robustness and generalizability of the proposed approaches.\n\n5. Lack of Multimodal Integration: While the paper discusses the potential extension of multimodal SLR models to become handshape-aware, it does not explore the integration of other modalities beyond handshapes. Considering that SLR can benefit from multiple modalities like RGB videos, human body key points, and optical flow, the exclusion of multimodal integration may limit the overall effectiveness of the proposed methodologies.\n\n6. Absence of Real-World Impact Evaluation: The paper primarily focuses on the technical contributions and improvements in SLR accuracy. However, there is limited discussion about the real-world impact of the proposed methods in practical sign language recognition systems. Without a comprehensive evaluation of the practical usability and efficiency of the proposed approaches, it remains uncertain how these methods would perform in real-world applications.", "questions_to_authors": "1. Considering the challenges in finding DGS experts as annotators, how did you address potential biases or inconsistencies in the annotations? Did you perform any inter-annotator agreement analysis to assess the reliability of the handshape labels?\n\n2. While the PHOENIX14T-HS dataset focuses on handshape labels, sign language recognition involves multiple parameters like orientation, location, movement, and facial expressions. How do you plan to address the exclusion of these other important parameters in future work, and how might it affect the performance of the proposed methodologies in more comprehensive sign language recognition tasks?\n\n3. The paper discusses the potential extension of multimodal SLR models to become handshape-aware. Can you elaborate on the advantages and challenges of integrating other modalities beyond handshapes, and how this integration could impact the overall performance and robustness of the proposed approaches?\n\n4. In the evaluation, the paper primarily focuses on accuracy improvements in SLR. Have you considered conducting experiments to evaluate the real-world impact and practical usability of the proposed methods in sign language recognition systems? If not, do you have any plans for such evaluations in future work?\n\n5. The paper emphasizes the performance on the PHOENIX14T-HS dataset. How well do the proposed methodologies generalize to other sign language datasets with different sign languages, domains, and sizes? Have you considered conducting multi-dataset evaluations to assess the generalizability of the methods?", "paper_topic_and_main_contributions": "This paper addresses the problem of handshape-aware Sign Language Recognition (SLR) and makes several key contributions in this area. \n\nHandshape-Enriched Dataset: The paper introduces a new dataset called PHOENIX14T-HS, which is an extension of the PHOENIX14T dataset. PHOENIX14T-HS includes handshape labels derived from the SignWriting dictionary and manual labeling. The dataset features German Sign Language (DGS) videos with annotations for glosses and translations into spoken German, along with the newly added handshape labels for both hands.\n\nHandshape-Inclusive SLR Methods: The paper proposes two handshape-inclusive SLR methods, referred to as Model I and Model II. These methods explicitly utilize handshape information and are designed to predict both sign glosses and handshapes concurrently. Model II, in particular, utilizes dual encoders, one for glosses and one for handshapes, and employs a joint head for generating gloss predictions. This approach outperforms existing single-modality SLR models, ranking as the top performer among them.\n\nThe main focus of this paper is to promote the integration of handshape information within SLR systems, enrich the existing SLR datasets with handshape labels, and propose effective handshape-inclusive SLR methods. The contributions aim to advance the understanding and development of SLR models that can take into account the phonological features of sign languages.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ucxJHCVobP", "reviewer": "Reviewer_NMb4", "summary": "", "questions": "A. The best model is the Model II variant, where there are two heads for left and right hands. However, Table 3 indicates low gains from left head (22.07 vs 22.56). This is consistent with the statement that right-handed signers were in the dataset (line 271). Why not remove the left head entirely to save space and improve the training speed?\n\nB. Please specify who and how you could benefit from your paper. (For example: Anyone who is using PHOENIX dataset and works on an SLR task) ", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n# In response to your questions:\n\n> A.*The best model is the Model II variant, where there are two heads for left and right hands. However, Table 3 indicates low gains from left head (22.07 vs 22.56). This is consistent with the statement that right-handed signers were in the dataset (line 271). Why not remove the left head entirely to save space and improve the training speed?*\n\nThank you for your astute observation and suggestion. While the gains from adding an extra left-hand head appear to be minimal, as indicated by the comparison of 22.07 vs. 22.56 in Table 3, our intention is to provide a comprehensive examination of the effects of incorporating handshape information into SLR systems. For this reason, we have chosen to report the results for both cases, thereby allowing readers to make an informed decision on whether to include the left-hand head in their own SLR system development.  \n\n>B.*Please specify who and how you could benefit from your paper. (For example: Anyone who is using PHOENIX dataset and works on an SLR task)*\n\nOur paper is designed to benefit a broad audience, including both users of the PHOENIX dataset and researchers engaged in SLR tasks.\n\nFor those utilizing the PHOENIX dataset, our handshape labels provide a valuable resource for deeper analysis and experimentation. Specifically, our labels enable users to conduct linguistic analyses, gather statistics on handshape distribution, and assess the dataset's suitability for real-world applications. Additionally, users can perform error analyses to identify patterns in the misrecognition of signs with certain handshapes and use the label information to develop more advanced systems. For instance, the labels can facilitate contrastive learning by enabling the creation of negative examples using signs with similar handshapes.\n\nFor SLR practitioners, our paper serves as a comprehensive reference and guide for creating new datasets and developing SLR systems for real-world applications. By demonstrating the benefits of handshape-aware SLR, we hope to encourage practitioners to consider handshape labeling as a valuable component of their work. Ultimately, we believe that the insights and strategies presented in our paper will be valuable takeaways for practitioners seeking to advance the field of SLR.\n\n# In response to the critiques:\n> *The handshape predictors are valuable contributions. However, the main stated focus is on the SLR task, where the overall performance gain is not substantial (21.8 vs. sota 22.4). That limits the perceived contribution of the work.*\n\nWe appreciate the feedback and understand that the improvement from 22.4 to 21.8 may not seem substantial at first glance. However, it is important to consider this advancement in the context of recent developments in the field of SLR. As indicated in Table 2 of our paper, the state-of-the-art (SOTA) performance has been incrementally improving over time. For instance, MMTLB (Chen et al., NeuRIPS, 2022a) improved the SOTA from 23.9 (SignBT, Zhou et al., CVPR 2021a) to 22.5, and SignBT improved the previous SOTA (CMA, Papastratis et al., IEEE 2020) from 24.0 to 23.9.\n\nWhile these improvements may seem incremental, they represent important progress in a field where gains are often hard-fought and incremental. Our work contributes to this ongoing effort to push the boundaries of what is currently achievable in SLR. Additionally, our paper introduces the concept of handshape awareness, which we believe is a valuable contribution that opens up new avenues for exploration and improvement in SLR systems.\n\n# In response to the concerns of the reproducibility:\n\nEnsuring reproducibility is of utmost importance to us, and we have taken several steps to facilitate this. As stated in the abstract, we have committed to making our dataset and code publicly available. Additionally, we have made a concerted effort to provide detailed information about the system configuration and experimental setups in the paper.\n\nWe understand that reproducibility encompasses various aspects, and we are open to making any additional modifications that may be necessary. If there are specific elements that you feel are missing or could be further elaborated upon to enhance reproducibility, we would greatly appreciate your guidance. We are fully committed to addressing any gaps and providing all the necessary information to ensure that our work can be accurately replicated by others.\n\nWe hope that our responses have sufficiently addressed the concerns you raised. Please take our explanations into consideration and reevaluate our paper. We appreciate the time and effort you have invested in reviewing our work. We firmly believe that this research has the potential to make a meaningful impact on the SLR community, and we are grateful for your consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work is well written, and the reporting on the study is sufficient to support its claims. \nContributions of the work are relevant to the ongoing research competition in SLR on the PHOENIX dataset. \nThe authors claim to enrich the PHOENIX sign language dataset with handshape annotation.", "reasons_to_reject": "The handshape predictors are valuable contributions. However, the main stated focus is on the SLR task, where the overall performance gain is not substantial (21.8 vs. sota 22.4). That limits the perceived contribution of the work.\n\nIt is implemented for well-researched datasets, and the authors did not show how other sign language datasets can directly benefit from their work. That limits the perceived contribution of the work.\n\nThe handshape \"awareness\" of the proposed models does not substantially increase the performance compared to its own vanilla \"unaware\" approach (22.07 vs. 23.69). However, the increase in memory and training speed is substantial. That diminishes the main contribution claim.", "questions_to_authors": "A. The best model is the Model II variant, where there are two heads for left and right hands. However, Table 3 indicates low gains from left head (22.07 vs 22.56). This is consistent with the statement that right-handed signers were in the dataset (line 271). Why not remove the left head entirely to save space and improve the training speed?\n\nB. Please specify who and how you could benefit from your paper. (For example: Anyone who is using PHOENIX dataset and works on an SLR task) ", "paper_topic_and_main_contributions": "The work introduces two handshapes \"aware\" model architectures for sign language recognition (SLR) tasks on the PHOENIX dataset. ", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "500 Out -> Our ? "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting paper and will be useful in the relatively new Sign Language Understanding subfield. The new dataset the authors propose, especially integrating a new style of modality in Sign Language (handshapes) would be impactful for multimodal tasks other than Sign Language Understanding as well. 

Pros:
- New innovative dataset annotated and publicly available as part of this work with good potential for future work
- Interesting step forward for Sign Language Recognition research

Cons:
- Dataset might have noisy labels due to single annotator
- Lack of Multimodal integration
Decision: Accept-Findings