Two-Stream Network for Sign Language Recognition and Translation
Authors: Yutong Chen
Ronglai Zuo
Fangyun Wei
Yu Wu
Shujie LIU
Brian Mak
Venue: NeurIPS 2022 Accept (2022)
Keywords: sign language recognition
sign language translation

Abstract: Sign languages are visual languages using manual articulations and non-manual elements to convey information. For sign language recognition and translation, the majority of existing approaches directly encode RGB videos into hidden representations. RGB videos, however, are raw signals with substantial visual redundancy, leading the encoder to overlook the key information for sign language understanding. To mitigate this problem and better incorporate domain knowledge, such as handshape and body movement, we introduce a dual visual encoder containing two separate streams to model both the raw videos and the keypoint sequences generated by an off-the-shelf keypoint estimator. To make the two streams interact with each other, we explore a variety of techniques, including bidirectional lateral connection, sign pyramid network with auxiliary supervision, and frame-level self-distillation. The resulting model is called TwoStream-SLR, which is competent for sign language recognition (SLR). TwoStream-SLR is extended to a sign language translation (SLT) model, TwoStream-SLT, by simply attaching an extra translation network. Experimentally, our TwoStream-SLR and TwoStream-SLT achieve state-of-the-art performance on SLR and SLT tasks across a series of datasets including Phoenix-2014, Phoenix-2014T, and CSL-Daily.

Two-Stream Network for Sign Language Recognition and Translation
Yutong Chen'
Ronglai Zuo Fangyun Weil Yu Wu' Shujie Liu' Brian Mak?
"Microsoft Research Asia "The Hong Kong University of Science and Technology
chenytjudy8@mail com {rzuo, mak)@cse www.se.sst.....eeeee.eeeee ust.hk {fawe, yuwul, cmrcoooof...ooooooooooooooooooo.
Abstract
Sign languages are visual languages using manual articulations and non-manual elements to convey information. For sign language recognition and translation, the majority of existing approaches directly encode RGB videos into hidden represen- tations. RGB videos, however, are raw signals with substantial visual redundancy, leading the encoder to overlook the key information for sign language understand- ing. To mitigate this problem and better incorporate domain knowledge, such as handshape and body movement, two separate streams to model both the raw videos and the keypoint sequences generated by off- aff-the-shelf keypoint estimator. To make the two streams interact with each other, we explore a variety of techniques, including bidirectional lateral connection, sign pyramid network with auxiliary supervision, and frame-level self- distillation. The resulting model is called TwoStream-SI.R which is competent for sign language recognition (SIR). TwoStream-SV SR.R is extended to sign language translation (SLT) model, TwoStream-SLe SLT, by simply attaching an extra translation network. Experimentally, our TwoStream-SL. and TwoStreammSSi achieve state- of-the-art performance on SLR and SLT tasks across a series of datasets including Phoenix-2014, Phoenix-2014T, and CSL-Daily.
introduce dual visual encoder containing
Introduction
Sign languages, which are the primary means of communication among the deaf and hard-of-hearing people, have been studied for long time 46 45 4 In linguistic terms, sign languages are as rich and complex as any spoken language [2] and their word-order typology may differ from the spoken languages. Moreover, there are limited parallel data for sign-to-text, and the gap between the visual modality and the language modality poses another challenge to developing a well-performing sign language translation (SLT) system [6]. To mitigate the problem, SLT often requires an intermediate representation between an input visual signal and the output text, which is a sequence of glosses to achieve satisfactory translation results. The process of generating gloss sequences from given sign videos is termed as sign language recognition (SLR) [3], which does not have the word ordering problem in SLT. Figure Ta illustrates the relationship between SLR 27
14 and
SLT 8 6
31
55
10,
56.. In this paper, we work on both tasks.
The key to tackle SLR and SLT is to build a visual encoder to embed visual signals into hidden representations. Inspired by the promising progress of action recognition [22
48 9 51
35
3
majority of works [36] 20 41 37 5 explore to directly model RGB videos to understand sign
languages. However, a major problem with this kind of models their robustness, and the models
'Equal contribution.
Corresponniig author. Glosses are the word-for- for-word transcription of sign language where each gloss is unique label sign.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Sign Video
Dual Visual Encoder
Bidirectional Lateral Connection
Sign Pyramid Network
RGB Video
Frame-Level Delt-iistilltiion
Keypoint Estimator
TwoStream-She Network
Translation
Network
SLR
Gloss:
KUESTE (Coast)
WIND (Wind)
HABEN (Have)
Gloss Supervision
Text Supervision
SLT
Keypoint Sequence
Text: an den küsten ist es windig (it is windy on the coasts)
TwoStream- -Str
TwoStream-Se SLT
+
(a) Illustration of SLR and SLT.
(h) Overview of TwoStream--SIRR and TwoStream-SI.T.
Figure I: (a) We select sign video from the Phoenix-20141 66] dataset and visualize its gloss sequence and the corresponding text. Sign language recognition (SLR) seeks to train a model to recognize gloss sequence from sign video with weak sentence-level gloss annotations (i.e., gloss temporal boundaries are unknown). In contrast, sign language translation (SLT) aims to directly generate spoken languages (texts) with or without intermediate representations (glosses). (b) TwoStream-SLT is built upon TwoStream-SL. to enable SLT. We use HRNet [50]] trained on COCO-WholeBody Body [23] to extract keypoints of the face, hands, and upper body. Keypoints represented by heatmaps.
suffer from dramatic performance degradation when the background or signer is mismatched between training and testing. To alleviate the problem, we consider injecting proper domain knowledge (characteristic of sign languages) into learning. Sign languages use two types of visual signals to convey information [26]: manual elements that include handshape, palm orientation, etc., and non non-mand manumal markers such as facial expressions and movement of the body, head, mouth, eyes, and eyebrows. In this paper, advocate involving keypoints of the face, hands, and upper body in the learning, and introduce a novel two-stream network named TwoStream-S"i videos and keypoint sequences for sign language recognition (SLR). The proposed TwoStream-SS can be extended to handle sign language translation (SLT) by attaching an extra translation network. The resulting model is termed TwoStream-SLT, An overview of TwoStream-SLR and TwoStream- SLT is shown in Figure [b] Our contributions mainly lie in the proposed TwoStream-SLLP we summarize design principles and key components of TwoStream-Si
to model both RGB
follows:
1. Dual Visual Encoder. We use two separate S3D [5 backbones with lightweight head networks to encode RGB videos and keypoint sequences which are represented by a set of heatmaps. Since most sign language datasets do not provide keypoint annotations,
off-the-shee keypoint estimator, HRNet [50] which is trained on COCO- WholeBody [23], to generate pseudo keypoints of face, hands, and upper body for each frame. Both streams supervised by the well-known connectionist temporal classification (CTC) loss [9]. In contrast to existing methods which either utilize keypoints crоp concerned areas on feature maps /originaa videos 38), or provide supervision for multi-task learning [56] 11 we directly apply convolutional neural network to model keypoint sequences to avoid ad-hoc design 225 47). Our dual encoder architecture is differ- ent from two-stream networks for action recognition which encode either image (video)/optical flow or two videos of different frame rates [77. We fuse different streams late stage by averaging their predicted frame- wise gloss probabilities before feeding them to CTC decoder to produce the final gloss sequences.
2. Information Interaction via Bidirectional Lateral Connection. Constrained by the photo- graphic apparatus, motion blur heavily exists in sign videos 22 6). resulting in inaccurate keypoint estimation. This motivates us to use the information extracted by the video stream to alleviate the negative impacts caused by inferior keypoints when modeling keypoint sequences. On the other hand, heavy redundancy and irrelevant information (e.g., background and appear- ance of signer) in sign videos may lead the model to overlook the salient information. Thus, we propose to integrate the features extracted by the keypoint stream into the video stream as supplementary information. Thanks to the dual encoder design, intermediate representations

---

of video and keypoint sequence streams could be easily integrated via a bidirectional lateral connection module [1] for information exchange.
3. Alleviate Data Scarcity via Sign Pyramid Network and Auxiliary Loss. Both SLR and SLT greatly suffer from data scarcity. For example, there are only around 7K parallel training samples in the widely used Phoenix-20147 [6] dataset. In contrast, training an effective neural machine translation model usually requires a corpus of 1M parallel data [43]. Besides, similar to actions with different visual tempos in the action recognition field 152). glosses also have different temporal spans [42]. Thus, we present sign pyramid network (SPN), which is inspired by the feature pyramid network [3] in object detection and the temporal pyramid network (52) in action recognition, on top of the dual encoder to better capture glosses of various temporal spans in the low-data regime. The fused features yielded by SPN are further supervised by auxiliary CTC losses, which enable the shallow layers to learn meaningful features.
4. Frame-Level self-Distillation. We treat the averaged ensemble predictions as pseudo-targets and propose an additional self-distillation loss calculated by the KL-divergence between each stream's ' predictions and pseudo-targets the frame level. The self-distillation feeds rich knowledge in the late ensemble predictions back into each stream's 's learning. Moreover, compared with the CTC loss which only provides nenennce-level supervision without temporal boundary information, the self- celf-distillation loss is computed per frame to facilitate training with extra pseudo frame-level supervision.
To enable sign language translation, we append a translation network to the TwoStream-SLA yielding our translation model named TwoStream-SLTT The proposed TwoStream-SSLR and TwoStream-SS achieve sfue-off-thearr performance on both SLR and SLT across series of benchmarks including Phoenix-2014 27), P004 [6). and the recently published CSL-Daily (55).
2 Related Work
Sign Language Recognition and Translation. Sign language recognition (SLR) aims to transcribe an input sign video into gloss sequence. An SLR model usually consists of two components,
video encoder that extracts frame-level features from an input video, and a decoder (or head network) that outputs gloss sequences from the extracted features. The video encoder is usually based on
CNNs including 3D-CNNs [10 40 [1J, 2D-CNN (5] 37], and 2D+ ID CNNs [56][4]. and both single-stream
10 40 31 and multi-streat architectures [56] 14 38 have been adopted. In this work,
we utilize two separate S3D [51] backbones to model both RGB videos and keypoint sequences. In
the design of gloss decoder, all recent works use either HMM 28
30 CTC [1] 56 36 following
their success in automatic speech recognition. We adopt CTC due to its simplicity. Since CTC loss
just provides weak sentence-level supervision, [14 [54 propose to iteratively generate fine-grained pseudo labels from CTC outputs to introduce stronger frame-level supervision, while (36] achieves frame-Ievel knowledge distillation between the entire model and the visual encoder. In this work, as a byproduct of the two-stream architecture, our frame-level distillation imparts the final ensemble knowledge into each individual stream and enhances the interaction and consistency between the two streams. Sign language translation (SLT) directly predicts texts given sign vidcos. Most existing approaches formulate this task as a neural machine translation (NMT) problem by employing visual encoder as tokenizer to extract visual features and forwarding them to a translation network for spoken text generation [10 g 31 55 53 6). We use mBART [34] as our translation network due to its excellent SLT performance 10. To achieve satisfactory results, gloss supervision is commonly
utilized SLT by pretraining the visual encoder on SLR g 56, 55 and jointly training SLR and
SLT (56 (8)
Introduce Keypoints into SLR and SLT. How to leverage keypoints to boost the performance of SLR and SLT is still an open problem. (56) and (38) utilize estimated keypoint coordinates to crop feature maps and RGB videos to process each cue (hands, face, and body) independently. Some other works (25 7 47 model keypoints from coordinates. For example, the SLT system in 22 feeds keypoint coordinates into an MLP followed by recurrent neural networks. 447 represents keypoints as skeleton graphs, which are further modeled by graph convolutional networks (GCNs). However, all of the existing works treat keypoints as a set of coordinates, which is so sensitive to noise that a small perturbation may lead to wrong predictions [15.. Besides, extra efforts need to be paid to devise dedicated modules, e.g., GCN. this work, we represent keypoints as heatmaps, which are robust to noise and can simply share the same architecture with the video stream.
3

---

A uxilliary CTC Loss
A Axxiliary CTC Loss
Stop Gradient
TI2HAW/4
Sign Pyramid Network
Loss
CTC Loss
S3D-Video
Video
S3D-B1
S3D-B2
S3D-B3
S3D-B4
Posling
Head
RGB Video
C1
rocking
CTC Loss
Distillation Loss
1/2/2//22
Joint
CTC
B Bidirectional Lateral Connection
Concatenate
Head
Decoder
1/2///001
C1
rocking
Distillation Loss
Distillation Loss
K Keypoint Head
S3D-BB
S3D-B2
S3D-B3
S3D-B4
S3D-Keypoint
Keypoint Sequence
CTC Loss
Gioss: KUESTE/WIND/HABBN
Sign Pyramid Network
[Coost/Wind/Have]]
1/2H74474
1/24/5.78
A uuiiliry CTC Loss
- Auxiliary CTC Loss
which is composed of: 1) video encoder; 2) keypoint sequence encoder; 3) a joint head; 4) a bidirectional lateral connection module; 5) two sign pyramid networks. The whole network is jointly supervised by the CTC losses and the frame-level self- distillation losses. Keypoint sequences are represented by heatmaps.
Figure 2: Overview of TwoStream-SLe
Multi-Stream Networks. Multi-Stream networks are widely explored in the action recognition field 44 16 57 17]. For example, I3D [9] builds a two-stream 3D-CNN architecture and takes RGB video and optical flow as imputs. SlowFast [17] encodes videos with different frame rates. As for SLR and SLT, DNF [14] follows the idea of I3D to fuse information of RGB videos and optical flow, while STMC (50] models the multi-cue property of sign languages with multi-stream architecture that takes cropped feature maps as inputs. In this work, our approach directly models RGB videos and keypoint heatmaps via dual encoder. Besides, to alleviate the data scarcity issue and better capture glosses of various temporal spans, we present sign pyramid network equipped with auxiliary supervision to drive the shallow layers to learn meaningful representations. How to model the interactions between different streams non-trivial. I3D ? uses a late fusion strategy by simply averaging the predictions of two streams. Another way is to fuse the intermediate features of each stream in early stage by lateral connections [17). concatenation [56). or addition [4]. In this work. extend lateral connections in [17] to bidirectional ones to make two streams complement each other. addition, our self-distillation integrates the knowledge of both streams to generate tarudo-targets, which can also be regarded as kind of interaction.
3 Method
In this section, we introduce our TwoStream-SSI.R and TwoStream-SLT for the SLR and SLT tasks, respectively. Given sign video V (U1, ..., UT) with T frames, our goal is optimize TwoStream- SLR that can predict the gloss sequence G = (g1. .... gu) with U glosses (i.e., SLR). Appending translation network onto TwoStream-SLR yields TwoStream-SLT, a model which capable of predicting associated spoken language sentence S = ($1,... ..., sw) with W words (i.e., SLT) from sign videos. In general, U + W. The proposed TwoStream-SI and TwoStream-Si are introduced in Section 3.1 and Section 3.2 respectively.
3.1 TwoStream-SLR
Figure 2 shows the overview of our TwoStream-SLa. which consists of five parts: 1) video encoder; 2) a keypoint sequence encoder; 3) joint head; 4) a bidirectional lateral connection module; 5) two sign pyramid networks with auxiliary supervision, model RGB videos and keypoint sequences.
Video Encoder. We use S3D 51 with a lightweight head network as our video encoder. Only the first four blocks of S3D are used since our goal is to extract dense representations along the temporal

---

dimension. We feed each T x H x W x 3 video into the encoder to extract its features, where T denotes the frame number, H and W represent the height and width of the sign video. We set and W as 224 by default. The output feature of the last S3D block is spatially pooled into the size of T/4 x 832 before is fed into the head network. The goal of the head network is to further capture the temporal context. It consists of a temporal linear layer, a batch normalization layer, ReLU layer, as well as a temporal convolutional block which contains two temporal convolutional layers with a temporal kernel size of 3 and a stride of 1, a linear translation layer, and ReLU layer. The output feature which is named as gloss representation has a size of T/4 x 512. Then a linear classifier and a Softmax function applied to extract frame-level gloss probabilities. At last, use connectionist temporal classification (CTC) loss LorC to optimize the video encoder.
Keypoint Encoder. To model keypoint sequences, we first utilize the HRNet [50] which is trained on Boy (23] to generate 42 hand keypoints, 68 face keypoints covering the mouth, eyes, and face contour, and 11 upper body keypoints covering shoulders, elbows, and wrists per frame. We empirically found that using only a subset of 26 face keypoints (10 mouth keypoints and 16 for other parts) performs well while saving computational resources. In total, 79 keypoints are used. We use heatmaps to represent the keypoints. Concretely, denoting the keypoint heatmap sequence with size of T x H' X W' x K as G. where H' and W" represent the spatial resolution of each heatmap, and K is the total number of keypoints, the elements of G are generated by a Gaussian function: (r....A) exp (-I(i )² + (j- yt ) where (x.f, vf") denotes the coordinates of the k-th keypoint in the t-th frame. and α controls the scale of keypoints. We set a = 4 by default and H' W 112 to reduce computational cost. The network architecture of the keypoint encoder is the same as the video encoder, except for the first convolutional layer which is substituted to fit the keypoint input. Note that weights of the video encoder and keypoint encoder are not shared. Similarly, a CTC loss is utilized to train the keypoint encoder, which is denoted as Lere-
Bidirectional Lateral Connection. To fuse the information of two streams, we propose lateral connection, which is explored in action recognition [7 12 16 [5] and object detection [ The lateral connection is implemented as an element-wise add operation between two feature maps of the same resolution. We apply lateral connection on the features (C1, C22 and C3) generated by the first three blocks (B1, B2, and B3) of the two S3D backbones. Since the spatial resolutions of interme- diate features extracted by the two streams are different, we use spatially strided convolution and transposed convolution align their spatial resolutions. For implementation, the lateral connection is bidirectional, and other variants such as unidirectional lateral connection are studied in Section [.2
Joint Head and Late Ensemble. Both the video encoder and keypoint encoder have their own head networks. To fully explore the potential of our dual encoder architecture, we present an additional head network named joint head, which takes the concatenation of outputs of the two S3D networks as inputs. The architecture of the joint head is the same as the video head and keypoint head. The joint head is supervised by CTC loss well, which is denoted as LLro. We average the frame-wise gloss probabilities predicted by the video head, keypoint head, and joint head and feed them to a CTC decoder to generate the gloss sequence G. This late ensemble strategy fuses multi-stream results and improves over single-stream predictions as shown in our experiments.
Sign Pyramid Network. To better capture glosses of different temporal spans and efficiently supervise the shallow layers to learn meaningful representations, we build sign pyramid network (SPN) with auxiliary supervision upon the dual visual encoder. The architecture of SPN is illustrated in Figure 3al Specifically, we denote outputs of the last three blocks of the S3D backbone as C2, Cz. and C₁, respectively. Similar to [52], the construction of the sign pyramid involves top-down pathway and a lateral connection. We use element-wise add operation to fuse features extracted by adjacent S3D blocks, and the fused features are termed as P2 and P3 (see Figure Bap. We use transposed convolution to match both temporal and spatial dimensions of two feature maps before element- wise addition. Then two separate head networks with the same architecture as the used in the dual encoder are applied on P2 and P3 to extract frame-level gloss probabilities. Similarly, CTC losses are adopted to provide auxiliary supervision. Without loss of generality, we two independent SPNs for the video and keypoint stream. The auxiliary CTC losses of two streams are denoted as EXcre and CK L'CTe. respectively. SPN is dropped in the inference stage.
Frame-Level Seff-Distillation. Existing datasets only provide sentence-level gloss annotations, where gloss temporal boundaries are not labeled. Thus, CTC [19] loss is widely adopted to leverage this kind of weak supervision. However, once well optimized, the visual encoder is able to generate
5

---

C4
KUESTEWIINDHABEN (CoistWWindHHvve)
S3D.B4
Trans. Conv
Element- A dd
TwoStreem-
Recognition Loss
RGB Video
SLR
P3
Auxiliary CTC Loss
S3D. B3
Head
MLP
1/2HA8W/8
1/2/////wr
Trans. Conv
Translation Network
Translation
Keypoint Sequence
C2
Auxiliary CTC Loss
Loss
S3D.B2
Head
Text: an den kiston windig (itis windy on the toasts)
(a) Architecture of sign pyramid network (SPN).
(b) Overview of TwoStream- SLT.
Figure 3: (a) The sign pyramid network takes the features (C2, C3. and C4) of the last three blocks of S3D backbone as inputs and generate the fused features P2 and P3. The construction of the sign pyramid involves top-down pathway and lateral connection where the transposed convolution is used to match both temporal and spatial dimensions of two feature maps. Two separate head networks of the same architecture are applied on P2 and P3 to generate frame-level gloss probabilities. We use the CTC loss as auxiliary supervision. Each stream in our dual encoder has an independent SPN. (b) Simply appending a translation network onto our TwoStream-Sii -LR yields TwoStream-SS) SLT, framework for SLT. TwoStream-SLT is jointly supervised by the recognition loss and the translation loss.
frame-wise gloss probabilities from which of glosses. This inspires us to use the predicted frame-wise gloss probabilities as pseudo-targed to provide extra fine-gran ane-grained supervision in addition to the coarse-grained CTC loss. In accordance with our two-stream design, we use the averaged gloss probabilities from the three head networks as pseudo-targets to guide the learning of each stream. Formally, we minimize the KL-divergence between the pseudo-targets and the predictions of the three head networks. We call it frame-level seff-distillation loss Dist as it not only provides frame-level supervision but also distills knowledge in the late ensemble back into each individual stream.
estimate the approximate temporal boundaries
Loss Function. The overall loss of TwoStream-SLL is composed of three parts: 1) the CTC losses applied on the outputs of the video encoder (CYTe), keypoint encoder (CŌre). and joint head (LoTC); 2) the auxiliary CTC losses (CXoTe and LK ((CTC) applied on the outputs two sign pyramid networks; 3) the distillation loss (Liss)). We formulate the recognition loss as follows:
CK LSLK CYTC C + CCre + >y + > A LDist
(1)
where by and A denote loss weights of the auxiliary CTC loss of the video stream and keypoint stream. Up now, we have introduced all components of TwoStream-SI.R Once the training is finished, TwoStream-SLLR is capable of predicting a gloss sequence by averaging predictions from the three head networks.
3.2 TWoStream-SLT
formulate the SLT task as a neural machine translation (NMT) problem, where the input of the translation network is the output of the visual encoder. We follow this paradigm and append an MLP with two hidden layers and a subsequent translation network onto SwoStream-SLA to enable SLT. The resulting network is called TwoStream-SLT, which illustrated in Figure 3b We adopt mBART (34 as our translation network due to its excellent SLT performance [10]. To take advantage of our TwoStream design, append an MLP and translation network to each of the three heads of our TwoStream-SL -.R.. The inputs of each MLP are the features (i.e., gloss representations defined in Section 3.I encoded by the corresponding head network. The translation loss is a standard loss Lr is the sum of three translation losses. TwoStream-SLT is jointly supervised by the recognition loss LSLR defined in Eq. and the translation loss Lr, which can be formulated as:
Previous approaches
56 55 10
pruuence-to-cqqeene
cross-entropy loss 44.. The overall translation
CSLT CSLR + Lr.
During inference, we adopt the fusion strategy for multi-source translation ensemble 18 to combine predictions of the three translation networks. More details can be found the supplementary materials.

---

Table Comparison with previous works on Sign Language Recognition (SLR). WER is adopted
as the evaluation metric. Previous best results are underlined. denotes methods which adopt other modalities besides RGB videos such as pose keypoints [56], optical flow [14], or hand/mouth
shape 30
The results of [ 21
g on CSI--Daily are reproduced by SignBT [55]
Damenix-20144
Phoenix- 20144
CSL-Daily
Method
Dev
Test
Dev
Test
Dev
Test
SubUNets 5
40.8
40.7
41.4
41.0
LS-HAN 21
39.0
39.4
IAN 40
37.1
36.7
ReSign 28
27.1
26.8
CNN-LSTM- "uutii-ttremm"
30
26.0
26.0
22.1
24.1
SFL [37
24,9
25.3
25.1
26.1
DNF (RGB)
14
23,8
24.4
32.8
32.4
FCN [
23.7
23,9
23.3
25.1
33.2
33.5
DNF (RGB+Flow)
23.1
22.9
Joint-SLRT
24.6
24.5
33.1
32.0
VAC 36
21.2
22.3
LCSA 59
21.4
21.9
CMA 41
21.3
21.9
SignB'I [55]
22.7
23.9
33.2
33.2
MMTLB [10
21.9
22.5
SMKD 20
20.8
21.0
20.8
22.4
STMC-R (RGB+Pose))
56
21.1
20.7
19.6
21.0
CLR (RGB+Pose)'
558
20.5
20.4
20.2
20.4
TwoStream-SI (Ours)"
18.4
18.8
17.7
19.3
25.4
25.3
Experiment
Implementation Details. The S3D backbone is first pretrained on Kinetics-400 224.. Then we separately pretrain the video and keypoint encoder without the sign pyramid network under the supervision of a single CTC loss. Finally. TwoStream-SHi -LR loads the pretrained weights of both encoders for SLR training with loss defined in Eq. [ Following [10), we initialize translation network with mBART- mBART-large-cc25 p on CC25 and freeze the S3D backbones during SLT training prevent overfitting. Unless otherwise specified,
set λv 0.2 and A 0.5 in Eq. [ beam width as 5 for the CTC decoder and the SLT decoder during inference. We use cosine annealing schedule of 40 epochs and Adam optimizer with weight decay le 3, initial learning rate le
for TwoStream- RLR and le 5 for the MLP and translation network TwoStream-Sh We train models on Nvidia V100 GPUs. Please refer the supplementary materials for more details.
Datasets. We use Phoenix-2014 (77), Phoenix-201 2014T [6, and CSL-Daily (55) to evaluate our method on SLR. while the last two datasets are also leveraged for SLT evaluation since they provide text annotations. All ablation studies are conducted on the Phocnix-20144 SLR task.
Phoenix-2014 is German SLR dataset with a vocabulary size of 1081 for glosses. It consists of 5672, 540, and 629 samples in the training, dev, and test set.
Phoenix-2014T is extension of Phoenix-2014 with vocabulary size of 1066 for glosses and 2887 for German text. There are 7096, 519, and 642 samples in the training, dev, and test set.
CSL-Daily is newly released large-scale Chinese sign language dataset with a vocabulary size of 2000 for glosses and 2343 for Chinese text. It consists of 18401, 1077, and 1176 samples in the training, dev, and test set.
Evaluation Metrics. Following previous works 56 for SLR evaluation, and BLEU [39] and ROUGE-L 32 better recognition performance. For BLEU and ROUGE-L L, the higher, the better.
10
8
we adopt word enror rate (WER) to evaluate SLT. Lower WER indicates
6
55
"htnps//huggingffccccoofcc htt.://wwww..cccccooooooonraaaae 'ttpss//commooccrwl.orr/

---

Table Comparison with previous works on Sign Language Translation (SLT). Sign2Gloss2Text indicates two-staged pipeline and Sign2Text indicates end-to-end Sign-to-text translation. Previous best results are underlined. denotes methods without using gloss annotations. (R: ROUGE, B:
BLEU.) The results of 6 g
CSL--Daily are reproduced by SignBT [5
Phoenix 2014T
Test
Sign2Gloss2Teext
R
B3
B4
B2
B3
St.-Luong 6
44.14
42.88
30.30
23.02
18.40
43.80
43.29
30.39
22.82
18.13
Joint-SLRT
47.73
34.82
27.11
22.11
48.47
35.35
27.57
22.45
SignBT [55
49.53
49.33
36.43
28.66
23.51
49.35
48.55
36.13
28.47
23.51
STMC -TMC-fransf
46.31
48.27
35.20
27.47
22.47
46.77
48.73
36.53
29.03
24.00
MMTLB 10
50.23
50.36
37.50
29.69
24.63
49.59
49.94
37.28
29.67
TwoStream-S5 (Ours)
52.01
52.35
39.76
31.85
26.47
51.59
52.11
39,81
32.00
26.71
Sign2Text
R
B1
B2
B3
B4
R
B1
B2
B3
B4
SL-Luung
31.80
31.87
19.11
13.16
9.94
31.80
32.24
19.03
12.83
9.58
TSPNet 51
34.96
36.10
23.12
16.88
13.41
Joint-SLRT
8
47.26
34.40
27.05
22.38
46.61
33.73
26.19
21.32
STMC-T 56
48.24
47.60
36.43
29.18
24.09
46.65
46.98
36.09
28.70
23.65
SignBT
50,29
51.11
29.80
24.45
49.54
50.80
37.75
29.72
24.32
MMTTL 10
53.10
53.95
41.12
33.14
27.61
52.65
53.97
41.75
33.84
28.39
TwoSSteeam-Si (Ours)
54.08
54.32
41.99
34.15
28.66
53.48
54.90
42.43
34.46
28.95
CSL-Daily
Dev
Test
Sign2Gloss2Text
R
B1
B2
B3
B4
B1
B2
B3
B4
SL-Luong 66
40.18
41.46
25.71
16.57
11.06
40.05
41.55
25.73
16.54
11.03
Joint-SL.R 8
44.18
46.82
32.22
22.49
15.94
44.81
47.09
32.49
22.61
16.24
SignBT (55
48.38
50.97
36.16
26.26
19.53
48.21
50.68
36,00
26.20
19.67
MMTLB 10
51.35
50.89
37.96
28.53
21.88
51.43
50.33
37.44
28.08
21.46
@wooStream-SI.T (Ours)
53.91
53.58
40.49
30.67
23.71
54.92
54.08
41.02
31.18
24.13
Sign2Text
R
B1
B3
B4
B1
B2
B3
SL-Luong
34.28
34.22
19.72
12.24
7.96
34.54
34.16
19.57
11.84
Joint-SLRT
37.06
37.47
24.67
16.86
11.88
36.74
37.38
24.36
16.55
11.79
SignBT [55]
49.49
51.46
37.23
27.51
20.80
49.31
51.42
37.26
27.76
21.34
MMTLB 10
53.38
53.81
40.84
31.29
24.42
53.25
53.31
40.41
30.87
23.92
TWooStreamSSI." (Ours)
55.10
55.21
42.31
32.71
25.76
55.72
55.44
42.59
32.87
25.79
4.1 Comparison with State-of-the-art Methods on SLR and SLT
For SLR, we compare our TwoStream- SLR with state-of state-of-the of-the-art methods on Phoenix-200 2014, Phoenix- 2014T, and CSL-Daily, as shown in Table Our recognition network achieves new state-of-the-an on all datasets and outperforms the previous best method by 1.6% on Phoenix-2014. 1.1% on Phoenix-201477 and 6.7% on CSL-Daily,, respectively. For SL.T, compare TwoStream-SI.T with state-of-the-arr methods on Phoenix-20144 4T and CSL-Daily as shown in Table Following common practice, evaluate method in settings: 1) Sign2Texx (as described in Section 3.2 which directly generates texts given sign videos; 2) Sign2Gloss2Tex where we first use recognition model predict gloss sequences from sign videos, and then translation model trained on gloss-text pairs to translate predicted gloss sequences to texts. Our TwoStream-SW surpasses all previous methods in both settings. We believe that our better SLT performance is mainly attributed to the superior visual representations encoded by TwoStream-SLR.
4.2 Ablation Study
Effects of Each Component. We first show the effects each proposed component of TwoStream- SLR in Table Without the dual architecture, the two single streams (one models RGB videos and the other models keypoint sequences) achieve
21.08% 21.08 and 14% WER on the Phoenix-201 20144T Dev set. By averaging the final predictions from two streams, the WER is reduced to 20.47%. The proposed bidirectional lateral connection brings information interaction between two streams, leading to the significant improvement of 1.44%. Introducing the sign pyramid network (SPN) with auxiliary

---

Table Study the effects of each component of TwoStream- SLR on the Phoenix-201 SLR task.
(V: video, K: keypoint, Bilateral: bidirectional lateral connection, SPN: sign pyramid network.)
V-Encoder K-Encoder Bilateral
SPN
Joint Head Distillation
Dev
Test
21.08
22.42
27.14
27.19
20.47
21.55
19.03
20.12
18.52
19.91
18.36
19.49
17.72
19.32
Table 4: Ablation studies of: (a) lateral connection; (b) sign pyramid network (SPN): (c) weights of the auxiliary CTC losses; (d) the weight of the distillation loss; (e) self-distillation strategies, on the mhoenix-20144 SLR task. See Section 3.1 for the definition of C2 and P.. (V: video stream, K: keypoint stream.)
V >
>V
Connection
Dev
Test
SPN-V
SPN-K
Level
Dev
Test
None
18.57
20.03
P2, Pa
17.99
19.39
C C2, C3
17.88
19.61
/
P2,
18.15
19.42
C₁ C2, C3
18.82
19.93
/
Pj, Ps
17.72
19.32
C₁ C2. Cs
17.72
19.32
/
Ps
18.07
19.35
C2 Co
17.91
19.54
/
Pa
17.96
19.51
Lateral connection.
(b) Sign pyramid network (SPN).
Xv
Ag
Dev
Test
Weight of Lpia
Dev
Test
0.1
0.5
17.85
19.42
18.41
Teacher
Students
Target
Dev
Test
0.2
0.5
17.72
19.32
0.5
18.20
19.63
Joint Head
V. K
Soft
18.82
19.93
0.5
0.5
17.99
19.30
1.0
17.72
19.32
Hasemble
Soft
0.2
0.1
17.80
19.14
1.5
18.28
19.93
Easemble
V. K.
Soft
17.72
19.32
0.2
0.2
17.72
19.49
2.0
17.83
19.28
Ensemble
V.K.
Hard
18.25
19.58
(c) Weights of the auxiliary
(d) The weight of the distillation
(e) Distillation strategies. "J" denotes joint
CTC loss.
loss.
head.
CTC losses facilitates intermediate layers to learn more meaningful features, further boosting the performance 18.52%. As described in Section 3.] besides the individual head networks, we present a joint head to further integrate the encoded features from two streams. Our approach equipped with the joint head achieves 18.36 WER. At last, by adding the auxiliary frame-wise self-distillation loss, our framework attains the best result, yielding the WER 17.72.
Study on Lateral Connection. The goal of lateral connection is to provide information interaction such that the video stream and keypoint stream can complement each other. Here we compare our default configuration where the bidirectional lateral connection is conducted on C1, C2, and C3 (see Section 3.1 for their definitions) of two streams, with other variants including unidirectional lateral connection and different connection strategies. The comparison is shown in Table Without the lateral connection, the bascline model achieves 18.57 WER on the Phoenix-20145 Dev set. Thanks to the information interaction, both unidirectional (video-<keypois and keypoint-vidcco) and bidirectional lateral connection strategies outperform the baseline. We adopt the bidirectional lateral connection performed on C₁, Cz. and C3 due to its best performance.
Study on Sign Pyramid Network and Auxiliary Supervisions. Sign language understanding suffers from data scarcity. To capture glosses of various temporal spans and drive intermediate layers to learn more meaningful features, we propose a sign pyramid network (SPN) with auxiliary CTC losses. Here we study three key factors: 1) applying SPN on a single stream or both streams; 2) the levels of SPN: 3) the loss weights of auxiliary CTC losses the two streams. The first two factors are studied in Table 4b while the last one in Table 4c
We observe that applying SPN on both video and keypoint

---

Table 5: Ablation studies of: (a) various combinations of keypoints as the inputs of our keypoint encoder; (b) the keypoint scale a of the Gaussian function and the resolution of the generated
heatmaps, on the Phoenix-20147 SLR task.
(II', W')
Dev
Test
Upper body
Hand
Mouth
Face
#Keypoints
Dev
Test
1
56
29.78
28.72 72.72
11
49.11
48.46
2
56
30.10
29.02
(+42)
37.15
36.88
2
112
27.22
27.52
63 (+10)
28.42
28.20
4
112
27.14
27.19
79 (+16)
27.14
27.19
112
27.94
27.10
(a) The effects of different combinations of keypoints inputs of the key- point encoder.
(b) Keypoint scale and heatmap resolution.
streams yields better results. We also find imposing extra CTC supervision very shallow layers (i.e., P1) hurts the performance, thus we advocate generating two levels of pyramid (Pz and P3) for each stream. We set λv and 0x in Eq. D as 0.2 and 0.5 as they give better performance.
Study on Self-Distillation Strategies. The proposed self-distillation loss provides frame-level supervision. There is a trade-off between the pseudo fine-ge fine-grained supervision selff-isttillatoon loss) and the coarse-graines supervision (CTC loss). Here we vary the weight of self-distillation loss Dist and show the results in Table Ad] The best performance is obtained when the weight is set to 1.0. Besides, we also study: 1) which prediction should be the pseudo ttrget (teacher) 2) which heads in our dual visual encoder should be taught by the pseudo-target (students); 3) whether to binarize the probabilities of the pseudo-target (soft target or hard target). Table shows the results. As described in Section 3.I we present three head networks in the dual visual encoder, namely the video head, keypoint head, and joint head. We observe that using averaged gloss probabilities from three heads (Ensemble) the pseudo-target outperforms using predictions of the joint head. We also find that applying the self-distillation loss on all three heads achieves better results than teaching only two heads, and using soft predictions
ppeudo-targets outperforms the one with hard pseudo-targets.
Study on Keypoint Inputs. Sign languages utilize multiple visual signals including handshape, facial expressions, the movement of body, head, mouth, and eyes, to convey information. To investigate the importance of various keypoints SLR, we train several single-stream keypoint encoders using different combinations of keypoints inputs and evaluate their performance on the Phoenix-20147 to generate 79 keypoints in total. These keypoints can be divided into 4 groups: 1) 11 upper body keypoints; 2) 42 hand keypoints; 3) 10 mouth keypoints; 4) 16 face keypoints (excluding the mouth). Step by step, we add each group of keypoints into model training and the results are shown in Table Sa It can be seen that all parts contribute to sign language understanding. As described in Section 3.1 we use a Gaussian function with a hyper-parameter a, which denotes keypoint scale, to generate a set of heatmaps, each of size H' x W' to represent keypoint sequences. Here we study the value of a and the resolution of heatmaps in Table 5b) We find that a 4 and H' W = 112 achieves the best performance.
SIR task. We use the HRNet trained on COCO-WholeBil
5 Conclusion
In this paper, we concentrate on how to introduce domain knowledge into sign language understanding. To achieve the goal, we present novel framework named TwoStream-SS.. which adopts two streams to model RGB videos and keypoint sequences for sign language recognition. A variety of techniques proposed to make the two streams interact with each other, including bidirectional lateral connection, sign pyramid network, and frame-level self-distillation. We further extend TwoStream-SLA to sign language translation model by attaching MLP and translation network, yielding the translation framework named TwoStream-SLL SLT. Our TwoStream-SLe and TwoStream- SLT achieve state-of-ther performance SLR and SLT tasks across series of datasets including Phoenix-2011 2iceni-220441 and CSL-Daily. We hope that our approach can serve as a baseline to facilitate future research.
Acknowledgements. The work described in this paper was partially supported by a grant from the Research Grants Council of the HKSAR, China (Project No. HKUST1620011)).
10

---

References
[1] Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. BSL-IK: Scaling up co-articulated sign language recognition using mouthing cues. In European Conference on Computer Vision, 2020.
[2] David F Armstrong. William Stokoe, and Sherman E Wilcox. Gesture and the nature of language. Cambridge University Press, 1995.
[3] Anurag Arnab, Mostafa Dehghani, Georg Heigold. Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit; A video vision transformer. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 6836-6846, 2021.
[4] Jan Bungeroth and Hermann Ney. Statistical sign language translation. In Workshop representation and processing of sign languages, The International Conference on Language Resources and Evaluation, volume 4, pp. 105-108, 2004.
[5] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, and Richard Bowden, Subunets: End-to-end hand shape and continuous sign language recognition. In 2017 IEEE International Conference on Computer Vision, 2017.
[6] Necati Cihan Camgoz. Simon Hadfield, Oscar Koller, Hermann Ney. and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. 2018.
[7] Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and Richard Bowden. Multi-channel transformers for multi-articulatoryy sign language translation. European Conference on Computer Vision 2020 Workshops, 2020.
[8] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
[9] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
[10] Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. arXiv preprint arXiv:2203. 04287, 2022.
[11] Ka Leong Cheng, Zhaoyang Yang, Qifeng Chen, and Yu- Yu-Wing Tai. Fully convolutional networks for continuous sign language recognition. In European Conference on Computer Vision. Springer International Publishing, 2020.
[12] R Christoph and Feichtenhofer Axel Pinz. Spatiotemporal residual networks for video action recognition. Advances in Neural Information Processing Systems, pp. 3468-3476, 2016.
[13] Helen Cooper, Brian Holt, and Richard Bowden. Sign Language Recognition, pp. 539-562. Springer London. London, 2011. ISBN 978 978-0-85729.997.4 doi: 10.10077978.0085522.99999 27. URL https //doi. 10. wtw.g//...000////000 100000000797111 0007//9777---8799---- 097-0_27
[14] Runpeng Cui, Hu Liu, and Changshui Zhang. A deep neural framework for continuous sign language recognition by iterative training. IEEE Transactions on Multimedia, 21(7):1880-18)) 2019. doi: 10./ TMM.2018.2888 8.2889663.
[15] Haodong Duan, Yue Zhao, Kai Chen, Dian Shao. Dahua Lin, and Bo Dai. Revisiting skeleton- basedd action recognition. arXiv preprint arXiv:2104.133 13586. 2021.
[16] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1933-1941, 2016.
[17] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, 2019.
[18] Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos Yarman Vural, and Kyunghyun Cho. Zero- resource translation with lingullingual neural machine translation. arXiv preprint arXiv:1606. 04164. 2016.

---

[19] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
[20] Aiming Hao, Yuecong Min, and Xilin Chen. Self-mutual distillation learning for continuous sign language recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 11303-
1131, Octoher 2021.
[21] Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, and Weiping Li. Video-based sign language recog- nition without temporal segmentation. In The Thirty-Second AAAI Conference on Artificial Intelligence. AAAI Press, 2018. ISBN 978- 17778-1-577 1-57735-800-8.
[22] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3D convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1):221-231, 2012.
[23] Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole- body human pose estimation in the wild. European Conference on Computer Vision, pp. 196 214. Springer, 2020.
[24] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vijyanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: arXiv:/705,0000 06950, 2017.
[25] Sang-Ki Ki Ko, Chang Jo Kim, Hyedong Jung, and Choongsang Cho. Neural sign language translation based human keypoint estimation. Applied Sciences, 9, 2019.
[26] Oscar Koller. Quantitative survey of the state of the art in sign language recognition. arXiv preprint arXiv:2008.00 099188 2020.
[27] Oscar Koller, Jens Forster, and Hermann Ney. Continuous sign language recognition: Towards large vocah- ulary statistical recognition systems handling multiple signers. Computer Vision and Image Understanding, December 2015.
141:108-125,
[28] Oscar Koller, Sepehr Zargaran, and Hermann Ney. Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent CNN- HMMs. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[29] Oscar Koller, Sepchr Zargaran, Hermann Ney, and Richard Bowden. Deep sign: Enabling robust statistical continuous sign language recognition via hyhrid CNN-HMMs. International Journal of Computer Vision, 121(12): 2018.
[30] Oscar Koller, Necati Cihan Camgoz, Hermann Ney, and Richard Bowden, Weakly supervised learning with multi-stream HMM- discover sequential parallelism in sign language videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
[31] Dongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang, Benjamin Swift, Hanna Suominen, and Hongdong Li. Tspnet: Hierarchical feature learning via temporal semantic pyramid sign language translation. In Advances in Neural Information Processing Systems, 2020.
[32] Chin- Yen- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
[33] Tsung-Yt Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117 2127, 2017.
[34] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 2020.
[35] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106.1 13230, 2021.
[36] Yuccong Min, Aiming Hao, Xiujuan Chai, and Xilin Chen. Visual alignment constraint for continuous sign language recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1542-11551, October 2021.
[37] Zhe Niu and Brian Mak. Stochastic fine-grained labeling of multi-state sign glosses for continuous sign language recognition. In European Conference Computer Vision, pp. 772-186.6 Springer. 2020.
12

---

[38] Katerina Papadimitriou and Gerasimos Potamianos. Multimodal Sign Language Recognition via Temporal Deformable Convolutional Sequence Learning. In Interspeech, pp. 2752-2756, 2020.
[39] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
[40] Junfu Pu, Wengang Zhou, and Houqiang Li. Iterative alignment network for continuous sign language recognition. In IEEE Conference Computer Vision and Pattrrn Recognition, CVPR, 2019.
[41] Junfu Pu, Wengang Zhou, Hezhen Hu, and Houqiang Li. Boosting Continuous Sign Language Recognition via Cross Modality Augmentation, pp. 1507-1505. Association for Computing Machinery, New York, NY, USA, 2020.
[42] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Progressive transformers for end-to-end sign language production. In European Conference on Computer Vision, pp. 687-705. Springer, 2020.
[43] Rico Sennrich and Biao Zhang. Revisiting low-resource neural machine translation: A case study. arXiv preprint arXiv: 1905. 1991, 2019.
[44] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[45] Thad Starner, Joshua Weaver, and Alex Pentland, Real-time American sign language recognition using desk and wearable computer based video. IEEE Transactions on pattern analysis and machine intelligence, 20(12):1371-1375 1998.
[46] Shinichi Tamura and Shingo Kawasaki. Recognition of sign language motion images. Pattern recognition, 21(4):343-33,, 1988.
[47] Shengeng Tang, Dan Guo, Richang Hong, and Meng Wang. Graph- basphd multimodal sequential embedding for sign language translation. IEEE Transactions on Multinedia, pp. 1-1, 2021. doi: 10.1109/TMM.200 3117124.
[48] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3D convolutional networks. In Proceedings of the IEEE international conference computer vision, pp. 4489-4497, 2015.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[50] Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43(10))3349-336 2020.
[51] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision, pp.
2018.
[52] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 591-600, 2020.
[53] Kayo Yin and Jesse Read. Better sign language translation with STMC-transformer. In Proceedings of the 28th International Conference on Computational Linguistics, 2020.
[54] Hao Zhou, Wengang Zhou, and Houqiang Li. Dynamic pseudo label decoding for continuous sign language recognition. In 20/9 IEFE International Conference on Multimedia and Expo (ICME), pp. 1282-1287. TEFF, 2019.
[55] Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign tack-tramslttion In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[56] Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporee multi-cue network for sign language recognition and translation. IEEE Transactions on Multimedia, 2021.
13

---

[57] Mohammadreza Zolfaghari, Gabriel L Oliveira, Nima Sedaghat, and Thomas Brox. Chained multi-stream networks exploiting pose, motion, and appearance for action classification and detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2904-2913, 2017.
[58] Ronglai Zuo and Brian Mak. C2SLR: Consistency- continuous sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5131-5140. 2022.
[59] Ronglai Zuo and Brian Mak. Local context-awas aware self atffeattenion for continuous sign language recognition. In Proc. Interspeech 2022, pp. 4810-4814, 2022.
Checklist
1. For all authors...
(a) Do main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you descrihe the limitations your work? [Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts your work? [Yes] See Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (cither in the supplemental material or as URL)? [Yes]
(h) Did you specify all the training details (e.g., data splits, hyperparamcters, how they were chosen)? [Ye]]
(c) Did you report hars (e.g., with respect the random seed after running experiments multiple times)? [No]
(d) Did you include total amount of compute and the type resources used (c.g., type GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g.. code, data, models) curating/releasing assets...
(a) your work uses cxisting assets, did you cite the creators? [Yes]
(b) Did you mention license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're us-
ing/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "XHTUPkHeJs", "reviewer": "Reviewer_rULN", "summary": "This paper proposes a two-stream model including the raw videos and the keypoint sequences for sign language recognition and sign language translation. The two-stream model also utilize a variety of techniques, including bidirectional lateral connection, sign pyramid network, and frame-level self-distillation to further improve their model. Empirically, the two-stream model outperforms the previous best method.", "questions": "1. It may be confused that why the keypoint sequences can achieve such an effect improvement.  \n2. This paper introduce the robustness of previous models, which suffer from dramatic performance degradation when the background or signer is mismatched between training and testing, but is the two-stream model can solve this problem?", "limitations": "The authors have addressed the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. This paper takes advantage of the keypoint sequences and a variety of techniques and achieve excellent results.\n2. This paper is well written.\nWeaknesses:\n1. It may be confused that why the keypoint sequences can achieve such an effect improvement.  \n2. This paper introduce the robustness of previous models, which suffer from dramatic performance degradation when the background or signer is mismatched between training and testing, but is the two-stream model can solve this problem?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Oc5t4lxC-A", "reviewer": "Reviewer_dyUD", "summary": "This paper proposes a two-stream network for sign language recognition and translation. The main idea behind this paper is using two separate S3D networks to encode RGB modality and human keypoint modality respectively according to the fact that sign languages use both manual articulations and non-manual elements to convey information. To make the two streams interact with each other, authors propose bidirectional lateral connection, sign pyramid network with auxiliary supervision, and a frame-level self-distillation strategy. Elaborate ablation studies have verified the effectiveness of each proposed component. The results are very solid. TwoStream-SLR which is designed for sign language recognition task, achieves 18.8 WER on Phoenix-2014, 19.0 on Phoenix-2014T and 25.3 on newly published CSL-Daily, which greatly outperforms prior methods by large margins. As for TwoStream-SLT, it also achieves  state-of-the-art performance on Phoenix-2014T and CSL-Daily datasets.", "questions": "1. Self-distillation loss brings improvement as shown in Table 3. Can authors study the weight of this distillation loss in Eq 1. \n2. As described in L169, authors utilize a Gaussian function to generate keypoint heatmaps. I don’t see studies about sigma, as well as the resolution of heatmaps. I hope authors could provide these studies since how to model keypoints is a key contribution, adding these experiments will make the submission stronger.\n", "limitations": "Due to the data bias and data scarcity issue, there are unpredictable recognition/translation errors as shown in Table 4 in Appendix. Authors also discuss the limitations and societal impact adequately.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThis paper is very well written and easy to follow. The motivation is very clear.\n2.\tThe proposed TwoStream network with bidirectional lateral connection, sign pyramid network, auxiliary supervision and frame-level self-distillation is technically sound. I believe this paper will facilitate this research direction.\n3.\tSystem-level experiments are very solid. This paper achieves state-of-the-art performance on two sign language understanding tasks, i.e. continuous sign language recognition and sign language translation across several datasets. It is worth mentioning that TwoStream-SLR outperforms previous best methods by large margins, especially on CSL-Daily dataset.\n4.\tA variety of ablation studies verify the effectiveness of each proposed component, as shown in Table 3 and 4.(a)-(d), as well as the Tables in appendix.\n\nWeakness:\n1.\tAuthors could move the formulation of CTC loss and translation loss from Appendix to the main paper to make the paper clearer. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HonYeZFzRXx", "reviewer": "Reviewer_Gxi4", "summary": "In the SLR task, the paper introduces two separate streams to model both the raw videos and the keypoint sequences. For the two streams to interact with each other better, the paper proposes a variety of techniques, including bidirectional lateral connection, sign pyramid network, and frame-level self-distillation. In the SLT task, the paper extends TwoStream-SLR to TwoStream-SLT by attaching an MLP and NMT. Experimental results show that TwoStream-SLR and TwoStream-SLT achieve SOTA performance on SLR and SLT tasks in three datasets.", "questions": "1. The logical coherence of the abstract needs strengthening. For example, in line4-line7, the link between the questions about visual redundancy and the need to incorporate domain knowledge is rather jumpy, thus affecting the fluency of the reading.\n2. In the 3.1 section, the details of the selection of the key point are not clear enough. Why were the 79 key points chosen and what were the reasons? The choice of key points was not explored further in the experiment part.\n3. In the SLT experiments, in past studies, the results of experiments with added gloss supervision were higher than the gloss-free method. However, the opposite results were obtained in this experiment. It is necessary to analyze and explain the reasons for this. \n4. Suggest that some methods of the visual field need to provide an explanation to increase the readability of the paper.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes Two-Stream Network, including bidirectional lateral connection, sign pyramid network, and frame-level self-distillation methods, to interact RGB videos and keypoint sequences for advancing SLR and SLT. \n2. The paper reported an improvement on SLR and SLT tasks.\n\nWeaknesses:\n\n1. These lack the evaluation of visual redundancy and interact with both the raw videos and the keypoint sequences.\n\n2. It is unclear about the usage of domain knowledge and key sequences.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept