Title: Towards Faithful Sign Language Translation
Venue: NeurIPS 2023
Accepted: no
Keywords: Sign Language Recognition; Sign Language Translation;
Abstract: Sign language translation (SLT) aims to translate perceived visual signals into spoken language. Recent works have achieved impressive performance by improving visual representations and adopting advanced machine translation techniques, but the faithfulness (\ie, whether the SLT model captures correct visual signals) in SLT has not received enough attention. In this paper, we explore the association among SLT-relevant tasks and find that the imprecise glosses and limited corpora may hinder faithfulness in SLT. To improve faithfulness in SLT, we first integrate SLT subtasks into a single framework named MonoSLT, which can share the acquired knowledge among SLT subtasks based on their monotonically aligned nature. We further propose two kinds of constraints: the alignment constraint aligns the visual and linguistic embeddings through a sharing translation module and synthetic code-switching corpora; the consistency constraint integrates the advantages of subtasks by regularizing the prediction consistency. Experimental results show that the proposed MonoSLT is competitive against previous SLT methods by increasing the utilization of visual signals, especially when glosses are imprecise.

Abstract

Sign language translation (SLT) aims to translate perceived visual signals into spoken language. Recent works have achieved impressive performance by improving visual representations and adopting advanced machine translation techniques, but the faithfulness ( i.e. , whether the SLT model captures correct visual signals) in SLT has not received enough attention. In this paper, we explore the association among SLT-relevant tasks and find that the imprecise glosses and limited corpora may hinder faithfulness in SLT. To improve faithfulness in SLT, we first integrate SLT subtasks into a single framework named MonoSLT, which can share the acquired knowledge among SLT subtasks based on their monotonically aligned nature. We further propose two kinds of constraints: the alignment constraint aligns the visual and linguistic embeddings through a sharing translation module and synthetic code-switching corpora; the consistency constraint integrates the advantages of subtasks by regularizing the prediction consistency. Experimental results show that the proposed MonoSLT is competitive against previous SLT methods by increasing the utilization of visual signals, especially when glosses are imprecise.

1 Introduction

Sign languages, as a typical visual language, fulfill the same social and mental functions within 17 the Deaf community effectively. Sign languages convey information through a unique physical 18 transmission system and the corresponding linguistic theory [1], which makes them differ greatly 19 from spoken languages. To bridge the communication gap between the Deaf and hearing communities, 20 vision-based Sign Language Recognition (SLR) [2, 3] and Sign Language Translation (SLT) [4-6] 21 have attracted much attention over several decades. Recent works often evaluate different aspects of 22 sign language understanding models on these two tasks: the effectiveness of the feature extraction [723 9] and the transferability from visual features to the target spoken language [10-12]. However, the 24 association between these two tasks has not been paid enough attention. 25

26

sequences play a critical role in both SLR and SLT. On one hand, recent SLR datasets [3, 11]

27

28

29

30

31

Gloss 1

have limited samples and only provide sentence-wise annotations (i.e., gloss sequences) due to the high cost of frame-wise annotations, and the monotonous alignment between the gloss sequence

and sign clips makes it possible to leverage Connectionist Temporal Classification (CTC) [13] to provide supervision. On the other hand, Gloss sequences are widely used as the input of Gloss2Text

(G2T) task to estimate the upper bound of Sign2Text (S2T) task [6, 12]. The relationship among

these tasks is illustrated in Fig. 1(a). Because glosses can be used to evaluate both SLR and SLT 32

33

models, it is logical to assume that the SLR model with a lower error rate (more accurate prediction of

34

35

36

glosses) can provide more accurate translation results. However, as a visual language, sign language conveys information through multiple visual signals and glosses are imprecise representations of sign

videos [10]. Many attempts [10, 14-16] have been done to improve the visual representations but

how to reduce effects from imprecise gloss has not attracted enough attention. 37

Text:

（

Partly sleet underneath.

）

Gloss:

桌子

下

猫

在

Gloss:      AUCH

DABEI

SCHNEE

REGEN

Sign Video

(also)

(there)

(snow)

(rain)

S2T

SLR

G2T

Visual

Module

Contextual

Module

Translation

Module

𝑇

（

Partly sleet underneath.

）

Gloss:  AUCH DABEI SCHNEE REGEN

V

V

V

V

V2T

C2T

G2T

Predict

S2T

C

C

C

C

(also there snow rain)

(a) Illustration of SLR, S2T and G2T

Figure 1: (a) An example from Phoenix14T [3]. The goal of SLR is to recognize a gloss sequence, which is monotonically aligned with sign clips, from the sign video. S2T and G2T aim to translate sign videos and gloss sequences into spoken language sentences, and G2T is often regarded as the 'upper bound' of S2T. (b) We decompose S2T into two subtasks based on the temporal receptive fields of source features: Vision-to-Text (V2T) and Context-to-Text (C2T), all SLT subtasks have monotonically aligned source features.

As shown in Fig. 1(a), S2T and G2T are similar translation tasks. Different from general multilingual 38 Neural Machine Translation (NMT) [17, 18], they have monotonically aligned source languages 39 (glosses and sign clips) and the same target language. Previous works attempt to improve SLT 40 performance by adopting large-scale pretrained LMs [12] and leveraging extra corpus [11, 19]. These 41 works are developed under the paradigm that 'improves G2T first and then transfers to S2T', which 42 greatly improve S2T performance but inevitably face the hallucination problem [20] ( i.e. , S2T models 43 tend to generate fluent but inadequate translations), and we attribute this problem to the lack of 44 faithfulness [21] ( i.e. , the S2T models cannot capture correct visual signals). Besides, the availability 45 of G2T corpora is also the bottleneck for the generalization of the pretrained model. 46

In this paper, we attempt to increase the utilization of visual signals in S2T to improve faithfulness, 47 especially when glosses are imprecise. We first decompose S2T into two subtasks based on the 48 temporal receptive fields of source features: Vision-to-Text (V2T) and Context-to-Text (C2T). As 49 shown in Fig. 1(b), from V2T to C2T to G2T, the degree of visual abstraction of source features 50 gradually increases, while the translation quality will get better generally. We revisit recent SLT 51 approaches [22, 12] and observe that it is hard for V2T models to find the corresponding visual clips 52 during training, while this is exactly the strength of G2T models. Moreover, improving the alignment 53 between visual clips and target words can improve the faithfulness of translation and relieve the 54 hallucination problem. Different from recent works [11, 12, 19] that attempt to improve the 'upper 55 bound' (G2T) of C2T, we focus on the association among these tasks and try to improve the 'lower 56 bound' (V2T) of C2T. 57

Specifically, we first integrate the learning of SLT subtasks into a single framework named MonoSLT 58 by sharing their translated modules, which can share the acquired knowledge among SLT subtasks 59 based on their monotonically aligned nature. We further propose two kinds of constraints to enhance 60 faithfulness in SLT. The alignment constraint implicitly aligns the visual and linguistic embeddings 61 through the shared translation module and synthetic code-switching corpora, which are generated 62 by replacing partial visual embeddings with their corresponding gloss embeddings. The consistency 63 constraint regularizes prediction consistency between different subtasks, which can improve both 64 training efficiency and translation quality. Experimental results show that the proposed approach can 65 surpass previous SLT methods on Phoenix14T by increasing the utilization of visual signals. 66

67

68

69

70

71

72

73

74

Our contributions can be summarized as follows:

⋄ Exploring the association among different relevant tasks about SLT and integrating SLT subtasks into a single framework named MonoSLT, which can share the acquired knowledge among SLT subtasks based on their monotonically aligned nature.

⋄ Proposing two kinds of constraints to enhance faithfulness in SLT. The alignment constraint aligns the visual and linguistic embeddings through a shared translation module and synthetic code-switching corpora, and the consistency constraint leverages the advantages of subtasks by regularizing the prediction consistency.

⋄ Showing the lack of faithfulness in recent SLT methods and verifying the effectiveness of 75 MonoSLT for the utilization of visual signals, especially when glosses are imprecise. 76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

2 Related Work

Sign Language Translation. With the development of vision and language understanding algorithms, SLT has progressed rapidly in recent years [6, 23, 22, 11, 12, 15]. Recent SLT methods can be roughly categorized into two categories: vision-based and language-based.

Vision-based SLT works devote to learning useful visual representations from videos. Considering the relationship with SLR, recent SLT solutions can be roughly divided into three categories: SLRpretrained, SLR-supervised, and SLR-free. SLR-pretrained solutions initialize the visual extractor with pretrained SLR models [12, 15] or directly adopt the pretrained SLR models to extract visual embeddings [10, 19]. SLR-supervised solutions [22, 14, 15] adopt the multi-task framework and leverage the supervision from both SLR and SLT. SLR-free solutions [23-27] attempt to tokenize visual information without gloss supervision and leverage more real-life data. Recent empirical results [15, 19] indicate that adopting more accurate SLR models in SLR-pretrained and SLRsupervised solutions often leads to better translation quality, but little work has been done to investigate the association between them.

On the other side, language-based SLT works focus on the linguistic difference between sign languages and spoken languages. The pioneering work [6] regards SLT as a typical NMT task and shows the potential of the encoder-decoder framework. Joint-SLRT [10] further adopts the transformer architecture [28] to integrate both SLR and SLT into a single framework. However, it is costly to collect large amounts of parallel corpora for SLT and recent works [11, 12, 19] reveals that data scarcity hinders the further development of SLT. To relieve this problem, Zhou et al . [11] leverage rich monolingual data and adopt back-translation to generate synthetic parallel data as a supplementary. Chen et al . [12] explore the potential of denoising auto-encoder that pretrained on large-scale multilingual corpora and progressively pretrain each task to achieve effective transfer in SLT. SLTUNet [19] proposes a unified model for multiple SLT-related tasks to further improve the translation. Our motivation is similar with [19] but we focus more on faithfulness, and leverage the monotonically aligned nature of SLT subtasks to align visual and linguistic embeddings.

Faithfulness in NMT. With the rapid development of the NLP techniques [28-31], the robustness and interpretability of NMT systems become a crucial issue. A good NMT model should produce translations that capture the intended meaning of the source language (faithfulness) while maintaining grammatical correctness and naturalness in the target language (fluency) [32, 33]. However, NMT models may generate hallucinations due to exposure bias [34], domain shift [35], lack of coverage [36], and other factors [20]. To enhance the faithfulness in NMT, Tu et al . [36] maintain a coverage vector to encourage NMT models to consider more source words, Wang and Sennrich [35] leverage minimum risk training to mitigate domain shift, and Feng [33] propose a faithfulness part to enhance the contextual representation of encoder output. Different from general NMT tasks, SLT models need to encoder source information from unsegmented video, which makes it harder to learn the correspondences between video and language and generate faithful translations. We focus on the relationship between visual and language translation tasks rather than diving into specific translation module designs, which makes the proposed method compatible with other NMT techniques.

3 Approach

In this section, we first introduce notation and background knowledge briefly. Then we explore the association among SLT-relevant tasks and present some empirical findings about the lack of faithfulness in SLT. After that, we propose a method to improve faithfulness in SLT.

3.1 Background

̸

Formally, given a sign sequence X = { x 1 , · · · , x T } with T frames, SLR aims to recognize its corresponding gloss sequence G = { g 1 , · · · , g N } with N glosses ( N ≤ T in general), which are monotonically aligned with sign clips S = { x η 1 , · · · , x η N } and η i is the corresponding frame indexes of gloss i . The SLR model is generally optimized by CTC, which leverages all possible alignments between X and G and can be written as L CTC = -log p ( G|X ) . Different from SLR, the objective of SLT is to translate X into spoken language sentence W = { w 1 , · · · , w M } with M words ( M = N in general), which often has different grammar and vocabulary, and the SLT model is optimized by minimizing the negative log-likelihood L SLT = ∑ M t =1 -log p ( w t |X , w <t ) .

138

139

𝑇

Translation

Module

V

V

V

V

Gloss: AUCH DABEI

SCHNEE REGEN

(also there snow rain)

MLP

𝒫

Pretrained Word

Embedding

ℰ

G

G

G

G

G

V

G

V

Translation

Module

GCN Module

Conv1D

BiLSTM

C

C

C

C

S

shared

Switch Code

𝐿𝑆𝐿𝑇

𝐶𝑆

𝐿𝑆𝐿𝑇

𝐶

𝐿𝑆𝐿𝑅

Shared Classifier

Baseline (C2T)

(a)

(b)

(c)

Recent SLT architectures [14, 9, 12] commonly contain three components: a visual module, a 129 contextual module, and a translation module. The basic architecture used in this paper is visualized 130 in Fig. 2. Considering the training efficiency, we use the coordinates of keypoint sequences as inputs. 131 As for the visual module, we adopt a lightweight GCN-based module and a two-layer temporal 132 convolution block (Conv1D). The outputs of the visual module V = { v 1 , · · · , v T } are fed into a 133 two-layer BiLSTM to obtain contextual features C = { c 1 , · · · , c T } . As mentioned in Fig. 1(b), all of 134 V , C , and G can be used as the source language for SLT, which are corresponding to V2T, C2T, and 135 G2T subtasks, respectively. Similar to V AC [9], we adopt two classifiers on both V and C to provide 136 supervision for SLR, and the basic supervision can be formulated as: 137

where the superscript indicates the input features of the loss function and λ C is the translation weight.

3.2 Exploring the Association among SLT-relevant Tasks

As shown in Fig. 2, the adopted baseline can learn SLR and SLT jointly, and the features of SLR are 140 further utilized by the translation module, which provides a sufficient basis to explore the relationship 141 between different SLT-relevant subtasks. We first train three individual models for V2T, C2T, and 142 G2T, respectively, and visualize the evaluation results during the training on Phoenix14T [3] in 143 Fig. 3(a) and 3(b). We can observe different convergence behaviors on SLT subtasks: the G2T model 144 converges faster at the beginning and achieves higher performance on the dev set, the S2T model 145 achieves comparable performance on the dev set but tends to overfit the training set, while the V2T 146 model encounters difficulties in converging. This observation indicates that the C2T model meets the 147 issue of overfitting before finding the correct visual signals, especially when adopting a powerful 148 translation module, and we identify this issue as the lack of faithfulness. 149

Moreover, we divide the dev set into several subsets based on SLR performance, and visualize the 150 relationship between SLT and SLR performance of the C2T model in Fig. 3(c). It is surprising 151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

to observe that there is no significant negative correlation ( i.e. , achieving higher BLEU scores on the subset with lower WER) between the performance of SLR and SLT, even though lower WER indicates the less accumulated error. We analyze results and find that C2T models tend to generate hallucinations [20], which are fluent but unrelated to source gloss sequences. This is another phenomenon that reflects a lack of faithfulness.

Based on the above observations, we conclude that enhancing the capability of SLT models to accurately identify visual signals is crucial, which can improve the faithfulness of SLT models. Besides, we assume imprecise gloss representations may hinder the further development of SLT models, and it is essential to increase the utilization of visual information. Different from recent works [12, 19] that explore the use of linguistic information to guide the learning of visual features, we prefer to take advantage of both modalities based on their different characteristics.

3.3 Improving Faithfulness in SLT

Previous works have shown remarkable success in modeling multi-lingual languages [37, 38] and cross-modal information [39] with a single transformer-based model [28], which verifies the capability of transformer-based models for aligning multi-source domains. Different from exploiting large-scale parallel corpora in a self-supervised way, we focus on how to make full use of existing supervised data in the low-resource setting. The proposed method includes a joint training scheme and two constraints.

Joint Learning of SLT Subtasks. All of the SLT subtasks (V2T, C2T, and G2T) are monotonically aligned and this characteristic of SLT indicates the acquired knowledge about translation can be shared across subtasks, which can not only control model complexity but also reduce overfitting. Therefore, we first integrate SLT subtasks into a single framework and learn them jointly. We adopt the pretrained word embedding of mBart [38] as previous work [15] to obtain the linguistic embedding sequence E ( G ) from a given gloss sequence G . To bridge the gap between visual and linguistic modalities, we use a two-layer MLP P to obtain the visual embedding sequences P ( V ) and P ( C ) . Besides, we share P and SLR classifiers for V and C to ensure the alignment between different kinds of visual features [40]. All of E ( G ) , P ( V ) , and P ( C ) are sent to the same translation module, and auxiliary translation losses are applied to the outputs of P ( V ) and E ( G ) for joint learning:

where λ G and λ V are hyperparameters to control the balance among subtasks.

Alignment Constraint. The joint learning scheme shares the translation module among subtasks, but it is hard to identify the relationships between multiple subtasks and we try to further simplify this scheme. Code-switching [41] is a phenomenon that the alternation of languages within a conversation or utterance, which occurs when speakers are multilingual and familiar with correspondences among languages. As mentioned in Fig. 1(b), the source features of SLT subtasks are monotonically aligned, which provides a sufficient basis to let the SLT learner train with a multilingual learner jointly and make the SLT learner aware of word alignment implicitly. As shown in Fig. 2, we only keep two branches for SLT: the primary branch is training for the C2T subtask, and the auxiliary branch needs to tackle code-switching translation.

To generate a synthetic code-switching corpus for the auxiliary branch, we first estimate the alignment path ˆ π = arg max π p ( π |X , G ) with the maximal probability [42] from the recognition prediction of the primary branch, and then obtain the corresponding frames indexes η i from ˆ π for each gloss i . The code-switched sentence embedding CS ( V , G ) is generated by replacing visual embeddings of each gloss in P ( V ) with the corresponding gloss embeddings E ( G ) (e.g., replacing P ( V ) η i with E ( G i ) for gloss i ) with a probability of β :

where m ( β ) is the mask vector for replacing and diag ( · ) convert a vector to the corresponding diagonal matirx. In addition to the above gloss-wise code-switching, we also propose a sentence-wise generation process, which simply mixes embedding sequences as Mixup [43]:

It is worth noting that β controls the ratio of gloss embeddings in the code-switched sentence, we adopt a larger β at the beginning to leverage the fast convergence of the gloss embedding and then

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

gradually decay. To balance all subtasks and prevent overfitting, we further adopt a cyclical annealing schedule [44], which gradually reduces β within each cycle:

where t is the epoch number, and M is the number of epochs for each cycle. We adopt a hyperparameter λ CS to weight the auxiliary translation loss and formulate the total process as:

Consistency Constraint. The alignment constraint implicitly aligns visual and linguistic embeddings by sharing the translation module and leveraging synthetic code-switching corpora. However, there is a certain degree of complementarity between different kinds of subtasks: G2T takes discrete gloss embedding as input, which can easily capture correspondences between source and target languages but may lose detailed visual information, while V2T and C2T take continuous embedding as input, which contains more useful information about the sign but struggles to converge. To better leverage the characteristics of different subtasks and balance the training processes, we further propose a consistency constraint to regularize the SLT predictions between two branches:

where p C and p CS are the predicted distribution over words based on features C and CS , respectively, and D KL ( · , · ) denotes Kullback-Leibler divergence. When applying both constraints, the consistency constraint encourages the C2T model to find correct correspondences at the beginning of each cycle and gradually improves the importance of visual information as β decays. The consistency constraint can also be explained from mutual learning [45] and learning from noisy labels [46].

Since the proposed method is based on the Mono tonically aligned nature of SLT subtasks, we named it MonoSLT and its final objective function is:

where λ c is the hyperparameter to balance constraints.

4 Experiments

4.1 Datasets and Evalution Metrics

Datasets. We evaluate MonoSLT on RWTHPHOENIX-Weather 2014T (Phoenix14T) and CSL-Daily datasets, and both datasets provide gloss and translation annotations.

⋄ Phoenix14T [6] is an extension of the previous SLR dataset [3] by redefining segmentation boundaries and providing parallel gloss annotation and German translation. It is collected from weather forecast broadcasts and manually annotated, which indicates the gloss annotations may be imprecise. It has 8,257 sentences signed by 9 signers with vocabularies of around 1k glosses and 3k German words. There are 7096, 519, and 642 samples in training, dev, and test sets.

⋄ CSL-Daily [11] is a Chinese sign language dataset with vocabularies of around 2k glosses and 2.3k Chinese characters. Different from Phoenix14T, CSL-Daily is collected by first designing the sign language corpus based on Chinese Sign Language textbooks and some Chinese corpora, and then inviting 10 signers to sign reference texts, which indicates the gloss annotations are quite precise. There are 18401, 1077, and 1176 samples in training, dev, and test sets.

Evalution Metrics. Similar to machine translation, BLEU [47] and ROUGH [48] scores (higher is better) are used to measure translation performance. We also report word error rate (WER, lower is better) to reflect the performance of SLR modules as previous works [3, 9, 15] do.

4.2 Implementation Details

For efficiency, we utilize MMPose [49] to estimate keypoint sequences from sign videos, and it generates 133 2D keypoints for each frame. We select 77 keypoints and divided them into five groups: 9 for body, 21 for each hand, 8 for mouth, and 18 for face. Group-wise modified ST-GCN [50] blocks are adopted to extract features from each group, and extracted features are projected to a vector of 1024 dimensions for each frame. For Conv1D, we adopt a 'C3-P2-C3-P2' structure, where C and P

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

denote 1D-CNN and max-pooling layer, respectively. Following [12], we utilize the official release of mBART-large-cc25 2 , which is pretrained on CC25 3 , as the initialization of the translation module. The default setting for hyperparameters: λ C , λ G , λ V are set to 1.0 and λ c is set to 0.1 for simplicity. The beam width for the CTC decoder and the SLT decoder are 10 and 4, respectively. We train each model for 80 epochs with the cosine annealing schedule and an Adam optimizer, and the initial learning rate for each module: 1e-3 for the MLP, 1e-5 for the translation module, and 3e-3 for others. Each experiment is conducted on a single NVIDIA GeForce RTX 3090 GPU. Other details can be found in the supplementary.

4.3 Comparison with State-of-the-art

Quantitative Comparison. We report the performance of our MonoSLT model and relevant methods on Phoenix14T in Table 1. Because this paper mainly focuses on improving faithfulness, we put results of the Sign2Gloss2Text task in the supplementary. As shown in Table 1, we adopt a strong baseline, and the proposed method can bring further improvement (+3.79 BLEU-4). Besides, the proposed MonoSLT is not the best SLR approach, but outperforms the previous SLT method [15] with the best SLR performance by 2.2% (WER: 21.4% vs. 19.3 %, BLEU-4: 31.15 % vs. 28.95%). MonoSLT also surpasses other previous methods with similar SLR performance, which indicates MonoSLT can increase the utilization of visual signals. This observation also reveals the lack of faithfulness in recent SLT methods, e.g., TwoStream-SLT [15] with multi-modality inputs (both skeleton sequence and video) achieve much better SLR performance than with skeleton sequence only (WER: 27.14% vs. 17.72%), but it achieves comparable SLT performance (BLEU-4: 28.23%

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

vs. 28.95%) under these two settings. Besides, the proposed method improves faithfulness through joint learning and two constraints, which can be applied to any SLR model and has the potential to achieve better translation performance with a more powerful SLR model.

To show the generalization of the proposed method, we also report relevant performance on CSL-Daily in Table 2. As mentioned in Sect. 4.1, the CSL-Daily dataset has more precise gloss annotations, which indicates models with lower SLR performance can often achieve better SLT results. The proposed MonoSLT achieves inferior SLR and SLT performance than previous works [12, 15] but is still better than other works. Besides, the proposed method achieves better SLT performance than the baseline, which indicates that although glosses are precise intermediate tokenization, the lack of faithfulness still exists.

Qualitative Comparison. To provide a more intuitive understanding of the proposed method, we present a translation example in Table 3. It can be observed that part of the translation cannot find corresponding glosses, which indicates glosses are imprecise representations. Besides, the baseline generates hallucination 'and northeast here and there', while the corresponding gloss 'kueste' is correctly recognized but ignored by the translation module. The proposed MonoSLT can improve faithfulness and translate 'on the coasts' correctly. More results can be found in the supplementary.

4.4 Ablation and Discussion

Ablation on Joint Learning. As we mentioned in Sect. 3.3, the acquired knowledge about translation can be shared across SLT subtasks. We first evaluate different combinations of subtasks and present results in Table. 4. We notice that learning V2T and C2T subtasks jointly obtain the most significant improvements (5.72% for V2T and 0.87% for C2T), which shows V2T can achieve comparable results to G2T with proper regularization and the performance of V2T and C2T can be mutually improved. This observation also reveals a clear difference between the SLTUNet [19] and the proposed method: we pay more attention to the association between visual translation subtasks. Moreover, simply sharing more subtasks can not bring further improvements, which indicates the importance of designing proper solutions to exploit SLT subtasks.

Ablation on Design Choices of MonoSLT. To investigate the effectiveness of the proposed method, we present the ablation results of each design in Table 5. Both token-wise and sentence-wise codeswitching achieve better performance than the best performance of joint learning, and we notice they can also accelerate training process and increase training stability. Adopting the cyclical annealing schedule can improve the G2T performance at the cost of a little performance loss of V2T and S2T, and combining it with the consistency constraint can bring further improvement. Besides, we can also observe that the proposed method can also improve the performance of G2T, which indicates S2T is also beneficial for G2T, and the previous 'G2T first' paradigm not fully exploits the potential of visual information.

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

Table 6: Ablation results (BLEU-4, %) of source features and frozen layers on Phoenix14T.

Source Feature

Features

Logits

✓

✓

✓

GCN module

Conv1D

✓

✓

✓

V2T

Dev

29.73

28.26

28.59

29.29

Test

30.03

29.55

29.73

31.45

C2T

Dev

29.96

28.99

29.20

29.68

Test

31.15

29.77

30.21

31.21

G2T

Dev

30.39

28.08

28.89

30.06

Test

30.20

27.55

29.66

30.98

Ablation on Other Designs. Compared to visual features, logits are a closer representation of glosses. As shown in Table 6, adopting logits as input leads to a little performance degradation, which also indicates glosses are imprecise representations of signs on Phoenix14T. To further explore the origin of performance gain, we evaluate the effects of frozen modules in Table 6. Frozing both the GCN-based module and Conv1D can achieve comparable results, which indicates the improvement mainly comes from making better use of existing features, rather than extracting new visual features. Adopting the frozen version of MonoSLT can also improve training efficiency.

Limitations and Discussions. Although the proposed MonoSLT achieves competitive results on two benchmarks, we notice several limitations of our model. First, the proposed method is motivated to solve the hallucination problem of SLT, however, we have not found proper metrics to quantitatively evaluate the faithfulness of SLT models and still use BLEU and ROUGE for evaluation. We believe faithfulness is important when SLT is applied in real life because an unfaithful SLT model may produce unexpected consequences. Secondly, although the proposed method can improve faithfulness in SLT, as shown in Table. 6, it does not extract new visual features, which indicates that expensive gloss annotations are still essential. Designing effective gloss-free is a fascinating route for SLT. Third, although we design several approaches to make the training stable, it still encounters difficulties in converging occasionally, and we will continue to enhance its stability.

5 Conclusion

Faithfulness is one of the desired criteria to evaluate the applicability of SLT models. In this paper, we explore the association among different SLT-relevant tasks and reveal that the lack of faithfulness exists in recent SLT methods. To improve faithfulness in SLT, we attempt to increase the utilization of visual signals in SLT and propose a framework named MonoSLT, which leverages the monotonically aligned nature of SLT subtasks to train them jointly. We further propose two kinds of constraints to align visual and linguistic embeddings and leverage the advantage of subtasks. Experimental results show that the proposed MonoSLT is competitive against previous SLT methods by increasing the utilization of visual signals, especially when glosses are imprecise. We hope the proposed method and empirical conclusions can inspire future studies on SLT and relevant tasks.

Broader Impact This paper focuses on improving faithfulness in SLT to bridge the communication gap between the Deaf and hearing communities. Although the MonoSLT has made some progress there still seems a long way to go. Please note this research is limited to public datasets which have limited samples and are collected under constrained conditions, and the findings may not directly transfer to other scenarios or domains. Domain expertise and human supervision are essential when using it to make critical decisions, which may generate erroneous or potentially harmful translations. Moreover, potential biases in the training data or method may introduce limitations or assumptions that need to be considered when using it.

✓

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

References

Wendy Sandler and Diane Lillo-Martin. Sign language and linguistic universals . Cambridge University Press, 2006.

Shinichi Tamura and Shingo Kawasaki. Recognition of sign language motion images. Pattern Recognition , 21(4):343-353, 1988.

Oscar Koller, Jens Forster, and Hermann Ney. Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers. Computer Vision and Image Understanding , 141:108-125, 2015.

Britta Bauer, Sonja Nießen, and Hermann Hienz. Towards an automatic sign language translation system. In Proceedings of the International Workshop on Physicality and Tangibility in Interaction: Towards New Paradigms for Interaction beyond the Desktop, Siena, Italy , 1999.

Xiujuan Chai, Guang Li, Yushun Lin, Zhihao Xu, Yili Tang, Xilin Chen, and Ming Zhou. Sign language recognition and translation with kinect. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition , volume 655, page 4, 2013.

Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7784-7793, 2018.

Runpeng Cui, Hu Liu, and Changshui Zhang. Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7361-7369, 2017.

Oscar Koller, Necati Cihan Camgoz, Hermann Ney, and Richard Bowden. Weakly supervised learning with multi-stream cnn-lstm-hmms to discover sequential parallelism in sign language videos. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(9):2306-2320, 2019.

Yuecong Min, Aiming Hao, Xiujuan Chai, and Xilin Chen. Visual alignment constraint for continuous sign language recognition. In Proceedings of the IEEE International Conference on Computer Vision , pages 11542-11551, 2021.

Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Multi-channel transformers for multi-articulatory sign language translation. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16 , pages 301-319. Springer, 2020.

Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1316-1325, 2021.

Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5120-5130, 2022.

Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the International Conference on Machine Learning , pages 369-376, 2006.

Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transactions on Multimedia , 24:768-779, 2021.

Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, LIU Shujie, and Brian Mak. Two-stream network for sign language recognition and translation. In Advances in Neural Information Processing Systems , 2022.

Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li. Signbert+: Hand-model-aware self-supervised pre-training for sign language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

Tomas Mikolov, Quoc V Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168 , 2013.

Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing , pages 1723-1732, 2015.

Biao Zhang, Mathias Müller, and Rico Sennrich. SLTUNET: A simple unified model for sign language translation. In International Conference on Learning Representations , 2023.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys , 55(12):1-38, 2023.

Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4198-4205, 2020.

Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 10023-10033, 2020.

Dongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang, Benjamin Swift, Hanna Suominen, and Hongdong Li. Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation. Advances in Neural Information Processing Systems , 33:12034-12045, 2020.

Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. Bsl-1k: Scaling up co-articulated sign language recognition using mouthing cues. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16 , pages 35-53. Springer, 2020.

Alptekin Orbay and Lale Akarun. Neural sign language translation by learning tokenization. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition , pages 222-228, 2020.

Liliane Momeni, Hannah Bull, KR Prajwal, Samuel Albanie, Gül Varol, and Andrew Zisserman. Automatic dense annotation of large-vocabulary sign language videos. In Computer VisionECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV , pages 671-690. Springer, 2022.

Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video. arXiv preprint arXiv:2205.12870 , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Rongxiang Weng, Heng Yu, Xiangpeng Wei, and Weihua Luo. Towards enhancing faithfulness for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages 2675-2684, 2020.

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

Yang Feng, Wanying Xie, Shuhao Gu, Chenze Shao, Wen Zhang, Zhengxin Yang, and Dong Yu. Modeling fluency and faithfulness for diverse neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 59-66, 2020.

Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In International Conference on Learning Representations , 2016.

Chaojun Wang and Rico Sennrich. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 3544-3552. Association for Computational Linguistics, 2020.

Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics , pages 76-85, 2016.

Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems , 32, 2019.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics , 8:726-742, 2020.

Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE International Conference on Computer Vision , pages 7464-7473, 2019.

Aiming Hao, Yuecong Min, and Xilin Chen. Self-mutual distillation learning for continuous sign language recognition. In Proceedings of the IEEE International Conference on Computer Vision , pages 11303-11312, 2021.

Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, and Alan W Black. A survey of code-switched speech and language processing. arXiv preprint arXiv:1904.00784 , 2019.

Runpeng Cui, Hu Liu, and Changshui Zhang. A deep neural framework for continuous sign language recognition by iterative training. IEEE Transactions on Multimedia , 21(7):1880-1891, 2019.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2017.

Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 240-250, 2019.

Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4320-4328, 2018.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE International Conference on Computer Vision , pages 322-330, 2019.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311-318, 2002.

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74-81, 2004.

MMPose Contributors. Openmmlab pose estimation toolbox and benchmark. https:// github.com/open-mmlab/mmpose , 2020.

Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In AAAI Conference on Artificial Intelligence , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel framework for sign language translation (SLT), which can integrate multiple SLT subtasks. The work is motivated by a series of experimental analysis, e.g., converging speed of different subtasks and relationship between SLT and sign language recognition (SLR) performance. Besides, two constraints are proposed to improve the faithfulness of the model and ease the model training. The method achieves SOTA performance on two widely adopted SLT benchmarks using only keypoint inputs.\n\nSTRENGTHS:\n1. The paper figures out two important problems: the lack of faithfulness in current SLT models and the inconsistent trend between SLR and SLT, which can inspire future works in this field.\n2. The method is well motivated by a series of in-depth analysis in Figure 3.\n3. The code-switching operation is interesting and novel, which can inspire future works on cross-modality modeling for SLT.\n4. SOTA performance are achieved with a lightweight model using only keypoint inputs.\n\nWEAKNESSES:\nMy major concerns come from method details and experiments.\n\nMethod:\n\n1. The MLP and classifier in Figure 2 should be shared or not? In VAC [9], two different classifiers are appended to the visual and contextual module, while SMKD [40] uses a shared classifier, and the paper follows the design of SMKD. Intuitively, different classifiers should be used to project two features from different spaces into a common space. More discussion is needed for the discrepancy.\n2. Gloss embeddings and mixup also appeared in a recent paper in the field of SLR [R1]. In [R1], the gloss embeddings are extracted by FastText, and the mixup is also achieved between visual and gloss embeddings. Some discussion or comparison should be added.\n3. What is the motivation to fulfill code switch between the visual and gloss embeddings? Is it possible to use it between the contextual and gloss embeddings?\n\nExperiments:\n\n4. In Table 5, the sentence-wise code-switching does not consistently outperform the token-wise counterpart. The authors may explain why the sentence-wise one performs better when not using annealing and consistency.\n5. As stated in line 299, logits are a closer representation of glosses. Also, [12] uses logits as the input for the translation module for CSL-Daily. Thus, it is not rigorous to conclude that adopting logits will degrade the performance since the ablation study is conducted on Phoenix14T.\n6. The paper focuses on improving the faithfulness of SLT. But there are not objective metrics mentioned to measure the faithfulness. \n\n[R1] Natural Language-Assisted Sign Language Recognition, CVPR 2023.\n\nQUESTIONS:\n1. How to obtain $m(\\beta)$? Is it right that: when $\\beta>0.5$, $m(\\beta)=1$?\n2. What is the difference between the consistency constraints in the paper with that in VAC?\n\nLIMITATIONS:\nThe authors have discussed the limitations and societal impact adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly discusses the challenges of improving faithfulness in sign language translation and proposes solutions. The researchers leverage rich monolingual data and adopt back-translation to generate synthetic parallel data, explore the potential of denoising auto-encoder, and propose the MonoSLT framework to improve the accuracy of sign language translation. They also emphasize the importance of alignment and consistency constraints to align visual and linguistic embeddings and improve faithfulness. This paper has important reference value for improving faithfulness in sign language translation.\n\nSTRENGTHS:\n1. Proposing a new unified framework, MonoSLT, which integrates subtasks of sign language translation into a single framework, allowing these subtasks to share acquired knowledge.\n\n2. Proposing two constraints: alignment constraint and consistency constraint, which help improve the faithfulness of translation.\n\n3. Experimental results show that the MonoSLT framework is competitive in improving the faithfulness of sign language translation and can increase the utilization of visual signals, especially when sign language vocabulary is imprecise.\n\nWEAKNESSES:\nThe paper does not explicitly address the handling of non-manual signals and sign language morphological changes, which are crucial factors influencing the faithfulness of sign language translation. \n\nThe experimental settings in the paper do not provide detailed explanations for many hyperparameters.\n\nQUESTIONS:\nWhy is the overfitting problem in the C2T model defined as a lack of faithfulness? \n\nUsing regularization to address overfitting seems to make sense, but a more common approach should be a suitable variant of Dropout. This requires further explanation. \n\nThe paper sets many hyperparameters, but lacks corresponding ablation studies, as well as the analysis of the mutual influences between different parameters.\n\nCan simply setting all hyperparameters to 1.0 or 0.1 yield better results than state-of-the-art (SOTA) methods? This question remains unanswered.\n\nLIMITATIONS:\nThe proposed method lacks proper metrics to quantitatively evaluate the faithfulness of Sign Language Translation models and continues to use BLEU and ROUGE for evaluation. \n\nThe paper's analysis and experimental results regarding faithfulness are not clearly defined.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the issue of faithfulness in sign language translation (SLT), which refers to whether the SLT model captures the correct visual signals. It is found that imprecise glosses and limited corpora can hinder faithfulness in SLT. To address this, the paper proposes MonoSLT, which integrates SLT subtasks into a single framework that can share knowledge among subtasks. Two kinds of constraints are proposed to improve faithfulness: the alignment constraint and the consistency constraint. Experimental results show that MonoSLT is competitive against previous SLT methods and can increase the utilization of visual signals, especially when glosses are imprecise.\n\nSTRENGTHS:\nThe method proposed in this paper outperforms multiple baseline methods, which is a promising contribution to sign language translation.\n\nWEAKNESSES:\n1. There is no comparison with [12,15] on the bug-free dataset, which is a concern. Although I understand that reproducing [15] would require additional effort, since your code is based on MMTLB, it would be reasonable to verify the effectiveness of MMTLAB on the bug-free data.\n\n2. The analysis of faithfulness and hallucination in the paper is not in-depth enough. There is no metric (either manual or automatic) to quantify faithfulness and hallucinations, and the improvement in BLEU is not sufficient to indicate that the faithfulness issue has been effectively addressed. The few cases presented in the paper are not enough to support the conclusions.\n\n3. The analysis in section 3.2 is not thorough enough, and the conclusions are somewhat forced. For example, the statement that overfitting is caused by faithfulness is not well-supported, and the conclusion that there is no obvious negative correlation between SLT and SLR in Figure 3(c) is due to hallucination lacks data support and quantitative analysis. The few examples presented in section 4 are not sufficient to demonstrate the issue of hallucination.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is dedicated to enabling the SLT model to capture correct visual signals (faithfulness in SLT). In order to improve faithfulness in SLT, the author integrates SLT subtasks into a single framework named MonoSLT, and based on this, proposes alignment constraints and consistency constraints. The former is used for aligning the visual and linguistic embeddings. The latter is used for integrating the advantages of subtasks. To demonstrate the effectiveness of the proposed method, the authors conduct experiments on two public datasets.\n\nSTRENGTHS:\n[1 - complete layout and detailed description]. The article has a relatively complete overall layout and a detailed work description.\n\n[2 – method novelty]. The author Introduced the code-switching phenomenon in the Alignment Constraint, mimicking the phenomenon of language alternation in conversations between multilinguals, and mixed visual embedding and gloss embedding as an input to the Translation Module.\n\n[3 – the rationality of alignment]. Implicitly align visual and linguistic embeddings through shared translation modules and synthetic code-switching corpora. Better utilization of the characteristics of different subtasks.\n\n[4 – method performance]. On the Phoenix14T dataset, the author's method only uses skeleton sequences as input, which improves performance (+2.2 BLUE-4) compared to the best method using RGB video as input.\n\nWEAKNESSES:\n[1 – Writing quality]. In section 3.2, some analysis is confusing, and the conclusion seems to be the author's subjective thoughts. And in the title of table 2, ‘the inconsistent punctation bug’ is confusing.\n\n[2 - method performance on CLS-Daily]. On the CLS-Daily dataset, MonoSLT performs poorly, lagging behind several sota models.\n\n[3 - Model evaluation issues]. The paper also mentions that although it alleviates the problem of faithfulness in SLT, there are no suitable metrics to measure it. The author still uses BLEU and ROUGE for evaluation\n\nQUESTIONS:\n1. At the end of line 145, S2T model should be C2T model. This may be a spelling error.\n\n2. In the title of table 2, the inconsistent punctation bug appeared, which confused me.  I don't know the meaning of this phrase.\n\n3. On the CSL-Daily dataset, the performance of MonoSLT is somewhat lacking. Can you further improve the performance of MonoSLT on CLS-Daily?\n\nLIMITATIONS:\n1.Find or create proper metrics to quantitatively evaluate the faithfulness of SLT models.\n\n2.You said that the CLS-Daily dataset provides more precise gloss annotations, which leads to other models with lower SLR performance being able to achieve better SLT results This also leads to your model not performing as well as some models on the CLS-Daily dataset. As the sign language dataset becomes larger and more accurate, your model may not be as good as other models. I think this is worth considering.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work mainly studies the faithfulness issue in SLT (i.e., whether the SLT model captures correct visual signals). The study identifies imprecise glosses and limited corpora as the main factors contributing to limited faithfulness. In order to mitigate this issue, this work proposes a framework called MonoSLT, which leverages the shared monotonically aligned nature among SLT subtasks. This framework incorporates alignment and consistency constraints. Experiments demonstrates the effectiveness of the proposed method.\n\nSTRENGTHS:\nThis paper is well-written and well-organized.\n\nThis work performs in-depth analysis on the previous works and the association among SLT-relevant tasks.\n\nThe overall performance is promising and shows notable performance gain over the baseline.\n\nWEAKNESSES:\nThe main focus of this work is on the concept of faithfulness in spoken language translation (SLT). However, a notable limitation of the study is the absence of quantitative metrics to evaluate faithfulness. While the authors acknowledge this limitation in the paper's discussion of limitations, it remains a drawback. It would be beneficial for the authors to provide further clarification on this issue, perhaps by suggesting potential quantitative metrics that could be used to assess faithfulness in future research.\n\nIt is suggested that the proposed framework be compared with VAC, as they share similar components such as consistency loss and visual module constraints.\n\nRegarding the discrepancy in length between the embeddings produced by the visual GCN module and the gloss module, it is essential to understand how the code-switching module handles this challenge. The authors should provide clarification or explanation on how the code-switching module addresses this issue.\n\nQUESTIONS:\nSee the comments in Weakness.\n\nLIMITATIONS:\nIt is better to design a suitable metric to evaluate faithfulness in SLT.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}