SlowGuess/NAACL_2022_OCR huggingface dataset:
like the task of G2T, in this comparison, we focus on the evaluation of the whole two-step pipeline, that is, obtaining spoken language sentences from sign language videos. Considering that only PH14 contains sign language videos, we thus conduct experiments on this dataset for S2G2T task, and the results are reported in Table 2. Note that, for the recognition step, we employ STMC model to realize vision-based sequence learning (Zhou et al., 2020). From the comparison we can see that, our TIN-SLT still outperforms existing approaches on most evaluation metrics.
5.3 Analysis and Discussions
Here, we conducted a series of detailed experiments to analyze our method and give some insights behind our network design.
(a) Comparing various  $\alpha$  s-trategies on PH14 dataset
(b) Comparing various  $\alpha$  strategies on ASLG dataset
(c) The learned value of  $\alpha$  on PH14 dataset
(d) The learned value of  $\alpha$  on ASLG dataset
(e) Effect of beam size
(f) Effect of layer number
(g) Effect of learning rate
Figure 5: Various analysis results. (a) & (b) present the results by using different feature fusion strategies on two datasets, respectively. (c) & (d) show our learned value of  $\alpha$  during the training process on the two datasets, respectively. (e)-(h) explore how beam size, layer number, learning rate, and dropout rate affect the model performance.
(h) Effect of dropout rate
Effect of learning-based feature fusion. In this work, we propose a learning-based strategy to set  $\alpha$  dynamically. Here, we conducted experiments by comparing this strategy with other four different strategies, including (1) cosine annealing (Loshchilov and Hutter, 2016), (2) cosine increment, (3) cosine decrement, and (4) constant value. The update of  $\alpha$  by the three cosine strategies are calculated as Eq. (11) with different settings of the epoch cycle coefficient  $T_{c}$ :
$$
\alpha_ {t + 1} = \alpha_ {\min } + \frac {1}{2} \left(\alpha_ {\max } - \alpha_ {\min }\right) \left(1 - \cos \left(\frac {T _ {t}}{T _ {c}} \pi + \gamma\right)\right) \tag {11}
$$
where  $\alpha$  is the fusion ratio,  $T_{t}$  is current epoch step, and  $\gamma$  is the time-shift constant. We set  $T_{c}$  as (25, 100, 100) and  $\gamma$  as  $(0,0,\pi)$  for cosine annealing, cosine decrement, and cosine increment, respectively. The minimum value  $\alpha_{min}$  and maximum value  $\alpha_{max}$  of  $\alpha$  are set to be 0 and 1.
Figures 5(a)-5(b) are the experimental results on the two datasets. We can observe that the learning-based strategy (red line) gets the best result on ASLG and comparable result with the constant setting  $(\alpha = 0.8)$  on PH14, but still better than other three cosine strategies. Moreover, we also visualize the learned value of  $\alpha$  during the training process as shown in Figures 5(c)-5(d) to find out the contribution ratio of the BERT model to the final performance. We can see that, the value of  $\alpha$  is gradually decreasing on PH14, meaning that
Model	Test Set
BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	METEOR
PHOENIX-2014-T Dataset Evaluation
Baseline	47.69	35.52	28.17	23.32	46.58	44.85
w/ DataAug	50.77	37.85	29.88	24.57	47.39	46.95
w/ Encoder	51.05	37.94	29.91	24.63	47.59	47.13
w/ Decoder	50.99	38.47	30.48	25.08	48.78	48.20
Full pipeline	52.77	40.08	32.09	26.55	49.43	49.36
ASLG-PC12 Dataset Evaluation
Baseline	92.98	89.09	85.63	82.41	95.87	96.46
w/ DataAug	92.60	89.15	85.80	83.05	95.08	95.33
w/ Encoder	92.77	89.22	86.23	83.40	95.22	96.87
w/ Decoder	93.15	89.80	86.49	83.89	95.34	95.67
Full pipeline	93.35	90.03	87.07	84.29	95.39	95.92
Table 3: Ablation analysis of our major network components on the G2T task.
the model depends more on the BERT pre-trained knowledge at the beginning of the training process and gradually inclines to our employed training corpus. The observation is just opposite on ASLG, since it is a much larger dataset than PH14 and our model relies more on BERT to further boost the performance near the end of training.
Analysis on major network components. In our TIN-SLT, there are two major components: the multi-level data augmentation scheme and the instruction module. To validate the effectiveness of each component, we conduct an ablation analysis on the G2T task with the following cases.
- Baseline: We use two layers Transformer (Yin and Read, 2020) without data augmentation and instruction module as baseline.
Model1	Size(MB)	Dataset	Gloss(%)	Text(%)	BLEU4
PHOENIX-2014-T Dataset Evaluation
Multilingual	641.10	PH14	59.96	74.62	25.48
Distilbert	257.30	PH14	44.50	71.15	24.73
Gbert	421.80	PH14	44.50	71.15	25.13
Dbmdz	421.80	PH14	73.72	88.13	26.55
ASLG-PC12 Dataset Evaluation
Base-Tiny	16.90	ASLG	76.77	96.35	82.44
Electra	51.70	ASLG	76.77	96.35	82.60
Distilbert	255.60	ASLG	76.77	96.35	83.06
Base-uncased	420.10	ASLG	76.77	96.35	84.29
- w/ DataAug: Based on the baseline, we add our data augmentation scheme back.
- w/ Encoder: Based on w/ DataAug, we fuse instruction module only into the encoder.
- w/ Decoder: Based on w/ DataAug, we fuse instruction module only into the decoder.
As a contrast, in our full pipeline, the instruction module is inserted into both encoder and decoder. Table 3 shows the evaluation results on both PH14 and ASLG. By comparing the results from Baseline and w/ DataAug, we can see that our data augmentation improves the translation performance, especially for the PH14 dataset. A reasonable interpretation is that the translation task on PH14 dataset is more difficult than on ASLG, thus our data augmentation contributes more. On the other hand, w/ Encoder, w/ Decoder and Full pipeline explore the best location to introduce PTM information into the model. Results in Table 3 show that our full model achieves the best performance. Particularly, by comparing the results from w/ Encoder and w/ Decoder against the results from SOTA methods (Tables 1 & 3), we can observe that as long as we employ the pre-trained model, no matter where it is inserted into the network, the performance is always better than existing methods.
Effect of different pre-trained models. We here explored the translation performance by using d-fifferent pre-trained models; see Table 4. We analyzed the model size and vocabulary coverage of the pre-trained model with gloss and text of our dataset. We can see that introducing a pre-trained model with larger vocabulary coverage of the target dataset will gain better performance, since a pre-trained model with larger vocabulary coverage can
Table 4: Comparing different pre-trained models in terms of BLEU-4.
Type	Content	BLEU-4
GT Gloss	X-IT BE DESC-UP TO X-YOU TO CONSIDER AND CHOOSE OUTCOME X-YOU WANT TO SEE .	100.00
GT Text	it is up to you to consider and choose the outcome you want to see .
Pred Text	it is up to you to consider and choose the outcome you want to see .
GT Gloss	X-I WANT IRELAND TO REMAIN AT HEART DECISION MAKE IN EUROPE .	57.58
GT Text	i want ireland to remain at the heart of decision making in europe .
Pred Text	i want ireland to remain at the heart of the decision made in europe .
GT Gloss	X-I WILL DESC-NEVER FORGET WHAT X-I EXPERIENCE . SHOULD BE ABOUT .	13.44
GT Text	that is what this european day of memorial should be about . i will never forget what i experienced .
Pred Text	i will never forget what i experienced .
Table 5: Qualitative evaluation of translation performance in different BLEU-4 scores on ASLG dataset.
inject more knowledge learned from another unlabeled corpus into the translation task. For ASLG, although the vocabulary coverage is the same, we can see that the bigger model has better performance since it can learn contextual representation better.
Analysis on hyper-parameters. To search the best settings of our hyper-parameters, we employed Neural Network Intelligence (NNI) (Microsoft, 2018), a lightweight but powerful toolkit. As shown in Figures 5(e)-5(h), we explored how beam size, layer number, learning rate and dropout rate affect the model performance on PH14 dataset. First, beam search enables to explore more possible candidates, but large beam widths do not always result in better performance as shown in Figure 5(e). We obtain optimal beam size as 10 on PH14. Second, the layer number decides the model size and capacity, where the larger model would overfit on a small dataset. In Figure 5(f), we find the optimal layer number to be 3 on PH14. Lastly, as shown in Figures 5(g) & 5(h), we adopt an early-stopping strategy to avoid overfitting and find the best learning rate and dropout rate are 0.0003 and 0.45, respectively.
Case study. Table 5 presents some intuitive translation results on ASLG by reporting the translated spoken sentences. Overall, the translation quality is good, even the translated sentences with low BLEU-4 still convey the same information. Also, we can observe that our translated sentences are basically the same with ground truth, although using different expressions, i.e., "decision making" vs. "decision made". The translation results on PH14 are reported in Appendix.
1The pre-trained models links are listed in Appendix.
In this paper, we proposed a task-aware instruction network for sign language translation. To address the problem of limited data for SLT, we introduced a pre-trained model into Transformer and designed an instruction module to adapt SLT task. Besides, due to the discrepancy between the representation space of sign glosses and spoken sentences, we proposed a multi-level data augmentation scheme. Extensive experiments validate our superior performance compared with state-of-the-art approaches. While there is obvious improvement among most evaluation metrics, the complexity of our models is also increased, causing a longer training period. In the future, we would like to explore the possibility of designing a lightweight model to achieve real-time efficiency.
We thank anonymous reviewers for the valuable comments. This work is supported by the China National Natural Science Foundation (No. 62176101 & No. 62106094) and Zhejiang Lab's International Talent Fund for Young Professionals.
Nikolaos Arvanitis, Constantinos Constantinopoulos, and Dimitrios Kosmopoulos. 2019. Translation of sign language glosses to text using sequence-to-sequence attention models. In 2019 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), pages 296-302. IEEE.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, et al. 2019. Sign language recognition, generation, and translation: An interdisciplinary perspective. In The 21st international ACM SIGACCESS conference on computers and accessibility, pages 16-31.
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7784-7793.
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023-10033.
Runpeng Cui, Hu Liu, and Changshui Zhang. 2019. A deep neural framework for continuous sign language recognition by iterative training. IEEE Transactions on Multimedia, 21(7):1880-1891.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pages 1243-1252. PMLR.
Huggingface-community. 2018. https://huggingface.co/models. Accessed: 2018-11-17.
Kenji Imamura and Eiichiro Sumita. 2019. Recycling a pre-trained bert encoder for neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 23-31.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Sang-Ki Ko, Chang Jo Kim, Hyedong Jung, and Choongsang Cho. 2019. Neural sign language translation based on human keypoint estimation. Applied Sciences, 9(13):2683.
Oscar Koller, Necati Cihan Camgoz, Hermann Ney, and Richard Bowden. 2019. Weakly supervised learning with multi-stream cnn-lstm-hmms to discover sequential parallelism in sign language videos. IEEE transactions on pattern analysis and machine intelligence, 42(9):2306-2320.
Oscar Koller, Sepehr Zargaran, and Hermann Ney. 2017. Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent cnn-hmms. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
E Kiran Kumar, PVV Kishore, M Teja Kiran Kumar, and D Anil Kumar. 2020. 3d sign language recognition with joint distance and angular coded color topographical descriptor on a 2-stream cnn. Neurocomputing, 372:40-54.
Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. 2020. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1459-1469.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983.
Microsoft. 2018. https://github.com/microsoft/nni. Accessed: 2018-09-20.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
A. Moryossef, K. Yin, G. Neubig, and Y. Goldberg. 2021. Data augmentation for sign language gloss translation.
Achraf Othman and Mohamed Jemni. 2012. English-asl gloss parallel corpus 2012: Aslg-pc12. In 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon LREC.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Zhaofan Qiu, Ting Yao, and Tao Mei. 2017. Learning spatio-temporal representation with pseudo-3d residual networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 5533-5541.
Shikhar Sharma and Krishan Kumar. 2021. Asl-3dcnn: American sign language recognition technique using 3-d convolutional neural networks. Multimedia Tools and Applications, pages 1-13.
Hassan S Shavarani and Anoop Sarkar. 2021. Better neural machine translation by extracting linguistic information from bert. arXiv preprint arXiv:2104.02831.
Ozge Mercanoglu Sincan and Hacer Yalim Keles. 2020. Autsl: A large scale multi-modal turkish sign language dataset and baseline methods. IEEE Access, 8:181340-181355.
Stephanie Stoll, Necati Cihan Camgoz, Simon Hadfield, and Richard Bowden. 2018. Sign language production using neural machine translation and generative adversarial networks. In Proceedings of the 29th British Machine Vision Conference (BMVC 2018). University of Surrey.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Chengcheng Wei, Wengang Zhou, Junfu Pu, and Houqiang Li. 2019. Deep grammatical multiclassifier for continuous sign language recognition. In 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), pages 435-442. IEEE.
Fang Yin, Xiujuan Chai, and Xilin Chen. 2016. Iterative reference driven metric learning for signer independent isolated sign language recognition. In European Conference on Computer Vision, pages 434-450. Springer.
K. Yin, A. Moryossef, J. Hochgesang, Y. Goldberg, and M. Alikhani. 2021. Including signed languages in natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Kayo Yin and Jesse Read. 2020. Better sign language translation with stmc-transformer. arXiv preprint arXiv:2004.00588.
Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. 2020. Spatial-temporal multi-cue network for continuous sign language recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13009-13016.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020. Incorporating bert into neural machine translation. arXiv preprint arXiv:2002.06823.
In this section, we will introduce two public benchmark datasets used in sign language translation tasks, namely PHOENIX-2014-T and ASLG-PC12. We conducted statistical analysis on the datasets and the results are shown in Table 6. It is obvious that PHOENIX-2014-T is a small-scale dataset, while ASLG-PC12 is a large-scale dataset.
Dataset	Gloss	Translation
Train	Dev	Test	Train	Dev	Test
PH14	Samples	7096	519	642	7096	519	642
Vocabs	1066	393	411	2887	951	1001
Words	67781	3745	4257	99081	6820	7816
ASLG	Samples	82709	4000	1000	82709	4000	1000
Vocabs	15782	4323	2150	21600	5634	2609
Words	862046	41030	10503	975942	46637	11953
Table 6: The descriptive statistics of PHOENIX-2014-T and ASLG-PC12 datasets. Samples row means the sample size of the dataset, Vocabs row represents the total vocabularies contained in the dataset, and Words row means the total words of the dataset.
A.2 PHOENIX-2014-T Qualitative Result
BE-SLT performance of G2T task on PHOENIX-2014-T is shown in Table 7, from which we can observe that sign language translation results are of good quality with different BLEU-4 scores and the predicted sentences can convey effective information even for low BLEU-4 scores.
A.3 Experiment Parameter
In order to help reproduce BE-SLT and its translation performance, as shown in Table 8 and 9, we list the hyper-parameters of the best results on two benchmark datasets. For G2T task on PHOENIX-2014-T, we list the best hyperparameter settings for the experiments which apply data augmentation scheme, or fuse BERT-attention module into encoder, decoder, and both respectively (namely,w/DataAug, w/Encoder, w/Decoder, w/All). W/All obtains the highest BLEU-4 using the initial learning rate of 0.00025, dropout rate of 0.45, beam search with width 5, and the max epoch size of 120. For G2T task on ASLG-PC12, we also list the hyper-parameter settings for the four
Type	Content	BLEU-4
Gloss	BERG ORKAN MOEGLICH
GT Text	auf den bergen sind orkanartige
bönen möglich .	100.00
Pred Text	auf den bergen sind orkanartige
bönen möglich .
Gloss	HEUTE NACHT ZWISCHEN NEUNZEHN ZWISCHEN
FUENFZEHN SUEDOST MAXIMAL ZWOELF
GT Text	heute nach werte zwischen neunzehn und fünfzehn	57.58
grad im Südosten bis zwölf grad .
Pred Text	heute nach neunzehn bis fünfzehn grad im Südosten bis zwölf grad .
Gloss	RUSSLAND IX TROCKEN HEISS SCHEINEN FUENF DREISSIG BIS VIERZIG GRAD
GT Text	ganz anders die trockene hitze über russland	13.44
mit fünfunddreißig bis vierzig grad .
Pred Text	aber bei uns wird es auch noch ein bisschen
heißer da sind es fünf bis vierzig grad .
Table 7: PHOENIX-2014-T: Qualitatively evaluation of translation performance in different BLEU-4 scores.
experiments that achieve significant results, listed in Table 9. For more experiment details, please refer to our code which will be published upon the publication of this work.
Parameter	PHOENIX-2014-T
w/DataAug	w/Encoder	w/Decoder	w/All
Embedding size	512	512	512	512
Hidden size	2048	2048	2048	2048
Head number	8	8	8	8
Encoder BERT gate	1	1	0	1
Decoder BERT gate	1	0	1	1
Optimizer	Adam	Adam	Adam	Adam
Learning rate	0.00025	0.00025	0.00025	0.0003
LR schedule	inverse sqrt	inverse sqrt	inverse sqrt	inverse sqrt
Weight decay	10-3	10-3	10-3	10-3
Drop out	0.45	0.45	0.45	0.45
Label smoothing	0.3	0.3	0.3	0.3
BERT ratio	-	0.6	0.6	0.65
Max epoch	120	120	120	120
BERT model	bert-base-german-dbmdz-uncased
Table 8: The hyper-parameters of the best results on PHOENIX-2014-T for the G2T task.
A.4 Alpha Strategy Settings
Here we introduce the  $\alpha$  value setting details corresponding to cosine strategy and constant strategy adopted in this work as shown in Formula 2 and Formula 4. The cosine annealing and cosine decrement strategies are calculated according to Formula 11. To simplify the calculation, the cosine increment strategy is calculated according to Formula
Parameter	ASLG-PC12
w/DataAug	w/Encoder	w/Decoder	w/All
Embedding size	512	512	512	512
Hidden size	2048	2048	2048	2048
Head number	8	8	8	8
Encoder BERT gate	1	1	0	1
Decoder BERT gate	1	0	1	1
Optimizer	Adam	Adam	Adam	Adam
Learning rate	0.00025	0.00025	0.00025	0.00045
LR schedule	inverse sqrt	inverse sqrt	inverse sqrt	inverse sqrt
Weight decay	10-3	10-3	10-3	10-3
Drop out	0.45	0.45	0.45	0.4
Label smoothing	0.3	0.3	0.3	0.1
BERT ratio	-	0.6	0.6	0.6
Max epoch	70	70	70	70
BERT model	bert-base-uncased
Table 9: The hyper-parameters of the best results on ASLG-PC12 for the G2T task.
12. In order to be more intuitive, we plotted the curve of  $\alpha$  value during the training process, as shown in Figure 6.
$$
\alpha_ {t + 1} = 1 - \alpha_ {\min } - \frac {1}{2} \left(\alpha_ {\max } - \alpha_ {\min }\right) \left(1 - \cos \left(\frac {T _ {t}}{T _ {c}} \pi\right)\right) \tag {12}
$$
(a) Cosine annealing strategy
(b) Constant strategy
(c) Cosine increment strategy (d) Cosine decrement strategy
Figure 6: The  $\alpha$  value during the training process in four setting strategies, namely cosine annealing, cosine increment, cosine decrement and constant.
A.5 Pre-trained Models Download
All BERT pre-trainied models adopted in Table 4 are published by (Huggingface-community, 2018). In order to help reproduce our work and use our
code easily, we summarize the download links of the pre-trained models as follows.
PHOENIX-2014-T Dataset
- Multilingual: bert-base-multilingual-uncased https://huggingface.co/ bert-base-multilingual-uncased
Distilbert: distilbert-base-german-cased https://huggingface.co/distilbert-base-german-cased
- Gbert: gbert-base
https://huggingface.co/deepset/gbert-base
- Dbmdz: bert-base-german-dbmdz-uncased https://huggingface.co/ bert-base-german-dbmdz-uncased
ASLG-PC12 Dataset
- Base-Tiny: bert-tiny
https://huggingface.co/prajjwallbert-tiny
- Electra: electra-small-discriminator
https://huggingface.co/google/electra-small-discriminator
Distilbert: distilbert-base-uncased https://huggingface.co/distilbert-base-uncased
- Base-uncased: bert-base-uncased https://huggingface.co/ bert-base-uncased