SlowGuess/NAACL_2022_OCR huggingface dataset:
Yong Cao $^{1*}$ , Wei Li $^{2*}$ , Xianzhi Li $^{1*}$ , Min Chen $^{1\#}$ , Guangyong Chen $^{3}$
Long Hu $^{1}$ , Zhengdao Li $^{4}$  and Hwang Kai $^{4}$
<sup>1</sup>Huazhong University of Science and Technology, <sup>2</sup>Nanchang University  
<sup>3</sup>Zhejiang University, Zhejiang Lab, <sup>4</sup>The Chinese University of Hong Kong, Shenzhen  
{yongcao_ epic, weili_ epic, xzli, minchen2012} @hust.edu.cn, gychen@zhejianglab.com  
hulong@hust.edu.cn, zhengdaoli@link.cuhk.edu.cn, hwangkai@cuhk.edu.cn
Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos and then employs a translation module to translate glosses into spoken sentences. Most existing works focus on the recognition step, while paying less attention to sign language translation. In this work, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation, by introducing the isntruction module and the learning-based feature fuse strategy into a Transformer network. In this way, the pre-trained model's language ability can be well explored and utilized to further boost the translation performance. Moreover, by exploring the representation space of sign language glosses and target spoken language, we propose a multi-level data augmentation scheme to adjust the data distribution of the training set. We conduct extensive experiments on two challenging benchmark datasets, PHOENIX-2014-T and ASLG-PC12, on which our method outperforms former best solutions by 1.65 and 1.42 in terms of BLEU-4. Our code is published at https://github.com/yongcaoplus/TIN-SLT.
Sign language recognition and translation aims to transform sign language videos into spoken languages, which builds a bridge for communication between deaf and normal people. Considering the unique grammar of sign languages, current effective recognition and translation systems involve two steps: a tokenization module to generate glosses from sign language videos, and a translation module to translate the recognized glosses into spoken natural languages. Previous works (Li et al., 2020; Sincan and Keles, 2020; Sharma and Kumar, 2021; Kumar et al., 2020; Camgoz et al., 2020)
Figure 1: Comparing the sign language translation performance on two challenging datasets, i.e., PHOENIX-2014-T (blue) and ASLG-PC12 (gray), in terms of BLEU-1 and BLEU-4 metrics. Clearly, our approach achieves the highest scores on both datasets compared with others. The experiments section contains more results and analysis.
have proposed various solutions to address the first step, but paid less attention to the translation system. Hence, this paper aims to solve the problem of sign language translation (SLT) with the goal of translating multiple recognized independent glosses into a complete sentence.
To do so, most existing works (Ko et al., 2019; S-toll et al., 2018) directly apply advanced techniques, e.g., Seq2Seq model (Sutskever et al., 2014) or Transformer (Vaswani et al., 2017), from neural machine translation to SLT. However, different from the lingual translation task in neural machine translation, SLT poses several unique challenges. First, it is hard to collect and annotate a large amount of sign language corpus. It is still an open question that how to explore more guidance and external information for SLT task by incorporating the pre-trained language models based on masses of unlabeled corpus. Second, since sign languages are developed independently from spoken languages with quite different linguistic features, the discrep
* Equal Contribution. # Corresponding author: Min Chen.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2679 - 2690 July 10-15, 2022 ©2022 Association for Computational Linguistics
ancy of representation space between glosses and spoken sentences is significant, thus increasing the translation difficulty.
To address the above issues, we propose a novel task-aware instruction network, called TIN-SLT for sign language translation, further enhanced with a multi-level data augmentation scheme. Our TIN-SLT is capable of encoding pre-trained language model's ability into the translation model and also decreasing the discrepancy between the representation space of glosses and texts.
To begin with, we leverage the extracted hidden features from the pre-trained model as extra information to guide the sign language translation. Besides, we apply an instruction module to transform general token features into task-aware features. In this way, we can fully utilize the language skills originating from the external world, thus reducing the demand for sign language training data.
Next, to better inject the information from pretrained model into the SLT model, we design a learning-based feature fusion strategy, which has been analyzed and validated to be effective compared with existing commonly-used fusion ways.
Finally, considering the large difference between the sign language glosses and texts in terms of the representation space, we propose a multilevel data augmentation scheme to enrich the coverage and variety of existing datasets.
In summary, our contributions are threefold: (i) a novel TIN-SLT network to explore more guidance of pre-trained models, (ii) a learning-based feature fusion strategy, and (iii) a multi-level data augmentation scheme. Extensive experiments on challenging benchmark datasets validate the superiority of our TIN-SLT over state-of-the-art approaches; see Figure 1 for example results.
Methods for sign language recognition. SLR task mainly focuses on the extraction of extended spatial and temporal multi-cue features (Zhou et al., 2020; Koller et al., 2017). Most existing works (Yin et al., 2016; Qiu et al., 2017; Wei et al., 2019; Cui et al., 2019) study the strong representation of sign language videos such as multi-semantic (Cui et al., 2019) and multi-modality (Koller et al., 2019) analysis. Although extracting representative features from sign language videos is fully explored, how to effectively conduct the subsequent translation by considering the unique linguistic features of sign
(a) Vocabulary distribution on PH14 dataset
(b) Vocabulary distribution on ASLG dataset
Figure 2: Comparing the sample distribution between the input sign glosses (yellow dots) and the output translated texts (red dots) on two datasets.
language is often ignored in these SLR works.
Methods for sign language translation. Early approaches for SLT rely on seq2seq model and attention mechanism (Arvanitis et al., 2019), while facing the limitation of long-term dependencies. Later, motivated by the ability of the Transformer (Vaswani et al., 2017), many researchers utilize it to effectively improve SLT performance. For example, the work in Camgoz et al. (2020) tried to use Transformer for both recognition and translation, and promote the joint optimization of sign language recognition and translation. The subsequent work (Yin and Read, 2020) proposed the STMC-Transformer network which first uses STM-C networks (Zhou et al., 2020) to achieve better results for SLR, and then exploits Transformer for translation to obtain better SLT performance.
General neural machine translation. Broadly speaking, sign language translation belongs to the field of neural machine translation, with the goal of carrying out automated text translation. Earlier approaches deployed recurrent network (Bahdanau et al., 2014), convolutional network (Gehring et al., 2017), or Transformer (Vaswani et al., 2017) as encoder-decoder module. Among them, Transformer has achieved state-of-the-art results, but the translation performance still needs to be improved due to the limited training corpus. In addition, there are some explorations in bringing the pre-trained models into neural machine translation (Imamura and Sumita, 2019; Shavarani and Sarkar, 2021; Zhu et al., 2020).
The goal of this work is to translate the recognized multiple independent glosses (network input) into a complete spoken sentence (expected output). Compared with general neural machine translation tasks, SLT faces two main challenges:
Figure 3: Network architecture of TIN-SLT. As shown in the bottom row, we first employ STMC model (Zhou et al., 2020) to recognize sign language videos to independent glosses. Next, we design a multi-level data augmentation scheme to enrich existing data pool for better feature embedding from glosses. Then, we design a task-aware instruction network with a novel instruction module to translate glosses into a complete spoken sentence.
Limited annotated corpus: Compared with natural languages, the data resources of sign languages are scarce (Bragg et al., 2019). As a result, the SLT models trained on limited data often suffer from the overfitting problem with poor generalization (Moryossef et al., 2021; Yin et al., 2021).
Discrepancy between glosses (input) and texts (output): Figure 2 shows the representation space of sign glosses (yellow dots) and translated texts (red dots) using Word2Vec (Mikolov et al., 2013) on two different datasets. We can observe that the representation space of sign glosses is clearly smaller than that of the target spoken language, thus increasing the difficulty of network learning.
To address the above challenges, we propose TIN-SLT by effectively introducing the pre-trained model into SLT task and further designing a multi-level data augmentation scheme. Figure 3 depicts the detailed network architecture. In the following subsections, we will firstly introduce the network architecture of TIN-SLT, followed by our solutions to address the above two challenges.
4.1 Network Architecture of TIN-SLT
Given a sign language video  $\mathcal{V} = \{V_1,\dots ,V_T\}$  with  $T$  frames, like existing approaches, we also adopt a two-step pipeline by first (i) recognizing  $\mathcal{V}$  into a sequence  $\mathcal{G} = \{g_1,\ldots ,g_L\}$  with  $L$  independent glosses and then (ii) translating  $\mathcal{G}$  into a complete spoken sentence  $\mathcal{S} = \{w_{1},\dots ,w_{M}\}$  with  $M$  words, but we pay more attention to solve step
(ii). Hence, for step (i), as shown in the bottom-left part of Figure 3, we empirically use the spatial-temporal multi-cue (STMC) network (Zhou et al., 2020), which consists of a spatial multi-cue module and a temporal multi-cue module. For more technical details of STMC, please refer to (Zhou et al., 2020). Below, we shall mainly elaborate on the details of addressing step (ii).
After obtaining the sequence  $\mathcal{G}$  of sign glosses, considering that the representation space of glosses is much smaller than that of texts (see Figure 2), we thus design a multi-level data augmentation scheme to expand the gloss representation space; see the top-left part of Figure 3 as an illustration and we shall present its details in Section 4.3.
Next, as shown in the bottom-middle part of Figure 3, the key of our design is a task-aware instruction network, where we adopt Transformer as the network backbone consisting of several encoder and decoder layers, whose objective is to learn the conditional probabilities  $p(S|\mathcal{G})$ . Since SLT is an extremely low-data-resource task as we have discussed in Section 3, we thus focus on exploring more task-aware guidance by learning external world knowledge, which is dynamically incorporated into the Transformer backbone via our designed task-aware instruction module. We shall present its details in Section 4.2.
Lastly, the outputs of last decoder are passed through a non-linear point-wise feed forward layer and we can obtain the predicted sentence  $S$  by a linear transform and softmax layer.
4.2 Task-aware Instruction Module
As is shown in Figure 3, our task-aware instruction network is composed of a series of encoder and decoder layers. To handle the limited training data, we propose to leverage the learned external knowledge from natural language datasets to guide the learning of sign languages. More specifically, we design a task-aware instruction module to dynamically inject external knowledge from pre-trained models into our encoder and decoder. Below, we shall present the details.
Encoder. Given the recognized glosses, let  $H_{I}$  denotes the instruction features encoded by the pre-trained model (PTM),  $H_{E}$  and  $H_{E}^{\prime}$  denotes the input and output of encoder which is randomly initialized. As shown in Figure 4,  $H_{I}$  and  $H_{E}$  are fed into the task-aware instruction module for feature fusing. Then, the output of the instruction module is fed into residual connection (Add&Norm) and feed forward network (FFN).
The light yellow box of Figure 4 shows the detailed design of task-aware instruction module. Specifically, we feed  $H_{E}$  into a self-attention module to learn the contextual relationship between the features of glosses, while  $H_{I}$  is fed into a PTM-attention, which is the same architecture as self-attention. Different from existing work which employ PTM in general neural network (Zhu et al., 2020), we insert an adaptive layer to fine-tune PTM-attention output for SLT task, to transform general gloss features into task-aware features.
$$
h _ {i} = \sigma \left(A t t n _ {I} \left(h _ {t}, H _ {I}, H _ {I}\right)\right) \tag {1}
$$
where  $\sigma (\cdot)$  denotes the adaptive layer (we set it as fully connection layers here), and  $h_t$  denotes the gloss features at time step  $t$ . Then, the output of two modules are combined via  $\alpha$  strategy. The whole process is formulated as follows:
$$
\hat {h} _ {t} = (1 - \alpha) A t t n _ {E} (h _ {t}, H _ {E}, H _ {E}) + \alpha h _ {i} \quad (2)
$$
where  $Attn_{E}$  and  $Attn_{I}$  are two attention layers with different parameters, which follow (Vaswani et al., 2017). The way of setting an optimal  $\alpha$  will be introduced later.
Decoder. Let  $S_{D}$  and  $S_{D}^{\prime}$  denotes the input and output of decoder,  $s_t$  denote the hidden state at time step  $t$ , and  $s_0$  denotes the beginning token of a sentence, i.e.,  $< bos >$ . The hidden states are passed to a masked self-attention ensuring that each token may only use its predecessors as follows:
Figure 4: Details of Encoder layer, Decoder layer, and and Instruction Module.
$$
\tilde {s} _ {t} = \operatorname {A t t n} _ {D} \left(s _ {t}, s _ {1: t}, s _ {1: t}\right) \tag {3}
$$
Representations  $H_{E}^{\prime}$  and  $H_{I}$  extracted from encoder and PTM are fed into the decoder-attention and PTM-attention module, respectively, as shown in the right part of Figure 4. Similar to Encoder, we formulate this decoding output as:
$$
\hat {s} _ {t} = (1 - \alpha) A t t n _ {D} (\tilde {s} _ {t}, H _ {E} ^ {\prime}, H _ {E} ^ {\prime}) + \alpha h _ {i} \quad (4)
$$
where  $Attn_{D}$  represent decoder-attention, and  $\hat{s}_{t}$  is the output of decoder instruction module.
Learning-based feature fusion. As shown in Eq. (2), representations extracted from both PTM- and self- attention are fused via a parameter  $\alpha$ . How to set a reasonable and optimal  $\alpha$  will directly affects the learning performance, which is a problem worthy of exploration. Instead of manually setting a constant  $\alpha$ , we propose a learning-based strategy to encourage the network to learn the optimal  $\alpha$  by itself for better feature fusion.
Specifically, learning-based strategy means that we adopt the back-propagation learning algorithm to update  $\alpha$  during the network training process:
$$
\alpha_ {t + 1} = \Gamma \left(\alpha_ {t}, g _ {t}\right) \tag {5}
$$
where  $g_{t}$  indicates the gradient and  $\Gamma(\cdot)$  represents the optimization algorithm. Though the idea of self-learning is straightforward, we shall show in the experiment section that it is quite effective compared with many other strategies.
4.3 Multi-level Data Augmentation
To decrease the discrepancy between glosses (input) and texts (output), we propose a multi-level data augmentation scheme. Our key idea is that, besides existing gloss-text pairs, we use upsampling as our data augmentation algorithm and generate text-text pairs as extended samples to introduce texts information into glosses, thus enlarging the feature distribution space of glosses.
Actually, there is a trade-off between augmentation and overfitting, which means the upsampling ratio  $\Phi_{upsamp}$  should be determined by the degree of gloss-text difference. We here propose four factors  $\phi = [\phi_v,\phi_r,\phi_s,\phi_d]$  to calculate the difference in terms of token, sentence and dataset level, and set weighted  $\phi$  as  $\Phi_{upsamp}$ .
Token level. Vocabulary Different Ratio (VDR,  $\phi_v$ ) is used to measure the difference of gloss vocabulary space and text's, as calculated by Eq. (6).
$$
\phi_ {v} = 1 - \frac {\left| W _ {\mathcal {G}} \right|}{\left| W _ {\mathcal {G}} \cup W _ {\mathcal {S}} \right|} \tag {6}
$$
where  $W_{\mathcal{G}}$  and  $W_{\mathcal{S}}$  represent gloss and text vocabularies, and  $|\cdot|$  denotes the size of set.
We present Rare Vocabulary Ratio (RVR,  $\phi_r$ ) to calculate the ratio of the rare words:
$$
\phi_ {r} = 1 - \frac {\sum_ {\mathcal {G} \in W _ {\mathcal {G}}} \# (C o u n t e r (\mathcal {G}) <   \tau_ {r})}{| W _ {\mathcal {G}} \cup W _ {\mathcal {S}} |} \tag {7}
$$
where  $\# (\cdot)$  is 1 if the value is true, else 0, Counter  $(\mathcal{G})$  is to calculate the gloss vocabulary frequency, and  $\tau_{r}$  means the empirical thresh frequency determined by the vocabulary frequency, which is empirically set to be 2.
Sentence level. We propose Sentence Cover Ratio (SCR,  $\phi_s$ ) to compute the gloss-text pair similarity and covered ratio, calculated as:
$$
r _ {i} = \frac {\left| \mathcal {G} _ {i} \cap \mathcal {S} _ {i} \right|}{\left| \mathcal {S} _ {i} \right|}, \quad \phi_ {s} = 1 - \frac {1}{N} \sum_ {i, r _ {i} > \tau_ {c}} r _ {i} \tag {8}
$$
where  $r_i$  denotes the covered ratio of gloss-text pair  $\mathcal{G}_i$  and  $\mathcal{S}_i$ , while  $\tau_c$  means the empirical thresh (set  $\tau_c = 0.5$ ). We labeled gloss-text pairs which satisfy  $r_i > \tau_c$  as candidates  $\mathcal{C}$ .
Dataset level. We use Dataset Length-difference Ratio (DLR,  $\phi_d$ ) to calculate the length of sentence distance, calculated as:
$$
\phi_ {d} = 1 - \frac {\sum_ {i} | \mathcal {G} _ {i} |}{\sum_ {i} | \mathcal {S} _ {i} |} \tag {9}
$$
Then we can get the upsampling ratio by:
$$
\Phi_ {u p s a m p} = \theta * \phi \tag {10}
$$
where the weight matrix  $\theta$  is empirically set as [0.1, 0.1, 0.6, 0.2], corresponding to the weight of  $[\phi_v, \phi_r, \phi_s, \phi_d]$ , as we suppose the sentence level matters the most and the weight of token level is the same as dataset level. Lastly, we obtain the upsampling ratio and use upsampling strategy among all candidates  $\mathcal{C}$  to enrich the dataset.
Datasets. We conduct our experiments on two popular benchmark datasets of different languages and scales, including PHOENIX-2014-T (Camgoz et al., 2018) dataset and ASLG-PC12 (Othman and Jemni, 2012) dataset.
Specifically, PHOENIX-2014-T, i.e., PH14, is an open-source German sign language dataset, recorded from broadcast news about the weather. This dataset contains parallel sign language videos from 9 different signers, gloss annotations with a vocabulary of 1066 different signs, and their translations with a vocabulary of 2887 different words.
ASLG-PC12, i.e., ASLG, is a parallel corpus of English written texts and American Sign Language (ASL) glosses, which is constructed based on rule-based approach. It contains more than one hundred million pairs of sentences between English sentences and ASL glosses.
Evaluation metrics. To fairly evaluate the effectiveness of our TIN-SLT, we follow (Yin and Read, 2020) to use the commonly-used BLEU-  $N$  (Ngrams ranges from 1 to 4) (Papineni et al., 2002), ROUGE-L (Lin, 2004) and METEOR (Banerjee and Lavie, 2005) as the evaluation metrics.
Experimental setup. The experiments are conducted on Ubuntu 18.04 system with two NVIDIA V100 GPUs. Our Transformers are built using 2048 hidden units and 8 heads in each layer. Besides, we adopt Adam (Kingma and Ba, 2014) as optimization algorithm with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.998$  and use inverse sqrt learning rate scheduler with a weight decay of  $10^{-3}$ . Please refer to Appendix for more hyper-parameter settings.
Model	Dev Set	Test Set
BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	METEOR	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	METEOR
PHOENIX-2014-T Dataset Evaluation
Raw Data (Yin and Read 2020)	13.01	6.23	3.03	1.71	24.23	13.69	11.88	5.05	2.41	1.36	22.81	12.12
Seq2seq (Camgoz et al. 2018)	44.40	31.93	24.61	20.16	46.02	-	44.13	31.47	23.89	19.26	45.45	-
Transformer (Camgoz et al. 2020)	50.69	38.16	30.53	25.35	-	-	48.90	36.88	29.45	24.54	-	-
Transformer (Yin and Read 2020)	49.05	36.20	28.53	23.52	47.36	46.09	47.69	35.52	28.17	23.32	46.58	44.85
Transformer Ens. (Yin and Read 2020)	48.85	36.62	29.23	24.38	49.01	46.96	48.40	36.90	29.70	24.90	48.51	46.24
DataAug (Moryossef et al. 2021b)	-	-	-	-	-	-	-	-	-	23.35	-	-
TIN-SLT(Ours)	52.35	39.03	30.83	25.38	48.82	48.40	52.77	40.08	32.09	26.55	49.43	49.36
ASLG-PC12 Dataset Evaluation
Raw data (Yin and Read 2020)	54.60	39.67	28.92	21.16	76.11	61.25	54.19	39.26	28.44	20.63	75.59	61.65
Preprocessed data (Yin and Read 2020)	69.25	56.83	46.94	38.74	83.80	78.75	68.82	56.36	46.53	38.37	83.28	79.06
Seq2seq (Arvanitis et al. 2019)	-	-	-	-	-	-	86.70	79.50	73.20	65.90	-	-
Transformer (Yin and Read 2020)	92.98	89.09	83.55	85.63	82.41	95.93	92.98	89.09	85.63	82.41	95.87	96.46
Transformer Ens.(Yin and Read 2020)	92.67	88.72	85.22	81.93	96.18	95.95	92.88	89.22	85.95	82.87	96.22	96.60
TIN-SLT (Ours)	92.75	88.91	85.51	82.33	95.17	95.21	93.35	90.03	87.07	84.29	95.39	95.92
5.2 Comparison with Others
To compare our TIN-SLT against state-of-the-art approaches on sign language translation task, we conducted two groups of experiments, Gloss2Text (G2T) and Sign2Gloss2Text (S2G2T).
Evaluation on G2T. G2T is a text-to-text translation task, whose objective is to translate ground-truth sign glosses to spoken language sentences. In specific, for PH14 dataset, we should output German spoken language sentences; while for ASLG dataset, we should output English sentences. Table 1 summarizes the comparison results. Clearly, our TIN-SLT achieves the highest values on most evaluation metrics with a significant margin. Particularly, the superiority of our method on PH14 dataset is more obvious, where almost all the evaluation values are the highest. Thanks to our multi-level data augmentation scheme, the integrity of translated sentences has been improved, which is reflected in the significant improvement of BLEU-N metric. In addition, the strong guidance from external knowledge also encourages our network to generate translated sentences in correct grammar, consistent tense and appropriate word order. For the lower ROUGE-L metric, we think that although the instruction module obviously help improve the accuracy and fluency of translation results, it leads to a slight decrease of continuous texts' recall rate in this task.
Evaluation on S2G2T. S2G2T is an extended task beyond G2T, which aims to recognize sign language videos to sign glosses, and then translate the recognized glosses to spoken sentences. Hence, un
Table 1: Comparing the translation performance of TIN-SLT against state-of-the-art techniques on PHOENIX-2014-T and ASLG-PC12 datasets. Clearly, our TIN-SLT achieves the best performance on most metrics.
Model	Test Set
BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGL-L	METEOR
G2T	44.13	31.47	23.89	19.26	45.45	-
S2G-G2T	41.54	29.52	22.24	17.79	43.45	-
S2G2T	43.29	30.39	22.82	18.13	43.80	-
Sign2	46.61	33.73	26.19	21.32	-	-
Bahdanau	47.53	33.82	26.07	21.54	45.50	44.87
Luong	47.08	33.93	26.31	21.75	45.66	44.84
Transformer Ens.	50.63	38.36	30.58	25.40	48.78	47.60
TIN-SLT (Ours)	51.06	38.85	31.23	26.13	48.56	47.83
Table 2: Comparing the S2G2T performance by using our TIN-SLT and state-of-the-art techniques on PHOENIX-2014-T dataset. The results of G2T, S2G-G2T and S2G2T are from (Camgoz et al., 2018). The results of Sign2 are from (Camgoz et al., 2020). The results of Bahdanau, Luong, and Transformer Ens. are from (Yin and Read, 2020). Clearly, our TIN-SLT achieves the highest values on most metrics.