SlowGuess/ACL_2020_OCR huggingface dataset:
Canasai Kruengkrai Thien Hai Nguyen Sharifah Mahani Aljunied Lidong Bing
DAMO Academy, Alibaba Group
canasai@gmail.com,
{thienhai.nguyen,mahani.aljunied,l.bing}@alibaba-inc.com
Exploiting sentence-level labels, which are easy to obtain, is one of the plausible methods to improve low-resource named entity recognition (NER), where token-level labels are costly to annotate. Current models for jointly learning sentence and token labeling are limited to binary classification. We present a joint model that supports multi-class classification and introduce a simple variant of self-attention that allows the model to learn scaling factors. Our model produces  $3.78\%$ ,  $4.20\%$ ,  $2.08\%$  improvements in F1 over the BiLSTM-CRF baseline on e-commerce product titles in three different low-resource languages: Vietnamese, Thai, and Indonesian, respectively.
Neural named entity recognition (NER) has become a mainstream approach due to its superior performance (Huang et al., 2015; Lample et al., 2016; Ma and Hovy, 2016; Chiu and Nichols, 2016; Akbik et al., 2018). However, neural NER typically requires a large amount of manually labeled training data, which are not always available in low-resource languages. Training neural NER with limited labeled data can be very challenging. In this paper, we consider bridging multi-task learning (MTL) (Caruana, 1993; Ruder, 2017) and pretraining (Peters et al., 2018; Devlin et al., 2019) to leverage training signals of an auxiliary task that has a sufficiently large number of labeled data.
Researchers have investigated a wide variety of auxiliary tasks and resources to boost the performance of neural NER, e.g., training coarse-grained NER (Aguilar et al., 2017), fine-tuning bilingual word embeddings (Wang et al., 2017), applying language models (Rei, 2017), integrating part-of-speech (POS) tagging (Lin et al., 2018), using cross-lingual knowledge (Feng et al., 2018), and learning paraphrases (Watanabe et al., 2019).
Figure 1: Examples of product titles with NER annotation in Vietnamese. Product categories are provided by sellers and can be used as sentence-level labels.
While most of the previous studies have exploited token-level information from auxiliary tasks, a few of them have tried to use sentence-level information (Rei and Søgaard, 2018; Devlin et al., 2019). Our work is closely related to the joint labeling framework in Rei and Søgaard (2019). However, they only focused on binary classification, while we attempt to handle multi-class classification on both sentence and token levels.
In this work, we focus on improving low-resource NER by exploiting large data, only having sentence-level labels. Figure 1 shows examples of product titles on an e-commerce website in Vietnamese. While the product titles with NER annotation done by our annotators are limited, those with product categories (e.g., ELECTRONICS) labeled by sellers are abundant, which can be used to train a sentence-level classifier. A key challenge is to pass useful training signals from the sentence-level classification to the token-level NER.
Our contributions are as follows. We present the joint sentence and token labeling framework that enables multi-class classification equipped with a pre-training strategy (§2.1). We show that the current attention mechanisms can produce suboptimal
1The sellers are required to assign a category when uploading the product, but such input could be noisy as well.
5898
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5898-5905 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: Architecture of our joint sentence and token labeling model. The attention layer is optional, which can be skipped or replaced with the desired approach.
results and propose a simple approach that allows the model to learn scaling factors to obtain a proper attention distribution (§2.2). Results on product title texts indicate that the proposed method is effective for low-resource NER across three different languages: Vietnamese, Thai, and Indonesian.
2 Proposed method
Figure 2 shows the architecture of our joint sentence and token labeling model. Our model is based on hard parameter sharing (Ruder, 2017) in which the hidden layers are shared between two tasks. The task-specific layers include a conditional random field (CRF) layer for NER and a linear layer for sentence classification.
Unlike the standard MTL, which trains multiple tasks at once and expects the model to perform well on all tasks (Hashimoto et al., 2017; Rei and Søgaard, 2019), the goal of our work is to improve the performance of the main task (NER) using the auxiliary task (sentence classification) for creating pre-trained representations and as a regularizer.
2.1 Joint learning framework for multi-class classification
Shared layers Let  $w_{1}, \ldots, w_{T}$  be an input token sequence, where  $w_{t}$  denotes the  $t$ -th token in the sequence. We represent each  $w_{t}$  using a pre-trained word embedding  $\mathbf{e}_{t} \in \mathbb{R}^{d_{e}}$ , where  $d_{e}$  is the dimensionality of word embeddings. We do not fine-tune word embeddings but project them into a new space
using  $\mathbf{x}_t = \mathbf{W}_1\mathbf{e}_t$  , where  $\mathbf{W}_1\in \mathbb{R}^{d_e\times d_e}$  is a trainable weight matrix. We then feed the projected embedding sequence  $\mathbf{X} = [\mathbf{x}_1,\dots ,\mathbf{x}_T]\in \mathbb{R}^{T\times d_e}$  to a bidirectional long short-term memory (BiLSTM) layer to obtain a forward hidden state sequence  $\vec{\mathbf{H}} = [\vec{\mathbf{h}}_1,\ldots ,\vec{\mathbf{h}}_T]\in \mathbb{R}^{T\times \frac{dh}{2}}$  and a backward hidden state sequence  $\vec{\mathbf{H}} = [\vec{\mathbf{h}}_1,\dots ,\vec{\mathbf{h}}_T]\in \mathbb{R}^{T\times \frac{dh}{2}}$  where  $d_{h}$  is the number of hidden units.
We concatenate the hidden states of both directions to obtain the final hidden representation  $\mathbf{H} = [\mathbf{h}_1, \ldots, \mathbf{h}_T] \in \mathbb{R}^{T \times d_h}$ , where  $\mathbf{h}_t = \mathrm{concat}(\vec{\mathbf{h}}_t, \vec{\mathbf{h}}_t) \in \mathbb{R}^{d_h}$ . We can either use  $\mathbf{H}$  for both the sentence classification and NER tasks directly or apply an attention mechanism on it to help the model focus on particular tokens (detailed in §2.2).
Sentence classification We create a fixed size vector by applying max-pooling (Collobert et al., 2011; Conneau et al., 2017) over  $\mathbf{H}$ , which encourages the model to capture the most useful local features encoded in the hidden states. We feed the fixed size global feature vector to a linear layer to obtain the unnormalized predicted scores for each class. Let  $K$  be the number of target classes,  $s_k$  be the  $k$ -th normalized predicted score after applying a softmax function, and  $\mathbf{t} \in \mathbb{R}^K$  be the one-hot encoded true label. To train the sentence classification model, we minimize the multi-class cross-entropy loss:
$$
\mathcal {L} _ {\mathrm {C}} = - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {K} t _ {k} ^ {(i)} \log \left(s _ {k} ^ {(i)}\right), \tag {1}
$$
where  $i$  denotes the sentence index, and  $N$  is the number of training examples.
We not only train the sentence classification and NER models jointly but also pre-train the sentence classification model using a sufficiently large number of training examples with sentence-level labels only. We expect that pre-trained hidden representations would help the model generalize better on our main task, as described below.
NER Following Huang et al. (2015); Lample et al. (2016), we feed  $\mathbf{H}$  to a CRF layer to obtain the probability of a label sequence  $\mathbf{y}$ . To train the NER model, we minimize the negative log-likelihood of the correct label sequences over the training set:
$$
\mathcal {L} _ {\mathrm {N E R}} = - \frac {1}{N} \sum_ {i = 1} ^ {N} \log p (\mathbf {y} ^ {(i)} | \mathbf {H} ^ {(i)}). \tag {2}
$$
2We use the term "sentence" to conform with the literature, although our data are not always complete sentences.
5899
Joint labeling objective Combining Eqs. (1) and (2), we obtain:
$$
\mathcal {L} _ {\text {J O I N T}} = \mathcal {L} _ {\text {N E R}} + \lambda \mathcal {L} _ {\mathrm {C}}, \tag {3}
$$
where  $\lambda$  is the balancing parameter. The  $\mathcal{L}_{\mathrm{C}}$  acts as a regularization term, which helps in reducing the risk of overfitting on our main task.
2.2 Revisiting attention mechanisms
We first consider a soft-attention mechanism (Shen and Lee, 2016), which is used in Rei and Søgaard (2018, 2019). This method is computationally efficient because the attention distribution  $\mathbf{a} \in \mathbb{R}^T$  over tokens in a sentence is computed from the final hidden representation without considering relationships between hidden states. Specifically, the new final representation  $\mathbf{H}' \in \mathbb{R}^{T \times d_h}$  can be derived as follows:
$$
\mathbf {H} ^ {\prime} = \mathbf {H} + \mathbf {H} \otimes \mathbf {a},
$$
$$
\mathbf {a} = \frac {\tilde {\mathbf {a}}}{\sum_ {j = 1} ^ {T} \tilde {a} _ {j}}, \tag {4}
$$
$$
\tilde {\mathbf {a}} = \sigma \left(\mathbf {w} _ {2} \mathbf {g} + b _ {2}\right),
$$
$$
\mathbf {g} = \tanh (\mathbf {W} _ {3} \mathbf {H} ^ {\top} + \mathbf {b} _ {3}),
$$
where  $\mathbf{w}_2\in \mathbb{R}^{d_h}$ ,  $b_{2}\in \mathbb{R}$ ,  $\mathbf{W}_3\in \mathbb{R}^{d_h\times d_h}$ ,  $\mathbf{b}_3\in \mathbb{R}^{d_h}$  are trainable parameters, and  $\otimes$  denotes the column-wise matrix-vector multiplication. We use a residual connection (He et al., 2016) between the input hidden representation and the attention output as shown in Figure 2.  $\mathbf{H}'$  can be fed to NER and sentence classification.
We further explore attention mechanisms that take into account the relationships between hidden states. In particular, we apply the multi-head self-attention mechanism in Transformer (Vaswani et al., 2017), which has shown promising results in many applications (Radford et al., 2018; Devlin et al., 2019). We replace Eq. (4) with:
$$
\mathbf {H} ^ {\prime} = \mathbf {H} + \operatorname {c o n c a t} \left(\operatorname {h e a d} _ {1}, \dots , \operatorname {h e a d} _ {n}\right) \mathbf {W} ^ {O},
$$
$$
\operatorname {h e a d} _ {j} = \operatorname {a t t e n t i o n} \left(\mathbf {Q} _ {j}, \mathbf {K} _ {j}, \mathbf {V} _ {j}\right), \tag {5}
$$
$$
\mathbf {Q} _ {j}, \mathbf {K} _ {j}, \mathbf {V} _ {j} = \mathbf {H} \mathbf {W} _ {j} ^ {Q}, \mathbf {H} \mathbf {W} _ {j} ^ {K}, \mathbf {H} \mathbf {W} _ {j} ^ {V},
$$
where  $\mathbf{W}_j^Q,\mathbf{W}_j^K,\mathbf{W}_j^V\in \mathbb{R}^{d_h\times \frac{d_h}{n}};\mathbf{W}^O\in \mathbb{R}^{d_h\times d_h}$  are trainable parameters, and  $n$  is the number
of parallel heads. The attention function can be computed by:
$$
\mathrm {a t t e n t i o n} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \mathrm {s o f t m a x} (\frac {\mathbf {Q} \mathbf {K} ^ {\top}}{\alpha}) \mathbf {V}. (6)
$$
We drop the head index  $j$  for simplicity and introduce the scaling factor  $\alpha \in \mathbb{R}$ . When setting  $\alpha = \sqrt{d_h / n}$ , Eq. (6) falls back to the standard scaled dot-product attention in Transformer. Yan et al. (2019) observed that the scaled dot-product attention produces poor results for NER and proposed the un-scaled dot-product attention, where  $\alpha = 1$ .
In this work, we consider  $\alpha$  as the softmax temperature (Hinton et al., 2015) that allows adjusting the probability distribution of a softmax output. Using a higher temperature yields a softer attention distribution. However, a sharper attention distribution might be more suitable for NER because only a few tokens in the sentence are named entities. Instead of setting  $\alpha$  to 1 or  $\sqrt{d_h / n}$ , we propose to learn the scaling factors  $\delta \in \mathbb{R}^T$  for each token. We modify Eq. (6) with:
$$
\operatorname {a t t e n t i o n} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \operatorname {s o f t m a x} \left(\frac {\mathbf {Q} \mathbf {K} ^ {\top}}{\delta}\right) \mathbf {V}, \tag {7}
$$
$$
\pmb {\delta} = \min (\mathrm {R e L U} (\mathbf {w} _ {4} \mathbf {H} ^ {\top} + b _ {4}), \sqrt {d _ {h} / n}) + 1,
$$
where  $\mathbf{w}_4\in \mathbb{R}^{d_h}$ ,  $b_{4}\in \mathbb{R}$  are the trainable parameters. Since the ReLU activation function produces output values in the range  $[0,\infty)$ , the  $t$ -th element of  $\delta$  is bounded in the range  $[1,1 + \sqrt{d_h / n}]$ . This allows the model to dynamically adapt  $\delta$  without increasing much computational cost.
The data used in our experiments are product titles obtained from major e-commerce websites in Southeast Asian countries during May-June, 2019. They cover three languages, including Vietnamese (VI), Thai (TH), and Indonesian (ID). A product title is a brief, information-rich description (less than 200 characters) written by the sellers. We hired annotators and linguists for each language to annotate the product titles based on our definitions and annotation guidelines.
After the annotation process, we obtained 2,000 product titles per language labeled with 6 product attribute NER tags, including PRODUCT, BRAND, CONSUMER_GROUP, MATERIAL, PATTERN, and
COLOR. For each language, we split the data into 1,000/500/500 - training/development/test sets. The statistics of NER tags can be found in Table 3 (see Appendix A).
For some NER tags, especially PRODUCT, the number of tags is much larger than the number of examples used. One reason is that the sellers writing a product title tend to include multiple different expressions referring to the same entity (near-synonyms), with the likely intention of acquiring more hits from potential customers. Using English to illustrate: "Genuine Leather Sling Bag Crossbody Bag Messenger bag for Men Women Office Laptop", the underlined elements are 3 PRODUCT and 2 CONSUMER_GROUP entities.
The other reason is that in one product title, it is common to find repeated identical expressions in the same language, as well as the same entity words appearing in English. Using a VI example to illustrate: “T-Shirt - Ao thun in phán quang - Ao thun Nam - Ao thun nu - Ao thun phong cach Nam Nu”, the underlined elements refer to the same product (t-shirt), appearing multiple times in VI and in English.
3.2 Training details
We implement our model on top of the Flair framework (Akbik et al., 2019), which has recently achieved state-of-the-art results in various sequence labeling tasks. Following Lample et al. (2016), we use the IOBES tagging scheme. We use the pretrained word embeddings of  $\text{fastText}^4$  (Bojanowski et al., 2016) with  $d_e = 300$  dimensions for each language and a single-layer BiLSTM with  $d_h = 512$  hidden units. We apply a locked dropout (Merit et al., 2018) with the probability of 0.5 before and after the BiLSTM layer and to the attention output before the residual connection. For the multi-head self-attention layer, we adapt the implementation of "The Annotated Transformer" (Rush, 2018) and use its default hyperparameters.
We train all models using Adam (Kingma and Ba, 2015) with the batch size of 32, the learning rate of 1e-3, and the gradient clipping of 5. We initialize all model parameters by sampling from  $\mathcal{U}(-0.1,0.1)$ . We set  $\lambda$  in Eq. (3) to 1. We use the same parameter setting for all languages. We apply early stopping in which the learning rate decays by
0.5 if the F1 score on the NER development set does not improve 3 times. We train until the learning rate drops below 1e-5, or the training epochs reach 100.
3.3 Pre-trained classification models
We collect unannotated product titles for each language and group them into 6 main categories, including FASHION, HEALTHBEAUTY, ELECTRONICS, HOME_FURNITURE, MOTORS, and OTHER. Since the number of product titles is different from one language to another, we can create  $360\mathrm{k} / 30\mathrm{k}$ $1.2\mathrm{M} / 60\mathrm{k}$ $864\mathrm{k} / 60\mathrm{k}$  - training/development sets for VI, TH, and ID, respectively. Since product titles are not segmented in TH, we segment them using a character cluster-based method simplified from the hybrid model of Kruengkrai et al. (2009). We implement our word segmenter based on CRFsuite (Okazaki, 2007) and train the model using the BEST corpus (Kosawat et al., 2009).
We pre-train the classification models for each language. Since our batch size is relatively small compared to the training data size, we find it suffices to train for 2 epochs. The F1 scores on the development sets are  $90.08\%$ ,  $89.79\%$ , and  $91.91\%$  for VI, TH, and ID, respectively. The pre-trained model parameters are used to initialize the projection and BiLSTM layers.
3.4 Main results
We run each experiment 10 times using different random seeds and report the average F1 score. All experiments are run on NVIDIA Tesla P100 GPUs. Table 1 shows the results of various models on the test sets. The Joint models consistently show improvements over the NER-only models, while the Joint + Pre-trained models further boost the F1 scores. These results suggest that the proposed framework is effective for all three languages. The Joint + Pre-trained model with the Self + Learned attention mechanism achieves the best F1 scores at  $62.16\%$ ,  $61.54\%$ , and  $76.10\%$  (i.e.,  $3.78\%$ ,  $4.20\%$ , and  $2.08\%$  improvements over the NER-only baselines) for VI, TH, and ID, respectively.
In addition, we experiment using simple data augmentation. The “+10k” and “+50k” rows in Table 1 indicate the number of additional training examples automatically labeled using a dictionary created from the training set. We do not observe any improvement in both the development and test
3For TH, 941 training examples remain after removing annotation errors.
<sup>4</sup>https://fasttext.cc/docs/en/crawl-vectors.html
<sup>5</sup>https://nlp.seas.harvard.edu/2018/04/03/attention.html
5901
Model	Attention	VI	TH	ID
NER-only (+10k)	-	53.47	52.47	74.22
NER-only (+50k)	-	51.12	50.35	71.60
NER-only	-	58.38	57.34	74.02
Soft	58.18	57.49	74.20
Self + Scaled	58.82	57.80	74.55
Self + Un-scaled	59.68	58.53	75.24
Self + Learned	60.18	58.63	74.83
Joint	-	59.47	58.81	74.67
Soft	59.50	58.82	74.88
Self + Scaled	59.34	58.46	75.03
Self + Un-scaled	60.58	59.56	75.66
Self + Learned	60.25	59.35	75.18
Joint + Pre-trained	-	61.26	60.27	75.86
Soft	61.05	60.50	75.80
Self + Scaled	61.80	61.32	75.90
Self + Un-scaled	62.09	61.45	76.01
Self + Learned	62.16	61.54	76.10
Table 1: F1 scores on the test sets. NER-only = baseline BiLSTM-CRF; Joint = joint labeling model; Joint + Pre-trained = Joint initialized with the pre-trained classification model; Soft = soft-attention (Shen and Lee, 2016; Rei and Søgaard, 2019); Self = multi-head self-attention described in §2.2, where Scaled = scaled dot-product (Vaswani et al., 2017), Un-scaled = unscaled dot-product (Yan et al., 2019), and Learned = our learned scaling factors.
Model	VI	TH	ID
Joint + Pre-trained & Self + Learned	62.16	61.54	76.10
w/o residual connection	61.28	61.52	75.74
w/o locked dropout	61.87	61.08	76.22
Table 2: Model ablations for our best configuration, the Joint + Pre-trained model with the Self + Learned attention mechanism.
results and hence do not pursue this idea further with the attention mechanisms.
Table 2 shows the model ablations for our best configuration, the Joint + Pre-trained model with the Self + Learned attention mechanism. Feeding the attention output to the CRF layer without the residual connection leads to a consistent drop in the F1 scores, although it shows a less pronounced effect on TH. The results indicate that the residual connection is a useful component in our architecture. Adding the attention output to the hidden representation without applying the locked dropout (i.e., setting the dropout probability to 0) hurts the F1 scores on VI and TH but shows an improvement on ID, suggesting that fine-tuning the dropout rate could help boost the F1 scores.
Our Self + Learned scaling approach shows the competitive results for the NER-only model and achieves the best results when training in tandem with the Joint + Pre-trained model. The Soft attention mechanism (Shen and Lee, 2016; Rei and Søgaard, 2019) shows slight or no improvements, suggesting that considering relationships between hidden states when computing the attention distribution is crucial for the NER task. The Self + Un-scaled approach (Yan et al., 2019) yields better F1 scores than the Self + Scaled approach (Vaswani et al., 2017) for all configurations, suggesting that a sharper attention distribution is helpful for the NER task.
Although VI, TH, and ID are used in Southeast Asia, they do not belong to the same language family and have different writing systems and scripts (i.e.,  $\mathrm{VI} =$  Austroasiatic;  $\mathrm{TH} =$  Kra-Dai;  $\mathrm{ID} =$  Austronesian). Handling these three languages without much engineering effort reflects the generalizability of our method. Furthermore, we examine whether our method still provides improvements, even if the NER training data size increases. We create an additional set of 2k labeled examples for VI and add them to the training set (3k in total). The baseline NER-only produces  $66.81\%$  F1, while Joint + Pre-trained with Self + Learned achieves  $69.26\%$  F1 (i.e.,  $2.45\%$  improvement).
We have shown that the proposed joint sentence and token labeling model is remarkably effective for low-resource NER in three different languages: Vietnamese, Thai, and Indonesian. Our model supports multi-class classification where the sentence and token labels can be weakly related, which indicates the potential of our model for many other real-world applications. Using a larger amount of general domain texts to build pre-trained representations (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Clark et al., 2020) can complement with our model and is one of the directions that we plan to take in future work.
We thank the anonymous reviewers for their constructive comments. Kruengkrai is grateful for support from National Institute of Informatics, Japan.
5902
Gustavo Aguilar, Suraj Maharjan, Adrian Pastor López-Monroy, and Thamar Solorio. 2017. A multi-task approach for named entity recognition in social media data. In Proceedings of ACL Workshop on Noisy User-generated Text, pages 148-153.
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of NAACL (Demonstrations), pages 54-59.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of COLING, pages 1638-1649.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv:1607.04606.
Richard Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of ICML, pages 41-48.
Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, pages 357-370.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pretraining text encoders as discriminators rather than generators. In Proceedings of ICLR.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., pages 2493-2537.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of EMNLP, pages 670-680.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171-4186.
Xiaocheng Feng, Xiachong Feng, Bing Qin, Zhangyin Feng, and Ting Liu. 2018. Improving low resource named entity recognition using cross-lingual knowledge transfer. In Proceedings of IJCAI, pages 4071-4077.
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In Proceedings of EMNLP, pages 1923-1933.