SlowGuess/NAACL_2021_OCR huggingface dataset:
Wenxi Li $^{1}$ , Yiyang Hou $^{1}$ , Yajie Ye $^{2}$ , Li Liang $^{1*}$  and Weiwei Sun $^{3}$
$^{1}$ Department of Chinese Language and Literature, Peking University  $^{2}$ Wangxuan Institute of Computer Technology, Peking University
$^{3}$ Department of Computer Science and Technology, University of Cambridge
{liwenxi,yiyang.hou,liangli15,yeyajie}@pku.edu.cn
ws390@cam.ac.uk
Universal Semantic Tagging aims to provide lightweight unified analysis for all languages at the word level. Though the proposed annotation scheme is conceptually promising, the feasibility is only examined in four Indo-European languages. This paper is concerned with extending the annotation scheme to handle Mandarin Chinese and empirically study the plausibility of unifying meaning representations for multiple languages. We discuss a set of language-specific semantic phenomena, propose new annotation specifications and build a richly annotated corpus. The corpus consists of 1100 English-Chinese parallel sentences, where compositional semantic analysis is available for English, and another 1000 Chinese sentences which has enriched syntactic analysis. By means of the new annotations, we also evaluate a series of neural tagging models to gauge how successful semantic tagging can be: accuracies of  $92.7\%$  and  $94.6\%$  are obtained for Chinese and English respectively. The English tagging performance is remarkably better than the state-of-the-art by  $7.7\%$ .
Developing meaning representations across different languages plays a fundamental and essential role in multilingual natural language processing, and is attracting more and more research interests (Costa-jussa et al., 2020). Existing approaches can be roughly divided into three categories: the crosslingual approach focuses on lending semantic annotation of a resource-rich language, such as English, to an under-resourced language (Wang et al., 2019; Blloshmi et al., 2020; Mohiuddin and Joty, 2020); the interlingual approach attempts to
provide a unified semantic framework for all languages (Abend and Rappoport, 2013; White et al., 2016; Ranta et al., 2020); the multilingual approach aims at developing comparable but not necessarily identical annotation schemes shared by different languages (Bond and Foster, 2013; Baker and Ellsworth, 2017; Pires et al., 2019).
In line with the interlingual approach, Universal Semantic Tagging (UST; Bjerva et al., 2016) develops a set of language-neutral tags (hereafter referred to as sem-tag) to annotate individual words, providing shallow yet effective semantic information. Semantic analyses of different languages utilise a same core tag set, but may also employ a few language-specific tags. Figure 1 presents an example.
English I/PRO had/PST repaired/EXT my/HAS watch/CON ./NIL
German Ich/PRO hatte/PST mein / HAS Armbanduhr/CON repariert/EXT./NIL
Italian Ho/NOW riparito/EXT il/DEF mio/HAS orologio/CON./NIL
Chinese 我/PRO 把/OBJ 我/PRO 的/MOD 手表/CON 修/EXT 好/EXT 了/PFT /NIL
Figure 1: An example of parallel sentences and their sem-tags. PRO: anaphoric & deictic pronouns; PST: past tense; EXT: untensd perfect; HAS: possessive pronoun; CON: concept; NOW: present tense; DEF: definite; OBJ: object; MOD: modification; PFT: perfect tense; NIL: empty semantics. All tags are universal, with the exception of non-core tags OBJ and MOD, which are newly created to annotate Chinese-specific linguistic phenomena that can not be represented by the existing system.
The idea of sem-tag is first applied to the Parallel Meaning Bank (PMB; Abzianidze et al., 2017), where a multilingual corpus, including Dutch, German and Italian, is semi-automatically built by projecting semantic tags from English sentences to
*This author is now working in Tencent.
<sup>1</sup> The terminology in the literature is quite diverse—the usages of “crosslingual”, “interlingual” and “multilingual” vary from author to author.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5554-5566 June 6-11, 2021. ©2021 Association for Computational Linguistics
their translated counterparts. However, it is insufficient to prove the feasibility of UST only through some cases of inflectional and genetically related languages, because one main challenge in developing interlingual meaning representations is unifying annotations related to different characteristics of different languages. We argue that two questions with regard to universality of UST are still unanswered. Firstly, homologous words in PMB languages facilitate the application of UST, but it is not clear whether UST is equally applicable to languages sharing little cognates, although UST employs a delexicalised method. Another concern is from typology: it still remains unknown whether word-level semantic tags are effective for annotating long "sentence-words" composing many morphemes which are common in agglutinative languages (e.g. Turkish and Japanese) and polysynthetic languages (e.g. Eskimo languages).
This paper takes Mandarin Chinese, a phylogenetically distant language from the Indo-European family, as an example to explore the effectiveness of UST as a universal annotation scheme. Considering the balance of Chinese-specific linguistic properties and universality, we present a more comprehensive tag set where six new tags are added, indicating most sem-tags are applicable to Chinese (§2). Based on the new tag set, we establish a parallel corpus by manually translating WSJ into corresponding Chinese sentences and annotating sem-tags for 1100 sentence pairs. It is a peer-reviewed corpus with  $92.9\%$  and  $91.2\%$  interannotator observed agreement of Chinese and English respectively (§3). This relatively successful practice of UST in Chinese suggests it keeps the balance between the depth of represented information and the breadth of its coverage of languages. In other words, shallow semantics of UST enables it to be extended to annotate diversified languages.
By means of the newly created corpus, we evaluate a series of neural sequence labeling techniques (§4). The results demonstrate that the proposed scheme is promising with the accuracy of Chinese achieving 92.7% and the accuracy of English 94.6% (§5). The English tagging performance is remarkably better than the state-of-the-art (Abzianidze and Bos, 2017) by 7.7%, even though the sentences in our corpus are much longer than PMB on average, with 25 tokens per sentence compared with 6 in PMB.
In order to analyse the divergence between an-
notations of English and Chinese data and the plausibility of developing universal semantic representation in general, we manually annotate word alignment for 500 sentences. By studying the aligned counterparts, we argue that universality is still threatened to some extent because there are  $37.0\%$  aligned tokens with mismatched sem-tags. This phenomenon is mainly due to grammatical divergence, information loss of translation and difference of annotation strategies. All the analyses based on word alignment suggest that even for a delexicalised, relatively shallow meaning representation scheme, it can still be problematic to ensure that semantic representations could be comparable in a word-to-word way.
2 Tailoring Tag Sets for Mandarin Chinese
Considering different linguistic ways to encode tense, aspect, prepositions, measure words, subordinate clauses and comparative expressions, we provide a tailored version of UST to handle Mandarin Chinese. We present the complete tailored tag set in the Appendix.
Events and tense/aspect Different from English as well as many other Indo-European languages, there are no inflection-style tense-markers in Mandarin. Therefore, the morphological tense-related labels, e.g. ENS and EPS, are removed. Alternatively, temporal interpretation of Chinese can be conveyed through function words, adverbials or shared understanding of the context in Chinese (Smith and Erbaugh, 2005). Apart from the last way, the previous two are encoded by sem-tags FUT and IST. As for aspect in Chinese, there are only four commonly recognized aspect markers, denoting the preceding verbs are actualized or ongoing—了/过 are perfective (PFT) and 在/着 are progressive (PRG) (Liu, 2015).
Preposition Prepositions of English and Chinese vary in their history origins though they have similar syntactic function at present. English prepositions are mainly created to replace the lost inflectional case markers (Mitchell, 1985). On the other hand, Chinese prepositions can be traced to verbs. Li and Thompson (1989) even go so far as to call them coverbs since some of them are like verbs and can be used as verbs that have similar meanings. This term can avoid labeling them either verbs or prepositions. In this regard, Chi-
EXS untensed simple: to walk, is eaten
ENS present simple:  $\text{we walk, he walks}$
EPS past simple: ate, went
EXG untensed progressive: is running
EXT untensed perfect: has eaten
EXS untensed simple: 走、跑、休息
EXG untensed progressive: 吃着、在看
EXT untensed perfect: 换了、见过
Table 1: EVE tags of English and Chinese.
NOW present tense: is  $\text{sking}$ ,  $\text{do}$ , has  $\text{skied}$ , now
PST past tense: was baked, had gone, did go
FUT future tense: will, shall
PRG progressive: has been being treated
PFT perfect: has been going/done
NOW present tense: 现在
FUT future tense: 将
PRG progressive: 在，着
PFT perfect: 了、过
Table 2: TNS tags of English and Chinese.
nese prepositions should not follow the practice on English because REL emphasizes grammatical relations between verbs and nouns while in Chinese the degree of grammarization of prepositions is not so far.
Consequently, we design a separate set of sem-tags for Chinese see prepositions by borrowing existing sem-tags (DXT/DXP/ALT) and adding some new sem-tags (MAN/RES/AIM/OBJ/COM).
Meaning	Sem-tag	Example
time & places	DXT / DXP	从、到、在、朝
manners	MAN	按照、用、被
reason & aim	RES / AIM	因、由于、为了
object	OBJ	对、和、替、连
comparative	COM	比、较
alternative	ALT	除、除了、除去
Table 3: Classification of Chinese prepositions and their corresponding sem-tags and examples.
Classifier Classifier is a Chinese-specific word class which is inserted between numerals and nouns to denote quantity. This category does not
exist in English so we generalize UOM over the unit of measurement since its function is quite similar to classifiers (Li and Thompson, 1989).
Subordinate clause Whether subordinate clauses exist in Chinese is controversial since not all the clauses meet the standard in a lower position than the main clause. Additionally, words corresponding to subordinate conjunctions of English such as因为 (because), 虽然 (although), etc, constitute a heterogeneous group and do not necessarily select a subordinating clausal complement (Paul, 2016). Given these two reasons, SUB is (temporarily) removed to avoid controversy.
Comparative expression UST designs a detailed label set to annotate comparative expressions in English. See Table 4. In particular, though expressions labeled as MOR/TOP and LES/BOT utilize exactly the same syntactic constructions, they are separated according to their meaning, in a way that is more oriented by applications. Different from English, Mandarin does not have morphological comparatives and superlatives. To express comparative-related meaning, adverbs more (roughly means more) and the most (roughly means most) are utilized and annotated as MOR and TOP respectively. Accordingly, LES and BOT are deleted.
EQU equative: as tall as John, whales are mammals
MOR comparative positive: smarter, more
LES comparative negative: less, worse
TOP superlative positive: smartest, most
BOT superlative negative: worst, least
ORD ordinal: 1st, 3rd, third
EQU equative: 这么、这样、和他一样高
MOR comparative positive: 更
TOP superlative positive: 最
ORD ordinal: 第一、首次
Table 4: COM tags of English and Chinese.
3 The Corpus
We introduce a new moderate-sized corpus containing high-quality manual annotations for English and Chinese, which is now available at https://github.com/pkucoli/UST.
To support fine-grained cross-lingual comparisons, the corpus includes 1100 parallel sentence pairs. We select 1100 sentences from the Wall Street Journal (WSJ) section of Penn TreeBank (PTB; Marcus et al., 1993). We choose it because it contains detailed semantic annotations and the sentences are relatively long, thus potentially carrying more complex information. It is noteworthy that various syntactic and semantic analyses of these English sentences have been built by multiple projects, e.g. DeepBank (Flickinger et al., 2012), PropBank (Palmer et al., 2005) and OntoNotes (Weischedel et al., 2013).
We then obtain Chinese counterparts of original English sentences by employing English-Chinese bilinguals to do literal translation. In addition, we also select 1000 sentences from Chinese TreeBank (CTB; Xue et al., 2005), where manual syntactic analyses are available.
One doctoral student and one undergraduate student, majoring in linguistics, annotate the pair sentences. The guideline for English annotation is derived from the universal semantic tag set (Abzianidze and Bos, 2017) with reference to data in PMB and Chinese is annotated based on the modified tag set in the appendix. The annotation process consists of three steps: firstly, annotators independently annotate 100 Chinese WSJ sentences, and later compare and discuss disagreements between the annotations. The conflicting cases are then analyzed to modify the specification. After some iterations, the consistency between annotators is significantly improved. Additionally, we find part-of-speech (POS) tags are quite useful to accelerate manual annotation. Therefore, we apply the Stanford CoreNLP tool (Manning et al., 2014a) to get automatically predicted POS tags for the translated Chinese sentences.
Quality of the corpus The observed inter-annotator agreement in annotating Chinese and English sub-corpus data achieves  $92.9\%$  and  $91.2\%$  for Chinese and English sentences respectively. A high consistency in the annotation of both sub-corpus is obtained, which, in our view, demonstrates that UST is feasible for Chinese and the adjustment of original tag set is relatively satisfactory.
Re-tagging In order to improve the quality of annotation, we leverage the re-tagging strategy (Ide and Pustejovsky, 2017). Specifically, we investigate disagreements between initial model predictions and manual tagging, and correct manual annotation errors. After a round of re-tagging and re-training, the disagreement between the gold and the output of the tagger reduces from  $10.3\%$  to  $7.9\%$  on Chinese and  $6.7\%$  to  $5.2\%$  for English.
3.3 Divergence between English and Chinese Annotations
As a multilingual annotation scheme, UST represents semantic information in an interlingual way. Therefore, we want to answer after the modification of tag set, how the retained cross-lingual syntax and semantic divergence between distant languages still threatens its universality. We leverage a token-level word alignment for 500 parallel sentence pairs and investigate sem-tag mismatching between aligned tokens. Of the total 7295 pairs of tokens aligned, tokens in 3392 pairs share matched semantic tags with their counterparts, with a matching rate of  $46.5\%$ . Note that punctuation and tokens tagged with NIL are excluded. Figure 2 shows an example of word alignment and sem-tag matching.
Our divergence analysis based on alignment is under the assumption that, as both the tasks of alignment and sem-tagging are concerning token-level semantic representation, the matched token pairs are expected to share the same sem-tags. Non-correspondence between aligned counterparts would therefore suggest divergence between the annotations in two languages, and further, may reveal problems caused by cross-lingual divergence.
Word alignment Word alignment between sentence pairs is firstly automatically acquired with Berkeley Aligner² and then manually corrected.
Matching rate and mismatches In general, aligned tokens are mostly entities or events, and among matches, the most frequent sem-tag is CON, followed by ORG and ROL. Other tags whose proportions in all matches exceed  $3\%$  are EXS, QUc, IST, PER and GPE. And the match per edge rates of these tags are also relatively high except for IST (see Table 5). However, since the mismatch phenomenon in CON, ORG and EXS are also
$^{2}$ https://code.google.com/archive/p/berkeleyaligner/
Figure 2: An example of alignment. Red lines show that some aligned words may have different tags. _ORG: organization; _EPS: past tense; _EXS: untensed simple; _ROL: role; _COO: coordination; _EXT: untensd perfect; _PFT: perfect; _HAS: possessive pronoun; _CON: concept; _REL: relation; _MAN: manner; _AND: conjunction & univ. quantif.; _UOM: unit of measurement; _GPE: geo-political entity; _DIS: disjunction & exist. quantif.; _QUC concrete quantity _NIL: empty semantics; _MOD: modification; _OBJ: object; _PRO: modification.
not rare, annotation divergence could probably exist. A linguistically-motivated analysis suggests the following important factors:
- Grammatical divergence: an example is EXS in Figure 2. As illustrated in §2, it is used to tag Chinese verbs that are non-progressive or non-perfect, while only limited to untensed simple for English. This grammatical difference leads to tag set modification and thus results in sem-tag mismatch.
- Information loss caused by non-literal translation: In the example in Figure 2, approved its acquisition is translated as 批准...对其进行收购, which cause mismatch between acquisition (noun, CON) and 收购 (verb, EXS).
- Different annotation strategy for MWE: Corp. is tagged ORG while in their Chinese counterparts公司 are tagged CON.
Sem-tag	Frequency	Correspondence
CON	34.9%	76.0%
ORG	8.7%	69.0%
ROL	7.2%	78.1%
EXS	6.0%	73,3%
QUC	5.8%	65.7%
IST	5.5%	31.7%
PER	4.4%	92.8%
GPE	4.4%	81.1%
Table 5: Frequency and correspondence rate of 8 sem-tags.
4 Tagging Models
Long Short-Term Memory (Hochreiter and Schmidhuber, 1997, LSTM) models have been
widely used in various sequential tagging tasks (Huang et al., 2015; Ma and Hovy, 2016; Bohnet et al., 2018) and have achieved the state-of-the-art performance for many popular benchmark datasets. In our paper, we use Bidirectional LSTM (BiLSTM) with and without a Conditional Random Field (CRF) inference layer to build baseline systems for our dataset. In the rest part of this section, we will briefly formulate our baseline tagging models and introduce some widely used techniques that may enhance prediction for some tagging tasks.
Model For a word  $w_{i}$  in an input sentence  $(w_{1},w_{2},\dots,w_{n})$ , we use dynamically learned word embeddings  $e_1$  summed with the feature vectors calculated by BERT/ELMo after a linear projection  $W_{e}$  as the input of BiLSTM. If the POS tag of word  $w_{i}$  is used as additional input, we extend  $x_{i}$  with the embedding  $p_i$  of the POS tag before passing it into the BiLSTM.
$$
\boldsymbol {x} _ {i} = \boldsymbol {e} _ {i} + \operatorname {B E R T} \left(w _ {1},..., w _ {n}\right) _ {i} \boldsymbol {W} _ {e}
$$
$$
\boldsymbol {f} _ {i}, \boldsymbol {b} _ {i} = \operatorname {B i L S T M} \left(\boldsymbol {x} _ {i} \oplus \boldsymbol {p} _ {i}, \dots \boldsymbol {x} _ {n} \oplus \boldsymbol {p} _ {n}\right) _ {i}
$$
After obtaining the contextual representations  $\pmb{f}_i$  and  $\pmb{b}_i$ , we pass the concatenation of  $\pmb{f}_i$  and  $\pmb{b}_i$  to a multilayer perceptron (MLP) to calculate the scores vector  $s_i$  over semantic tags.
$$
\boldsymbol {s} _ {i} = \operatorname {M L P} \left(\boldsymbol {f} _ {i} \oplus \boldsymbol {b} _ {i}\right)
$$
Finally, we feed  $s_i$  into a softmax layer to choose a tag with highest probability for each word independently, or a CRF layer which can select the tag sequence with highest probability for the whole sentence.
Subword/Character-level Models In order to solve the out-of-vocabulary (OOV) issues in sequence tagging tasks, many subword-level and character-level models are proposed (Akbik et al., 2018; Ling et al., 2015; Bohnet et al., 2018). We do not use these models for experiments, instead we leverage pretrained language models to handle OOV issues, such as BERT (Devlin et al., 2019) and ELMo (Peters et al., 2018). These pretrained language models are trained on large corpus and use a subword/character-level vocabulary, which provide better contextual word representations.
POS features POS categories can provide low-level syntax infomation which is beneficial for sem-tagging. In our experiments, we try to use POS tags as additional inputs for our baseline systems.
Multi-task Learning (MTL) Multi-task learning (MTL) is a widely discussed technique in the literature. Previous work (Changpinyo et al., 2018) shows that MTL can improve sequence tagging tasks in some cases. In our experiments, we try to jointly train a POS tagger and a semantic tagger which use a shared BiLSTM.
We conduct experiments on English and Chinese data separately. Since there are only about 2100 Chinese sentences and 1100 English sentences which are annotated, in order to achieve more stable tagging accuracy for future comparison, we randomly split the whole dataset into 5 folds. One fold is a test set and the remaining serves as the training set where our model is trained on  $85\%$  instances and model selection is judged by the performance on the rest  $15\%$  instances. And then the tagging accuracy will be calculated using the best model on the selected fold. Finally, we report the average accuracy on these 5 folds.
Built on the top of PyTorch (Paszke et al., 2017), we employ BiLSTM as our baseline model and all the models are trained for 8000 minibatches, with a size of 32. Using the Adam optimizer (Kingma and Ba, 2015) and a cosine learning rate annealing method, we train the model with an initial learning rate chosen from  $\{0.0001, 0.005, 0.001\}$ . The details of parameters setting in different models are as follow: 1) the dimension of the hidden states of LSTM is set to
128 for each direction and the number of layers is set to 1; 2) the embeddings of POS tags are randomly initialized and has a dimension of 32 while the embeddings of words have a dimension of 300 and are initialized by the GloVe vectors $^{3}$  (Pennington et al., 2014) and pre-trained word vectors $^{4}$  (Li et al., 2018) for English and Chinese respectively $^{5}$ ; 3) the parameters of BERT/ELMo are fixed during the training of our sequence tagging models; 4) for models with MTL, we directly optimize the sum of the losses for both POS tagging and universal semantic tagging.
Figure 3 shows the overall performance of different models. Gold POS tags bring significant performance improvements, which is also verified by Huo and de Melo (2020). However, MTL can only slightly improve the overall results. When pre-trained contextualized word embeddings are utilized, the gap between different models becomes insignificant. Additionally, the significant improvement of English accuracy over previous state-of-the-art is also attributed to the use of pretraining models: with the help of BERT, a simple BiLSTM tagger can be close to  $92.0\%$  -accurate for Chinese and  $94.6\%$  for English while without it, tagging accuracy of English data is around  $85\%$ .
(a) Chinese
(b) English
Figure 3: Averaged tagging accuracies.
3nlp.stanford.edu/projects/glove/   
4github.com/Embedding/   
Chinese-Word-Vectors   
5The embeddings missed in the pre-trained vectors are randomly initialized.
5.3 Comparative Analysis of Tagging Error
Empirical evaluation indicates competitive accuracy of our models. However, the result varies among different sem-tag categories and some of them remain at an extremely low level (Table 6).
To further improve the model's performance and have a better understanding of cross-lingual semantic representation, this section provides a fine-grained error analysis towards each underperforming sem-tag category.
Category of sem-tag	English	Chinese
ACT	speech act	96.7%	86.8%
DXS	deixis	64.9%	86.5%
ATT	attribute	86.5%	82.6%
COM	comparative	83.9%	88.2%
NAM	named entity	91.9%	89.6%