SlowGuess/NAACL_2021_OCR huggingface dataset:
Properties of Chinese adjectives The low predication accuracy of ATT is largely attributable to the difficulties in differentiating IST and SST, especially in the light of high frequencies of adjectives in Chinese, which are a more complicated case compared to English adjectives. Usages of Chinese adjectives and corresponding sem-tags are shown in Table 7:
Table 6: Tagging accuracies of five lowest sem-tag categories for English and Chinese
Usage	A	A+N	A+de+N
Narrow adjectives	IST	IST/SST	IST/SST
Distinct words	n.a.	IST	IST
We propose practical strategies to improve the performance of our tagging model on differentiating IST and SST in Chinese. The first method is to establish a lexicon, based on the fact that whether an adjective can be used as a predicate is an inherent property. Thus it is possible to distinguish the use of IST and SST by simply referring to a lexicon. Another strategy is rule-based: an adnominal adjective is tagged SST only when it obtains a gradable reading. We stipulate the follow
ing rules: if tokens preceded by attribute adjectives are tagged INT, EQU, MOR and TOP, adjectives should be marked as SST. After uploading the lexicon and rules, the tagging accuracy of IST and SST raise from  $68.8\%$  and  $63.1\%$  to  $81.4\%$  and  $77.9\%$ . Overall accuracies after uploading adjective lexicon and rules are shown in Table 8.
Table 7: Usages and sem-tags of Chinese adjectives. "A" denotes adjective; "N" denotes noun; "de" is a Chinese particle denoting modification. In Mandarin Chinese, there are two sub-types of broad-sensed adjectives: narrow adjectives can both be used as predicates and modifiers while distinct words are only modifiers.
Baseline	w/MTL	w/POS
+BERT	90.2%	90.3%	92.7%
+BERT+CRT	90.0%	90.2%	92.7%
Named entity Table 9 shows the accuracy of each of NAM (named entity) for English and Chinese. Although named entities are regarded as one of the most frequently corresponding concepts shared by various languages (see §3), marked differences still exist:
- The accuracies of each sem-tag of English are generally higher than those of Chinese $^{6}$ .
- English presents a lower diversity of performance (73.3%-98.0%) compared with Chinese (58.6%-97.9%).
Table 8: Averaged Chinese tagging accuracies after uploading adjective lexicon and rules.
Sem-tag	English	Chinese
PER	person	98.00%	95.8%
GPE	geo-political entity	92.1%	92.7%
GPO	geo-political origin	88.0%	76.2%
GEO	geographical location	73.3%	58.6%
ORG	organization	94.3%	86.6%
ART	artifact	76.1%	68.9%
HAP	happening	n.a.	24.2%
UOM	unit of measurement	93.2%	97.9%
Table 9: Accuracies of sem-tags under the NAM category for English and Chinese
We propose an explanation on why English and Chinese sem-taggers perform differently on NAM: named entities in English are identified by capitalization while Chinese not. Therefore, it is harder for Chinese to calculate the scope of proper names than English, and the overall accuracy is thus influenced. Moreover, it can also be inferred that Chinese is more sensitive to the length
$\underline{\mathrm{6}}_{\mathrm{HAP}}$  is not included and will be discussed in the next paragraph.
of named entities given its difficulties in judging scope: sem-tags (PER, GPE and UOM) whose accuracies are higher than the average level, are commonly used to annotate one-token units while other below-average tags (GPO, GEO, ORG and ART) annotate multi-word proper nouns. On the contrary, English, with certain markers of named entities, shows that the decrease of accuracy with length is not as prominent as it of Chinese.
Sparse data input DXD of DXS, ITJ, HES and GRE of ACT, EQU of COM and HAP of NAM, whose presences are not enough for training and learning, need more diverse data as input in further research.
6 On Annotating Semantics
6.1 Helpfulness of Syntactic Features
The high-quality manual annotation and automatic tagging both indicate the importance of POS tags in the UST—the inter-annotator agreement and tagging accuracies increase after applying POS tags. Huo and de Melo (2020) believe this is because POS tags may facilitate semantic disambiguation though the extra syntactic information. However, what is not revealed is the underlying mechanism under which a syntactic feature can contribute to semantic analysis.
To investigate the impact of POS tags, 50 new sentences of WSJ and their Chinese counterparts are selected for a pilot study. Two annotators are asked to annotate them with or without the assistance of POS tags. Table 10 shows that POS tags have an impact on the inter-annotator agreements. This tendency is observed for both English and Chinese data.
Language	Type	IAA
English	+POS	96.1%
-POS	95.2%
Chinese	+POS	93.6%
-POS	90.8%
Table 10: The changes of inter-annotator agreements before and after the introduction of POS tags.
After a detailed investigation, we summarize the influences of POS tags on inter-annotator agreements as two points: (i) Some tokens have multidimensional semantic features and POS tags are likely to make annotators choose sem-tags related to POS features. For instance, unable may be
annotated as NOT (negation) or POS (possibility). However, after the introduction of its POS tag, i.e. ADJ, two annotators are more likely to annotate it as IST, which is appropriate for most of adjectives, rather than NOT and POS; (ii) Gerunds which do not take arguments or are not modified by adverbs are more likely to bring challenges as it is difficult for annotators to determine whether event-related sem-tags or concept-related ones are more suitable for them. It is even more difficult for Chinese annotation in which verbs do not have inflected forms. All these can be easily solved by assigning POS tags.
In our view, the reason why POS contribute to semantic annotations can be traced to discussions of theoretical linguistics. Generally speaking, POS is category of words, whose identification has been a controversial problem for a long time in this area. Some linguists are in favor of a syntactic or distributional basis of POS (Harris, 1951; Edmonds, 1967) while others advocate a semantic or notional basis (Lyons, 1966). From a notion-based perspective, assigning forms to concepts, or POS tags and sem-tags to tokens, are all a process of categorizing and classifying objects referred by these tokens, which helps explain why POS tags have a significant influence on semantic sorts. In this regard, annotations are undoubtedly impacted by POS tags. Nonetheless, some researchers rebate it, believing that the notional definitions of POS are not applicable because of itsunclearness.According to them, distribution, morphological features, grammatical functions are all useful criteria for the identification of POS. In our view, contradiction between notion-based and distribution-based approach leads to some difficulties in annotation. To avoid this, we applied POS tags which are automatically-generated by the Stanford CoreNLP tool (Manning et al., 2014b) to assist manual annotation.
However, though POS tags actually improve the inter-annotator agreement by regulating manual annotations of sem-tags in two ways, it is not clear whether they improve the quality of annotations—the first one increases the possibility of one option while the second one directly makes choices for annotators. To what extent more coarse-grained annotating standards contribute to annotations needs further research.
6.2 Challenges of Multilingual Annotations
Building comparative semantic representations across languages has been an important topic in recent years as a strategy to both contribute to semantic parsing and syntactic analysis. Existing approaches towards it can be roughly divided into three categories. First, crosslingual approach is proposed, which lends semantic annotation of a resource-rich language to an under-resourced language; see e.g. Damonte and Cohen (2018). However, crosslingual divergence between the lender and the borrower is likely to be retained to a considerable extent, especially for the languages which are phylogenetically distant. Another widely-discussed multilingual approach aims to achieve the goal by developing a comparable scheme of annotations for different languages, such as multilingual FrameNet (Baker et al., 1998) and multilingual WordNet (Miller, 1995), whose main limitation is that the semantic information represented is at the risk of oversimplifying since many in-depth properties are language-specific. The third one, the interlingual approach aims to find universal semantic frameworks for all languages. Yet it can be fairly difficult to find such appropriate interlingual frameworks.
In our view, these strategies are employed by researchers to study the major challenge i.e., the divergence of languages, encountered in representing multilingual data. And UST, which is in line with interlingual method, attempts to address it by a relatively shallow scheme. Despite the high inter-annotator agreements and tagging accuracies, there are still some divergences, which requires more in-depth study of multilingual annotation.
UST is one of previous attempts of interlingua (Abzianidze and Bos, 2017), which is originally designed to provide necessary information for semantic parsing (Bjerva et al., 2016). Primary automatic sem-taggers are built using convolutional neural networks and deep residual networks (Bjerva et al., 2016). Later, in PMB project (Abzianidze et al., 2017), the authors propose a method of projecting automatically annotated semantic tags from a sentence to its sentence- and word-aligned counterparts. Following previous works, an updated universal semantic tagset is later proposed (Abzianidze and Bos, 2017), with a
modification of deriving the tagset in a data-driven manner to disambiguate categories. In this work, a tri-gram based tagging model, TnT tagger (Brants, 2000), is also initially explored for bootstrapping utilization. In a recent study built on Bjerva et al. (2016), employing sem-tag in multi-task learning is found to be beneficial to both sem-tag task and other NLP tasks including Universal Dependency POS tagging, Universal Dependency parsing, and Natural Language Inference (Abdou et al., 2018). Overall, these studies indicate that sem-tags are effective in conducting various NLP tasks.
In this paper, we take Chinese into account to provide a more comprehensive tag set based on which we establish a reliable manually-annotated corpus, and show that promising performance of automatic semantic tagging is obtained after employing MTL as well as gold POS tag and leveraging pre-trained models. The overall success of this approach prompts a reflection of universality of different languages and operability of multilingual meaning representation: 1) UST is plausible in general partly because it is delexicalised and can thus represent phylogenetically languages after some adoptions; 2) universality is threatened to some extent because there are aligned but mismatched tokens between English and Chinese, which are caused by grammatical divergence, information loss of translation and different annotation strategies for MWE; and 3) innate crosslingual divergences still exist even in NAM's thought to be the most consistent pairs, which needs further exploration.
Though our work demonstrates the plausibility of developing a shared delexicalised and shallow annotation scheme to mitigate divergences across languages, it seems that more in-depth semantic analysis, especially lexicalised ones, may not be possible to be unified. We think a wider range of languages can be annotated after some minor adaptations of scheme. But it is still unknown how to get deeper processing information on this basis and thus develop an enhanced understanding of multilingual meaning representation.
We thank the three anonymous reviewers for their helpful comments.
Mostafa Abdou, Artur Kulmizev, Vinit Ravishankar, Lasha Abzianidze, and Johan Bos. 2018. What can we learn from semantic tagging? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4881-4889, Brussels, Belgium. Association for Computational Linguistics.
Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228-238, Sofia, Bulgaria. Association for Computational Linguistics.
Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Ludmann, Duc-Duy Nguyen, and Johan Bos. 2017. The Parallel Meaning Bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 242-247, Valencia, Spain. Association for Computational Linguistics.
Lasha Abzianidze and Johan Bos. 2017. Towards universal semantic tagging. In IWCS 2017 — 12th International Conference on Computational Semantics — Short papers.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In COLING 2018, 27th International Conference on Computational Linguistics, pages 1638-1649.
Collin F. Baker and Michael Ellsworth. 2017. Graph methods for multilingual FrameNets. In Proceedings of TextGraphs-11: the Workshop on Graph-based Methods for Natural Language Processing, pages 45-50, Vancouver, Canada. Association for Computational Linguistics.
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 86-90, Montreal, Quebec, Canada. Association for Computational Linguistics.
Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic tagging with deep residual networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3531-3541, Osaka, Japan. The COLING 2016 Organizing Committee.
Rexhina Blloshmi, Rocco Tripodi, and Roberto Navigli. 2020. XL-AMR: Enabling cross-lingual AMR parsing with transfer learning techniques. In
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2487-2500, Online. Association for Computational Linguistics.
Bernd Bohnet, Ryan McDonald, Goncalo Simões, Daniel Andor, Emily Pitler, and Joshua Maynez. 2018. Morphosyntactic tagging with a metaBiLSTM model over context sensitive token encodings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2642-2652, Melbourne, Australia. Association for Computational Linguistics.
Francis Bond and Ryan Foster. 2013. Linking and extending an open multilingual Wordnet. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1352-1362, Sofia, Bulgaria. Association for Computational Linguistics.
Thorsten Brants. 2000. Tnt - A statistical part-of-speech tagger. CoRR, cs.CL/0003055.
Soravit Changpinyo, Hexiang Hu, and Fei Sha. 2018. Multi-task learning for sequence tagging: An empirical study. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2965-2977, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Marta R. Costa-jussa, Cristina Espana-Bonet, Pascale Fung, and Noah A. Smith. 2020. Multilingual and interlingual semantic representations for natural language processing: A brief introduction. Computational Linguistics, 46(2):249-255.
Marco Damonte and Shay B. Cohen. 2018. Crosslingual Abstract Meaning Representation parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1146-1155, New Orleans, Louisiana. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jack Edmonds. 1967. Optimum branchings. J. Research of the National Bureau of Standards, 71B:233-240.
Daniel Flickinger, Yi Zhang, and Valia Kordoni. 2012. Deepbank: A dynamically annotated treebank of the wall street journal. In Proceedings of the Eleventh International Workshop on Treebanks and Linguistic Theories, pages 85-96.
Zellig S Harris. 1951. Methods in structural linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9:1735-80.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Da Huo and Gerard de Melo. 2020. Inducing universal semantic tag vectors. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3121-3127, Marseille, France. European Language Resources Association.
Nancy Ide and James Pustejovsky. 2017. Handbook of Linguistic Annotation, 1st edition. Springer Publishing Company, Incorporated.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Charles N Li and Sandra A Thompson. 1989. Mandarin Chinese: A functional reference grammar, volume 3. Univ of California Press.
Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018. Analogical reasoning on Chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 138-143. Association for Computational Linguistics.
Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramón Fernández, Silvio Amir, Luís Marujo, and Tiago Luís. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1520-1530, Lisbon, Portugal. Association for Computational Linguistics.
Meichun Liu. 2015. Tense and aspect in mandarin chineshe. The Oxford Handbook of Chinese Linguistics, pages 274-289.
John Lyons. 1966. Towards a notional theory of the 'parts of speech'. Journal of linguistics, 2(2):209-236.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014a. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014b. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55-60.
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the penn treebank. Computational Linguistics, 19(2):313-330.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Bruce Mitchell. 1985. Old English Syntax: Concord, the parts of speech, and the sentence. Oxford University Press, USA.
Tasnim Mohiuddin and Shafiq Joty. 2020. Unsupervised word translation with adversarial autoencoder. Computational Linguistics, 46(2):257-288.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31:71-106.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
Waltraud Paul. 2016. Where “complex” sentences are not complex and “subordinate”. Coordination and subordination: Form and meaning—Selected papers from CSI Lisbon 2014, page 185.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the
Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Aarne Ranta, Krasimir Angelov, Normunds Gruzitis, and Prasanth Kolachina. 2020. Abstract syntax as interlingua: Scaling up the grammatical framework from controlled languages to robust pipelines. Computational Linguistics, 46(2):425-486.
Carlota S Smith and Mary S Erbaugh. 2005. Temporal interpretation in mandarin chinese. Linguistics, 43(4):713-756.
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual bert transformation for zero-shot dependency parsing. pages 5725-5731.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA.
Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016. Universal decompositional semantics on Universal Dependencies. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1713-1723, Austin, Texas. Association for Computational Linguistics.
Naiwen Xue, Fei Xia, Fu-dong Chiou, and Marta Palmer. 2005. The penn Chinese treebank: Phrase structure annotation of a large corpus. Natural Language Engineering, 11:207-238.
In the supplemental material, we present the complete tailored universal semantic tag set for Mandarin Chinese (see Table 11).
ANA anaphoric
PRO anaphoric & deictic pronouns: 他，她
DEF definite: 这个，那人
HAS possessive pronoun: 我弟弟，你学生
REF reflexive & reciprocal pron: 自己，对方
EMP emphasizing pronouns: 自己
ACT speech act
GRE greeting & parting: 你好，再见
ITJ interjections, exclamations: 啊、哎呀
HES hesitation: 额，{……}
QUE interrogative: 谁，什么，{？}
EVE events
EXG untensed progressive: 吃着、保持着
TNS tense & aspect
FUT future tense: 将，将来
PRG progressice: 在，着
DSC discourse
COO coordinate relations:  $\{\cdot ,\cdot ,\cdot ;\}$  、所以
APP appositional relations:  $\{\text{一} \} ,\{\cdot ,\}$
BUT contrast: 但是、然而
UNE unnamed entity
CON concept: 狗、人
ROL role: 学生、哥哥
GRP group: 等、张三{、}李四和王五
DXS deixis
DXP place deixis: 野外、沿着、前
DXT temporaldeixis:过去、自从、后
DXD discourse deixis: 首先、其次
LOG logical
ALT alternative & repetitions: 另、再
XCL exclusive: 只、仅仅
NIL empty semantics:  $\{\circ\}$ 、 $\{\langle\gg\rangle\}$
DIS disjunction & exist. quantif.: 或、某
IMP implication: 如果、除非、当
AND conjunction & univ. quantif.: 并且、所有
MOD modality
NOT negation: 不、没有
NEC necessity: 得、该
POS possibility: 能、可能、应该
ATT attribute
QUc concrete quantity: 二, 六百万
QUV vague quantity: 一些，几
COL color: 红，浅蓝
IST intersective: 公，大型
SST subjective: 高，热，可笑
PRI privative: 假，前，副
DEG degree: 2米高，8个月大
INT intensifier: 非常、很
SCO score: 3-0、100分
COM comparative
TOP superlative: 最
NAM named entity
PER person: 包拯、狄仁杰
GPE geo-political entity: 北京、日本
GPO geo-political origin: 华裔、东乡族
GEO geographical location:长江、尼罗河