SlowGuess/NAACL_2022_OCR huggingface dataset:
Knowledge-based semantic embedding for machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2245-2254.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funpowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201-206.
Hongming Zhang, Haoyu Wang, and Dan Roth. 2020a. Unsupervised label-aware event trigger and argument classification. CoRR, abs/2012.15243.
Hongming Zhang, Haoyu Wang, and Dan Roth. 2021. Zero-shot Label-aware Event Trigger and Argument Classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1331-1340, Online. Association for Computational Linguistics.
Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020b. Intent detection with WikiHow. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 328-333, Suzhou, China. Association for Computational Linguistics.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuaijiang Zhang, Xi Zhou, and Xiang Zhou. 2020c. Semantics-aware bert for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9628-9635.
A Modeling Details
All our models are implemented upon the Hugging-Face Transformers library, which provides streamlined model sharing functions. We will upload our models as "model cards" upon paper acceptance.
Both our predicate sense disambiguation (PSD) model and our argument classification (AC) model are based on RoBERTa-base with default RoBERTa tokenizers. Our PSD model is implemented as a RobertaForSequenceClassification object<sup>9</sup>, while
$^{9}$ https://huggingface.co/ transformers/model_doc/roberta.html#robertaforsquenceclassification
our AC and ACD models are implemented as a RobertaForTokenClassification object<sup>10</sup>. Details of configurations and hyperparameters can be found in their documentations.
We run all experiments on NVIDIA V100 GPUs with 16G memory. During training, we save the model with the best argument F1 on the development set each 100 training steps. We run each experiment (e.g. training ACD on CoNLL09) for up to 48 hours, or when the best model is not updated for 5,000 training steps, whichever is sooner, before evaluating the best model on the test set.
B Qualitative Example
While we have not found convincing patterns of examples that benefit from ACD, we showcase one such example with unconventional syntax.
In the CoNLL09 out-domain test set, one sentence (abridged) is
Any good decorator these days can [make] you a tasteful home.
with the word "make" as the predicate. Here, "make" has sense create, and thus "you" serves as an indirect object in a colloquial use, and the sentence is roughly equivalent to "...can make a tasteful home for you." This use case can easily be confused with the other sense of "make": cause to be (e.g. He makes me do all the work).
Predicate "make" with the correct sense "create" has 4 arguments from the frame file:
1. A0 creator, annotated as "decorator";
2. A1 creation, annotated as "home";
3. A2 created from, annotated as none;
4. A3 benefactive, annotated as "you".
With these definitions, the ACD model with gold sense correctly predicts all arguments, except missing A3. In contrast, given an incorrectly predicted of sense "cause to be" with the arguments from the frame file, the ACD model predicts:
1. A0 impeller to action correctly as "decorator";
2. A1 impelled agent incorrectly as "you";
3. A2 impelled action incorrectly as "home";
4. A3 , which is non-existent, incorrectly.
Identically, the AC model correctly predicts A0, which in most cases is the subject of the sentence, without much surprise. However, it incorrectly predicts A1 as "you" and A2 as "home". This example qualitatively provides evidence that with definitions of arguments for the correct predicate sense, the model is better at performing SRL on underrepresented or complex examples.
C Other Formats of Injecting Definitions
We demonstrate previously that we append the definitions to the end of a sentence, maintaining the token classification format. We have also attempted other formats which empirically perform worse. For example, the sentence
He [drills] three holes into the wall.
with the predicate "drill" with arguments A0 driller as "He" and A1 thing drilled, gaining holes as "wall", can be converted to the following formats.
Question answering. Similar to Du and Cardie (2020), in a classical SQuAD-style (Rajpurkar et al., 2016) format, the passage is the sentence. There are two questions: "What is the driller for 'drill'? with answer "He", and "What is the thing drilled, gaining holes for 'drill'? with answer "wall". In our experiments, we tried a variety of models such as RoBERTa, XLNet, etc. and were not able to have any of these converge on the training set.
Sentence completion. Similar to Schick and Schütze (2021), the input sentence with masked tokens is
In the sentence "He drills three holes into the wall," the driller for "drill" is...
with the answer "He". The example for "thing drilled" is omitted. In our experiments, we tried a variety of models such as RoBERTa, XLNet, etc. and were not able to have any of these converge on the training set.
Prompting. Similar to Brown et al. (2020), we were also able to convert our examples to a task description and some examples, using the two formats above, as input to models such as GPT-3. While this maneuver has potential, we do not access to the closed beta of GPT-3, and were not able to perform the experiments.
10https://huggingface.co/ transformers/model_doc/roberta.html#robertafortokenclassification
Percentile	N/A	10%	20%	30%	40%	50%	60%	70%	80%	90%	100%
% examples	1.1%	2.9%	3.2%	3.5%	3.8%	4.0%	5.2%	6.9%	9.2%	14.4%	45.7%
AC	77.6	82.5	82.6	86.2	87.0	87.5	90.4	89.2	88.9	90.6	91.2
ACD (gold)	82.0	86.5	85.0	86.9	87.3	89.9	90.1	89.2	88.8	89.7	90.9
ACD (pred)	80.8	85.0	83.5	86.1	87.0	89.7	89.8	89.1	88.5	89.5	90.8
Table 6: Argument F1 on subsets of CoNLL09 in-domain test set, bucketed by the percentile of predicate sense frequency in the training set. The "N/A" column refers to test examples with predicates absent in the training set.
Num. training examples	102	102.5	103	103.5	104	104.5	105
AC (in) mean	43.64	60.9	68.32	77.1	82.36	86.9	89.46
AC (in) SE	0.721526	0.258844	0.397995	0.246982	0.169115	0.070711	0.08124
ACD (in) mean	46.92	61.94	71.48	79.12	83.62	86.94	89.14
ACD (in) SE	0.921629	0.894763	0.349857	0.185472	0.115758	0.06	0.11225
AC (out) mean	42.82	58.34	64.2	71.78	75.98	80.08	82.08
AC (out) SE	0.938296	0.465403	0.564801	0.295635	0.558032	0.424735	0.243721
ACD (out) mean	45.82	60.04	67.28	74.94	78.44	81.58	82.54
ACD (out) SE	0.686586	0.612862	0.431741	0.304302	0.314006	0.341174	0.261916
Table 7: Mean and standard error of argument F1 over 5 runs of AC and ACD models trained on varying amount of randomly sampled examples, reported on CoNLL09 in- and out-domain test set.
D Multilingual Settings
We have also attempted leveraging argument definitions in non-English languages. Among the 6 languages present in CoNLL09, the frame files across them are structured very differently. We process those for Chinese whose frame files are formatted similar to those for English. Using a multilingual cased BERT (Devlin et al., 2019), we train AC and ACD models in the same fashion as English, and find that ACD underperforms AC on CoNLL09. Upon inspection, we find that argument labels for the Chinese frames are more terse and uninformative. For example, the definition "agent" and "entity" occupy more than  $50\%$  of all definition occurrences, corresponding to A0 and A1 most of the time. We hypothesize that these homogeneous definitions renders ACD performance lackluster.
We have also attempted a cross-lingual few-shot transfer setting, where a model is trained on the English training data with or without definition, and then continues to be trained on the Chinese training data without definition, before it is evaluated on the Chinese test set. We find that ACD "pre-training" also underperforms the AC counterpart.
E Risks and Biases
The potential risks and Biases of our work are minimal. Since we leverage the CoNLL09 the ProbBank datasets, though unlikely to exist, unsafe and unfair texts or those containing person-identifying information in these human-curated datasets may
propagate to the use of models trained on them.
F Licenses of Datasets Used
CoNLL09's licensing information cannot be found. PropBank is licensed under CC BY-SA 4.0.
The domain-specific PropBanks by IBM are licensed under CDLA-Sharing-1.0.