SlowGuess/NAACL_2022_OCR huggingface dataset:
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In International Conference on Learning Representations.
Elior Sulem, Jamaal Hay, and Dan Roth. 2022. Yes, No or IDK: The challenge of unanswerable Yes/No questions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), Online and Seattle, Washington. Association for Computational Linguistics.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5784-5789, Hong Kong, China. Association for Computational Linguistics.
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57:45.
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021a. Finetuned language models are zero-shot learners.
Kaiwen Wei, Xian Sun, Zequn Zhang, Jingyuan Zhang, Guo Zhi, and Li Jin. 2021b. Trigger is not sufficient: Exploiting frame-aware knowledge for implicit event argument extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4672-4682, Online. Association for Computational Linguistics.
Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 996-1005, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3914-3923, Hong Kong, China. Association for Computational Linguistics.
Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Universal natural language processing with limited annotations: Try few-shot textual entailment as a start. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8229-8239, Online. Association for Computational Linguistics.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Computational Linguistics.
Wenxuan Zhou and Muhao Chen. 2021. An improved baseline for sentence-level relation extraction.
Hyperparameter	EM	NLI	\( \mathrm{NLI}_{\text{MNLI only}} \)
\( \mathrm{N_E} / \mathrm{N_N} / \mathrm{N_C} \)	-	2/5/5	2/5/5
Batch size	32
Learning rate	\( 1 \times 10^{-5} \)	\( 4 \times 10^{-6} \)	\( 1 \times 10^{-5} \)
Seeds	{0,24,42}
Epochs	25 (*50)
Weight decay	0.01
Table 7: Hyperparameters of the trained systems. * indicates the difference between full-train and few-shot scenarios.
On this section we describe the hyperparameters we have used on our experiments. All the hyperparameters optimized on this work were optimized for the  $100\%$  split with the batch-size fixed to 32, and used on the rest. The Table 7 describes the hyperparameters used on EM, NLI and  $\mathrm{NLI}_{\mathrm{MNLI}}$  only variants, for the  $\mathrm{NLI + }$  the same hyperparameters as NLI were used. We have found that the same exact hyperparameters were the best on ACE, WikiEvents and TACRED datasets. For the future, we plan to test new hyperparameter sets that uses bigger batch-sizes, as recent works (Aribandi et al., 2022) suggest to be optimal for multi-task and -source learning experiments.
The pre-trained NLI models used on this work can be downloaded from the HuggingFace Models repository: NLI MNLIonly (roberta-large-mnli) and NLI (ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli).
The fine-tuned models derived from this work will be uploaded to HuggingFace Models repository. Check the GitHub repository for updated information.
B Multi-task in-depth analysis
The Figure 6 shows the per role absolute improvement obtained by training on different tasks over the  $0\%$  NLI system. Overall, we can see that training on ACE or WikiEvents improves almost all the roles and training on TACRED improves some and some others do not. A result that was unexpected is that there are few roles on WikiEvents that after training on WikiEvents become worse in contrary to training on ACE. This could be explained by the differences among the frequency distributions that the train, development and test sets of WikiEvents has. Moreover, there are some roles on WikiEvents
that decreases in all training scenarios, this suggests us that sequential fine-tuning might be not the best option for this type of multi-source learning and therefore further ways should be explored.
C ACE templates from both developers
The next table contains the templates written by both developers for the ACE arguments. We follow the notation introduced in Section 5.1. In addition, we also consider information from the event, such as the type on different granularity levels, including {trg_type} for the trigger type (e.g. Movement from Movement.Transform) and {trg_subtype} for the subtype of the trigger, e.g. Transport from Movement.Transform).
Figure 6: Absolute improvements over the NLI baseline using different tasks and sources. Rows indicates the testing data and columns the training data. Each bar indicates the F1-Score difference between the trained NLI system vs  $0\%$  NLI for a specific role.
Role	Main developer	Linguist
Adjudicator	{arg} tried the defendant.
{arg} convicted the defendant.
{arg} released the defendant.
{arg} sentenced the defendant.
{arg} acquitted the defendant.	{arg} convict someone.
{arg} sentence someone.
{arg} judge someone.
{arg} fine someone.
{arg} indict someone.
Agent	{arg} {trg} a person or organization.	{arg} do something.
{arg} select something.
{arg} carry out something.
{arg} create something.
{arg} give something.
Artifact	Someone {trg} the {arg}.
Someone moved {arg}.
Someone bought {arg}.
Someone sold {arg}.	{arg} be an object.
{arg} be a weapon.
Attacker	{arg} {trg} a person or organization.	{arg} assault someone.
{arg} aggress someone.
{arg} assault someone.
Beneficiary	The buyer bought to {arg} something.	{arg} get something.
{arg} be beneficiary.
{arg} benefit from something.
{arg} obtain something.
Buyer	{arg} bought something.	{arg} buy something.
{arg} possess something.
{arg} own something.
Defendant	{arg} was the defendant.	{arg} be accused of something.
{arg} be accused of a crime.
{arg} be judged.
Destination	Someone {trg_subtype} to {arg}.	{trg_type} go to {arg}.
{trg_type} finish in {arg}.
{trg_type} move to {arg}.
{arg} be a place.
{arg} be a location.
Entity	{arg} attended the demonstration.
{arg} met someone.
{arg} fired someone.
{arg} voted in the elections.
{arg} released the defendant.
{arg} was ordered to pay.	{arg} select something.
{arg} carry out something.
{arg} do something.
{arg} create something.
{arg} give something.
Giver	{arg} gave something to someone.	{arg} give something.
Instrument	Someone {trg_subtype} with {arg}.	{arg} be artifact.
{arg} be object.
{arg} be device.
{arg} cause harm.
Org {arg} organization declared bankruptcy. {arg} be in bankruptcy.
(continued on the next page)
Role	Main developer	Linguist
{arg} organization was dissolved.
{arg} organization was merged.
{arg} organization was launched.	{arg} be ended.
{arg} be merged.
{arg} be created.
{arg} be company.
{arg} be organization.
Origin	Someone {trg_subtype} from {arg}.	{arg} change location.
{arg} be location.
{trg_type} start in {arg}.
{trg_type} move from {arg}.
Person	{arg} was {trg}.	{arg} be person.
{arg} be living entity.
{arg} be born.
{arg} get married.
{arg} be married.
{arg} divorce.
{arg} 's marriage ended.
{arg} be hired.
{arg} start a job.
{arg} be fired.
{arg} end a job.
{arg} be nominated.
{arg} be elected.
{arg} be arrested.
{arg} be jailed.
{arg} be imprisoned.
{arg} be released.
{arg} be parole.
{arg} be executed.
{arg} be extradited.
Place	{trg} occurred in {arg}.	{arg} be a place.
{arg} be a location.
{arg} be a placement.
Plaintiff	{arg} filed suit against someone.	{arg} bring a lawsuit against someone.
{arg} bring a lawsuit against something.
{arg} sue someone.
{arg} sue something.
Prosecutor	{arg} indicted the defendant.
{arg} charged the defendant.	{arg} prosecute.
{arg} take somebody to court for a crime.
Recipient	{arg} received money from someone.	{arg} receive something.
{arg} get something.
{arg} get money.
Seller	{arg} sold something.	{arg} sell something.
Target	{arg} was {trg_subtype}.	{arg} be attacked.
{trg_type} 's target be {arg}.
Vehicle	{arg} was used as a vehicle.
(continued on the next page)	{arg} be a transport.
{arg} be a vehicle.
{arg} serve to move.
{arg} serve to change location.
{arg} serves as a means of transportation.
Victim	{arg} was {trg}.	{arg} be victim.
{arg} be injured.
{arg} be killed.
{arg} be harmed.
{arg} have a dead.
{arg} have a tragedy.
The templates written by both developers for ACE.