SlowGuess/NAACL_2022_OCR huggingface dataset:
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT 2016, pages 260-270.
Lukas Lange, Anastasiia Iurshina, Heike Adel, and Jannik Strötgen. 2020. Adversarial alignment of multilingual models for extracting temporal expressions from text. In Proceedings of Workshop on Representation Learning for NLP at ACL 2020, pages 103-109.
Egoitz Laparra, Dongfang Xu, and Steven Bethard. 2018. From characters to time intervals: New paradigms for evaluation and neural parsing of time normalizations. Transactions of the Association for Computational Linguistics, 6:343-356.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. In Proceedings of ACL 2019, pages 4487-4496.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of ICLR 2019.
Pawel Mazur and Robert Dale. 2010. Wikiwars: A new corpus for research on temporal expressions. In Proceedings of EMNLP 2010, pages 913-922.
Bhaskar Mitra, Nick Craswell, et al. 2018. An introduction to neural information retrieval. Now Foundations and Trends.
James Pustejovsky, Kiyong Lee, Harry Bunt, and Laurent Romary. 2010. Iso-timeml: An international standard for semantic annotation. In LREC, volume 10, pages 394-397.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Lance A Ramshaw and Mitchell P Marcus. 1999. Text chunking using transformation-based learning. In Natural language processing using very large corpora, pages 157-176. Springer.
Jannik Strötgen and Michael Gertz. 2013. Multilingual and cross-domain temporal tagging. *Language Resources and Evaluation*, 47(2):269-298.
Jannik Strötgen and Michael Gertz. 2015. A baseline temporal tagger for all languages. In Proceedings of EMNLP 2015, pages 541-547.
Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics, Volume 2: Proceedings of SemEval 2013, pages 1-9.
Sen Wu, Hongyang R Zhang, and Christopher Ré. 2020. Understanding and improving information transfer in multi-task learning. In Proceedings of ICLR 2020.
Xiaoshi Zhong, Aixin Sun, and Erik Cambria. 2017. Time expression analysis and recognition using syntactic token types and general heuristic rules. In Proceedings of ACL 2017, pages 420-429.
A Types of the Temporal Expressions
According to ISO-TimeML (Pustejovsky et al., 2010), the TEE dataset annotation guideline, there are four types of temporal expressions, i.e., Date, Time, Duration, and Set. Date refers to a calendar date, generally of a day or a larger temporal unit; Time refers to a time of the day and the granularity of which is smaller than a day; Duration refers to the expressions that explicitly describe some period of time; Set refers to a set of regularly recurring times (Pustejovsky et al., 2010).
B The Training Procedure
We adopt mini-batch-based stochastic gradient descent (SGD) to train XLTime, as shown in Algorithm 1. To concurrently train on the primary and secondary tasks, we split the training data of both tasks into mini-batches and randomly take one mini-batch at each step. We then calculate loss using that mini-batch and update the parameters of the shared backbone as well as the task-type-specific classifier. The classifier of the other task type is unaffected.
C Full Results for Low-resource Language TEE
Table 7 shows the full results for low-resource language TEE with/without considering the types of the temporal expressions. Note that the superiority of our proposed XLTime over the previous automatic SOTA still holds.
D Full Results for Low-resource Language TEE with Additional Source Languages
Tables 8 and 9 show the full results for low-resource language TEE with additional source languages.
E Comparative Error Analysis
This section qualitatively shows how the proposed XLTime framework transfers knowledge to the target language. Specifically, we show how the errors made by the vanilla multilingual models can be fixed by applying XLTime. We also show how applying XLTime on other languages in addition to English would help fix more errors.
We compare mBERT and XLTime-mBERT (transfer from EN) on FR TEE. Table 5 summarizes cases where mBERT fails while XLTime-mBERT gives correct predictions. We can tell that XLTime-mBERT learns 'hier (yesterday)', which is not understood by the mBERT model. XLTime-mBERT also learns to recognize vague time spans such as 'désomais (from now on)' and 'longtemps (long time)', which are missed by the mBERT model. Moreover, compared to mBERT, XLTime-mBERT understands FR grammar better, as it recognizes the roles of definite articles and adjectives, such as 'le (the)' and 'prochain (next)', in TEs. In a word, the proposed XLTime framework helps connect the concepts in EN to the corresponding ones in FR.
To show how applying XLTime on extra source languages would help fix more errors, we compare XLTime-mBERT (transfer from EN) and XLTime-mBERT (transfer from EN and ES) on FR TEE. Table 6 summarizes the TEs that the former fails while the latter gives correct predictions. We can tell that by leveraging ES as an additional source language, XLTime-mBERT better masters FR grammar. Specifically, it learns to recognize definite articles and prepositions that share similar (e.g., 'le/los') or identical (e.g., 'de' and 'en') forms in ES and FR. It can also better distinguish TEs of different types (e.g., it learns that 'quelques jours
Table 5: mBERT vs. XLTime-mBERT (transfer from EN) frequent (count  $\geq 10$ ) errors.
Error Desc.	FR TEs	EN translations	mBERT results (wrong)	XLTIme results (correct)	counts
fail to recognize ‘hier (yesterday)’	hier soit hier	last night yesterday	O B-TIME O	B-TIME I-TIME B-DATE	30
fail to recognize vague time span	désomais longtemps toute l'année	from now on long time all year	O O O	B-DATE B-DURATION B-SET I-SET	6
fail to recognize definite articles and adjectives	le 3 août la nuitundi prochain	August 3 the night next Monday	O B-DATE I-DATE O O B-DATE O	B-DATE I-DATE I-DATE B-TIME I-TIME B-DATE I-DATE	10
Table 6: XLTime-mBERT (transfer from EN) vs. XLTime-mBERT (transfer from EN and ES) frequent (count ≥ 8) errors.
Error Desc.	FR TEs	EN translations	EN results (wrong)	EN and ES results (correct)	counts
fail to recognize definite articles and prepositions	en été le 13 février de dimanche	in summer February 13 of Sunday	B-DATE I-DATE O B-DATE B-DATE I-DATE B-DATE I-DATE	O B-DATE B-DATE I-DATE O B-DATE	20
wrong token types	mardi quelques jours	Tuesday A few days	B-TIME B-DATE I-DATE	B-DATE B-DURATION I-DURATION	18
recognized extra TEs	quotidiens la saision	daily the season	B-SET B-DATE I-DATE	O O	8
(a few days) is a Duration, instead of a Date). One interesting fact is, when transferring solely from EN, the model recognizes some extra TEs that are not in the ground truth of the FR dataset. This is because of an inconsistency in data labeling: 'daily' is considered as a Set in the EN dataset, while its counterpart, 'quotidiens' is overlooked in the FR dataset. The proposed XLTime framework eliminates the needs of manually labeling multiple datasets and therefore, can be applied to minimize data label inconsistency.
F Language Models on English TEE
In our early experiments, we reexamine the language models on English TEE. This section presents the results.
F.1 Experimental Setup
We study BERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020) variants, RoBERTa (Liu et al., 2019b) and T5 Encoder (Raffel et al., 2019). We compare them to rule-based methods including HeidelTime (Strötgen and Gertz, 2013), SynTime (Zhong et al., 2017), and PTime (Ding et al., 2019), which report SOTA performances on Wikiwars, TE3, and Tweets, respectively. We experiment on both settings, i.e., "with type" and "without type", and report F1, precision, and recall in strict match (UzZaman et al., 2013). We use the data splits following Ding et al. (2019) and the experimental
settings introduced in Section 4.1.
F.2 Evaluation Results
Tables 10, 11, and 12 show the results. We observe: 1) When ignoring the types, the language models are inferior to SynTime on TE3, on par with or better than the rule-based methods on Wikiwars and Tweets. 2) When considering the types, the language models outperform the previous SOTAs by  $11 - 22\%$ ,  $18 - 21\%$ , and  $30 - 41\%$  in F1 on TE3, Wikiwars, and Tweets datasets.
Table 7: Multilingual TEE results (w/ type | w/o type).
w/ type	FR	ES	PT	EU
F1	Pr.	Re.	F1	Pr.	Re.	F1	Pr.	Re.	F1	Pr.	Re.
Automatic Baseline Models
HeidelTime-auto	0.53	0.63	0.46	0.41	0.56	0.32	0.49	0.66	0.39	0.15	0.60	0.09
BiLSTM+CRF	0.58	0.64	0.51	0.56	0.61	0.51	0.58	0.59	0.58	0.44	0.54	0.37
mBERT	0.56	0.61	0.51	0.56	0.62	0.51	0.60	0.56	0.64	0.59	0.64	0.55
XLMR-base	0.64	0.69	0.59	0.51	0.58	0.46	0.59	0.59	0.59	0.43	0.60	0.34
XLMR-large	0.69	0.70	0.68	0.68	0.71	0.66	0.71	0.69	0.73	0.66	0.70	0.63
Projection Method
TMP-mBERT	0.50	0.56	0.45	0.23	0.59	0.14	0.60	0.57	0.64	/	/	/
TMP-XLMRbase	0.50	0.60	0.43	0.23	0.57	0.14	0.61	0.58	0.64	/	/	/
TMP-XLMRlarge	0.52	0.61	0.46	0.24	0.59	0.15	0.61	0.58	0.63	/	/	/
Transfer from EN (Ours)
XLTTime-mBERT	0.62	0.62	0.62	0.65	0.70	0.61	0.61	0.58	0.66	0.68	0.72	0.65
XLTTime-XLMRbase	0.67	0.67	0.68	0.60	0.63	0.58	0.64	0.62	0.66	0.64	0.68	0.60
XLTTime-XLMRlarge	0.71	0.74	0.68	0.70	0.76	0.65	0.74	0.71	0.78	0.72	0.79	0.66
Transfer from EN and others (Ours)
XLTTime-mBERT	0.71	0.69	0.73	0.68	0.69	0.66	0.73	0.70	0.76	0.68	0.72	0.65
XLTTime-XLMRbase	0.70	0.67	0.74	0.65	0.69	0.62	0.66	0.64	0.68	0.70	0.76	0.65
XLTTime-XLMRlarge	0.75	0.72	0.78	0.70	0.76	0.65	0.81	0.79	0.84	0.74	0.79	0.69
Handcrafted Method
HeidelTime	0.80	0.81	0.79	0.85	0.90	0.80	0.57	0.60	0.53	/	/	/
w/o type	FR	ES	PT	EU
F1	Pr.	Re.	F1	Pr.	Re.	F1	Pr.	Re.	F1	Pr.	Re.
Automatic Baseline Models
HeidelTime-auto	0.55	0.65	0.47	0.42	0.58	0.33	0.50	0.67	0.39	0.17	0.66	0.10
BiLSTM+CRF	0.64	0.73	0.57	0.62	0.68	0.56	0.64	0.66	0.63	0.47	0.58	0.40
mBERT	0.63	0.70	0.58	0.62	0.69	0.56	0.66	0.63	0.69	0.65	0.71	0.60
XLMR-base	0.69	0.75	0.64	0.54	0.61	0.48	0.63	0.64	0.62	0.46	0.64	0.36
XLMR-large	0.75	0.78	0.73	0.72	0.75	0.69	0.75	0.74	0.76	0.70	0.74	0.67
Projection Method
TMP-mBERT	0.56	0.63	0.50	0.23	0.59	0.14	0.66	0.64	0.69	/	/	/
TMP-XLMRbase	0.55	0.67	0.47	0.23	0.57	0.14	0.64	0.61	0.67	/	/	/
TMP-XLMRlarge	0.56	0.66	0.50	0.24	0.59	0.15	0.65	0.61	0.68	/	/	/
Transfer from EN (Ours)
XLTTime-mBERT	0.73	0.73	0.72	0.71	0.77	0.66	0.67	0.64	0.71	0.76	0.81	0.71
XLTTime-XLMRbase	0.78	0.79	0.78	0.66	0.70	0.63	0.68	0.67	0.70	0.71	0.76	0.66
XLTTime-XLMRlarge	0.76	0.79	0.73	0.72	0.79	0.67	0.77	0.74	0.81	0.78	0.85	0.71
Transfer from EN and others (Ours)
XLTTime-mBERT	0.80	0.77	0.82	0.77	0.79	0.74	0.80	0.77	0.83	0.77	0.82	0.72
XLTTime-XLMRbase	0.82	0.79	0.86	0.72	0.78	0.68	0.73	0.72	0.75	0.79	0.86	0.73
XLTTime-XLMRlarge	0.84	0.82	0.86	0.75	0.79	0.71	0.84	0.82	0.87	0.79	0.84	0.74
Handcrafted Method
HeidelTime	0.86	0.87	0.85	0.86	0.91	0.81	0.60	0.64	0.57	/	/	/
Table 8: Low-resource language TEE with additional source languages (F1, precision, and recall scores w/ type). The blue cells are expected to, while the underlined cells actually outperform (by  $\geq 3\%$  ) using EN as the only source language.
F1
Target Language	FR	ES
Source Language(s)	EN	EN, EU	EN, PT	EN, ES	EN	EN, EU	EN, PT	EN, FR
XLTTime-mBERT	0.62	0.61	0.61	0.71	0.65	0.66	0.65	0.68
XLTTime-XLMRbase	0.67	0.67	0.66	0.70	0.60	0.61	0.64	0.65
XLTTime-XLMRlarge	0.71	0.73	0.73	0.75	0.70	0.68	0.69	0.68
Target Language	PT	EU
Source Language(s)	EN	EN, FR	EN, ES	EN, EU	EN	EN, PT	EN, ES	EN, FR
XLTTime-mBERT	0.61	0.72	0.59	0.73	0.68	0.66	0.66	0.68
XLTTime-XLMRbase	0.64	0.66	0.55	0.52	0.64	0.66	0.66	0.70
XLTTime-XLMRlarge	0.74	0.79	0.81	0.71	0.72	0.71	0.74	0.72
Precision
Target Language	FR	ES
Source Language(s)	EN	EN, EU	EN, PT	EN, ES	EN	EN, EU	EN, PT	EN, FR
XLTTime-mBERT	0.62	0.59	0.62	0.69	0.70	0.69	0.71	0.69
XLTTime-XLMRbase	0.67	0.66	0.67	0.67	0.63	0.64	0.67	0.69
XLTTime-XLMRlarge	0.74	0.72	0.76	0.72	0.76	0.65	0.73	0.68
Target Language	PT	EU
Source Language(s)	EN	EN, FR	EN, ES	EN, EU	EN	EN, PT	EN, ES	EN, FR
XLTTime-mBERT	0.58	0.68	0.56	0.70	0.72	0.70	0.69	0.72
XLTTime-XLMRbase	0.62	0.64	0.51	0.49	0.68	0.73	0.69	0.76
XLTTime-XLMRlarge	0.71	0.75	0.79	0.68	0.79	0.75	0.79	0.79
Recall
Target Language	FR	ES
Source Language(s)	EN	EN, EU	EN, PT	EN, ES	EN	EN, EU	EN, PT	EN, FR
XLTTime-mBERT	0.62	0.62	0.59	0.73	0.61	0.64	0.60	0.66
XLTTime-XLMRbase	0.68	0.67	0.64	0.74	0.58	0.59	0.61	0.62
XLTTime-XLMRlarge	0.68	0.73	0.71	0.78	0.65	0.71	0.65	0.67
Target Language	PT	EU
Source Language(s)	EN	EN, FR	EN, ES	EN, EU	EN	EN, PT	EN, ES	EN, FR
XLTTime-mBERT	0.66	0.75	0.62	0.76	0.65	0.63	0.64	0.64
XLTTime-XLMRbase	0.66	0.68	0.60	0.55	0.60	0.60	0.63	0.65
XLTTime-XLMRlarge	0.78	0.83	0.84	0.74	0.66	0.67	0.69	0.67
Table 9: Low-resource language TEE with additional source languages (precision and recall scores w/o type). The blue cells are expected to, while the underlined cells actually outperform (by  $\geq 4\%$  ) using EN as the only source language.
Precision
Target Language	FR	ES
Source Language(s)	EN	EN, EU	EN, PT	EN, ES	EN	EN, EU	EN, PT	EN, FR
XLTTime-mBERT	0.73	0.76	0.76	0.77	0.77	0.76	0.79	0.79
XLTTime-XLMRbase	0.79	0.77	0.81	0.79	0.70	0.72	0.75	0.78
XLTTime-XLMRlarge	0.79	0.81	0.84	0.82	0.79	0.70	0.79	0.74
Target Language	PT	EU
Source Language(s)	EN	EN, FR	EN, ES	EN, EU	EN	EN, PT	EN, ES	EN, FR
XLTTime-mBERT	0.64	0.77	0.67	0.77	0.81	0.78	0.79	0.82
XLTTime-XLMRbase	0.67	0.72	0.60	0.54	0.76	0.82	0.79	0.86
XLTTime-XLMRlarge	0.74	0.79	0.82	0.72	0.85	0.85	0.84	0.84
Recall
Target Language	FR	ES
Source Language(s)	EN	EN, EU	EN, PT	EN, ES	EN	EN, EU	EN, PT	EN, FR
XLTTime-mBERT	0.72	0.77	0.69	0.82	0.66	0.69	0.66	0.74
XLTTime-XLMRbase	0.78	0.76	0.75	0.86	0.63	0.64	0.68	0.68
XLTTime-XLMRlarge	0.73	0.81	0.77	0.86	0.67	0.75	0.71	0.72
Target Language	PT	EU
Source Language(s)	EN	EN, FR	EN, ES	EN, EU	EN	EN, PT	EN, ES	EN, FR
XLTTime-mBERT	0.71	0.83	0.74	0.83	0.71	0.69	0.70	0.72
XLTTime-XLMRbase	0.70	0.75	0.66	0.59	0.66	0.67	0.70	0.73
XLTTime-XLMRlarge	0.81	0.87	0.87	0.77	0.71	0.74	0.74	0.71
Table 10: Supervised English TEE on TE3 (w/ type | w/o type).
F1	Pr.	Re.
Rule-based Models
HeidelTime	0.77| 0.81	0.80| 0.84	0.75| 0.79
SynTime	0.65| 0.92	0.65| 0.91	0.66| 0.93
PTime	0.67| 0.85	0.68| 0.88	0.65| 0.83
Language Models
BERT-base	0.76| 0.82	0.78| 0.85	0.74| 0.80
BERT-large	0.79| 0.83	0.77| 0.82	0.80| 0.84
mBERT	0.79| 0.84	0.80| 0.86	0.77| 0.82
RoBERTa	0.78| 0.84	0.79| 0.86	0.77| 0.82
XLMR-base	0.79| 0.81	0.80| 0.82	0.77| 0.81
XLMR-large	0.78| 0.81	0.78| 0.82	0.78| 0.81
T5Encoder	0.79| 0.82	0.82| 0.85	0.78| 0.80
Table 11: Supervised English TEE on Wikiparts (w/ type | w/o type).
F1	Pr.	Re.
Rule-based Models
HeidelTime	0.80| 0.85	0.86| 0.92	0.75| 0.80
SynTime	0.79| 0.79	0.79| 0.79	0.79| 0.79
PTime	0.86| 0.86	0.87| 0.87	0.86| 0.86
Language Models
BERT-base	0.94| 0.94	0.95| 0.95	0.94| 0.94
BERT-large	0.95| 0.95	0.94| 0.94	0.96| 0.96
mBERT	0.97| 0.97	0.96| 0.96	0.97| 0.97
RoBERTa	0.95| 0.95	0.94| 0.94	0.97| 0.97
XLMR-base	0.97| 0.97	0.95| 0.95	0.98| 0.98
XLMR-large	0.96| 0.96	0.94| 0.94	0.97| 0.97
T5Encoder	0.96| 0.96	0.95| 0.95	0.97| 0.97
Table 12: Supervised English TEE on Tweets (w/ type | w/o type).
F1	Pr.	Re.
Rule-based Models
HeidelTime	0.80| 0.80	0.90| 0.90	0.72| 0.72
SynTime	0.63| 0.92	0.62| 0.91	0.65| 0.95
PTime	0.66| 0.95	0.65| 0.94	0.67| 0.96
Language Models
BERT-base	0.92| 0.94	0.90| 0.93	0.93| 0.95
BERT-large	0.86| 0.92	0.84| 0.92	0.88| 0.92
mBERT	0.87| 0.91	0.85| 0.88	0.90| 0.94
RoBERTa	0.91| 0.95	0.89| 0.93	0.94| 0.97
XLMR-base	0.90| 0.94	0.87| 0.92	0.93| 0.97
XLMR-large	0.93| 0.95	0.91| 0.93	0.95| 0.96
T5Encoder	0.87| 0.93	0.84| 0.91	0.91| 0.95