SlowGuess/NAACL_2022_OCR huggingface dataset:
Fig. 7 shows three examples of TD scores from POLIAFF (the first) and YELPCLS (the latter two). These illustrate cases with and without temporal sensitivity. In practice, most examples with deterioration showed a linear trend and thus the rate of degradation was suitable to be approximated by a line. The final TD score is averaged over all evaluation years  $\mathcal{T}'$ .
$$
\mathrm {T D} = \frac {\sum_ {t \in \mathcal {T} ^ {\prime}} \mathrm {T D} (\mathcal {T} \rightarrow t)}{| \mathcal {T} ^ {\prime} |}
$$
B Details of Model Development
Training Details for Temporal Adaptation We train GPT2 over each domain and timestamp for  $k$  steps using Huggingface's implementation of GPT2. Hyperparameter details can be seen in Table 4.
Hyperparameter	DAPT Assignment
Number of steps	10k
Batch size	32
Maximum learning rate	5e-05
Adam Epsilon	1e-08
Adam Beta	0.9. 0.999
Block size	1024
Table 4: Hyperparameters for temporal adaptation across the four domains.
Hyperparameter	Cls. Assign	Summ. Assign
Number of Epochs	50	10
Batch size	32	8
Max learning rate	2e-05	2e-05
Adam Epsilon	1e-08	1e-08
Adam Beta	0.9. 0.999	0.9. 0.999
top p (sampling)	-	0.05
top k	-	20
temperature	-	1
max length	-	512
Table 5: Hyperparameters for temporal finetuning across the eight tasks.
Training Details for Temporal Finetuning We use Huggingface's implementation of GPT2 for finetuning for both the classification and summarization tasks. We train on Quadro RTX 800 GPUs. See Table 5 for details.
We describe the postprocessing and data collection in greater detail. All data released is intended for non-commercial use.
POLIAFF We acquire a list of U.S. politician names and Twitter handles. $^{14}$  One of the authors manually annotated whether each politician was a Republican or Democrat. In addition, one volunteer double checked to ensure correctness. We discard any politician who changed parties between 2015 and 2020, any independents, and anyone suspended by Twitter (e.g., RealDonaldTrump).
$^{14}$ https://files.pushshift.io/twitter/US_PoliticalTweets.tar.gz
13See examples in Fig. 4.
Figure 7: Three example calculations of the TD score (left from POLIAFF and the center and right from YELPCLS). The annotated numbers are the raw evaluation scores  $S_{t' \to t}$  and the plotted markers represent the modified differences  $D(t' \to t)$  discussed in Section 2.3. For a particular plot, the red line is the line of best fit and its slope is the TD(t) score for evaluation timestep  $t$ . The final TD score is averaged between all evaluation timesteps for the particular task.
AIC We randomly sample science documents in Semantic Scholar's corpus. $^{15}$  Of those, we only keep documents that (1) are published in ICML or AAAI, (2) are classified as 'computer science' documents, and (3) have an abstract of at least 50 tokens.
Newsroom The following applies to the postprocessing and data selection for both supervised temporal finetuning and unsupervised temporal adaptation of PUBCLS and NEWSUM. We use the Newsroom dataset. $^{16}$  We only keep articles where (1) the year in the metadata also appears in the main text and (2) no future year is mentioned in the main text.
PUBCLS We carry out additional postprocessing and ensure that each of the three labels (Fox News, New York Times, and Washington Post) have an equal distribution across years. We do so by uniform-random downsampling.
We provide further results from our experiments described in Section 3.
Domain	Task (metric)	Pearson's r
Twitter	POLIAFF (F1)	0.84
TWIERC (F1)	0.51
Science	SCIERC (F1)	0.72
AIC (F1)	0.79
News	PUBCLS (F1)	0.65
NEWSUM (Rouge-L)	0.72
MFC (F1)	0.80
Reviews	YELPCLS (F1)	0.14
Table 6: Pearson  $r$  correlation coeffecients between the word overlap and performance of each task.
Word Overlap Correlation with Performance In addition to measuring vocabularies' change over time in Section 3.2, we find correlations between the word overlap and model performance of each task in Table 6.
Finetuning Results We provide the full results from our fientuning experiments in Section 3.1 in Fig. 8. These results are for downstream tasks with no domain adaptation.
Finetuning with Temporal Domain Adaptation We provide the full results from our finetuning with temporal domain adaptation in Section 3.2 in Fig. 7.
$^{15}$  https://api(semanticscholar.org/corpus/; licensed under an ODC-BY
16 https://lil.nlp.cornell.edu/newsroom/
Train Year
Figure 8: Temporal misalignment in finetuning affects task performance (§3.1). In all cases, higher scores are better. The heatmap is shaded per column, i.e., the darkest shade of orange in a cell means the cell has the highest score in that column. Mismatch between the training and evaluation data can result in massive performance drop; degree varies by task. For example, YELPCLS, MFC, and TwIERC show minimal degradation. In contrast, POLIAFF and NEWSUM major deterioration over time.
Domain (Task)	Finetune Year	Evaluation → Pretrain ↓	2015	2020	Domain (Task)	Finetune Year	Evaluation → Pretrain ↓	2014	2019
Twitter (PoliAft) F1	2015	Default	91.4	48.4	Twitter (TwiERC) F1	2014	Default	74.3	68.9
Default → 2015	92.2	47.5	Default → 2014	76.1	69.6
Default → 2020	90.9	50.8	Default → 2019	74.1	68.9
2020	Default	45.8	78.0	2019	Default	71.0	74.6
Default → 2015	47.2	76.9	Default → 2014	73.1	75.2
Default → 2020	44.2	78.3	Default → 2019	73.7	75.8
Domain (Task)	Finetune Year	Evaluation → Pretrain ↓	2009-11	2018-20	Domain (Task)	Finetune Year	Evaluation → Pretrain ↓	1980-1999	2010-2016
Scientific (AIC) F1	2009-2011	Default	79.0	72.0	Scientific (SciERC) F1	1980-1999	Default	67.9	57.2
Default → 2009-2011	94.5	68.8	Default → 1980-1999	73.2	66.4
Default → 2018-2020	88.4	86.0	Default → 2010-2016	73.7	66.8
2018-2020	Default	72.0	85.0	2010-2016	Default	60.3	72.5
Default → 2009-2011	87.2	65.2	Default → 1980-1999	63.4	75.0
Default → 2018-2020	86.8	79.4	Default → 2010-2016	64.8	76.0
Domain (Task)	Finetune Year	Evaluation → Pretrain ↓	2009-2010	2015-2016	Domain (Task)	Finetune Year	Evaluation → Pretrain ↓	2009-2010	2015-2016
News (MFC) F1	2009-2010	Default	27.0	26.0	News (PubCls) F1	2009-2010	Default	94.1	52.4
Default → 2009-2010	30.6	31.8	Default → 2009-2010	95.4	54.0
Default → 2015-2016	29.8	30.0	Default → 2015-2016	95.4	53.5
2015-2016	Default	23.8	33.4	2015-2016	Default	71.3	88.2
Default → 2009-2010	29.7	41.6	Default → 2009-2010	80.4	90.7
Default → 2015-2016	32.7	41.9	Default → 2015-2016	78.7	91.1
Domain (Task)	Finetune Year	Evaluation → Pretrain ↓	2009-2010	2015-2016	Domain (Task) ↓	Finetune Year ↓	Evaluation → Pretrain ↓	2014	2019
News (NewsSum) Rouge-L	2009-2010	Default	36.4	29.0	Food Reviews (Yelp) F1	2013	Default	58.6	58.3
Default → 2009-2010	36.4	29.1	Default → 2013	63.3	60.1
Default → 2015-2016	36.1	28.9	Default → 2019	60.2	62.3
2015-2016	Default	27.8	31.8	2019	Default	58.3	58.3
Default → 2009-2010	28.2	31.8	Default → 2013	60.2	62.3
Default → 2015-2016	27.8	31.6	Default → 2019	60.8	62.3
Table 7: Combination of temporal adaptation and finetuning (§3.2) on our tasks. The row labeled "Default" corresponds to a model that has not been adapted (uses the default pretraining). The color coding is proportional to the magnitude of the performances of each task (darker shade of orange indicates higher scores). We see that models that were finetuned on similar time periods performed similarly, no matter how their DAPT conditions differed.