Mitigating Temporal Misalignment by Discarding Outdated Facts
Authors: Michael JQ Zhang
Eunsol Choi
Venue: EMNLP 2023 Main (2023)
Keywords: Question Answering
Temporal

Abstract: While large language models are able to retain vast amounts of world knowledge seen during pretraining, such knowledge is prone to going out of date and is nontrivial to update. Furthermore, these models are often used under temporal misalignment, tasked with answering questions about the present, despite having only been trained on data collected in the past. To mitigate the effects of temporal misalignment, we propose fact duration prediction: the task of predicting how long a given fact will remain true. In our experiments, we demonstrate that identifying which facts are prone to rapid change can help models avoid reciting outdated information and determine which predictions require seeking out up-to-date knowledge sources. We also show how modeling fact duration improves calibration for knowledge-intensive tasks, such as open-retrieval question answering, under temporal misalignment, by discarding volatile facts.

Mitigating Temporal Misalignment by Discarding Outdated Facts
Michael J.Q. Zhang and Eunsol Choi
Department of Computer Science The University of Texas at Austin {mjqzhang, eunsol]@utexas. edu
Abstract
Question Asked in =0021:
Who sang the American Anthem at the Super Bowl?
What's the tallest building iin the world of all time?
While large language models are able retain vast amounts of world knowledge seen during pretraining, such knowledge prone to going out of date and is nontrivial to update. Fur- thermore, these models are often used under temporal misalignment, tasked with answer- ing questions about the present, despite having only been trained on data collected in the past. To mitigate the effects of temporal misalign- ment, we propose fact duration prediction: the task predicting how long given fact will demon-
System trained in 1212:
Retrieved Evidence from 1 2018
The Burj Khalifa the tallest building...
Pink sang the American...
Confidence: 90%
Confidence: 80%
Fact Duration (d)
Misalignment (m) years
Prediction:
ipd S m)=
p(d 5%
90%
Pred Duration: 10 years
Pred Dur: =11 years
Confidence Adjusted for
Conf. Adjusted for Misalignment: 8%
remain true. In experiments, strate that identifying which facts are prone to rapid change can help models avoid reciting outdated information and determine which pre- dictions require seeking out up-to-date knowl- edge sources. We also show how modeling fact duration improves calibration for knowledge- intensive tasks, such as open-retrieval question answering. under temporal misalignment, by discarding volatile facts. Our data and code are released publicly at https: //gi thub. com/ mike mikejqzhang mi tigatinggmiiiii www.mmssllllgn igmment.
Misalignment: 85%
Figure 1: We depict the critical timestamps play in open-retrieval QA systems. In the example on the left, the temporal misaligmment between when the system was trained and evaluated has affect the answer. On the right, the answer has changed, causing the sys- tem to output an outdated answer with high confidence. To account for this, apply our fact duration prediction system to adjust the system's 's confidence accordingly.
Prior work has attempted to address these is- sues by updating the knowledge stored within the parameters of an existing pretrained model (Cao et al., 2021; Mitchell et al., 2022; Onoe et al., 2023). Another line of work has proposed us- ing retrieval-based systems, which utilize parametric corpus of facts that can be updated over time (Karpukhin et al., 2020; Guu al., 2020; Lewis et al., 2021). Both methods, however, are incomplete solutions as they rely on an oracle to identify which facts need to be updated and to con- tinuously curate corpus of up-to-date facts.
1 Introduction
A core challenge in deploying NLP systems lies in managing temporal misalignment, where model that is trained on data collected in the past is eval- uated data from the present (Lazaridou al.,
2021). Temporal misalignment causes performance degradation in variety of NLP tasks (Luu al., 2021; Dhingra et al., 2022; Zhang and Choi, 2021). This is particularly true for knowledge-intensse tasks, such as open-retrieval question answering (QA) (Chen et al., 2017), where models must make predictions based on world knowledge which can rapidly change. Furthermore, such issues are only exacerbated as the paradigm for creating NLP sys- tems continues shift toward relying on large pre- trained models (Zhang et al., 2022; Chowdhery et al., 2022) that are prohibitively expensive to re- train and prone to reciting outdated facts.
Given the difficulty of keeping existing mod- els up-to-date, we propose an alternative solution where we abstain from presenting facts that we predict are out of date.! To accomplish this, we
'Some modern language model assistants (e.g.. ChatGPT) have demonstrated a similar ability to abstain from responding to questions with temporally-de dependent answers. We compare our methods against such system in Section 5.3.
imporallyddopendemt

---

introduce fact duration prediction, the task of predicting how frequently a given fact changes, and establish several classification and regression- based baselines. We also explore large-scale sources of distant supervision for our task, in- cluding fact durations extracted from temporal knowledge bases (Chen et al., 2021b) and duration- related news text (Yang et al., 2020). We provide rich discussion this challenging task, exploring the relationship between fact duration prediction and temporal commonsense (Zhou et al., 2020).
Eval.
EM
AUR-
ECE
RCW55
Model
Year
OC t
(A 4)
2018 ! TS
36,0
0.82
0.13
55.0 (0.0)
DPR
37.9
0.74
0.22
55.0 (0.0)
! T5
17.4
0.77
0.26
27.2 (27.8)
2021
2 DPR
17.1
0.63
0.43
22.4 (32.6)
Adjusting Confidence With Oracle Misalignment Info.
TS
17.4
0.75
0.12
47.8 (7.2)
2021
DPR
17.1
0.71
0.17
38.7 (16.3)
Table 1: NQ-Open QA performance evaluated on an- swers from 2018 (from NQ-Open) and 2021 (from Situ- atedQA). Confidence estimates are taken from calibra- tion models that have been trained for each QA system. All models are trained on 2018 answers from NQ-Open. On the bottom, we compare against an oracle system which zeros the confidence predictions whose an- swers have changed between 2018 and 2021.
We provide two sets of evaluations for our fact duration prediction systems. First, as intrinsic eval- uations, we report how close systems' duration estimates are to ground truth labels. We find that models that are trained with only distant supervi- sion can predict the duration of 65% of temporally dependent facts from real search queries in Natu- ralQuestions (Kwiatkowski et al., 2019) to within 3 years, compared to 11% from a simple average- duration baseline. Second, in extrinsic evaluations, we measure how systems' duration estimates can improve open-retrieval QA system's predictions under temporal misalignment. We mainly focus on improving calibration (as depicted in Figure 1). Our approach can reduce expected calibration error by 50-60% over using system confidence alone on two QA systems (Roberts et al., 2020; Karpukhin et al., 2020) on the SituatedQA dataset (Zhang and Choi, 2021).
ing outdated answers for some questions whose answer has changed in the meantime.
Table 1 reports the QA performance of existing systems SituatedQA (Zhang and Choi, 2021), a subset of questions from NQ-Open (Kwiatkowski et al., 2019; Lee et al., 2019) that has been re- annotated with the correct answer as of 2021. this dataset, 48% of questions are updated within the temporal gap (2018 to 2021). We can see that the current models, without considering temporal misalignment, experience performance degradation on both answer accuracy (EM) and calibration. ?
Lastly, we also explore other ways of applying our fact duration systems in QA. We experiment with adaptive inference in ensembled open/closed- book QA systems, using duration prediction to de- cide when retrieval is necessary due to temporal misalignment. We also apply fact duration pre- diction in scenario where retrieval is performed over heterogeneous corpus containing both out- dated and recent articles, and systems must weigh the relevance of an article against its recency. In summation, we present the first focused study on mitigating temporal misalignment in QA through estimating the duration of facts.
In this table, we also explore using an oracle that identifics which answers have changed and zeroes the QA system's confidence in such predictions. While this does not change the system's 's accuracy, it helps models identify incorrect predictions, im- proving calibration metrics across the board. In real- world scenarios, however, we do not know which facts are outdated. Thus, in this work we build fact duration model which predicts facts that are likely outdated and use it to adjust the con- fidence of the QA model. We introduce our fact duration prediction and QA settings in detail below.
2.1 Fact Duration Prediction
2 Settings
We define the fact duration prediction task as fol- lows: given a fact f. systems must predict its duration d, the amount of time that the fact re- mained true for. We consider datasets that represent facts in a variety of formats: QA pairs, statements,
We aim to address temporal misalignment (Luu al., 2021) in knowledge-intensive tasks, such as open-retrieval QA. Figure illustrates our setting, We assume a QA model that is developed in the past and is evaluated on a query from a later date. This system suffers from temporal misalignment, return-
"QA and calibration model details can he found in Sec- tion 5.1 and calibration metrics are explained in Section 2.2.

---

knowledge-base relations. For modeling purposes, we convert all facts to statements. For example, the fact f ="The last Summer Olympic Games were held in Athens." has a duration of d = 4 years.
Calibration Metries Even without temporal mis- alignment, models will not always know the cor- reet answer. Well calibrated model predictions, however, allow us to identify low-confidence pre- dictions and avoid presenting users with incorrect information (Kamath et al., 2020). Under temporal misalignment, calibration further requires identi- fying which predictions should receive reduced confidence because the answer has likely changed. We consider following calibration metries:
Error Metrics We evaluate fact duration systems by measuring error compared to the gold reference duration: Year MAE is the mean absolute error in their predictions in years and Log-Sec MSE is mean squared error in log-seconds.
2.2 QA under Temporal Misalignment
AUROC: Area under the ROC curve evaluates calibration system's performance at classifying correct and incorrect predictions over all possible confidence thresholds (Tran al., 2022).
The open-retrieval QA task is defined follows: Given question q', system must produce the cor- responding answer a, possibly relying on retrieved knowledge from an evidence corpus E. When tak- ing temporal misalignment into consideration, sev- eral critical timestamps can affect performance:
Expected Calibration Error (ECE): Com- puted by ordering predictions by estimated confi- dence then partitioning into 10 equally sized buck- cts. ECE is then macro-averaged absolute error each bucket's s average confidence and accuracy.
Model Training Date (far): When the train- ing data for M was collected annotated. Evidence Date (Lg): When E was authored,"
Risk Control (RC@ XX): Uncertainty esti- mates are often used for pelective-prediction, where models withhold low-confidence predictions below some threshold (< r), where is set to achieve a target accuracy (XX%) on some evaluation set. We measure how well r generalizes to a new dataset (Angelopoulos et al.. 2022). To compute RC@ XX, we set based on predictions from tM. then com- pute the accuracy on predictions from lg with con- fidence 2 In the ideal case, the difference (|Al) between RC@ XX and XX should be zero
Query Date (lg): When q was asked.
For studying QA under temporal misalignment, we further specify that systems must produce appro- priate answer at the time of the query ot,- For example, the question = ="Where are the next Summer Olympics?" asked at lg 2006 has answer 02006 = "Beijing". We define the mag- nitude of the temporal misalignment (m) to be the amount of time between model's training date and the query date (m tas - tq). We will compare this with the duration of the fact being asked - f(g, at.). m > d, we should lower the confidence of the model on this question.
3 Data
We first describe the datasets used for evaluation, split by task. We then describe our two large-scale sources for distant supervision. Appendix B con- tains further prepossessing details and examples.
For simplicity, we do not take an answer's ' start date into account. Ideally, determining whether a given QA pair (g, a) has gone out of date should also consider the answer's start date (t,) and model's training date (lm). and confidence can be lowered f. + < tm + m. While we expect this approximation to have less of an impact when set- tings where the misalignment period small with respect to the distribution of durations, we perform error analysis on examples where considering start date hurts performance in Appendix C.
3.1 Evaluation Datasets
QA under Misalignment Our primary evalua- tions are on SituatedQQ (Zhang and Choi, 2021), a dataset of questions from NQ-Open (Kwiatkowski et al., 2019) with temporally or geographically de- pendent answers. We use the cemporrlly-dependeee subset, where each question has been annotated with a brief timeline of answers that includes the correct answer as of 2021, the prior answer, and the dates when each answer started to be true. We evaluate misalignment between f. 2018 and 1g 2021 using the answers from NQ-Open for 02018 and answers from SituatedQA as a2021
"We primarily study settings where the evidence corpus does not change berween training and inference (lg = ( ) but also explore the effects of updating the evidence corpus (f& = fu) in Section 6. Training corpora and evidence corpora can contain documents authored over a span of time. For Lar. we use the date its latest document was authored. For, fr. we studying using Wikipedia as our evidence corpus, and therefore all evidence is up-to-date as of tg.
While several recent works have proposed new

---

QA Calibration Under Temporal Misalignment
Total (Ch. / Unch. between tar 2018 and f4 = 2021)
Due to the differences described above, we do not use temporal commonsense datasets for evaluat- ing fact duration prediction. We, however, still eval- uate on them to explore how these tasks compare. In particular, we evaluate our fact duration systems on the event duration subset of MCTACO (Zhou et al., 2019). Each MCTACO example consists of multiple-choice question about the duration of some event in context sentence, which we convert into duration statements. We evalute using the met- rics proposed by the original authors. Following Yang et al. (2020), we select all multiple choice op- tions whose duration falls within a tuned threshold of the predicted duration. EM measures accuracy, evaluating whether the gold and predicted answer sets exactly match. Fl measures the average Fl between the gold and predicted answer sets.
SituatedQA
322 (157 165)
Duration Prediction
# Train Dev / Test
SituatedQA
/ 322
MC-TACO (Duration)
/ 1,075 / 2,899
TimeQA
11,708 2,499 2,461 24,089 / 5,686
TimePre
Table 2: Dataset statistics for our QA misalignment cal- ibration and duration prediction tasks. We report the number of examples used in our QA calibration exper- iments along with how many examples have answers that have changed/unchanged between 2018 and 2021.
datasets for studying temporal shifts in QA (Kasai al., 2022; Livska et al., 2022), these works focus on questions about new events, where answers do not necessarily change (c.g.. "How much was the deal between Elon Musk and Twitter worth?"). We do not study such shifts in the input distribution over time. We, instead, study methods for manag- ing the shift in the output distribution (i.e., answers changing over time). Adjusting model confidence due changes in input distribution has been ex- plored (Kamath et al., 2020); however, to the best of our knowledge, this the first work on calibrat- ing over shifts in output distribution in QA.
3.2 Distant Supervision Sources
Temporal Knowledge Bases have been used in numerous prior works for studying how facts change over time. TimeQA (Chen et al., 2021b) is one such work that curates a dataset of 70 different semporally-dependerr relations from Wikidata and uses handcrafted templates to convert into decon- textualized QA pairs, where the question specifies a time period. To convert this dataset into fact- duration pairs (f,d), we first convert their QA pairs into factual statements by removing the date and using QA-to-statement conversion model (Chen et al., 2021 1a). We then determine the duration of each facts be the length of time between the start date of one answer to the question and the next.
Fact Duration Following suit with the QA evalu- ations above, we also evaluate fact duration predic- tion on SituatedQA. To generate fact-duration pairs, we use the annotated previous answer as of 2021, converting the question/answer pair into statement using existing T5-based (Raffel et al., 2020) con- version model (Chen et al., 2021 a). We then use distance between the 2021 and previous answer's start date as the fact's duration, d.
News Text contains vast array of facts and rich temporal information. Time-Aware Pretrain- ing dataset (TimePre) (Yang et al., 2020) curates such texts from CNN and Daily Mail news articles using regular expressions to match for duration- specifying phrases (e.g., "Crystal Palace goal- keeper Julian Speroni has ended the uncertainty over his future by signing new 12 month con- tract to stay at Selhurst Park."). Pretraining this dataset has previously been shown to improve performance on temporal commonsense tasks.
Temporal Commonsense Temporal common- sense focuses on inferences about generic events (e.g., identifying that glaciers move over centuries and a college tours last hours). In contrast, fact du- ration prediction requires making inferences about specific entities. For instance, determining the du- ration of an answer to question like "Who does Lebron James plays for?" requires entity knowl- edge to determine that Lebron James is basketball player and commonsense knowledge to determine that basketball players often change teams every few years. Previous work (Onoe et al., 2021) has demonstrated the non- non-trivial nature of combining entity-specific and commonsense knowledge.
3.3 Dataset Summary
Table 2 reports data statistics and Figure 2 presents the distribution of durations from each dataset. While most facts in SituatedQA and TimeQA change over the course of months to decades, facts in MCTACO and TimePre cover a wider range.

---

SituatedOA
28
11
26
Regression Model uses the mean squared error loss in units of log seconds, where the output from the hidden layer predicts a scalar value.
MCTACO
23
17
11
14
2
1
2
3
7
TimeQA
35
15
13
16
TimePre
4.2 Results
Figure 2: Duration statistics each dataset's develop- ment set. Columns represent different duration classes used by our classification model, with units abbreviated Seconds, Minutes, Days, Weeks, Months, Years, Decades, and Centuries. Cells contain the % of exam- ples in each dataset in the column's duration class.
We experiment with training on TimeQA and TimePre individually and on the union of both datasets. Figure reports duration prediction and temporal commonsense performance. Overall, we find that our trained systems outperform simple random and average baselines on SituatedQA. This is indicative of strong generalizability from our distannly-superviiedd fact-duration systems, even when baselines benefit from access to the gold label distribution. We also provide histogram of errors from systems in Figure 3 where that over 60% of our classification- based system's predictions are within 3 years of the gold duration, while predicting the exact duration remains chal- lenging. Below, we reflect on the impact of our different modeling choices and research questions.
4 Fact Duration Prediction
4.1 Comparison Systems
Here, we describe our fact duration prediction sys- tems. We include two simple lowerbound baselines: Random samples a duration and Average uses the average duration from each dataset.
see
temporal common sense reasoning (Yang et al., 2020), we develop BERT-based (Devlin et al., 2018) models. We frame fact duration prediction as cloze questions to more closely match the system's 's pretraining ohjec- tive (Schick and Schütze, 2020). To this end, we append " lasting [MASK][MASO] eliciting the model to fill in the masked tokens with a duration. We use two mask tokens as typically duration information requires at least two tokens, for value and another for unit. For our TimePre and MCTACO datasets, we similarly replace the target durations with two mask tokens. Table 8 in Appendix contains examples. Predictions are made by averaging the encoded representations of the two "[MASK]" tokens, then using this represen- tation as an input to a single hidden layer network. Using this same representation, we train two mod- cls with regression-basee and dassification-based learning objectives described below.
Following prior work
Regression vs. Classification Regression- based models tend to outperform their classification- based counterparts. The instances where this is not true can be attributed to an insufficient amount of training data. In Figure 3, we can see the dif- ferent types of errors each model makes. The clas- sification system predicts duration within I year more frequently, but the regression system predicts duration within years more frequently.
onto each fact,
Supervision from KB vs. News Text We find that training on temporal knowledge-basc relations (TimeQA) alone vastly outperforms training on news text (TimePre) alone for fact-duration predic- tion; however, the opposite true when compar- ing performance on temporal commonsense (MC- TACO). Training on both datasets tends to improve our regression-bact based system, but yields mixed re- sults for our classification based system. We hy- pothesize that the closeness in label distribution (see Figure between the training and evaluation sets impacts the performance significantly.
Classification Model frames the task as 13-way classification task where each class corresponds duration in Figure 2.5 We train using cross entropy loss, selecting the closest duration class as the pseudo-gold label for each fact. Because this model can only predict a limited set of durations, we report its upperbound by always selecting the class closest to the gold duration.
Fact Duration vs. Temporal Commonsense While fact duration prediction and temporal com- monsense are conceptually related, we find that strong performance either task does not neces- sarily transfer to the other. As discussed above. this can be attributed to differences in label distri- butions; however, label distribution also serves as a proxy variable for the type of information being
*We explore using other pretrained models in Appendix C. *Wo select these duration classes hased on frequeney across all datasets.

---

SituatedQA
TimeQA
MCTACO
100
Average
Model
Training Data
MSE
MAE
MSE
MAE
EM
F1
50
(LS,
(Y. 4)
(LS.
(Y. J)
(t)
Random
7.14
13.44
2.22
6.96
20.3
38.1
0
Average
5.42
10.61
1.65
5.41
23.8
41.3
Requency
100
Classification
Oracle
0.28
4.18
0.12
1.66
8.0
37.2
50
Classification
TimeQA
4.20
8,45
1.55
4.80
20.3
41.3
TimePre
40.51
8.77
9.36
7.85
28.3
57.7
0
Time(QA+Pra
6.28
8.37
1.70
4.66
28.3
57.2
100
Regression
TimeQA
3.75
8.40
0.97
4.22
21.2
41.0
Regression
TimePre 38,48
8.99
32.10
5.70
33.1
57.8
Time(QA+Pre)
3.58
8.15
0.97
4.19
31.8
57.8
10
15
Error (Years)
Figure 3: Fact Duration Prediction Results. On the left, we report full results, with performance split by model type and training data. Performance on SituatedQA and TimeQA are given as the mean average error in years (Y) and mean squared error years log-seconds (LSS), the same as regression system training loss. On the right, depict error histograms evaluated on SituatedQA, with systems trained on TimeQA.
queried for either task. Commonsense knowl- edge primarily differentiates events that take place over different orders of magnitude of time (e.g..
be the CDF of this distribution evaluated at m:
G. -(q.a) Eagm P(dlg, a).
Regression: Here, the output is single pre- dicted duration d. We set the discount factor to the binary value indicating whether or not the misalignment period has exceeded the predicted duration: Cn. c(q. a) a) < m)
seconds versus years). Differentiating whether an event takes place over finer-grained ranges (e.g.. versus two years), however, cannot be resolved with commonsense knowledge alone, and further require fact retrieval. We find that NQ contains queries for facts that change over smaller range durations (between 1-10 years), and, therefore, commonsense knowledge alone is insufficient.
As classification systems predict a distribution over fact durations, we are able use the CDF this distribution to make granular adjustments to confidence over time. In contrast, our regression systems predict a single value for fact duration, and confidence adjustments over time are abrupt. leaving confidence unchanged or setting it to zero.
5 Calibrating QA under Temporal Misalignment
5.1 Models
Here, we return our motivating use-case of us- ing fact duration prediction to calibrate open- retrieval QA systems under temporal misalignment. We assume an access to base calibration system, c(g, a) [0, 1] that has the same training date as the QA system its calibrating. We then use fact duration generate misalignment-awarn confi- dence score cm through simple post-hoc augmenta- tion, scaling down the original confidence score by a discount factor based on the degree of misalign- ment and the predicted fact duration. We compute this factor differently for each of our fact duration systems.
Base QA and Calibration Systems We experi- ment with three QA systems throughout our study:
! T5: We use T5-large (Roberts et al., 2020) which has been trained with salient-span- masking and closed-book QA on NQ-Open.
2 DPR (l.=2018): We use DPR (Karpukhin al., 2020), open-book system which trieves passages from te 2018 Wikipedia snapshot and is also trained on NQ-Open.
N DPR (t.=2021): We use the same model as 2. but swap the retrieval corpus with an updated Wikipedia snapshot that matches query times-
"We experiment with using our classification-ban sys- tem's 's predicted class or the expected duration as the duration for adjusting confidence, but both methods underperform com- pared to using the system's 's CDE We include these results in Appendix C.
Classification: Here, the system's output probability distribution over different duration classes, p(d q. a). We set the discount factor to

---

QA Model
2018 2021 EM
Dur. Model
AUROCT
ECE
RC@55 ((A) ()
Avg Conf
N/A Oracle
0.766
0.265
27.2 (27.8)
0.0
0.749
0.116
47.8 (7.2)
-25.8
! T5
36.0 + 17.4
Regression
0.709
0.185
32,4 (22.6)
-23.2
Classification
0.765
0.131
29.3 (25.7)
-15.1
N/A Oracle
0.629
0.433
22.4 (32.6)
0.0
0.708
0.172
38.7 (16.3)
-3699
DPR (fe- 2018)
37.9 17.1
Regression
0.601
0.268
26.1 (28.9)
-3..0
Classification
0.654
0.235
43.5 (11.5)
-20.9
N DPR (L. 2021)
19.6
N/A
0.636
0.370
25.4 (29.6)
-3.8
Table 3: Results for calibrating QA under temporal misalignment SituatedQA. All systems' training dates are 2018 and evaluation dates are 2021. We report each system's 's EM accuracy, evaluated against the answers from 2018 and 2021. We also report how much model confidence changes average (Avg Conf % A) with each adjustment method (for DPR with Le 2021 we compare average confidence against using = 2018).
Model
Adj.
AUROC
ECE
RC@ (lAI)
gression and classification duration predictions lower the confidence of models, improving cal- ibration metrics across the board. We find that our classification- model consistently outper- forms our regression-basse model on our calibra- tion task, despite the opposite being true for our fact-duration evaluations. We attribute this behav- ior classification-batt based system' 's error distri- bution, it gets more examples correct within year (Figure 3). Classification- system also hedge over different duration classes by predicting a distribution, which we use to compute the CDE
Uniform
0.757
0.180
30.5 (24.5)
① T5
Per-Ex
0.765
0.131
29.3 (25.7)
Uniform
0.627
0.259
25.6 (29.4)
DPR
Per-Ex
0.654
0.235
43.5 (11.5)
Table 4: Ablating per-example calibration: We first ad- just confidence Per-Example, which is our full system. We then adjust confidence Uniformly across all exam- ples, such that the net decrease in confidence across the entire test set is equivalent.
tamp (he 2021) following Zhang and Choi (2021), which showed partial success in return- ing up-to-date answers.
Retrieval-Bact QA: Update or Adjust In Ta- ble 3, we compare the performance of DPR with static and hot-swapped retrieval corpora from le = 2018 and le 2021. While updating the retrieval corpus improves EM accuracy, adjusting model static corpus
For each QA system, we train a calibrator that pre- dicts the correctness of the QA system's ' answer. We follow Zhang et al. (2021) for the design and input features to calibrator, using the model's 's pre- dieted likelihood and encoded representations of the input (details Appendix A).
confidence using fact duration performs better all calibration metries. This suggests that, when users care about having ac- curate confidence estimates or seeing only high- confidence predictions, confidence adjustments can be more beneficial than swapping retrieval corpus.
Fact Duration Systems For both our regression and classification based models, we use the systems trained over both with TimeQA. We also include results using an oracle fact duration system, which zeroes the confidence for all questions that have been updated since the training date.
Ablations: Per-Example vs Uniform Adjust- ment We compare our system, which adjusts con- fidence on per-example basis, against one that uni- formly decreases confidence by the same value v across the entire test set: C.. max(c(g, These ablations still depend our fact-duration systems determine the such that the total con- fidence over the entire test set is the same for both methods. Table reports the results from this ab- lation study. We find that uniformly adjusting con- fidence improves ECE, which is expected given
5.2 Results
0).
Table 3 reports the results from calibration periments. Both QA models suffer from temporal degradation, and zero-ing out the confidence of outdated facts with oracle information improves the calibration performance. Using model pre- diction durations shows similar gains. Both re-

---

the decrease in the QA systems EM accuracy af- ter misalignment. We find, however, that our per- example adjustment methods outperform uniform confidence adjustments.
Inference Ensemble
EM
1 Ts
17.4
0.0
2 DPR L. -2018
17.1
0.0
N DPR t. 2021
19.6
100.0
! T5 / & DPR I, 2021
20.5
45.7
5.3 Comparisons against Prompted LLMs
DPR t, = 2018 S DPR t. 2021
19.3
45.0
While we primarily focus on calibrating fine-tuned QA models, recent work has also explored calibrat- ing prompted large language models (LLMs) for QA (Si et al., 2022; Kadavath et al., 2022; Cole al., 2023). Furthermore, recent LLMs (c.g.. ChatGPT) have demonstrated the abil- ity abstain from answering questions on the basis that their knowledge cutoff date is too far in the past; however, it is not publicly known how these systems exhibit this behavior.
Table 5: Adaptive inference for temporal misalignment results: we use our duration prediction decide whether to use the prediction from model with newer corpus (DPR f. 2021). I). In this data (SituatedQA), 48.8% of examples requires up-to-date knowledge.
prediction system using this answer. We then use the CDF of the predicted duration distribution determine whether it is at least 50% likely that the fact has changed: Edsm P(d a) 2 0.5. so, we then run retrieval with DPR using the updated corpus t. = 2021 and present the predicted answer. We report our results in the first row of Table 5, which shows that this outperforms either system on its own, while running retrieval on less than half of all examples.
In this experiment, investigate such sys- tem, GPT-4 (OpenAI, 2023), and its ability ab- stain from answering questions with rapidly chang- ing answers. We prompt GPT-4 to answer ques- tions from SituatedQA, and find that abstains from answering 86% of all questions (95% of ques- tions whose answers have been updated between 2018 and 2021 and on 79% of examples that have not). Overall, this behavior suggests that GPT-4 overestimates how frequently it must to abstain from answering user queries. Furthermore, GPT-4 does not provide how frequently the answer is ex- pected to change. Collectively, GPT-4's tendency to over-abstain and its lack transparency limits its usefulness to users. In contrast, our approach provides users with an duration estimate indicating why prediction may not be trustworthy. Further experimental details and example outputs are re- ported in Appendix D.
Two Corpora: Relevancy vs. Recency (2 While most work for QA have focused on retriev- ing over Wikipedia, many questions require retriev- ing over other sources such as news or web text. One challenge in moving away from Wikipedia lies in managing temporal heterogeneity across dif- ferent articles. Unlike Wikipedia, news and web articles generally not maintained to stay current, requiring retrieval-based QA systems to identify out-of-date information in articles. Systems that retrieve such resources must consider the trade-off between the recency versus relevancy of an article. In these experiments, we experiment with using fact duration prediction as a method for weighing this trade-off in retrieval.
6 Beyond Calibration: Adaptive Inference
Our experimental setup is as follows: instead of computing misalignment from the model's train- ing date, we compute relative to when the article was authored (m years for 2018 Wikipedia and m years for 2021 Wikipedia). After per- forming inference using both corpora, we re-rank answers according to their misaligoment-adjussedd confidence estimates. We report results in Table 5, We find that our method is able to recover compara- ble performance to always using up to-dete articles, while using it just under half the time.
In this section, we explore using misalignment- aware confidence scores to decide how answer a question. Below, we motivate and describe two adaptive inference scenarios where systems may choose between two methods for answering ques- tion using their fact duration predictions.
Hybrid: Closed Open (++)): Besides the computational benefits from not always having to use retrieval, forgoing retrieval for popular ques- tions can also improve answer accuracy (Mallen et al., 2022). We use our fact duration predictions to decide when retrieval is necessary: we first pre- dict an answer using T5 and run our fact duration
Related Work
Commonsense and Temporal Reasoning Re- cent works have proposed forecasting bench-

---

marks (Zou et al., 2022; Jin et al., 2021a) related our fact duration prediction task. While our task asks models to predict when a fact will change, these forecasting tasks ask how a fact will change. Qin et al. (2021) studies temporal commonsense reasoning in dialogues. Quantitative reasoning has been explored in other works quantitative re- lations between nouns (Forbes and Choi, 2017; Bagherinezhad et al., 2016), distributions over quantitative attributes Elazar al. (2019), and rep- resenting numbers in language models (Wallace et al., 2019).
diction task, alongside several datasets and base- line systems for it. Future work may build upon this evaluation framework to further improve QA calibration under temporal misalignment. For in- stance, future work may examine modeling differ- ent classes distributions of fact duration distribu- tions, like modeling whether a fact changes after regular, periodic time interval.
Limitations
We only evaluate temporal misalignment between 2018 and 2021, three-year time difference, on relatively small scale SituatedQA dataset (N=322). This is mainly due to lack of benchmark that sup- ports studying temporal misalignment. Exploring this in more diverse setup, including different lan- guages, text domains, wide range of temporal gaps, would be fruitful direction for future work.
Calibration Abstaining from providing QA system's s answers has been explored in several re- cent works. Chen et al. (2022) examines instances where knowledge conflicts exist between model's memorized knowledge and retrieved documents. As the authors note, such instances often arise due temporal misalignment. Prior work (Ka- math al., 2020; Zhang et al., 2021; Varshney and Baral, 2023) has explored abstaining from answer- ing questions by predicting whether or not the test question comes from the same training distribution of the QA system. While fact duration also pre- dicts a shift in distribution, fact duration focuses on predicting a shift in question's outpur distribu- tion of answers instead of a shift in input distribu- tion of questions; therefore, these two systems addressing orthogonal challenges in robustness to distribution shift and are complementary.
As is the case with all systems that attempt faithfully relay world knowledge, treating model predictions as fact runs the risk of propagating misinformation. While the goal of our fact du- ration prediction systems is to prevent models from reciting outdated facts, it does not always succeed and facts may change earlier than expected. Even though a given fact may be expected to only change once every decade, improbable outcome may oceur and the fact changes after only year. In such an event, our misalignment- system may erroneously maintain high confidence in the outdated answer.
calibration
Keeping Systems Up-to-Date Several works have explored continuing pretraining to address temporal misalignment in pretrained models (Dhin- gra et al., 2022; Jin et al., 2021b). Other works have explored editing specific facts into models (Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022). These works, however, have only focused synthetic settings and assume access the up- dated facts. Furthermore, such systems have yet be successfully applied to new benchmarks for measuring whether language models have acquired emergent information (Onoe et al., 2022; Padman- abhan et al., 2023). Recent works on retrieval- based QA systems have found improved adapta- tion when updated with up-to-date retrieval cor- pora (Izacard et al., 2022; Lazaridou et al., 2022).
Furthermore, our system, as it stands, does not take the answer start date into account. Our system also can make errors due to changes in the typical duration of a given fact. For instance "What's 's the world's tallest building?" changes more frequently over time as the rate of technological advances also increases. We provide examples of such system errors in Appendix C.
Acknowledgements
The work was partially supported by Google Re- search Award, a gift from HomeDepot, and grant from UT Machine Learning Lab. This work was in part supported by Cisco Research. Any opinions, findings and conclusions, recommendations ex- pressed in this material are those of the authors and do not necessarily reflect the views of Cisco Research. The authors would like to thank the members of the UT Austin NLP community and Jordan Boyd-Grahe for helpful discussions.
8 Conclusion
We improve QA calibration under temporal mis- alignment by introducing the fact duration pre-

---

References
Jacob Devlin, Ming-Wor King-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre- tre-training of deep bidirectional transformers for language understand- ing. NAACL.
Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. 2022. Conformal risk control. arXiv preprint arXiv:2208. 02814.
Bhuwan Dhingra, Jeremy Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Transactions the Association for Computational Linguistics, 10:257- 273.
Hessam Bagherinezhad, Hannaneh Hajishirzi, Yejin Choi, and Ali Farhadi. 2016. Are elephants bigger than butterflies? reasoning about sizes of objects. In AAAL
Nicola De Cao, W. Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In EMNLP.
Yanai Elazar, A. Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. 2019. How large are lions? inducing distributions over quantitative attributes. In ACL.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. In ACL.
Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In ACI.
Hung-Ting Chen, Michael JQ Zhang, and Eunsol Choi. 2022. Rich knowledge sources bring complex knowl- edge conflicts: Recalibrating models to reflect con- flicting evidence. arXiv preprini urXiv:2210. 137.11...
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International Con- ference on Machine Learning, pages 3929-3938. PMI..
Jifan Chen, Eunsol Choi, and Greg Durrett. 2021a. Can nli models verify qa systems' predictions? arXiv preprint arXiv:2104.08%
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785- 794.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with cradient-disentaaeee embedding shar- ing. arXiv preprint arXiv:2111 11..94431.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot Learning with Retrieval Aug- mented Language Models.
Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021b. A dataset for answering timeseensitvee ques- tions. ArXiv, abs/2108 00022008.063114 043.4.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garefa, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez. Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee, Zong- wei Zhou, Xuezhi Wang. Brennan Sacta. Mark Díaz. Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-HHllstern, Douglas Eck, Jeff Dean, Slay Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. ArXiv, abs/2204.0234
Woojeong Jin, Suji Kim, Rahul Khanna, Dong-Ho Lee, Fred Morstatter, A. G. Galstyan, and Xiang Ren. 202 la. Forecastqa: A question answering challenge for event forecasting with temporal text data. In ACL.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arool,, and Xiang Ren. 202 Lifelong pretraining: Continu- ally adapting language models to emerging corpora. arXiv preprint 10.05:2110.08534.
Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethann Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran- Johnson, Scott Johnston, Sheer EI-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. ArXiv, 0als/220..0522))
Jeremy R Cole, Michael JQ Zhang, Daniel Gillick, Ju- lian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. Selectively answering ambiguous questions. arXiv preprint 2305 146.1333....
Amita Kamath, Robin Jia, and Percy Liang. 2020. Se- lective question answering under domain shift, In ACL.

---

Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov. Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Inves- tigating effectiveness and limitations of paramet- ric and non- con-paamettric memories. arXiv preprint arXiv:2212. 10511.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov, 2022. Locating and Editing Factual Asso- ciations in GPT.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi. Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentarou Inui. 2022. Realtime qu: What's the answer right now? ArXiv, abs/2207.13332
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022. Fast model editing at scale. In International Conference on Learning Representations.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Iilia Polosukhin. Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics.
Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, and Greg Durrett. 2021. Creak: A dataset for commonsense reasoning over entity knowledge. abs/2109.01653.
Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity cloze by date: What Ims know about unseen entities. arXiv.
Yasumasa Onoe, Michael J.Q. Zhang, Shankar Padman- abhan, Greg Durrett, and Eunsol Choi. 2023, Can Ims learn new entities from descriptions? challenges in propagating injected knowledge. abs/2305.01651.
Angeliki Lazaridou, Elena Gribovskaya, Wojeiech Stokowiec, and Nikolai Grigorev. 2022. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint 15Xiv:2203.05/15.
OpenAL. 2023, Gpt-4 technical report. arXiv preprint urXiv:2303.08774
Angeliki Lazaridou, Adhiguma Kuncoro, Elena Gri- bovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomás Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 1.21. Mind the gap: Assessing temporal generalization in neural language models. In NeurIPS.
Shankar Padmanabhan, Yasumasa Onoe, Michael JQ Zhang, Greg Durrett, and Eunsol Choi. 2023. Propa- gating knowledge updates to Ims through distillation. arXiv preprint 20t6:2306.09000
Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021. Timedial: Temporal commonsense reasoning in dialog. In ACL.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. ArKiv, 1000000000
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res,, 11(14
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Min- ervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. Paq: 65 mil- lion probably- asked questions and what you can do with them. Transactions of the Association for Com- putational Linguistics, 9:1098-11 1015
Adam Roberts, Colin Raffel, and Noam M. Shazeer. 2020. How much knowledge can you pack into the parameters of language model? ArXiv, abs/2002.08910.
Adam Livska, Tom' avs Kovcisk's y, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cy- prien de Masson d' Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-McMahon, Sophia Austin. Phil Blunsom, and Angeliki Lazari- dou. 2022. Streamingqa: A benchmark for adapta- tion to new knowledge over time in question answer- ing models. International Conference on Machine Learning.
Timo Schick and Hinrich Schüizee 2020. Exploit- ing cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.cm 007666.
Chenglei Si, Zhe Gan. Zhengyuan Yang, Shuohang Wang, Jianfeng Wang. Jordan Boyd-Graber, and Li- juan Wang. 2022. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2200.09
Dustin Tran, Jeremiah Liu, Michael W Dusenberry. Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. 2022. Plex: Towards reliability using pretrained large model ex- tensions. arXiv preprint 22v:2207.074111
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Kar- ishma Mandyam, and Noah A. Smith. 2021. Time waits for no one! analysis and challenges of temporal misalignment. ArXiv, abs/21 1.07408.

---

Neeraj Varshney and Chitta Baral. 2023. Post- abstention: Towards reliably re-attempting the ab- stained instances in qa. ArXiv, abs/2305.01 812
A.2 Calibration Models
We implement trained calibration systems ing XGBoost (Chen and Guestrin, 2016) using the features for T5 and DPR outlined in Zhang et al. (2021). For T5, we concatenate (1) the averaged, encoded representations of the input question, and (2) the model likelihood. For DPR, we concatenate (1) the averaged, encoded representations of the in- put question and selected passage, (2) the averaged, encoded representations of the start and end tokens of the selected answer span, and (3) the likelihood of the answer span, computed the product of the likelihoods of selecting the start index, end index, and passage index.
Erie Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019, Do NLP models know num- bers? probing numeracy in embeddings. In Empiri- cal Methods in Natural Language Processing.
Zonglin Yang, X. Du, Alexander M. Rush, and Claire Cardie. 2020. Improving event duration prediction via time-aware pre-training. ArXiv, abs/201 1 1006611 10.
Michael J.Q. Zhang and Eunsol Choi. 2021. Situatedqa: Incorporating extra-linguistic contexts into qa.
Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Knowing more about questions can help: Im- proving calibration in question answering. ArXiv, abs/2106.01494.
We train systems on NQ-Open on randomly sampled 60/40 training and development splits, fol- lowing Zhang et al. (2021). We maximum depth of 10 and experiment with several values for the learning rate {0.01, 0.1, 0.2, 0.5} and column sub-sampling ratio {0.0, 0.1, 0.9),, which we keep the for sampling by tree, level, and node. We train with early stopping after 10 epochs without improvement and select the best performing system as evaluated on the development split.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoerr. 2022. Opt: Open pre-trained transformer language models. ArXiv, abs/2205.0100 00066...
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "going on vacation" takes longer than "go- ing for a walk": A study of temporal commonsense understanding. ArXiv, abs/ 199.0059..
A.3 Fact Duration Prediction Models
We use BERT-base from the transformers library for all duration prediction baselines, trained with batch size in (32, 613 and learning rate in (le 5, 5e 5]. We train until convergence and select the best checkpoint as determined by development set performance. Due to computational resource constraints, we do not further tune hyperparameters. All models are trained once and results reflect a single run. All experiments took were performed Quadro RTX 8000 gpus and required less than one week's s worth of GPU hours.
Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with min- imal supervision. ahs/2005.04304.
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. 2022. Forecast- ing future world events with neural networks. ArXiv, abs/2206.1 15474.
Implementation Details
B Datasets
A.1 QA Models
We provide examples from each dataset and our pre- possessing pipeline in Table 8. We provide futher preprocessing details below.
We use the T and S checkpoints that have been finetuned on NQ-Open's 's training set from the transformers library model hub.9 9 For DPR, we use
B.1 Fact Duration Dataset Preprocessing
the retrieval corpora from December 20, 2018 for tg 2018 and February 20, 2021 for tg 2021, following Zhang and Choi (2021).
In TimeQA, several examples have answers that are simply the empty string. We remove all such exam- ples from our preprocessed dataset. In Situated several examples have answers that begin and end in the same year, without further annotation deter- mining the exact number of days or months. We
base
"hanss/huuggingfaecccommadde

---

SituatedQA
TimeQA
MCTACO
Model
Training Data
LS-MSE
Y-MAE
LS-MSE
Y-MAE
F1
TimeQA
8.5
4.18
4.7
1.47
20.6
41.2
Classification
TA Pretrain
9.0
55.33
6.0
22.05
28.9
53.8
TimeQA TA Pretrain
7.58
5.1
1.56
28.0
56.9
TimeQA
8.4
3.76
4.6
1.18
25.1
41.3
Regression
TA Pretrain
11.6
44.56
20.6
27.59
33.1
58.3
TimeQA TA Pretrain
8.7
7.66
4.6
1.23
32.2
57.3
Table Fact Duration Prediction Results using DeBBiRaa-v3-hase instead of BERRT-hase. All other settings are the same as Figure 3.
QA Model
Misalingmnent Adj.
AUROC f
ECE 4
RC@55 (IAI))
Avg Conf A
CDF
0.765
0.131
29.3
25.7
15.1
C T5 36.0 - 17.4
Argmax
0.762
0.249
28.1
26.9
2.6
Expectation
0.708
0.169
37.5
17.5
35.0
CDF Argmax Expectation
0.654
0.235
43.5
11.5
20.9
(1e 2018)
0.650
0.403
23.8
31.2
3.9
37.9 + 17.1
0.641
0.177
38.2
16.8
49.2
Table 7: Additional calibration results on SituatedQA, comparing different methods of adjusting model confidence for temporal misalignment using the output of our classification-bat based fact-duration system. All other settings are the same as in Table 3.
simply assume that all such question answer pairs instances have a duration of month.
classification based system, we also experiment with using the expected duration as our discount- ing factor. We incorporate this by zeroing the confidence estimate if the expected duration is ex- ceeded by the degree of misalignment: f(q. a)
B.2 Temporal Commonsense Datasets Preprocessing
As we noted above, each MCTACO example con- sists a multiple-choice question about the du- ration of some event a provided context sen- tence. During prepossessing, we use the same ques- tion conversion model as above to transform each QA pair into statement and prepend the context sentence onto each question. We use the metrics proposed by the original authors, and select all multiple choice options whose duration falls within some absolute threshold of predicted dura- tion, measured in log seconds (Yang et al., 2020). This threshold is selected based on development set performance.
II 1 d2m d P(d a)}.
In Table we report additional results on our calibration evaluation. We include calibration per- formance of our best performing fact duration mod- els finetuned on SituatedQA: trained only on Sit- uatedQA for our dlassification-basee model and first trained on TimeQA + TA Pretrain for our regression-based model.
Error Analysis In Table 9, we highlight sampled errors from our fact duration system and discuss their causes and impact.
D ChatGPT and GPT-4 outputs
Additional Results
Table 10 includes two examples of ChatGPT in- forming users that the answers to a given question may have changed. It, however, does not provide users with an estimate of how likely it has changed, or how often the answer is expected to change. This lack of duration estimate results in lesser transparency and interpretatbility for users. To get results on SituatedQA, prompt GPT-4 with the following system prompt (recommended by their documentation): "You are ChatGPT, a large lan- guage model trained by OpenAL. Answer as con-
Different Pretrained Models for Fact Duration Prediction In Table report our results from DeBERTa-v3-base (He et al., 2021) on our fact duration prediction system. We also experiment with using the large variants of both BERT and DeBERTa, but do not find substantial improvement.
Adjusting Confidence with Expected Duration addition adjusting confidence using the CDF of the predicted duration distribution from our

---

Dataset
Example
Q: Who the judges on Asia Got Talent? / A: Vanness Wu / Start: 2015, End: 2017
SituatedQA
MI: Vanness Wu the judge Asia Got Talent, lasting [MASK] [MASK] ./ TD:: Years
Context: About 30% of Ratners's ' profit already is derived from the U.S.
MCTACO
Q: How long did it take to make profit/ A: 3 Months
MI: About 30% of Ratners's 's profit already is derived from the U.S. It took [MASK] [MASK] make
profit. / TD: 3 Months
Subj: Patrick Burss (businessmann) Rel:
Lives in Obj: Oshawa, Ontario Start: 1856 End: 1878
TimeQA
MI: Patrick Burns (businessman) lived in Oshawa, Ontario, lasting [MASK] [MASK] ./ TD:: 22 years
TA-Pretrain
MI: Jorge Ramos has been the face of Univision's 's News broadcast for [MASK] [MASK]. TD: 24 Years
Table 8: Fact duration prediction input examples. We standardize formats to predict target duration (TD) from the masked input (MI). The top row(s) in each cell represents the original data, and the bottom row shows our setting.
Quesiton (2018 Answer / 2021 Answer)
(G/P) Dur
User: Who is the current US president?
As an Al language model, I don't 't have real- time information or the ahility to browse the internet, My knowledge was last updated in September 2021, and at that time, the President of the United States was Joseph R. Biden Jr. However, please note that the political landscape can change, and there may have been a new election or other developments since then. To obtain the most accurate and up-to-date information, I recommend checking reliable sources such as official government websites or reputable news outlets.
ChatGPT:
When did last volcano erupt in Iceland? (2010/ / March 19, 2021)
How many episodes of Touching Evil are there? (16/ 16)
I
Who got the most passing yards in the NFL? (Peyton Manning Drew Brees)
3Y/IY
Table Example fact duration predictions errors from SituatedQA. We include the answers from Lyr 2018 and tg = 2021, and the gold / predicted durations ((G P) Dur). In the first example, tem correctly identified volcanic eruptions as an infre- quent event; however, our system misses that answer has changed within the misalignment period because it doesn't 't have access to the answer start date. In the sec- ond example, our system fails no longer changes since the show ended. In the last example, our model underes- timates the fact's duration, but still correctly identifies that the answer has changed.
Who won the last NBA Champtionship? ChatGPT: As an Al language model, I don't 't have real- time data, and my knowledge was last updated in Septem- ber 2021. Therefore, cannot provide information on the winner of the most recent NBA Championship. How- ever, as of my last update, the Milwaukee Bucks were the reigning NBA Champions. having won the champi- onship in the 2020-2021 scason. recommend checking reliable news sources or conducting quick online search to find the current champion.
User:
see that our sys-
Table 10: Example ChatGPT outputs to two temporally dependent questions with different answer durations. Predictions are take from the May 12, 2023 ChatGPT version.
cisely as possible. Knowledge cutoff: September 2021. Current date: May 12, 2023." We then present GPT-4 with the user's question from Situal- edQA. We determine whether systems abstains from given prediction if it references its knowl- edge cutoff from the prompt September 2021 if it mentions real-time information in its generated output.
of all these resources are in compliance with their licenses.
F Ethical Considerations
We do not collect any data in this paper. While we do generate a dataset, we visually inspect generated examples and do not find any cases of harmful or offensive content. The existing datasets used in this work have previously been vetted by their authors prior work. We also note that our work is only applied English datasets and models.
E Licensing
Dataset Licensing TimeQA is licensed under the BSD 3-Clause License. SituatedQA and Wikipedia are both distributed under the CC BY-SA 4.0 li- cense. MCTACO and TA-Pretrain are both unli- censed.
Model Licensing The transformers library. BERT model, T5 model, and question conver- sion model are all licensed under the Apache Li- cense v2.0. DPR is licensed under the Attribution- NonCommercial 4.0 International license. Our use


Reviews:
[{"review_id": "c7dwEiNAre", "reviewer": "Reviewer_Cm26", "summary": "", "questions": "Question Cm26_A: In the last experiment ( Two Corpora: Relevancy vs. Recency): Why would one want to perform inference based on two corpora (and double the inference costs), while using only one corpus (the most recent) gives better results ? \nQuestion Cm26_B: L447-452: how is this calibrator used with the fact duration system?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and notes on presentation! We plan to incorporate your suggestions in our revisions. Regarding your specific questions, we respond to each below:\n\nQuestion A: “In the last experiment (Two Corpora: Relevancy vs. Recency): Why would one want to perform inference based on two corpora (and double the inference costs), while using only one corpus (the most recent) gives better results?”\n\n> We agree that in the ideal scenario for retrieval would be to have one corpus where each article is constantly updated and outdated information is removed. This, however, is not the case for most corpera outside of Wikipedia. News articles, for instance, are usually written once and are never edited after they are published. News corpera contain articles written over a large period of time, where older articles contain a mix of correct facts, which have not changed, and outdated facts, which newer articles have updated. In using news articles as a knowledge corpus, we cannot completely throw out old articles without losing valuable information. \nThis experiment simulates a setting when we have two retrieval corpus, each from a different time period. In this setting, only using the most recent corpus is akin to the oracle setting, where we are able to remove all facts from the older corpus that have been restated or updated in future articles. Here, we demonstrate that relying on fact duration prediction to determine when facts from an older article can be trusted recovers similar performance to this oracle. We will clarify our motivations for this scenario in our writing. \n\n\nQuestion B: “L447-452: how is this calibrator used with the fact duration system?”\n\n> This process is described in the equations on L425 and L430. In these equations, we use the calibrator described on L447-452 to get the confidence score, c(q, a). This confidence score can be interpreted as predicting “the likelihood that this answer is true as of 2018”. We then use our fact duration predictions (which are made independently from a separate model) to adjust this score based on the temporal misalignment. We use this adjusted confidence estimate as our misalignment-aware confidence score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper addresses an interesting problem from a new perspective \n* The proposed solution can help make QA-systems more robust by making them more \"aware\" of the validity of their answers with respect to time. \n*  Several scenarios are considered in the evaluation", "reasons_to_reject": "NA ", "questions_to_authors": "Question Cm26_A: In the last experiment ( Two Corpora: Relevancy vs. Recency): Why would one want to perform inference based on two corpora (and double the inference costs), while using only one corpus (the most recent) gives better results ? \nQuestion Cm26_B: L447-452: how is this calibrator used with the fact duration system?", "paper_topic_and_main_contributions": "The paper addresses the problem of temporal misalignment in a QA setting. Language models can be trained to answer questions based on their pre-training data. However, this knowledge becomes outdated as time passes. The paper proposes predicting for how long would a fact remain true to mitigate temporal misalignment. By estimating when a fact would \"expire\", the models can be made less confident about their predictions. The paper introduces two fact duration prediction systems based on BERT and evaluate these. Furthermore, the paper combine these systems with QA-models with goal of calibrating their predictions (making them less confident about outdated answers). Additionally, the paper uses fact duration prediction to combine open- and closed-book QA system, falling back on the open-book system, when the closed-book system provides outdated answers. The benefit of the proposed system is also shown in re-ranking answers based on two corpora with different dates in an open-QA system. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* It is not clear what is the idea behind ECE \n* It is confusing to look at Table 1, when starting to read the paper \n* Table 5: having the baseline you compare against in the same table would make the results easier to compare\n* L 268: them explore: \"to\" missing \n* L286: ... 70 different: \"of\" missing \n* L364: reflect the: \"on\" missing \n* L447-448: predict predicts"}}, {"review_id": "EWQ6B60Kxa", "reviewer": "Reviewer_BbjM", "summary": "", "questions": "- If possible, it would be interesting to see whether LLMs have solved the calibration issue or are simply abstaining.\n- I'm curious about the relationship to the parallel question of mathematical reasoning, as fact duration prediction could be viewed as a rudimentary type of mathematical question. It would be informative to include results with math QA models.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review!\n\n\nRegarding your comment about whether existing LLMs have already solved the issue of temporal misalignment:\n\n> We noticed newer proprietary LLMs (e.g., ChatGPT) are equipped with an ability to abstain from answering with rationale of temporally dependent questions. Yet, from our investigation (L634-653) their ability is fairly brittle and the problem remains unresolved. Specifically, we find that LLMs tend to over-abstain from answering, abstaining from 79% of questions whose answers did not change between 2018 and 2021 (and 95% of temporally dependent questions).\n\n> Furthermore, unlike our systems, they do not provide an estimate of how likely it is that its predicted answer is out-of-date. This piece of information can be important for users deciding whether or not to trust the given answer.\n\n> Lastly, as far as we are aware, none of the open-sourced LLM show such abstaining functionality, adding value to our study. \n\n\nRegarding connections to mathematical reasoning:\n\n> Great point! We have also noted there is a relationship between fact duration prediction and other quantitative reasoning tasks (L556-571). We did not include systems designed for mathematical-reasoning tasks (e.g., DROP) as baselines as these systems are designed to perform complex reasoning using only the values given within the context. Fact duration prediction, however, is a task that only requires very simple mathematical reasoning using values that are recalled from real-world facts and commonsense knowledge(e.g., the US president is re-elected every 4 years, news anchors usually last many years). Therefore, the challenges these two tasks pose and the methods developed for each are quite orthogonal. We will add this discussion to the final draft of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting and somewhat overlooked problem in pretrained language models. The distantly supervised model shows moderate performance in fact duration prediction, showing there is a large room for improvement, which can encourage more research in this area. \nThe paper's exploration of temporal degradation in relation to QA model calibration should be commended.", "reasons_to_reject": "The scope of the problem is limited, and while LLMs do not entirely resolve the problem, LLMs are able to abstain from answers with potential temporal misalignment.", "questions_to_authors": "- If possible, it would be interesting to see whether LLMs have solved the calibration issue or are simply abstaining.\n- I'm curious about the relationship to the parallel question of mathematical reasoning, as fact duration prediction could be viewed as a rudimentary type of mathematical question. It would be informative to include results with math QA models.", "paper_topic_and_main_contributions": "This paper proposes a method to deal with temporal misalignment in open-retrieval QA systems. To do so, a \"fact duration prediction\" task is proposed, where a model is trained to predict the lifespan of facts. The model is trained using large-scale distant supervision, either as a classification or regression. The trained model then can be used to aid the calibration of QA models by rescaling model confidence using the fact duration error measured by the duration prediction model. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XTm33BqnA5", "reviewer": "Reviewer_TZQm", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review! We agree that the edits you suggested for Table 5 would clarify the comparisons made in our work. We plan to include these edits in our revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem the submission tackles, i.e., temporal misalignment between LM training and use, is highly relevant. Of course, one paper alone will not solve this problem, but I think it makes a valuable contribution.\n\n- The proposed task, fact duration prediction, is well-defined, feasible, and looks like it could have other uses besides those described in the submission.\n\n- While experiments on more than the two models used in the submission would have made a stronger case, the experiments look like a solid proof-of-concept. The different ways of using the temporal misalignment prediction for improving calibration (showcased in the second half of the paper) appear very useful.\n\n- The submission is well-written overall and easy to understand.", "reasons_to_reject": "- none", "questions_to_authors": "", "paper_topic_and_main_contributions": "A problem stemming from the currently dominant way of pretraining language models (LMs) on a data that was collected up to a certain point in time is that the resulting model quickly becomes out-of-date. Prior work has shown that temporal misalignment, i.e., the temporal distance between the training data cutoff and actual model usage correlates with task performance (of course, this applies only to temporally-sensitive tasks).\n\nWithin this context, the submission proposes a new auxiliary task, namely fact duration prediction, and shows that (1) models can be trained to predict the duration of facts expressed in text and (2) that such predictions can be used to improve the calibration of question answering models when dealing with temporally-sensitive facts, such as \"Where will the next Summer Olympics be held?\".\n\nThe basic idea is that one first uses the proposed method to predict the duration of fact like \"The next Summer Olympics are going to be held in Athens\" and then uses that prediction to judge whether there is any temporal misalignment, either with the model's training data (in a closed-book QA setting) or the corpus used for retrieval (in an open-book QA setting). If deemed a temporal misalignment, there are several options, such as reducing the confidence of model predictions post-hoc, or dynamically selecting wether to answer a question closed-book (i.e., no retrieval) or open-book (retrieval). Results show that the prediction of temporal misalignment is accurate enough to improve model calibration when used in this way. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 5 should include scores for systems without adaptive inference (if I understand correctlly, that would be the EM scores in Table 3) so that the reader doesn't have to backtrack in order to understand what is being refered by \"either system on its own\" (lines 529-530):\n\n> We report our results in the first row of Table 5, which shows that this outperforms either system on its own"}}]

Author rebuttal:

Meta-review: This paper addresses an important question that when an LLM is trained with outdated information, it fails to generate correct responses for time-sensitive questions. The paper introduces a factual direction prediction component to determine whether there is a temporal mismatch between the training data and test data. All reviewers appreciated the clarity of the problem space and solution, and solid experiments.
Decision: Accept-Main