Multi-granularity Temporal Question Answering over Knowledge Graphs
Year: 2023

Abstract:
Recently, question answering over temporal knowledge graphs (i.e., TKGQA) has been introduced and investigated, in quest of reasoning about dynamic factual knowledge. To foster research on TKGQA, a few datasets have been curated (e.g., CRONQUESTIONS and ComplexCRONQUESTIONS), and various models have been proposed based on these datasets. Nevertheless, existing efforts overlook the fact that real-life applications of TKGQA also tend to be complex in temporal granularity, i.e., the questions may concern mixed temporal granularities (e.g., both day and month). To overcome the limitation, in this paper, we motivate the notion of multi-granularity temporal question answering over knowledge graphs and present a largescale dataset for multi-granularity TKGQA, namely MULTITQ. To the best of our knowledge, MULTITQ is among the first of its kind, and compared with existing datasets on TKGQA, MULTITQ features at least two desirable aspects—ample relevant facts and multiple temporal granularities. It is expected to better reflect real-world challenges, and serve as a test bed for TKGQA models. In addition, we propose a competing baseline MultiQA over MULTITQ, which is experimentally demonstrated to be effective in dealing with TKGQA. The data and code are released at https: //github.com/czy1999/MultiTQ.

heading: 1 Introduction
text: In real-life applications factual knowledge is apt to evolve over time (Nonaka et al., 2000; Roddick and Spiliopoulou, 2002; Hoffart et al., 2011; Gottschalk and Demidova, 2018); for instance, The host city of the Winter Olympic Games in 2018 was South Korea, while in 2022 it was Beijing.In this connection, there is a current trend to investigate knowledge graphs (KGs) involving time, and these KGs are coined as temporal knowledge graphs (TKGs). In a
∗Corresponding author.
TKG, fact triplets are equipped with temporal information (e.g., timestamps), and a temporal fact can be stated in the form like “(Beijing, held, Winter Olympic Games, 2022)”.
To exploit the value of TKGs, recent research effort has been devoted to process natural language questions over TKG, i.e., question answering over TKG (TKGQA in short) (Saxena et al., 2021). Given a question and a background TKG, it retrieves from the TKG an answer to the question. To foster research on TKGQA, several datasets have been introduced, among which CRONQUESTIONS (Saxena et al., 2021) is by far the largest. We explain the task with a sample question in CRONQUESTIONS.
Example 1 In the upper part of Figure 1, the agent is supplied with the question “What award did Carlo Taverna receive in 1863?” as well as a TKG. By considering the semantic relevance of the facts, the agent locates the candidate fact “(Carlo Taverna, receive, Order of Saints Maurice and Lazarus, 1863)”, and hence, Order of Saints Maurice and Lazarus is returned as the answer.
Specifically, CRONQUESTIONS comprises 410k temporal questions, each of them has a temporal constraint, e.g., “in 1863” in the example above. Albeit large scale, the questions in CRONQUESTIONS tend to be “pseudo-temporal” (Chen et al., 2022). By looking into the construction of CRON-
11378
QUESTIONS, we find that most of the questions are related to, respectively, only one fact, which can be well located without enforcing the temporal constraint in the question; for example, Carlo Taverna only received one award, which was Order of Saints Maurice and Lazarus. In this case, the temporal constraint does not further restrict the candidate facts to answering the question, and the question is essentially atemporal in the context of the given KG. Moreover, in CRONQUESTIONS, questions and the TKG are designed to be both described in the temporal granularity of year. This simplification, however, is less practical, since questions and knowledge in the real world are not limited to the time frame of years. For instance, as shown in the lower part of Figure 1, the agent is likely to be given a question in the granularity of month, which is common in the real world. In short, these two important aspects are not well attended by existing TKGQA datasets, which thus may be insufficient in evaluating TKGQA models.
In this research, we are motivated to address the shortcomings by presenting a new dataset for TKGQA, namely MULTITQ. MULTITQ is a largescale dataset featuring ample relevant facts and multiple temporal granularities (comparison of statistics in Table 1). To avoid the pseudo-temporal issue, we intentionally generate temporal questions that are relevant to more than one fact triplet, such that the temporal constraint is always necessary to correctly locate the answer. This characteristic is of importance to evaluating TKGQA models, since temporal reasoning is a unique challenge arising out of the task. Further, MULTITQ features multiple temporal granularities, which is largely overlooked by existing datasets. We resort to a template-based question generation method, which automatically constructs question templates (and hence questions) of multiple temporal granularities. In this way, MULTITQ is expected to serve as a test bed for evaluating TKGQA models, especially in reasoning with temporal constraints and coordinating between temporal granularities. In addition, to provide a competing baseline on MULTITQ, we propose a transformer-based model for multi-granularity TKGQA, namely MultiQA.
In summary, our contribution is three-fold:
• To the best of our knowledge, we are among the first to elicit the notion and motivate the challenges of multi-granularity TKGQA.
• We present a multi-granularity TKGQA
dataset MULTITQ. Besides multiple temporal granularities, the dataset is also prominent in its large scale with ample relevant facts regarding each questions therein.
• We propose MultiQA, a strong baseline to handle multi-granularity TKGQA, the performance of which is demonstrated by comprehensive experiments on MULTITQ.

heading: 2 Related Work

heading: 2.1 Datasets for TKGQA
text: TEMPQUESTIONS (Jia et al., 2018a) is one of the first publicly available TKGQA datasets consisting of 1,271 questions. SYGMA (Neelam et al., 2021) introduced a subset of TEMPQUESTIONS that can be answered over Wikidata called TEMPQAWD. Previous collections on temporal questions contain only about a thousand questions and are not suitable for building neural models. TIMEQUESTIONS (Jia et al., 2021) searches through eight datasets of question answering over conventional KGs for time-related questions and contains 16k questions. CRONQUESTIONS (Saxena et al., 2021) is another TKGQA dataset that uses its KG drawn from Wikidata, which comprises a total of 410k questions. While it alleviates problem of incomplete learning of large models due to small amount of data, CRONQUESTIONS contains a large number of pseudo-temporal questions (Chen et al., 2022). This reduces the applicability of CRONQUESTIONS for evaluating the temporal reasoning capability of TKGQA models.
Since these datasets focus on single-time granularity, consistent with the KGs, they do not reflect the real-world challenges of multi-granularity temporal question answering. It motivates us to close the gap by presenting a novel dataset for TKGQA.

heading: 2.2 TKGQA Models
text: There are two streams of approaches to tackle TKGQA. The first decomposes the original question into several non-temporal questions and time constraints. Then models designed for question answering over conventional KGs are applied to answer these questions, and time constraints finally compare and select the most proper answer, e.g., TEQUILA (Jia et al., 2018b). However, this approach needs handcrafted decomposition rules and cannot cope with complex questions (Jia et al., 2021).
The methods in the second stream try to acquire TKG embedding to calculate the semantic similarities for the answer determination. CronKGQA (Saxena et al., 2021) provides a learnable reasoning process for TKGQA, which does not rely on handcrafted rules. Although CronKGQA performs well in answering simple questions, it fails to solve complex questions requiring inference of certain time constraints. TempoQR (Mavromatis et al., 2021) introduces time scope information for each question and employs EaE method (Févry et al., 2020) to enhance the semantic information of the question representation.
However, limited by the single granularity of available datasets, none of these methods have considered the multi-granularity problem, making them lacking in real-world applications. In this paper, we address the challenges by proposing multigranularity temporal QA methods, MultiQA.

heading: 2.3 Analysis for Temporal Questions
text: It is noted that temporality also gains attention in community question answering (CQA) and multimodal question answering (MQA).
Models (Duan et al., 2018; Wu et al., 2017; Zhang et al., 2020) and datasets (Pal et al., 2012; Figueroa, 2010; Figueroa et al., 2016, 2019) for temporal community questions are emerging in recent years. There are two viewpoints for temporality across CQA sites: 1) a measure of the usefulness of the answers (Pal et al., 2012), and 2) the recurrent attention given to questions during different time-frames (Figueroa et al., 2016). Based on these two viewpoints, a new set of time-frame specific categories are proposed (Figueroa et al., 2019).
In the field of multimodal question answering, a series of temporal question answering datasets integrating audio and video have been proposed (Lei et al., 2020; Fayek and Johnson, 2020; Jang et al., 2017). Techniques such as spatio-temporal attention (Jang et al., 2017), motion-appearance memory (Gao et al., 2018), spatio-temporal grounded audio-visual network (Li et al., 2022) and spatiotemporal graph models (Cherian et al., 2022) have been proposed and demonstrated their effectiveness on different VideoQA and AudioQA datasets.

heading: 3 The MULTITQ Dataset
text: MULTITQ is a new complex temporal question answering dataset with multi-granularity temporal information. Compared to existing datasets,
our dataset features in a few advantages, including large scale, ample relations and multiple temporal granularity, which hence better reflects real-world scenarios, as shown in Table 1.

heading: 3.1 Analysis of KG
text: Most TKGQA datasets use Wikidata as the KG. However, Wikidata suffers from relationsparsity problem. Specifically, for each entity in the KG, the number of relation types involved is fairly homogeneous. We define semantic complexity.
Definition 1 For a TKG K := (E ,R, T ,F), where E ,R, T denote entities, relations, and timestamps respectively. KG semantic complexity SCK is defined as the average of the number of relation types involved in each entity:
SCK = 1 |E| ∑
ei∈E N
rtype ei , (1)
where N rtypeei is the number of relation types involved in ei.
A larger SCK indicates a richer KG K in terms of relation semantic information. For example, CRONQUESTIONS uses a subset of Wikidata as the KG, but the SC value of this KG is only 1.32, i.e., an average of 1.32 types of relation per entity, which is more descriptive, with over 80% of entities having only one relation type and 99% of entities having no more than two relation types. Thus, even though it contains a rich number of entities and relations, the KG of CRONQUESTIONS is fairly sparse at semantic level.
Unlike previous datasets, we take ICEWS05-15 (García-Durán et al., 2018), a subset from the Integrated Crisis Early Warning System (ICEWS) database, as the KG for MULTITQ. ICEWS captures and processes millions of pieces of data from digital news media, social media and other sources, with a wealth of dynamic semantic information that provides an adequate KG for temporal question answering. As shown in Table 2, ICEWS05-15 is rich in semantic information with SC value 7.05. The richness of relation types makes it more in line with real-life scenarios.

heading: 3.2 Question Construction
text: Following CRONQUESTIONS, we filter through ICEWS05-15 to find 22 most frequent relations to build templates and generate questions.
Firstly, 246 unique core templates are constructed by five experts in social computing based on the 22 most frequently occurring relations. Taking the relation ‘make a visit’ as an example, human experts have constructed several core templates based on their expert knowledge, e.g., “Who first visited {tail}”. Next, the core template will be expanded by the question category (cf. Section 3.2.2). Time constraints and multi-granularity temporal information are added to the core template, as shown in Table 3, enriching and diversifying the semantics of templates. Finally, we ended up with 7,334 templates. Each of these templates has a corresponding procedure that could be executed over the TKG to extract all possible answers for that template. These templates were then filled using entity aliases from ICEWS to generate 500k unique question-answer pairs.

heading: 3.2.1 Multi-Granularity Temporal Questions
text: Time is naturally multi-granular, a property that previous models have ignored. The motivation for proposing Multi-Granularity Temporal Questions is to drive the attention of temporal questions rea-
1This abnormality is brought by some science fiction-type knowledge, and some erroneous time information.
soning on multi-granular time. ICEWS provides time information at a day granularity, which allows us to generate higher granularity information, such as year and month granularity. Questions in MULTITQ contain three temporal granularities, i.e., day, month and year. In order to generate multi-granular time information, we have designed a time generation module that can randomly generate different formats and types of year-month-day granularity time expressions from the daily granularity time according to syntactic criteria, as shown in Figure 2, effectively increasing the variety and complexity of the question texts.

heading: 3.2.2 Question Categorization
text: To make the problem more challenging, we propose the concept of multiple temporal reasoning questions, where there are multiple temporal constraint words in one question and the QA model needs multiple complex reasoning to obtain final. We categorize questions into “Single questions” and “Multiple questions”. Please refer Table 5 for examples of these questions.
Single questions. These questions contain a single temporal constraint, where the answer can be either an entity or a time instance. For example, the question “Who visited the United States in 2008?” requires a single temporal constraint to answer the question, namely Equal. In our dataset, single questions are further categorized into three types: Equal, Before/After and First/Last.
Multiple questions. These questions contain multiple temporal constraints to answer and can be more varied. For example “Which country first visited United States in 2015?” This requires reasoning over multiple temporal constraints, including Equal and First. In our dataset, Multiple questions are further categorized into three types: Equal multi, Before last and After first.

heading: 3.2.3 Question Filtering and Splitting
text: We follow CRONQUESTIONS and ensure that there is no entity overlap between train questions and test questions. This policy ensures that models are doing temporal reasoning rather than guessing from entities seen during training. Specifically, we split the ICEWS05-15 into train/dev/test folds without entity overlap, and then perform question generation protocol on each divided TKG.
Automatic question generation via templates may lead to some questions with low quality, including pseudo-temporal questions and nonsensical questions. To compensate for these shortcomings, we follow Chen et al. (2022) and eliminate all pseudo-temporal questions, making the dataset more challenging. Furthermore, due to the factual sparsity of the KG, automatic generation through templates may result in questions such as "Who visited the United States in 2005?" where there may be hundreds of answers. To avoid this, we eliminate questions with more than 20 answers to ensure that the questions in the dataset are of practical importance.
Finally, we get train/dev/test folds with a ratio of roughly 8:1:1, and 500k questions in total. Dataset statistics are shown in Table 4. We believe that providing entity and time annotations directly would significantly affect the performance of the model, reducing reasoning on simple questions to a KG query task. Therefore, we do not provide corresponding entity and time annotations in our dataset. Summarizing, each of our examples contains a natural language temporal question and a set of ‘gold’ answers (entity or time).

heading: 3.3 Statistics of MULTITQ
text: We summarize the number of questions in MULTITQ across different types in Table 4, and the core statistics of each split in Table 6. In Table 5, we present sample questions from MULTITQ as per question type, time granularity and answer type.
Overall, the resulting MULTITQ dataset contains 500k questions from 22 relations (More statistic are listed in Appendix A.3). In Figure 3, we show how questions in our benchmark are distributed by length (in words), and contrast this with CRONQUESTIONS and TEMPQUESTIONS. Questions in our benchmark are between 4 and 35 words long, and the average question length is 13.01 words. The figure shows that a good proportion of questions in MULTITQ are relatively verbose, implying increased parsing difficulty for QA systems.
4 The MultiQA Model
We overview the model architecture in Figure 4.

heading: 4.1 Question Pre-processing
text: To obtain the entity and time information in the question, we use named entity recognition (NER) and time extraction tools to enable the model to more accurately access the information in the KG by aligning it with the entities and times in the KG (See Appendix A.2).
We obtain the semantic information of questions by a pre-trained language model. Specifically, the natural language form of the question qtext is transformed into a semantic matrix QR by the pretrained RoBERTa (Liu et al., 2019).
QR = WR RoBERTa (qtext) , (2)
where QR = [qCLS,qR1 , ...,qRN ] is a D × L embedding matrix. L is the number of tokens and D is the dimensions of the TKG embeddings. WR is a D ×Droberta projection matrix where Droberta is the dimension of the RoBERTa embeddings. The finial question representation q = qCLS.

heading: 4.2 Multi-Granularity Time Aggregation
text: As the facts provided by the KG are all at day granularity, e.g., 2008-03-19, the TKG embedding thus trained contains only semantic information at day granularity (See Appendix A.1). However, the question contains reasoning about year and month granularity, and no semantic information can be obtained directly from the pre-trained TKG embeddings. To solve this problem and obtain time embeddings at a coarser granularity, we propose a multi-granularity time aggregation module. Taking the example of month granularity time aggregation, we want to aggregate all related day information to get that of month granularity in the question.
Specifically, For the month granularity time m in the question, we first extract all contained day timestamps d1, d2, ..., dN and their TKG embeddings td1 , td2 , ..., tdN , which are rich in temporal
information. N is the number of related days. To obtain the time representation at month granularity, we construct the temporal semantic matrix Td,
Td = [td1 , td2 , ..., tdN ], (3)
where Td ∈ RN×D is a matrix containing all day embeddings for month m.
Time as an ordering sequence has an inherent similarity to the positions of words in the text, so we enrich its sequential property by employing a sinusoidal position encoding method (Vaswani et al., 2017; Jia et al., 2021). Here, the kth position in Td will be encoded as:
PE(k, j) =    sin ( k/10000 2i D ) , if j = 2i cos ( k/10000 2i D ) , if j = 2i+ 1 (4)
where j is the (even/odd) position in the Ddimensional vector. Further, we get T′d by adding positional embedding to Td. Adding positional embedding ensures sequential ordering among the timestamps, which is vital for reasoning signals like before and after in temporal questions.
Next, we propose an information fusion layer to fuse the information into a single time representation tm. Following Févry et al. (2020), we use an information fusion layer that consists of a dedicated learnable encoder Transformer(·) which consists of 2 Transformer encoding layers (Vaswani et al., 2017). This encoder allows the time tokens to attend each other, which fuses all days’ embeddings into a single month embedding. The final token embedding matrix Tm is calculated as
Tm = Transformer(T ′ d), (5)
where Tm = [tCLS, tm1 , ..., tmN ], and the finial question representation tm = tCLS.
Repeating the aggregation, we obtain a time representation of year granularity ty. The final time representation is tτ for the question at τ .

heading: 4.3 Answer Scoring Module
text: Finally, we get the scores of the candidate answers, consisting of all entities and timestamps,
max (ϕ (es,Weq, eϵ, tτ ) ,
ϕ (eo,Weq, eϵ, tτ )) +⃝ϕ (es,Wtq, eo, tτ ) . (6)
where s, o and τ are the annotated subject, object and timestamp, respectively. ϵ represents candidate answers (all entities in the TKG). We and Wt are D ×D learnable matrix specific for entity predictions and time predictions respectively. ϕ denotes
the score function in TComplEx (Lacroix et al., 2020). We treat the annotated subject and object interchangeably, and max(·) function ensures that we ignore the scores when s or o is a dummy entity.
During training, softmax is used to calculate probabilities over this combined score vector, and cross-entropy loss is employed.

heading: 5 Experiments
text: We experimentally evaluates MultiQA against five baselines. In the interest of space, experiment settings are in Appendix A.6.

heading: 5.1 Baseline Methods
text: • Pre-trained LMs: To evaluate BERT (Devlin et al., 2019), DistillBERT (Sanh et al., 2019) and ALBERT (Lan et al., 2020), we generate their LM-based question embedding and concatenate it with the entity and time embeddings, followed by a learnable projection. The resulted embedding is scored against all entities and timestamps via dot-product.
• EmbedKGQA (Saxena et al., 2020) is designed with static KGs. To deal with multiple temporal granularities, timestamps are ignored during pre-training and random time embeddings are used.
• CronKGQA (Saxena et al., 2021) is designed for single temporal granularity. To deal with multiple granularities, time embeddings at the
year/month granularity are drawn at random from corresponding day embeddings.

heading: 5.2 Overall Results
text: Table 7 shows the results of our method compared to other baselines on MULTITQ. First, by comparing EmbedKGQA to pre-trained LMs (BERT, DistillBERT, ALBERT), we see that introducing KG representations with score function significantly improves the model’s reasoning ability, even without providing any temporal information. We hypothesize that this is because KG embeddings specific to the TKG helps the model to focus on those entities.
Since EmbedKGQA has non-temporal embeddings, its performance on questions where the answer is a time is very low. By comparing CronKGQA to EmbedKGQA, we see that introducing a pre-trained time representation it refers significantly helps in answering temporal questions. In this case, the absolute improvement for all questions is 7% and 15% at Hits@1 and Hits@10, respectively. Further, we see the benefit of multi-granular time aggregation to the question, which effectively improves the inference on multi-granularity temporal questions (cf. Section 5.3). The absolute improvement of MultiQA over CronKGQA is 1% at Hits@1.
With the results of the paired t-test, we find that the MultiQA outperforms the best baseline significantly in most tasks, which demonstrates that multi-granular time aggregation is an effec-
tive solution to improve the overall performance of multi-granularity TKGQA. As shown in Figure 5, MultiQA achieves a better performance than CronKGQA in most categories of questions.

heading: 5.3 Results on Multi-Granular Time
text: To verify the effectiveness of the model on multigranularity temporal reasoning, we experiment on multi-granularity temporal questions.
First, by comparing CronKGQA to EmbedKGQA, paradoxically, CronKGQA, while outperforming EmbedKGQA in overall results, is rather less effective at multi-granularity temporal reasoning. We argue that CronKGQA’s introduction of a single granularity time representation improves inference at the corresponding time granularity, but
misleads inference at the other granularities, causing the results to fall instead on multi-granularity TKGQA. This also highlights previous models’ lack of inference capability for multi-granularity temporal question answering.
In addition, due to the multi-granularity aggregation module, MultiQA improves significantly at month and year granularity. Specifically, it outperforms by 7% at month and year granularity on before/after types, respectively. Similar pattern is also observed on the other types.

heading: 5.4 Effect of Training Dataset Size
text: Although the dataset is constructed from templates and is semantically narrowed, the large
dataset is still effective in improving model effectiveness. Figure 6 shows the effect of training dataset size on model performance. As we can see, for MultiQA, increasing the training dataset size from 10% to 100% steadily increases its performance for both single and Multiple reasoning type questions. We hypothesize that this is because the large number of entities and facts in the KG and the large number of model trainable parameters. These results affirm the hypothesis that having a large, even if synthetic, the dataset is useful for training temporal reasoning models (Saxena et al., 2021).

heading: 6 Conclusion and Limitation
text: In this paper, we introduce the concept of multigranularity temporal question answering and construct a benchmark dataset MULTITQ, which features ample relevant facts and multiple temporal granularities. We also propose a multi-granularity temporal question Answering model MultiQA, serving as a strong baseline for follow-up research.
Limitation. The main drawback of our data creation protocol is that the question/answer pairs were generated automatically, leading the question distribution to be artificial from a semantic perspective. In addition, the KG adopted in the research focuses on a single event domain, and extending the dataset to multiple domains is planned as future work.

heading: Acknowledgement
text: The authors would like to thank the anonymous reviewers for their insightful and constructive comments, which greatly contributed to improving the quality of the paper. This work was partially supported by National Key R&D Program of China No. 2020AAA0108800, NSFC under grants Nos. 62272469 and 71971212, and The Science and Technology Innovation Program of Hunan Province under grant No. CX20220067.

heading: A Appendix
text: A.1 TKG Embeddings A TKG K := (E ,R, T ,F) is a multi-relational directed graph with time-stamped edges between entities. A fact in K can be formalized as (s, r, o, τ) ∈ F , where s, o ∈ E denote the subject and object entities, r ∈ R denotes the relation between them, and τ ∈ T is the timestamp associated with that relation. TKG embedding methods learn a K-dimensional vector eϵ,vr, tτ ∈ RK of each ϵ ∈ E , r ∈ R and τ ∈ T in K, such that each fact (s, r, o, τ) ∈ F has a higher score than the one (s′, r′, o′, τ ′) /∈ F through a scoring function ϕ(·), formally ϕ (es,vr, eo, tτ ) > ϕ (es′ ,vr′ , eo′ , tτ ′) . TComplEx (Lacroix et al., 2020) is an extension of ComplEx (Trouillon et al., 2016) considering time information, which encodes each entity, relation and timestamp to complex vector. The score function ϕ(·) of TComplEx is defined by
ϕ (es,vr, eo, tτ ) = Re (⟨es,vr ⊙ tτ , eo⟩) , (7)
where Re(·) denotes the real part, (·) is the complex conjugate of the embedding vector and ⊙ is the element-wise product.
We train TComplEx on ICEWS05-15 with the TKG completion task. We learn the entity and relation representations in the complex space Cd, where d denotes the dimension of the complex vectors.
A.2 Entity and Time Retrieval Unlike previous QA datasets, our dataset does not contain entities and time annotations, so the only information the QA model can use is the text of the questions and the corresponding KG information. This is also in line with the TKGQA task in a practical application scenario. Due to the lack of entity linking tools for the ICEWS, we first used a pre-trained generic NER tool (Schweter and Akbik, 2020) to extract the question text, filter out the entity names in it, and then match it with the entities in the KG through fuzzy matching to find the most similar entity as the entity result for subsequent inference.
Qentity = FuzzyMatch (Qner, E) , (8)
where Qner is the list of identified entities, and we fuzzy match the identified entities with the entities E in the KG by calculating the similarity,
Sim(e1, e2) = 2 · Me1e2
Le1 + Le2 , (9)
where Le is the text length of entity e, Me1e2 is the the maximum length that can be matched between e1 and e2. Entity with highest similarity in E will be added to entity linking set Qentity.
As the expression of time is more fixed, we adopt a rule-based method to extract time information from the question text for subsequent reasoning. Specifically, a series of regular expressions based on common time formats have written to extract the corresponding time information in the question.
A.3 More Statistics of MULTITQ We summarize the statistics of different time granularities in Table 9 and distribution of relations in Figure 7 in MULTITQ.
A.4 Results on Single-Granularity Time We conduct an additional experiment to analyze the behavior of baseline models on single-time granularity datasets. We partition MULTITQ by time
granularity, ensuring that there is only single granularity of time in each divided dataset (Day, Month, and Year). At the setting of single-day, since the temporal granularity of KG coincides with that of the dataset, our model degenerates to CronKGQA. From the experiment results at a single granularity, even if the time granularity of the KG is kept consistent with that of the questions, the existing model still struggles to achieve excellent results as that on previous datasets (e.g., CRONQUESTIONS), mainly because our proposed dataset has more complex question types and KG with higher semantic complexity, which hence better reflects real-world scenarios.
As observed in Table 10, we can see that the introduction of time information on fine-grained questions can significantly improve the performance of the temporal QA system. Consistent with the observations on the multi-granularity experiments discussed in Section 5.3, MultiQA is able to achieve substantially improved performance at coarse-grained timescales thanks to the multigranularity time aggregation module. This further validates the efficacy of this module.
A.5 Error Analysis For error analysis, we randomly sample 100 error instances from the test set and summarized the following three types of typical errors: (1) Retrieving irrelevant entities , meaning the model obtained wrong entities from the KG; Although our entity linking model can achieve a high prediction accuracy, wrong entities still exist in some questions. (2) Wrong reasoning at the semantic level, meaning the model failed to obtain the entities related to the semantics of the question. Limited by the representation of the question and the reasoning ability, even when the time constraint is not taken into account,
there are still cases where the reasoning yields irrelevant entities or times. Such a phenomenon is especially common in complex questions. (3) Lacking the ability of reasoning about complex temporal constraints, meaning the model design cannot support complex temporal constraints. The inference ability of MultiQA comes from the complementary inference ability obtained in the pre-training of TKG Embedding, which is limited to simple temporal inference. This prevents our model from achieving efficient reasoning about complex constraints such as First, before, etc.
This demonstrates more efforts are needed to strengthen the model’s reasoning capability, especially in semantic reasoning and complex temporal constraints reasoning. Also, using more advanced NEL models would be an effective direction for enhancement.
A.6 Reproducibility In this section, we report more experimental details to ensure the reproducibility of this paper.
The model is implemented with PyTorch (Paszke et al., 2019). We use TComplEx (Lacroix et al., 2020) as our TKG embeddings, and their dimensions D = 512. We use BERT-base, DistillBERT-base and ALBERT-base in our implementation. Both LM’s parameters and the TKG embeddings are not updated during the training. We set the number of transformer layers of the encoder Transformer(·) to l = 2 with 4 heads per layer. The model’s parameters are updated with Adam (Kingma and Ba, 2015) with a learning rate of 0.0002. All the experiments are conducted on a server that has an Intel(R) Core(TM) i9-10900K@3.70GHz CPU and a 24-GB Nvidia RTX 3090 GPU. The operating system is Ubuntu 20.04. More details about the implementation, e.g., dependency libraries, can be found in the README file of the software.
In addition, our model has about 195M parameters. And the average training time is 2.5h.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 6 Conclusion and Limitation

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 6 Conclusion and Limitation

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1 introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: ICEWS dataset has been sanctioned by the U.S. Government for public release.and is allowed for science research[1].ICEWS is open with the CCO License allowing free access[2]. [1] https://www.lockheedmartin.com/en-us/news/features/2016/ICEWs-10000-dataset-download.html [2] https://dataverse.harvard.edu/dataverse/icews

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? ICEWS is open with the CCO License allowing free access[1]. [1] https://dataverse.harvard.edu/dataverse/icews
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The ICEWS dataset contains information on significant events such as national institutions and does not involve personal private information.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3 and Appendix A.3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 and Appendix A.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.6

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A.6

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.6
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In addition, the KG adopted in the research focuses on a single event domain, and extending the dataset to multiple domains is planned as future work