Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Authors constructs a new time sensitive QA dataset. They use facts from WikiData for construct this dataset. They take help of crowd workers to filter out noisy facts. Authors claim that this dataset pose QA challenges in terms of both temporal understanding and temporal reasoning. The accuracy using long document QA system FiD for this dataset is far behind human performance.
strengths: - A new  dataset for training and evaluation of time sensitive QA systems.
- Baseline results using SOTA models for QA over long text.
weaknesses: - Manual templates for generating questions after fact extraction and verification 
- Relation distribution is skewed around 2-3 relation, so a simple template based QA baseline might do great as well.
- The dataset is very similar to TORQUE. Long text can be transformed into short text by extracting relevant context from wiki page.
correctness: seems ok.
clarity: Overall paper is well written.
relation to prior work: Yes discussed.
additional feedback: - Probably a comparison with template based QA system is needed.
-How does BigBird and FiD perform on TORQUE/ other timeQA dataset?
review_title: Time sensitive QA dataset based on facts from WikiData

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors describe the lacking aspect of current SOTA NLP models, specifically, the inability to handle questions related to timing and time relations. The paper proposes a two (easy, difficult) Wikipedia-based QA dataset with variable degrees of question complexity to measure this challenging type of reasoning. The authors take extra care to curate the datasets by providing manual labels, performing quality control, down-sampling of the most dominant types of questions, providing human performance, etc. Finally, two strong models are validated as a baseline on these tasks (BigBird and FiD) to prove enough space for future improvement.
Even though the paper's main contribution is focused on temporal reasoning, the dataset may pose interesting challenges due to the fact that the input documents are long.
strengths: The paper is an excellent example of how to describe the dataset construction process, starting from the analysis of the current models to the problem definition and dataset synthesis.
The analysis of results concerning the input length, the type of relation, and output consistency to the input perturbations may help future researchers deeper understand the topic's nuances.

weaknesses: I can only see one minor issue with the dataset: 
1. Authors perform costly manual labeling and quality control, but the underlying text is still sourced from Wikipedia. As there are multiple wiki-based datasets, one can ask how much harder it would be to label the real-world data coming from news articles? As of now, I think this is the main limitation of the work. However, considering that only Wikipedia has a tightly coupled wikidata database, I could also see that this may be the only reasonable approach. Can you motivate the choice of Wikipedia and state that, e.g., sourcing and labeling real-world articles related to temporal reasoning is an order of magnitude more expensive, or that aligning facts with the texts helped to drastically increase the size of the dataset? These questions are not meant to downsize the quality of your work but rather to help establish the position that domain diversity is an essential factor in measuring NLP progress. I believe future readers would appreciate adding a short note to the paper to motivate why Wikipedia was chosen.

correctness: The steps presented in the paper that relate to the construction of the dataset are sound and comprehensive.
clarity: The content of the paper is written clearly and in a concise manner. In addition, the structure of the article is excellent, and the graphics are informative.
relation to prior work: Yes, the relation to prior works is a very extended and informative section.
additional feedback: See weaknesses.
review_title: Interesting idea and great execution

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents TimeQA, a QA dataset that requires models to do temporal understanding and temporal reasoning on free-form texts. It collects time-evolving facts from WikiData, aligns them with Wikipedia documents using both automatic method and human efforts, and synthesizes high-quality temporal questions. The paper also sets up benchmarks on this dataset with SOTA reading comprehension models, and finds that these models face a significant challenge in TimeQA and thus warrants future endeavors. 
strengths: * The paper sets forth a criterion for what truly challenging temporal questions should look like (Section 1 Paragraph 2). Namely, modifying the time constraint should change the answer, and requiring temporal reasoning. The authors then carried out a data collection process that fulfills these requirements. 

* The dataset is collected with strict verification and quality control. 

weaknesses: I’m not sure about the ontology for time-evolving facts proposed in this paper. According to Section 2.1 Paragraph 2 Line 9, the format
	 (subject, relation, {t1→t2:object1,t2→t3:object2, ... tk→tk+1:objectk})
seems to attach time modifiers exclusively to the object, but time should really be a qualifier for the entire subject-relation-object triplet. 

Following the aforementioned asymmetry of temporal ontology, it seems that the questions are synthesized to ask exclusively about the object, given a subject and a time specifier. However, some time-evolving facts may be centered around the object -- for example, consider the following triples:
	({t1→t2:Tang Dynasty,t2→t3:Song Dynasty, ... tk→tk+1:Qing Dynasty}, rules, China)
And we should be able to ask questions about the subject, like “Which dynasty ruled China from t_i to t_(i+1)?” And likewise, we also might want to ask about the time specifier, such as “The Tang Dynasty ruled China between which years?” It seems that the question synthesis schema proposed in this paper lacks the ability to generate these types of questions, thereby limiting the diversity and coverage of the TimeQA dataset. 

On another note, there lacks a discussion on how the granularity of time is handled, which is a crucial topic when working with temporal reasoning. For truly advanced temporal reasoning, coarse-grained and fine-grained timestamps can be mixed in the document and the question, and the dataset should reflect that. An incomplete list of question types I can think of: 
* The time in the question can be more coarse-grained than in the document/fact. For example, the document may tell you “Alice played for TeamA from April 2020 to August 2020”, and the question asks “Which team did Alice play for in 2020?”
* Vice versa, The time in the question can be more fine-grained than in the document/fact. For example, the document may tell you “Alice played for TeamA in 2020”, and the question asks “Which team did Alice play for between April 2020 and August 2020?” In this case, it might be insufficient to only extract an answer span from the document, as the document’s fact has more uncertainty; the model will probably need to re-interpret the query so that it can provide grounded answers, e.g. “I know that Alice played for TeamA for at least some part of 2020.”
The WikiData likely annotates time mostly with the granularity of year, month, or day, but it’s worth to include time with other granularities (e.g. century, hour, minute) as part of the dataset, so it has more comprehensive coverage. 

It is fine if handling granularity is beyond the scope of this project. But still it is worth noting your assumptions about what granularity you choose to use, and mentioning this as a limitation/future work. 

correctness: In Section 2.1, why did the number of (text, time-evolving fact) pairs grow after human verification? You say you provide 5K such pairs to AMT workers but finally yield 5060 pairs. 

In Section 3.2 Paragraph 1, there seems to be a discrepancy in the description of FiD model. Equation (3) says T5 encodes $\hat{P_i}$ into $R_i$. If so, then Line 5 should probably read “$\hat{P_i}$ is fed to the T5 encoder …” and “$R_i \in R^{(K+M) \times D}$” because the length of $Q$ and $P_i$ concatenated is $(K+M)$. Or is it the other way around -- T5 actually encodes $P_i$ only -- but then how would the question $Q$ be taken into account? 

clarity: The paper is mostly clear and easy to follow. 

The criteria for classifying proposed questions as “easy” and “hard” is not clear to me. (Section 2.2 Paragraph 2) Does the difficulty level depend on the particular instantiation of the time specifier? For example, for the same question in Figure 3, “in June 1778” would be “easy” but “in July 1776” would be “hard”?
relation to prior work: Related works are thoroughly discussed. Existing work on MRC/QA does not pose real temporal reasoning challenges; whereas most of the temporal reasoning and enhancement techniques are developed around KG instead of unstructured texts. This work fills in the gap by testing deep temporal reasoning in the unstructured text setting. 
additional feedback: Your submission pdf may have been rendered in camera-ready mode and does not contain line numbers, so it’s hard to locate exact lines when communicating detailed feedback.
review_title: Good dataset, but with a bit limited diversity and coverage

Meta-review: This paper presents a very interesting dataset for time-sensitive reasoning in question answering. All of the reviews are positive, and my reading was also positive. The community, especially those who are interested in NLP and QA, would be quite fascinated to hear about this dataset and get their hands on it.
Decision: Accept