Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Strengths:
- Clear, well explained motivations
- Positive results compared to baseline RoBERTa methods
- Ablation study to show the benefits of the proposed model

Weaknesses
- Gains are not huge for EM and F1, especially for the test set

This paper presents an intuitive technique with positive results, I am a bit concerned that the observed gains might be due to noise/randomness, but at least the consistency score and low-data results seems to have been improved by a substantial margin.

Question:
- Were the results averaged over multiple trials? It would help substantially to show averaged results for at least the validation set, especially since some of the margins are small
- Are these results statistically significant given the size of the test/validation sets?
-  Is the MLP from equation 7 used at test time? If not, it seems like that would limit the benefit of the contrastive loss. Is there a way to share parameters between the MLP in the contrastive loss and the logistic regression model in the event detection loss so the contrastive loss more directly effects the model's output?

Presentation:
- Equation 2: Should there be commas between the h and i in the W subscript?
- Table 2: 50.9 is bolded even though it is not the highest score in that column, is that intentional?
- Equation 8: “is a binary event indicator (1 or 0) if” -> “is a binary event indicator (1 or 0) and”
review_title: Intuitive method with positive results.

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**
Authors introduce a temporal question answering (TQA) model called OTR-QA, which decomposes a given question into a question event and a temporal relation. Their main contribution is to learn open temporal relations that do not require a predefined set of relations. The architecture is simple and trained with multiple hyperplanes. Answer event prediction loss is further assisted by two auxiliary losses (event detection loss and contrastive loss). Problem definition is straightforward (i.e., extract all the answer entities given the question entity and the relation) and the model is evaluated on TORQUE (Ning et al., 2020). Compared to the RoBERTa-large baseline by Ning et al., 2020, OTR-QA achieves +1.1% F1, -0.2% EM, and +3% C score (consistency). Authors also show that their model is more data-efficient and modeling the open temporal relations is indeed the key to achieve strong performance.

**Strengths**
* The paper is well-structured and easy to follow.
* The ablation study in Table 3 is helpful to understand why their model works well.
* Their model shows significant improvement over RoBERTA-base in the low-resource setting.

**Weaknesses**
* The performance of the RoBERTa-large baseline in the TORQUE leaderboard is (F1=0.7562, EM=0.5076, C=0.3691), which largely closes the gap with OTR-QA, especially in C score. I understand that authors reported the RoBERT-large performance from Ning et al., 2020, but it would be nice if authors can explain why this happens and report the performance of OTR-QA evaluated on the official leaderboard.
* Their main result (Table 3) might have omitted comparison with previous works on TORQUE (e.g., DEER: A Data Efficient Language Model for Event Temporal Reasoning. Han et al., 2020) but only includes the baselines provided by Ning et al., 2020. It would be better to include them to put their numbers in context.
* In Figure 3, it is not very clear that OTR-QA has better representations of the relations and the color scheme is also confusing.

Overall, I find the paper interesting and clear, but the improvement over previous models seems marginal. It would be nice if authors can provide clarification on these weaknesses.
review_title: Solid study on open temporal relations for QA, but the empirical result seems marginal

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper reformulates TORQUE questions (temporal-ordering-focused reading comprehension questions)
as open temporal relation extraction.  Questions are decomposed as (source event, relation, ?)
queries, where the ? corresponds to another event in the passage.  After this decomposition, events
from the passage and the extracted relation are modeled in a way similar to knowledge base
completion, using multiple hyperplane projections.  An additional contrastive loss is added, which
basically amounts to normalizing event answer scores across contrasting questions, when it is known
a priori that an event has exactly one answer among the set of questions.  Experimental results show
that this model outperforms prior work on TORQUE, and good ablations are done to show the impact of
various modeling decisions that were made.

Overall, I think the paper presents a straightforward but novel idea, executes this idea well, and
evaluates it thoroughly.  I think the ideas and experiments here are interesting and timely.  The
results with a contrastive loss and open relation modeling add to the body of evidence of the
utility of these methods generally.  There are a few things that I would recommend improving in the
paper, but it's already useful as it is.


More detailed comments:

Related work on contrastive losses is missing.  Asai and Hajishirzi, 2020, "Logic- guided data
augmentation and regularization for consistent question answering", is a good starting place for
finding related work in this area.  The question-conditional contrastive estimation method from
Dheeru Dua et al. 2021, "Learning with Instance Bundles for Reading Comprehension" seems like it is
basically identical to the loss presented in this paper (this second one is a recent enough paper
that a comparison is not expected, and a missing citation is not surprising, I'm just providing it
FYI).

Do you know what percentage of the training data is not well captured by an (event, relation, ?)
query?  That is, how often does your decomposition fail?

3.2.1 - it's not clear what exactly is being averaged to compute v_q - all of the words in the
question?  Just the event word in the question?  The event word in the question and the
corresponding event word(s) in the passage?

Using a fixed set of temporal relations is a nice ablation, I like including that experiment quite a
bit.

Section 4.4 - t-SNE is generally not very informative, and I disagree with the text in this section
that says that the OTR model has a better clustering than the baseline.  It does not seem nearly so
clear cut to me as is stated in the text.  I think this section would be better to just be removed
from the paper.  It doesn't really add anything that's scientifically useful.
review_title: Review

Meta-review: The paper proposes to reformulate temporal question answering as open temporal relation extraction task. All reviewers agree that this is a fine paper which executes the idea well and evaluates it thoroughly, including ablation experiments. A good and timely piece of work. The approach works well for temporal QA without the need of explicit timestamps: the experiments show that the decomposition generalize better and models better different temporal relations with the contrastive loss.


Decision: Accept