TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.
PROBLEM: Because our approach is specifically intended to yield sentences that are grammatical, we additionally consider the following log ratio (i.e., the grammatical phrase over the ungrammatical phrase): 
 Question: How do they measure grammaticality?

SOLUTION: by calculating log ratio of grammatical phrase over ungrammatical phrase

PROBLEM: A meta-research study found only 1 in 9 qualitative papers in Human-Computer Interaction reported inter-rater reliability metrics BIBREF31. Another related area are meta-research and methods papers focused on identifying or preventing low-effort responses from crowdworkers — sometimes called “spam” or “random” responses, or alternatively ”fraudsters” or ”cheaters.” Rates of “self-agreement” are often used, determining if the same person labels the same item differently at a later stage. 
 Question: What are the key issues around whether the gold standard data produced in such an annotation is reliable? 

SOLUTION:  only 1 in 9 qualitative papers in Human-Computer Interaction reported inter-rater reliability metrics low-effort responses from crowdworkers

PROBLEM: We explored three different aspects of inconsistency and designed metrics for their measurements.  A word level feature using tf-idf BIBREF22 is added for robustness. 
 Question: What features do they extract?

SOLUTION:
Inconsistency in Noun Phrase Structures  Inconsistency Between Clauses Inconsistency Between Named Entities and Noun Phrases Word Level Feature Using TF-IDF