Definition: In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.

Positive Example 1 -
Input: We evaluate the proposed approach on the Chinese social media text summarization task, based on the sequence-to-sequence model. Large-Scale Chinese Short Text Summarization Dataset (LCSTS) is constructed by BIBREF1 . The dataset consists of more than 2.4 million text-summary pairs in total, constructed from a famous Chinese social media microblogging service Weibo.  
 Question: Are results reported only for English data?
Output: No

Positive Example 2 -
Input: Since INLINEFORM0 is constant, we only need to minimize INLINEFORM1 , therefore the loss function becomes: DISPLAYFORM0 
 Question: How is the expectation regularization loss defined?
Output: DISPLAYFORM0

Negative Example 1 -
Input: The minimal change operation substantially changed the meaning of the sentence, and yet the embedding of the transformation lies very closely to the original sentence (average similarity of 0.930). 
 Question: Do they do any analysis of of how the modifications changed the starting set of sentences?
Output: No

Negative Example 2 -
Input: Looking at the current WinoGrande leaderboard, it appears that the previous state of the art is based on RoBERTa BIBREF2, which can be characterized as an encoder-only transformer architecture. 
 Question: What is the previous state of the art?
Output: GPT-3

Now complete the following example -
Input: In this section, we recommend a set of evaluation design changes that we believe are needed for assessing human–machine parity, and will strengthen the human evaluation of MT in general.

Recommendations ::: (R1) Choose professional translators as raters.
In our blind experiment (Section SECREF3), non-experts assess parity between human and machine translation where professional translators do not, indicating that the former neglect more subtle differences between different translation outputs.

Recommendations ::: (R2) Evaluate documents, not sentences.
When evaluating sentences in random order, professional translators judge machine translation more favourably as they cannot identify errors related to textual coherence and cohesion, such as different translations of the same product name. Our experiments show that using whole documents (i. e., full news articles) as unit of evaluation increases the rating gap between human and machine translation (Section SECREF4).

Recommendations ::: (R3) Evaluate fluency in addition to adequacy.
Raters who judge target language fluency without access to the source texts show a stronger preference for human translation than raters with access to the source texts (Sections SECREF4 and SECREF24). In all of our experiments, raters prefer human translation in terms of fluency while, just as in BIBREF3's BIBREF3 evaluation, they find no significant difference between human and machine translation in sentence-level adequacy (Tables TABREF21 and TABREF30). Our error analysis in Table TABREF34 also indicates that MT still lags behind human translation in fluency, specifically in grammaticality.

Recommendations ::: (R4) Do not heavily edit reference translations for fluency.
In professional translation workflows, texts are typically revised with a focus on target language fluency after an initial translation step. As shown in our experiment in Section SECREF24, aggressive revision can make translations more fluent but less accurate, to the degree that they become indistinguishable from MT in terms of accuracy (Table TABREF30).

Recommendations ::: (R5) Use original source texts.
Raters show a significant preference for human over machine translations of texts that were originally written in the source language, but not for source texts that are translations themselves (Section SECREF35). Our results are further evidence that translated texts tend to be simpler than original texts, and in turn easier to translate with MT. 
 Question: What recommendations do they offer?
Output: 



Answer:  Choose professional translators as raters  Evaluate documents, not sentences Evaluate fluency in addition to adequacy Do not heavily edit reference translations for fluency Use original source texts