Definition: In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.

Positive Example 1 -
Input: Questions are gathered from anonymized, aggregated queries to the Google search engine. Queries that are likely to be yes/no questions are heuristically identified: we found selecting queries where the first word is in a manually constructed set of indicator words and are of sufficient length, to be effective.

Questions are only kept if a Wikipedia page is returned as one of the first five results, in which case the question and Wikipedia page are given to a human annotator for further processing.

Annotators label question/article pairs in a three-step process. First, they decide if the question is good, meaning it is comprehensible, unambiguous, and requesting factual information. This judgment is made before the annotator sees the Wikipedia page. Next, for good questions, annotators find a passage within the document that contains enough information to answer the question. Annotators can mark questions as “not answerable" if the Wikipedia article does not contain the requested information. Finally, annotators mark whether the question's answer is “yes" or “no". Annotating data in this manner is quite expensive since annotators need to search entire Wikipedia documents for relevant evidence and read the text carefully.
Output: how was the dataset built?

Positive Example 2 -
Input: Using this annotation model, we create a new large publicly available dataset of English tweets.
Output: In what language are the tweets?

Negative Example 1 -
Input: Using this annotation model, we create a new large publicly available dataset of English tweets.
Output: In what language is tweets?

Negative Example 2 -
Input: Questions are gathered from anonymized, aggregated queries to the Google search engine. Queries that are likely to be yes/no questions are heuristically identified: we found selecting queries where the first word is in a manually constructed set of indicator words and are of sufficient length, to be effective.

Questions are only kept if a Wikipedia page is returned as one of the first five results, in which case the question and Wikipedia page are given to a human annotator for further processing.

Annotators label question/article pairs in a three-step process. First, they decide if the question is good, meaning it is comprehensible, unambiguous, and requesting factual information. This judgment is made before the annotator sees the Wikipedia page. Next, for good questions, annotators find a passage within the document that contains enough information to answer the question. Annotators can mark questions as “not answerable" if the Wikipedia article does not contain the requested information. Finally, annotators mark whether the question's answer is “yes" or “no". Annotating data in this manner is quite expensive since annotators need to search entire Wikipedia documents for relevant evidence and read the text carefully.
Output: What is the size of the dataset?

Now complete the following example -
Input: To compare the conversions between USVC and PitchNet, we employed an automatic evaluation score and a human evaluation score. The automatic score roughly followed the design in BIBREF13. The pitch tracker of librosa package BIBREF18 was employed to extract pitch information of the input and output audio. Then the output pitch was compared to the input pitch using the normalized cross correlation (NCC) which would give a score between 0 and 1. The higher the score is, the better the output pitch matches the input pitch. We conducted the evaluation on USVC (our) and PitchNet. The evaluated automatic scores on conversion and reconstruction tasks are shown in Tab. TABREF14. Our method performed better both on conversion and reconstruction. The scores of reconstruction are higher than conversion since both models were trained using a reconstruction loss. However, the score of our method on conversion is even higher than the score of USVC (Our) on reconstruction. Mean Opinion Score (MOS) was used as a subjective metric to evaluate the quality of the converted audio. Two questions were asked: (1) what is quality of the audio? (naturalness) (2) How well does the converted version match the original? (similarity) A score of 1-5 would be given to answer the questions. The evaluation was conducted on USVC (Our) and PitchNet. Besides, the converted samples provided by BIBREF0 was also included to give a more convincing evaluation. As shown by Tab. TABREF15, the naturalness and similarity of our method are both higher than the other two ones. Our implementation of USVC performed slightly lower than the original author's because we cannot fully reproduce the results of them. Next we qualitatively analyze the influence of input pitch in our method. We used different pitch as input to observe how the output pitch would change along with the input pitch. The input pitch was multiplied by 0.7, 1.0 and 1.3 respectively. And the output pitch was also extracted by the pitch tracker of the librosa package. Fig. FIGREF16 plots the pitch of input audio and output audio with different pitch as input while keeping the target singer the same. As shown by Fig. FIGREF16, the output pitch changes significantly along with the input pitch. The examples are also presented at our website.
Output: 



Output: How is the quality of singing voice measured?