instruction:
In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.
question:
To collect article-level labels, we utilized a platform in the company that has been used by the market research team to collect surveys from the subscribers of different news publishers. The survey works as follows: The user is first presented with a set of selected pages (usually 4 pages and around 20 articles) from the print paper the day before. The user can select an article each time that he or she has read, and answer some questions about it. We added 3 questions to the existing survey that asked the level of partisanship, the polarity of partisanship, and which pro- or anti- entities the article presents. We also asked the political standpoint of the user. The complete survey can be found in Appendices. 
 Question: Did they crowdsource the annotations?
answer:
Yes


question:
Experimental Setup 
 Question: what english datasets were used?
answer:
Answer with content missing: (Data section) Penn Treebank (PTB)


question:
A meta-research study found only 1 in 9 qualitative papers in Human-Computer Interaction reported inter-rater reliability metrics BIBREF31. Another related area are meta-research and methods papers focused on identifying or preventing low-effort responses from crowdworkers — sometimes called “spam” or “random” responses, or alternatively ”fraudsters” or ”cheaters.” Rates of “self-agreement” are often used, determining if the same person labels the same item differently at a later stage. 
 Question: What are the key issues around whether the gold standard data produced in such an annotation is reliable? 
answer:

 only 1 in 9 qualitative papers in Human-Computer Interaction reported inter-rater reliability metrics low-effort responses from crowdworkers