SlowGuess/NAACL_2021_OCR huggingface dataset:
Fleiss Kappa: Let  $\mathbf{N}$  be the total number of subjects, let  $\mathbf{n}$  be the number of ratings per subject, and let  $\mathbf{k}$  be the number of categories into which assignments are made. Let  $n_{ij}$  represent the number of raters who assigned the i-th subject to the j-th category. The kappa agreement is calculated as
$$
p _ {i} = \frac {1}{n (n - 1)} \sum_ {j = 1} ^ {k} n _ {i j} \left(n _ {i j} - 1\right)
$$
$$
\bar {P} = \frac {1}{N} \sum_ {i = 1} ^ {N} p _ {i}
$$
$$
p _ {j} = \frac {1}{N n} \sum_ {i = 1} ^ {N} n _ {i j}
$$
$$
P _ {e} = \sum_ {j = 1} ^ {k} p _ {j} ^ {2}
$$
$$
\kappa = \frac {\bar {P} - P _ {e}}{1 - P _ {e}}
$$
B Example Screenshots from ThePerspective Website and MTurk Annotation Interface
In this section, we show example screenshots from the website where we extract the data, www.theperspective.com/perspectives, and example screenshots of our data annotation process.
For the three data annotation tasks using Mechanical Turk, stance annotation, implicit reference resolution, and topic annotation, we present
the Turkers with definitions and instructions of the tasks that we require them to do, and 3-6 example questions with our expected answers. We ask them to read and comprehend our instructions before annotating, and use random control sets to filter out invalid annotations. More details can be found in the screenshots below.
Figure 6: An example of a query and its two perspectives in ThePerspective website
Figure 7: An example of two abstracts and their links to news editorials in ThePerspective website
Six months after the last big lockdown, it is clear that a new one is needed, believes German Lopez of Vox. Daily new cases of COVID-19 have risen as high as 180,000, and hospitals all over the country are nearing full capacity. Over 1,000 Americans are dying every day, which may rise to 2,000 and 3,000 before the end of the year. This fast spread has made containing the virus much harder. Past lockdowns were highly effective at stopping the virus from spreading and helped the country stabilize the infection rate. They proved to be effective in other countries as well. A new lockdown, with financial support for businesses, is direly needed to curtail further disaster.
$\ll \ll$  KEEP ON READING AT VOX
While the COVID-19 pandemic has to be taken serious, violating citizens' civil liberties is not acceptable, suggests Jonathan S. Tobing of The New York Post. The government has already gone too far regarding excessive and obstructive rules. Head-scratching measures like closing churches while leaving open casinos, which happened in Nevada, show how poorly thought out much of this strategy is. That pandemic is being politicized, as certain gatherings, such as religious ones, are cracked down upon, while Black Lives Matter protests go unpunished. We need to remember to respect individual freedoms, first and foremost, and not allow unlimited government control.
KEEP ON READING AT THE NEW YORK POST>
THE PERSFOTIVE
Become a Patron!
BIZ/TECH ENTERTAINMENT LIVING POLITICS SPORTS THE BIG DEBATES
OW FORMATS
Tasks description and Instructions
This is important and please finish this before you start the tasks.
For each question, we will give you three different sentences, one is a question, and the other two are sentences trying to answer the question.
For examples,
Question: Is Trump Right To Criticize Mail-In Voting?
Sentence 1: It's far too risky for an election
Sentence 2: Mail-in voting is far too risky for an election
You can see that the first sentence is trying to answer the Question. However, it has some implicit references to the subjects in the question that if you don't know what the question is, you will not be able to understand it. For instance, the word "it" in sentence 1 refers to Mail-In Voting in the question. And if we don't give you the question, you will never be able to know what is the it representing.
Therefore, to mitigate this problem, we manually edited and modified sentence 1 and wrote sentence 2 that replaces words like it in this case with the subject it is referring to. So sentence 2 is almost the same as sentence 1, but has some few words modified/added to make sure people can understand what is it discussing even without the question.
Figure 8: A screenshot of the MTurk annotation instruction for implicit reference resolution part1
Here we want you to help us verify that the sentence 2 that we wrote does not introduce any error.
So we want you to answer the question, Will changing sentence 1 to sentence 2 introduce any error? i.e., does sentence 2 distort the original meaning in sentence 1 or introduce any grammatical problem?
And you should either answer "Yes, changing sentence 1 to sentence 2 will cause problems" or "No, it will not cause any problem."
Examples of a bad sentence 2 that changes the original meaning
Question: are biden's and kavanaugh's sexual abuse cases comparable?
Sentence 1: They are similar despite media bias
Sentence 2: Biden and kavanaugh are similar despite media bias
You should answer: "Yes, changing sentence 1 to sentence 2 will cause problems"
The reason that you should answer yes here is that, by replacing "They" in sentence 1 with "Biden and Kavanaugh" changes the original idea sentence 1 is trying to express.
Sentence 1 is meant to say their cases are different, but not they are different. So here sentence 2 takes the wrong subject They is referring to in the question, and therefore you should answer "Yes".
A better sentence 2 that does not have this problem would be Biden's and kavanaugh's sexual abuse cases are similar despite media bias
Figure 9: A screenshot of the MTurk annotation instruction for implicit reference resolution part2
Choose "Yes" if sentence 2 changes the original meaning in sentence 1 or sentence 2 has a grammatical error.
Chosse "No" if sentence 2 means exactly the same thing as sentence 1 and has no grammar error.
To better make sure that you will answer them seriously, a few examples in the dataset are set as control questions. If you can not get them correct, we will have no choice but to reject your submission. So please be patient and honest. Don't put in random selections or we will not be able to pay you. Thank you for answering the question!
Note: Make sure you have answered all the questions before you click any submit button so we can record your results.
Figure 10: A screenshot of the MTurk annotation instruction for implicit reference resolution part3
Instructions Shortcuts Problem with Sentence 2?
"Will changing sentence 1 to sentence 2 introduce any error? i.e., does sentence 2 distort the original meaning in sentence 1 or introduce any grammatical problem?"
Question: is trump's economic policy good for america?
Sentence 1: His ideas are terrible for the country
Sentence 2: Trump's ideas are terrible for the country
Please be patient and honest when answering the question. Remember, we will not be able to pay you if you are randomly choosing the answers.
Yes, changing sentence 1 to sentence 1  
2 will cause problems.
No, it will not cause any problem. 2
Figure 11: A screenshot of the MTurk annotation example for implicit reference resolution
Important and Must Read
In each HIT, there are two questions that we know the answers for. These two are the easiest questions that as long as you carefully finish reading the instruction and examples, you should be able to answer them correctly. We set them as the criteria of approving your submission. We will only approve your submission and pay you if you answer both of them correctly. So please be patient and honest when answering the questions.
Instruction and Examples
In this HIT, we will ask you one same question for ten different argument-query pairs, that is "What stance does this argument take on this query, support or oppose?"
A query is a question about a news topic
An argument is an argumentative answer to the query that expresses the side it takes on the query.
Figure 12: A screenshot of the MTurk annotation instruction for stance annotation part1
Examples and Explantions
Question: "What stance does this Argument take on this Query, support or oppose?"
Query: Is trump handling the coronavirus response well?
Argument: Trump is protecting Americans
Answer: Yes, the Argument supports the Query.
The reason is that, the argument agrees with the statement "Trump is handling the coronavirus response well" by saying "Trump is protecting Americans".
Another example argument that disagrees with this query could be "He is making things far worse."
Figure 13: A screenshot of the MTurk annotation instruction for stance annotation part2
Question: "What stance does this Argument take on this Query, support or oppose?" i.e., does this argument support or oppose the query?
Query: should the 2nd amendment cover assault weapons?
Argument: They're too dangerous to be allowed
Note that we are not asking you to answer "yes" or "no" to the query, but instead is asking you whether the argument supports or opposes the query.
Please be patient and honest when answering the question. Remember, we will not be able to pay you if you are randomly choosing the answers.
If you find it difficult to decide, or the argument is not clear enough, please read the below paragraph that explains the argument in more details. It may help you choose the right answer, especially the first sentence of the paragraph.
Paragraph: Assault rifles' great potential to kill large groups of people in a short amount of time
Yes, the argument supports the query.
No, the argument opposes the query. 2
Figure 14: A screenshot of the MTurk annotation example for stance annotation. Note that there will be more sentences shown in the Paragraph line if the user scrolls down.
In this HIT, we will ask you one same question for ten different sentences, that is "Which category of news topics is discussed in this sentence?"
Each sentence will discuss a popular news topic in everyday life. We will need you to help us decide which category of news topic is the sentence discussing, e.g., US Politics, Covid-19, World News, Entertainment, Gender Equality, etc.
Definitions and examples of the categories
US Politics: if you see US foreign policies, US politician names, political events, or political parties, etc, you consider it belongs to the US Politics category. However, select this only if the only topic discussed in the sentence is about politics but no other categories are involved.
Example: did president trump obstruct justice?
COVID-19: If you see it discusses anything about pandemic, covid-19/coronavirus, or lockdown, etc, you consider it belongs to the COVID-19 category.
Example: should we end covid-19 security measures soon?
Figure 15: A screenshot of the MTurk annotation instruction for topic annotation part1
World News If you see it discusses anything or anyone that's outside of the United States, e.g., other countries' politics, new, etc, you consider it belongs to the World News category.
Example: is brexit justified?
Entertainment: If you see it discusses anything about a song, a movie, a video game etc, you consider it belongs to the Entertainment category.
Example: is a star is born worth all the hype?
Sports: If you see it discusses anything about a type of sport, a sport's league, or a sport player, etc, you consider it belongs to the Sports category.
Example: who will win the 2018 nba playoffs?
Gender Equality: If you see it discusses anything about feminism, gender quality, equal pay, etc, you consider it belongs to the Gender Equality category.
Example: are conservatives using feminism to their own advantage?
Economy: If you see it discusses anything about a country's economy, a trade war, business, companies, anything about money, etc, you consider it belongs to the
Economy category.
Figure 16: A screenshot of the MTurk annotation instruction for topic annotation part2
Racial Equality: If you see it discusses anything about black voices, racial discrimination, racial equality, etc, you consider it belongs to the Racial Equality category.
Example: was the Starbucks arrest racially motivated?
Other Topics If you see it discusses anything about topics other than the ones above, you consider it belongs to the Other Topics category.
Example: was elon musk's space x project justified?
If you think that it is discussing topics in more than one categories, e.g., it discusses topics in both US Politics and COVID-19, choose the one that you think it is emphasizing more and is the key point that it is discussing... Think about which category will this news headlines appear on CNN, Fox News, NBC, etc.
Question: "Which category of news topics is discussed in this sentence?"
Answer: COVID-19
We see that here it discusses topics both in US Politics and in COVID-19. However, we believe that the emphasis is more on COVID-19 because it only mentions one
Figure 17: A screenshot of the MTurk annotation instruction for topic annotation part3
Question 1: Which category of news topics is discussed in this sentence?
Sentence: should trump choose the new cfpb head?
US Politics
COVID-19 2
World News 3
Entertainment 4
Sports 5
Gender Equality 6
Economy 7
Racial Equality 8
Other topics 9
Figure 18: A screenshot of the MTurk annotation example for topic annotation