Question: What about the inter annotator agreement for the human evaluation?
Evidence: Posts on Reddit capture debate and discussion across a diverse set of topics, making them a natural starting point for untangling domain-independent linguistic features of dogmatism.
Data collection. Subreddits are sub-communities on Reddit oriented around specific interests or topics, such as technology or politics. Sampling from Reddit as a whole would bias the model towards the most commonly discussed content. But by sampling posts from individual subreddits, we can control the kinds of posts we use to train our model. To collect a diverse training dataset, we have randomly sampled 1000 posts each from the subreddits politics, business, science, and AskReddit, and 1000 additional posts from the Reddit frontpage. All posts in our sample appeared between January 2007 and March 2015, and to control for length effects, contain between 300 and 400 characters. This results in a total training dataset of 5000 posts.
Dogmatism annotations. Building a useful computational model requires labeled training data. We labeled the Reddit dataset using crowdworkers on Amazon Mechanical Turk (AMT), creating the first public corpus annotated with levels of dogmatism. We asked crowdworkers to rate levels of dogmatism on a 5-point Likert scale, as supported by similar annotation tasks in prior work BIBREF2 . Concretely, we gave crowdworkers the following task: 
Given a comment, imagine you hold a well-informed, different opinion from the commenter in question. We'd like you to tell us how likely that commenter would be to engage you in a constructive conversation about your disagreement, where you each are able to explore the other's beliefs. The options are:
(5): It's unlikely you'll be able to engage in any substantive conversation. When you respectfully express your disagreement, they are likely to ignore you or insult you or otherwise lower the level of discourse.
(4): They are deeply rooted in their opinion, but you are able to exchange your views without the conversation degenerating too much.
(3): It's not likely you'll be able to change their mind, but you're easily able to talk and understand each other's point of view.
(2): They may have a clear opinion about the subject, but would likely be open to discussing alternative viewpoints.
(1): They are not set in their opinion, and it's possible you might change their mind. If the comment does not convey an opinion of any kind, you may also select this option.
To ensure quality work, we restricted the task to Masters workers and provided examples corresponding to each point on the scale. Including examples in a task has been shown to significantly increase the agreement and quality of crowdwork BIBREF3 . For instance, here is an example of a highly dogmatic (5) comment: 
I won't be happy until I see the executive suite of BofA, Wells, and all the others, frog-marched into waiting squad cars. It's ALREADY BEEN ESTABLISHED that...
And a minimally dogmatic (1) comment: 
I agree. I would like to compile a playlist for us trance yogi's, even if you just would like to experiment with it. Is there any preference on which platform to use?
Each comment has been annotated by three independent workers on AMT, which is enough to produce reliable results in most labeling tasks BIBREF4 . To compute an aggregate measure of dogmatism for each comment, we summed the scores of all three workers. We show the resulting distribution of annotations in Figure 1 .
Inter-annotator agreement. To evaluate the reliability of annotations we compute Krippendorff's $\alpha $ , a measure of agreement designed for variable levels of measurement such as a Likert scale BIBREF5 . An $\alpha $ of 0 indicates agreement indistinguishable from chance, while an $\alpha $ of 1 indicates perfect agreement. Across all annotations we find $\alpha =0.44$ . While workers agree much more than chance, clearly dogmatism is also subjective. In fact, when we examine only the middle two quartiles of the dogmatism annotations, we find agreement is no better than chance. Alternatively, when we measure agreement only among the top and bottom quartiles of annotations, we find agreement of $\alpha =0.69$ . This suggests comments with scores that are only slightly dogmatic are unreliable and often subject to human disagreement. For this reason, we use only the top and bottom quartiles of comments when training our model.
Inter-annotator agreement. To evaluate the reliability of annotations we compute Krippendorff's a, a measure of agreement designed for variable levels of measurement such as a Likert scale (Hayes and Krippendorff, 2007). An a of 0 indicates agreement indistinguishable from chance, while an a of 1 indicates perfect agreement. Across all annotations we find a = 0.44. While workers agree much more than chance, clearly dogmatism is also subjective. In fact, when we examine only the middle two quartiles of the dogmatism annotations, we find agreement is no better than chance. Alternatively, when we measure agreement only among the top and bottom quartiles of annotations, we find agreement of a = 0.69. This suggests comments with scores that are only slightly dogmatic are unreliable and often subject to human disagreement. For this reason, we use only the top and bottom quartiles of comments when training our model.
Answer: To evaluate the reliability of annotations they compute Krippendorff's a, a measure of agreement designed for variable levels of measurement such as a Likert scale (Hayes and Krippendorff, 2007). An a of 0 indicates agreement indistinguishable from chance, while an a of 1 indicates perfect agreement. Across all annotations they find a = 0.44. While workers agree much more than chance, clearly dogmatism is also subjective. In fact, when they examine only the middle two quartiles of the dogmatism annotations, they find agreement is no better than chance. Alternatively, when they measure agreement only among the top and bottom quartiles of annotations, they find agreement of a = 0.69. This suggests comments with scores that are only slightly dogmatic are unreliable and often subject to human disagreement. For this reason, they use only the top and bottom quartiles of comments when training their model.