SlowGuess/ACL_2020_OCR huggingface dataset:
Barbara Mellers, Eric Stone, Terry Murray, Angela Minster, Nick Rohrbaugh, Michael Bishop, Eva Chen, Joshua Baker, Yuan Hou, Michael Horowitz, et al. 2015b. Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspectives on Psychological Science.
Barbara Mellers, Lyle Ungar, Jonathan Baron, Jaime Ramos, Burcu Gurcay, Katrina Fincher, Sydney E. Scott, Don Moore, Pavel Atanasov, Samuel A. Swift, Terry Murray, Eric Stone, and Philip E. Tetlock. 2014. Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25(5):1106-1115. PMID: 24659192.
Dong Nguyen, Rilana Gravel, Dolf Trieschnigg, and Theo Meder. 2013. "how old do you think i am?" a study of language and age in twitter. In Seventh International AAAI Conference on Weblogs and Social Media.
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10. Association for Computational Linguistics.
Gregory Park, H Andrew Schwartz, Maarten Sap, Margaret L Kern, Evan Weingarten, Johannes C Eichstaedt, Jonah Berger, David J Stillwell, Michal Kosinski, Lyle H Ungar, et al. 2017. Living in the past, present, and future: Measuring temporal orientation with language. Journal of personality.
Verónica Pérez-Rosas and Rada Mihalcea. 2015. Experiments in open domain deception detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Roser Saurí and James Pustejovsky. 2012. Are you sure that this happened? assessing the factuality degree of events in text. Computational Linguistics.
Alan G. Sawyer, Juliano Laran, and Jun Xu. 2008. The readability of marketing journals: Are award-winning articles better written? Journal of Marketing, 72(1):108-117.
H Andrew Schwartz, Gregory Park, Maarten Sap, Evan Weingarten, Johannes Eichstaedt, Margaret Kern, David Stillwell, Michal Kosinski, Jonah Berger, Martin Seligman, et al. 2015. Extracting human temporal orientation from facebook language. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
H. Andrew Schwartz, Masoud Rouhizadeh, Michael Bishop, Philip Tetlock, Barbara Mellers, and Lyle Ungar. 2017. Assessing objective recommendation quality through political forecasting. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2348-2357, Copenhagen, Denmark. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Youngseo Son, Nipun Bayas, and H Andrew Schwartz. 2018. Causal explanation analysis on social media. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Sandesh Swamy, Alan Ritter, and Marie-Catherine de Marneffe. 2017. "I have a feeling trump will win....": Forecasting winners and losers from user predictions on twitter. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1583-1592, Copenhagen, Denmark. Association for Computational Linguistics.
Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(2):267-307.
Chenhao Tan, Lillian Lee, and Bo Pang. 2014. The effect of wording on message propagation: Topic- and author-controlled natural experiments on twitter. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 175–185, Baltimore, Maryland. Association for Computational Linguistics.
Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th International Conference on World Wide Web, WWW '16, pages 613-624, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee.
Yla R. Tausczik and James W. Pennebaker. 2010. The psychological meaning of words: Liwc and computerized text analysis methods. Journal of Language and Social Psychology.
Philip Tetlock. 2005. Expert political judgment: How good is it? how can we know?
Oren Tsur and Ari Rappoport. 2009. Revrank: A fully unsupervised algorithm for selecting the most helpful book reviews. In Third International AAAI Conference on Weblogs and Social Media.
Rob Voigt, Nicholas P Camp, Vinodkumar Prabhakaran, William L Hamilton, Rebecca C Hetey, Camilla M Griffiths, David Jurgens, Dan Jurafsky, and Jennifer L Eberhardt. 2017. Language from police body camera footage shows racial disparities in officer respect. Proceedings of the National Academy of Sciences.
Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. Rtgender: A corpus for studying differential responses to gender. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
Svitlana Volkova, Glen Coppersmith, and Benjamin Van Durme. 2014. Inferring user political preferences from streaming communications. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Zijian Wang and David Jurgens. 2018. It's going to be okay: Measuring access to support in online communities. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 33-45, Brussels, Belgium. Association for Computational Linguistics.
Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing.
Diyi Yang, Jiaao Chen, Zichao Yang, Dan Jurafsky, and Eduard Hovy. 2019. Lets make your request more persuasive: Modeling persuasive strategies via semisupervised neural nets on crowdfunding platforms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
A Additional Experiments on Good Judgment Open Dataset
A.1 Differences Between Top and Bottom Forecasters?
Figure 2 presents calibration curves and averaged standardized Brier scores across years for the top and bottom 500 forecasters. We observe the differences between these two groups are persistent over time. Controlled lab experiments from psychology have also demonstrated that top forecasters ranked by Brier scores consistently have better forecasting performance than bottom forecasters (Mellers et al., 2015a).
(a) Calibration curves by using all forecasts.
(b) Aggregated forecasting performance across years.
Figure 2: Comparison of forecasting skill between the top 500 and bottom 500 forecasters ranked by averaged standardized Brier scores. (a) Calibration curves for each group calculated using all forecasts (with and without justifications). The diagonal dotted line indicates a perfect calibration. (b) Trends of average standardized Brier scores over years. Negative values indicate better forecasting skill.
A.2 Additional Metrics and Examples for Linguistic Analysis
Uncertainty. We present examples of sentences with uncertainty scores from our dataset in Table 9. Discourse connectives. We further investigate the portion of discourse connectives used between sentences within each group. For this purpose, we use a lexicon developed by Das et al. (2018), which
collects connectives from PDTB corpus connective list, RST Signalling Corpus and RST-DT relational indicator list. The lexicon contains 149 English connectives, divided into 4 categories: comparison, contingency, expansion, and temporal.[12] Our results show that skilled forecasters tend to use discourse connectives more frequently compared to unskilled forecasters, which may indicate that they tend to make more coherent arguments.
Thinking style. Analytical thinking score in LIWC (Tausczik and Pennebaker, 2010) ranks the level of a person's thinking skill. A high score correlates with formal, logical, and hierarchical thinking, while low scores are associated with informal, and narrative thinking. As shown in Table 8, good forecasters appear to demonstrate better analytical thinking skills.
Metric	p	Bonferroni
Discourse connectives
Comparison	↑↑↑	*
Contingency	↑↑
Expansion	↑↑	*
Temporal	↑↑↑	*
Thinking style
Analytical thinking	↑↑	*
Table 8: Comparison of various metrics computed over text written by the top 500 and bottom 500 forecasters.  $p$ -values are calculated by bootstrap hypothesis test. The number of arrows indicates the level of  $p$ -value, while the direction shows the relative relationship between top and bottom forecasters,  $\uparrow \uparrow \uparrow$ : top group is higher than bottom group with  $p < 0.001$ ,  $\uparrow \uparrow$ :  $p < 0.01$ ,  $\uparrow$ :  $p < 0.05$ . Tests that pass Bonferroni correction are marked by *.
A.3 Linguistic Cues over Time
We are interested in whether our observed linguistic differences are consistent over time. To answer this question, we select the top 500 and bottom 500 forecasters based on their final ranking and evaluate aggregated metrics for the two groups in different years. Our results are shown in Figure 3. We observe the same pattern for all linguistic metrics. For example, skilled forecasters consistently exhibit a higher level of uncertainty and past temporal orientation, and a lower readability compared to unskilled forecasters.
<sup>12</sup>As some connectives are listed under more than one category, we restrict the list to those belonging to one or two categories.
Sentence	Uncert. Score
Merkel is probably least prone to political scandals among the Western leaders and candidates.	1.00
It seems unlikely that the court would transfer the terms of that contract to Uber.	0.99
My assumptions: - Sturgeon will not set a date for indyref2 before the UK elections on June 8.	0.05
To date, Toyota has distributed only 100 of the 300 Mirais preordered in California ...	0.02
Table 9: Examples of sentences in our dataset with uncertainty scores estimated by the model proposed by Adel and Schütze (2017). A higher uncertainty score indicates a higher level of uncertainty.
(a) Readability (Dale)
(b) Emotion
(c) Parts of Speech (noun)
(d) Parts of Speech (verb)
(e) Discourse connectives (comparison)
(f) Discourse connectives (temporal)
(g) Uncertainty
(h) Thinking style (analytical score)
(i) Temporal orientation (focus on past)
Figure 3: Linguistic features in different years for top 500 and bottom 500 forecasters. The plots show how readability (Dale), emotion, Parts of Speech (noun and verb), discourse connectives (comparison and temporal), uncertainty, thinking style (analytical score), and temporal orientation (focus on past) change in different years. We observe nearly consistent trends for all metrics over time, which indicates that linguistic differences are stable. Error bars represent standard errors.
B Experimental Details on Companies' Earning Forecasts
B.1 Extracting Numerical Forecasts from Text
Not all analysts' notes in our dataset are associated with structured earnings forecasts (in tables). Instead, the analysts' numerical predictions for future earnings are directly reported in the text of their notes, which also contain additional language justifying their predictions. Therefore, our first goal
is to extract structured representations of analysts' EPS estimates in a  $\langle \mathrm{TIME}, \mathrm{VALUE} \rangle$  format. We noticed that analysts have a highly consistent style when writing this section of the report, we therefore use a set of lexico-syntactic patterns to extract the forecasts from text; as described below. We found this approach to have both high precision and high recall.
We randomly sampled  $60\%$  of the notes in our dataset for developing patterns. Before generating the rules, we replaced entities indicating time
Sentence Pattern Extracted	We trim our 12-month target price to $20 from $23, 10X our '16 EPS estimate of $2.01 -LRB- trimmed today from $2.10 -RRB-.
<TIME> EPS estimate of <MONEY>
<'16, $2.01'
Sentence Pattern Extracted	We raise '18 and '19 EPS estimates by $4.61 and $5.72 to $19.85 and $25.95 .
<TIME> and <TIME> EPS estimates <BY-MASK> to <MONEY> and <MONEY>
<'18, $19.85>, <'19, $25.95'
Sentence Pattern Extracted	We raise our FY 17 EPS estimate to $3.23 from $2.96 and set FY 18 ’s at $3.43 .
<TIME> EPS estimate to <MONEY> <FROM-MASK> and set <TIME> at <MONEY>
<FY 17, $3.23>, <FY 18, $3.43'
Table 10: Examples of earnings forecasts extracted from analysts' notes. Only sentences mentioning the earnings forecast are shown; the notes also contain additional analysis to justify the forecast. All sentences from notes are used to classify accurate versus inaccurate forecasts as described in §3.2.
and money with special  $\langle \mathrm{TIME} \rangle$  and  $\langle \mathrm{MONEY} \rangle$  tokens. To evaluate the generalization of our patterns, we randomly sampled 100 sentences containing 136 numerical forecasts from the remaining  $40\%$  of notes and manually checked all of them. We estimate that our pattern-based approach extracts numerical forecasts with 0.91 precision and 0.82 recall. Table 10 shows examples of numerical forecasts extracted using our approach. In a few cases we found that an analyst's note can contain more than one forecast. For simplicity, we only consider the earliest forecast that is made within the 2014-2018 time range.