SlowGuess/ACL_2020_OCR huggingface dataset:
Shi Zong<sup>1</sup> Alan Ritter<sup>1</sup> Eduard Hovy<sup>2</sup>
$^{1}$ Department of Computer Science and Engineering, The Ohio State University
$^{2}$ Language Technologies Institute, Carnegie Mellon University
{zong.56, ritter.1492}@osu.edu, hovy@cs.cmu.edu
People vary in their ability to make accurate predictions about the future. Prior studies have shown that some individuals can predict the outcome of future events with consistently better accuracy. This leads to a natural question: what makes some forecasters better than others? In this paper we explore connections between the language people use to describe their predictions and their forecasting skill. Datasets from two different forecasting domains are explored: (1) geopolitical forecasts from Good Judgment Open, an online prediction forum and (2) a corpus of company earnings forecasts made by financial analysts. We present a number of linguistic metrics which are computed over text associated with people's predictions about the future including: uncertainty, readability, and emotion. By studying linguistic factors associated with predictions, we are able to shed some light on the approach taken by skilled forecasters. Furthermore, we demonstrate that it is possible to accurately predict forecasting skill using a model that is based solely on language. This could potentially be useful for identifying accurate predictions or potentially skilled forecasters earlier.<sup>1</sup>
People often make predictions about the future, for example meteorologists tell us what the weather might look like tomorrow, financial analysts predict which companies will report favorable earnings and intelligence analysts evaluate the likelihood of future geopolitical events. An interesting question is why some individuals are significantly better forecasters (Mellers et al., 2015b)?
Previous work has analyzed to what degree various factors (intelligence, thinking style, knowledge
of a specific topic, etc.) contribute to a person's skill. These studies have used surveys or psychological tests to measure dispositional, situational and behavioral variables (Mellers et al., 2015a). Another source of information has been largely overlooked, however: the language forecasters use to justify their predictions. Recent research has demonstrated that it is possible to accurately forecast the outcome of future events by aggregating social media users' predictions and analyzing their veridicality (Swamy et al., 2017), but to our knowledge, no prior work has investigated whether it might be possible to measure a forecaster's ability by analyzing their language.
In this paper, we present the first systematic study of the connection between language and forecasting ability. To do so, we analyze texts written by top forecasters (ranked by accuracy against ground truth) in two domains: geopolitical forecasts from an online prediction forum, and company earnings forecasts made by financial analysts. To shed light on the differences in approach employed by skilled and unskilled forecasters, we investigate a variety of linguistic metrics. These metrics are computed using natural language processing methods to analyze sentiment (Pang et al., 2002; Wilson et al., 2005), uncertainty (de Marneffe et al., 2012; Saurí and Pustejovsky, 2012), readability, etc. In addition we make use of word lists taken from the Linguistic Inquiry and Word Count (LIWC) software (Tausczik and Pennebaker, 2010), which is widely used in psychological research. By analyzing forecasters' texts, we are able to provide evidence to support or refute hypotheses about factors that may influence forecasting skill. For example, we show forecasters whose justifications contain a higher proportion of uncertain statements tend to make more accurate predictions. This supports the hypothesis that more open-minded thinkers, who have a higher tolerance for ambiguity tend to make
1We provide our code and dataset descriptions at: https://github.com/viczong/measuring_forecasting_skill_from_text.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5317-5331 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
better predictions (Tetlock, 2005).
Beyond analyzing linguistic factors associated with forecasting ability, we further demonstrate that it is possible to identify skilled forecasters and accurate predictions based only on relevant text. Estimating the quality of a prediction using the forecaster's language could potentially be very beneficial. For example, this does not require access to historical predictions to evaluate past performance, so it could help to identify potentially skilled individuals sooner. Also, forecasters do not always provide an explicit estimate of their confidence, so a confidence measure derived directly from text could be very useful.
2 Linguistic Cues of Accurate Forecasting
In this section, we are interested in uncovering linguistic cues in people's writing that are predictive of forecasting skill. We start by analyzing texts written by forecasters to justify their predictions in a geopolitical forecasting forum. Linguistic differences between forecasters are explored by aggregating metrics across each forecaster's predictions. In §3, we analyze the accuracy of individual predictions using a dataset of financial analysts' forecasts towards companies' (continuous) earnings per share. By controlling for differences between analysts and companies, we are able to analyze intra-analyst differences between accurate and inaccurate forecasts.
2.1 Geopolitical Forecasting Data
To explore the connections between language and forecasting skill, we make use of data from Good Judgment Open, $^{2}$  an online prediction forum. Users of this website share predictions in response to a number of pre-specified questions about future events with uncertain outcomes, such as: "Will North Korea fire another intercontinental ballistic missile before August 2019?" Users' predictions consist of an estimated chance the event will occur (for example,  $5\%$ ) in addition to an optional text justification that explains why the forecast was made. A sample is presented in Figure 1.
Preprocessing. Not all predictions contain associated text justifications; in this work, we only consider predictions with justifications containing more than 10 tokens. We ran langid.py (Lui
Question: Will Kim Jong Un visit Seoul before 1 October 2019?
Estimated Chance: 5%
Forecast Justification: No North Korean leader has stepped foot in Seoul since the partition of the Koreas at the end of the Korean War. ...
Figure 1: A sample prediction made by a user in response to a question posted by the Economist.
and Baldwin, 2012) to remove forecasts with non-English text, and further restrict our data to contain only users that made at least 5 predictions with text.
In our pilot studies, we also notice some forecasters directly quote text from outside resources (like Wikipedia, New York Times, etc.) as part of their justifications. To avoid including justifications that are mostly copied from external sources, we remove forecasts that consist of more than  $50\%$  text enclosed in quotation marks from the data.
Dataset statistics. We collected all questions with binary answers that closed before April 9, 2019, leading to a total of 441 questions. 23,530 forecasters made 426,909 predictions. During preprocessing steps, 3,873 forecasts are identified as heavily quoted and thus removed. After removing non-English and heavily quoted forecasts, forecasts with no text justifications or justifications less than 10 tokens, in addition users with fewer than 5 predictions with text, 55,099 forecasts made by 2,284 forecasters are selected for the final dataset.
The distribution of predictions made by each forecaster is heavily skewed.  $8.0\%$  of forecasters make over 50 forecasts. On average, each forecaster makes 10.3 forecasts, excluding those who made over 50 predictions. In Table 1, we also provide breakdown statistics for top and bottom forecasters.
2.2 Measuring Ground Truth
In order to build a model that can accurately classify good forecasters based on features of their language, we first need a metric to measure people's forecasting skill. For this purpose we use Brier score (Brier, 1950), a commonly used measure for evaluating probabilistic forecasts. For questions
3In our dataset, forecasters could even make over 1,000 forecasts with justifications.
4Other possible scoring rules exist, for example ranking forecasters by log-likelihood. For a log-likelihood scoring rule, however, we need to adjust estimates of 1.00 and 0.00, which are not uncommon in the data, to avoid zero probability events. There are many ways this adjustment could be done and it is difficult to justify one choice over another.
$^{2}$ https://www.gjopen.com/
with binary answers, it is defined as:
$$
\text {F o r e c a s t e r ' s B r i e r S c o r e} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left(f _ {i} - o _ {i}\right) ^ {2}
$$
Here  $f_{i}$  is the forecaster's estimated probability,  $o_{i}$  is a binary variable indicating the final outcome of the event, and  $N$  is the total number of forecasts. Brier scores can be interpreted as the mean squared error between the forecast probability and true answer; lower scores indicate better forecasts.
Ranking forecasters. Directly comparing raw Brier scores is problematic, because users are free to choose questions they prefer, and could achieve a lower Brier score simply by selecting easier questions. To address this issue, we standardized Brier scores by subtracting the mean Brier scores and dividing by the standard deviation within questions (Mellers et al., 2015a).
We construct a set of balanced datasets for training and evaluating classifiers by choosing the top  $K$  and bottom  $K$  forecasters respectively. In our experiments, we vary  $K$  from 100 to 1,000; when  $K = 1,000$ , the task can be interpreted roughly as classifying all  $\sim 2k$  users into the top or bottom half of forecasters.
2.3 Linguistic Analysis
In §2.2, we discussed how to measure ground-truth forecasting skill by comparing a user's predictions against ground-truth outcomes. In the following subsections, we examine a selected series of linguistic phenomenon and their connections with forecasting ability. Statistical tests are conducted using the paired bootstrap (Efron and Tibshirani, 1994). As we are performing multiple hypothesis testing, we also report results for Bonferroni-corrected significance level 0.05/30.
As discussed in §2.1, the distribution of forecasts per user is highly skewed. To control for this, we compute averages for each forecaster and use aggregate statistics to compare differences between the two groups at the user-level. Analyses are performed over 6,639 justifications from the top 500 forecasters and 6,040 from bottom 500.
2.3.1 Textual Factors
Length. We first check the average length of justifications from different groups and report our results
in Table 1. We observe that skilled forecasters normally write significantly longer justifications with more tokens per sentence. This suggests that good forecasters tend to provide more rationale to support their predictions.
Metric	Top 500	Btm 500	p
Forecasters statistics
# users making ≥ 50 forecasts	20	14	-
Avg. forecasts (w/o above users)	9.4	9.2	-
Length & word counts
Avg. # tokens per user	69.1	47.0	↑↑↑
% answers ≥ 100 tokens per user	18.5	8.3	↑↑↑
Avg. # tokens per sentence	20.9	19.2	↑↑↑
Table 1: Statistics of our dataset.  $p$ -values are calculated by bootstrap test. ↑↑↑:  $p < 0.001$ .
Readability. We compute two widely used metrics for readability: (1) Flesch reading ease (Flesch, 1948) and (2) Dale-Chall formula (Dale and Chall, 1948). Table 2 summarizes our results on average readability scores. We find good forecasters have lower readability compared to bad forecasters.
It is interesting to compare this result with the findings reported by Ganjigunte Ashok et al. (2013), who found a negative correlation between the success of novels and their readability, and also the work of Sawyer et al. (2008) who found award winning articles in academic marketing journals had higher readability. Our finding that more accurate forecasters write justifications that have lower readability suggests that skilled forecasters tend to use more complex language.
Emotion. We also analyze the sentiment reflected in forecasters' written text. Rather than analyzing sentiment orientation ("positive", "negative", or "neutral"), here we focus on measuring sentiment strength. We hypothesize that skilled forecasters organize their supporting claims in a more rational way using less emotional language. Many existing sentiment analysis tools (e.g., Socher et al. (2013)) are built on corpora such as the Stanford Sentiment Treebank, which are composed of movie reviews or similar texts. However, justifications in our dataset focus on expressing opinions towards future uncertain events, rather than simply expressing preferences toward a movie or restaurant, leading to a significant domain mismatch. In pilot studies, we noticed many sentences that are marked as negative by the Stanford sentiment analyzer on our data do not in fact express a negative emotion. We thus use Semantic Orientation CALculator (SO
Readers may wonder if there do exist differences between top and bottom forecasters. We provide justifications for our ranking approach in Appendix A.1.
Metric	p	Bonferroni
Textual Factors
Readability
Flesch reading ease	↓↓
Dale-Chall	↑↑↑	*
Emotion
Absolute sentiment strength	↓↓↓	*
Parts of Speech
Cardinal	↑↑↑	*
Noun	↑↑
Preposition	↑↑↑	*
Pronoun	↓↓↓	*
1st personal pronoun	↑
Verb	↓↓↓	*
Cognitive Factors
Uncertainty
% uncertain statements	↑↑↑	*
Tentative (LIWC)	↑↑↑	*
Thinking style
% forecasts with quoted text	↑↑↑	*
Temporal orientation
Focus on past	↑↑
Focus on present & future	↓↓↓	*
Table 2: Comparison of various metrics computed over text written by the top 500 and bottom 500 forecasters. Good forecasters tend to exhibit more uncertainty, cite outside resources, and tend toward neutral sentiment; they also use more complex language resulting in lower readability and focus more on past events.  $p$ -values are calculated by bootstrap test. The number of arrows indicates the level of  $p$ -value, while the direction shows the relative relationship between top and bottom forecasters,  $\uparrow \uparrow \uparrow$ : top group is higher than bottom group with  $p < 0.001$ ,  $\uparrow \uparrow \uparrow$ :  $p < 0.01$ ,  $\uparrow$ :  $p < 0.05$ . Tests that pass Bonferroni correction are marked by *.
CAL), a lexicon-based model proposed by Taboada et al. (2011) which has been demonstrated to have good performance across a variety of domains. The model generates a score for each justification by adding together semantic scores of words present in the justification, with a 0 score indicating a neutral sentiment. We then take the absolute values of scores from the model and calculate averages for each group. Results in Table 2 show that the top 500 forecasters have a significantly lower average sentiment strength compared to bottom 500 forecasters, indicating statements from skilled forecasters tend to express neutral sentiment.
Parts of Speech. As shown in Table 2, we observe that top forecasters use a higher percentage of cardinal numbers and nouns, while higher numbers of
verbs are associated with lower forecasting ability. $^{6}$
We also note the bottom 500 use a higher percentage of pronouns when justifying their predictions. To investigate this difference, we further separate first person pronouns from second or third person pronouns. As presented in Table 2, first person pronouns are used more often by the top forecasters.
2.3.2 Cognitive Factors
We now evaluate a number of factors that were found to be related to decision making processes based on prior psychological studies (e.g., Mellers et al. (2015a)), that can be tested using computational tools. A number of these metrics are calculated by using the Linguistic Inquiry and Word Count (LIWC) lexicon (Tausczik and Pennebaker, 2010), a widely used tool for psychological and social science research.
Uncertainty. To test the hypothesis that good forecasters have a greater tolerance for uncertainty and ambiguity, we employ several metrics to evaluate the degree of uncertainty reflected in their written language. We use the model proposed by Adel and Schütze (2017) to estimate the proportion of uncertain statements made by each forecaster in our dataset. It is an attention based convolutional neural network model, that achieves state-of-the-art results on a Wikipedia benchmark dataset from the 2010 CoNLL shared task (Farkas et al., 2010); we use the trained parameters provided by Adel and Schütze (2017). After the model assigns an uncertainty label for each sentence, we calculate the percentage of sentences marked as uncertain. Results of this analysis are reported in Table 2; we observe that the top 500 forecasters make a significantly greater number of uncertain statements compared to the bottom 500, supporting the hypothesis mentioned above.
Thinking style. In §2.1, we discussed the issue that many forecasts contain quoted text. Although we removed posts consisting of mostly quoted text as a preprocessing step, we are interested in how people use outside resources during their decision making process. We thus calculate the portion of forecasts with quotes for the two groups. We notice skilled forecasters cite outside resources more frequently. This may indicate that skilled forecasters tend to account for more information taken from external sources when making predictions.
POS tags were obtained using Stanford CoreNLP.
7"I, "me", "mine", "my" and "myself".
Temporal orientation. We make use of the LIWC lexicon (Tausczik and Pennebaker, 2010) to analyze the temporal orientation of forecasters' justifications. We notice good forecasters tend to focus more on past events (reflected by tokens like "ago" and "talked"); bad forecasters pay more attention to what is currently happening or potential future events (using tokens like "now", "will", and "soon"). We conjecture this is because past events can provide more reliable evidence for what is likely to happen in the future.
2.4 Predicting Forecasting Skill
In §2.3, we showed there are significant linguistic differences between justifications written by skilled and unskilled forecasters. This leads to a natural question: is it possible to automatically identify skilled forecasters based on the written text associated with their predictions? We examine this question in general terms first, then present experiments using a realistic setup for early prediction of forecasting skill in §2.5.
Models and features. We start with a log-linear model using bag-of-ngram features extracted from the combined answers for each forecaster. We experimented with different combinations of n-gram features from sizes 1 to 4. N-grams of size 1 and 2 have best classification accuracy. We map n-grams that occur only once to a  $\langle \mathrm{UNK} \rangle$  token, and replace all digits with 0. Inspired by our findings in §2.3, we also incorporate textual and cognition factors as features in our log-linear model.
We also experiment with convolutional neural networks (Kim, 2014) and BERT (Devlin et al., 2019). The 1D convolutional neural network consists of a convolution layer, a max-pooling layer, and a fully connected layer. We minimize cross entropy loss using Adam (Kingma and Ba, 2015); the learning rate is 0.01 with a batch size of 32. We fine-tune BERT on our dataset, using a batch size of 5 and a learning rate of 5e-6. All hyperparameters were selected using a held-out dev set.
Model performance. Results are presented in Table 3. As we increase the number of forecasters  $K$ , the task becomes more difficult as more forecasters are ranked in the middle. However, we observe a stable accuracy around  $70\%$ . All models consistently outperform a random baseline (50% accuracy), suggesting that the language users use to describe their predictions does indeed contain information that is predictive of forecasting ability.
The n-grams with largest weights in the logistic regression model are presented in Table 4. We find that n-grams that seem to indicate uncertainty, including: "it seems unlikely", "seem to have" and "it is likely" are among the largest positive weights.
K	100	200	300	500	1000
LR	Bag-of-ngrams	69.5	74.2	72.5	69.2	64.8
Textual	66.0	60.8	62.0	59.3	57.4
Cognitive	69.0	68.0	67.3	65.5	61.0
All above	70.5	73.5	73.3	69.8	64.7
Neural	CNN	71.5	75.0	72.0	69.6	64.0
BERT-base	74.5	77.3	74.3	69.7	65.1
Table 3: Accuracy  $(\%)$  on classifying skilled forecasters when choosing the top  $K$  and bottom  $K$  forecasters. For logistic regression (LR), we experiment with different sets of features: bag of  $\{1,2\}$ -grams, textual factors in §2.3.1, cognitive factors in §2.3.2, and combination of all above. For neural networks (Neural), we use convolutional neural network (CNN) and BERT-base. All results are based on 5-fold cross validation.
Top15
(High-weight)	in the next / . also , / . however , / based on the / there are no / . according to / of time . / . based on / they wo n’t / there is no / it seems unlikely / do n’t see / it is likely / more of a / seem to have
Bottom15
(Low-weight)	will continue to / it will be / the world . / . it ’s / there is a / is not a / the west . / to be on / to be the . / yes , / he ’s a / there will be / in the world / will still be / . he will
Table 4: High and low-weight n-gram features from the logistic regression model trained to identify good forecasters ( $K = 500$  with only 3-gram features for interpretability). Positive features indicate some uncertainty (e.g., "it is likely", "seem to have", "it seems unlikely"), in addition to consideration of evidence from many sources (e.g., "based on the", ". according to").
2.5 Identifying Good Forecasters Earlier
With the model developed in §2.4, we are now ready to answer the following question: using only their first written justification, can we foresee a forecaster's future performance?
Setup. Our goal is to rank forecasters by their performance. We first equally split all 2,284 forecasters into two groups (top half versus bottom half) based on their standardized Brier scores. We then partition them into  $60\%$  train,  $20\%$  validation, and  $20\%$  test splits within each group. We combine all justifications for each forecaster in the training set. For forecasters in the validation and test sets,
we only use their single earliest forecast.
We use forecasters' final rank sorted by averaged standardized Brier score over all forecasts as ground truth. We then compare our text-based model to the following two baselines: (1) a random baseline  $(50\%)$  and (2) the standardized Brier score of the users' single earliest forecast.
Results. We calculate the proportion of good forecasters identified in the top  $N$ , ranked by our text-based model, and report results in Table 5. We observe that our models achieve comparable or even better performance relative to the first prediction's adjusted Brier score. Calculating Brier scores requires knowing ground-truth, while our model can evaluate the performance of a forecaster without waiting to know the outcome of a predicted event.
P@10	P@50	P@100
Brier score	60	64	62
Text-based (LR)	70	70	65
Text-based (CNN)	90	68	64
Text-based (BERT-base)	80	70	67
Table 5: Precision@  $N$  of identifying skilled forecasters based on their first prediction.
3 Companies' Earnings Forecasts