Forecasting Uncertainty in Conversations using Large Language Models (2024)

Abstract: Effective interlocutors account for the uncertain goals, beliefs, and emotions of others. But even the best human conversationalist cannot perfectly anticipate the trajectory of a dialogue. How well can language models represent inherent uncertainty in conversations? We propose FortUne Dial, an expansion of the long-standing “conversation forecasting” task: instead of just accuracy, evaluation is conducted with uncertainty-aware metrics, effectively enabling abstention on individual instances. We study two ways in which language models potentially represent outcome uncertainty (internally, using scores and directly, using tokens) and propose fine-tuning strategies to improve calibration of both representations. Experiments on eight difficult negotiation corpora demonstrate that our proposed fine-tuning strategies (a traditional supervision strategy and an off-policy reinforcement learning strategy) can calibrate smaller open-source models to compete with pre-trained models 10x their size.

heading: 1 Introduction
text: Implicit Uncertainty Direct Uncertainty

⚖ 🔬 ✅
👨⚖ Well… Your Honor 👩⚖ And it's just too bad for the U.S. 👨⚖ -- No, to the contrary, the U.S. claim
is preserved. They can prove that. 👩⚖ But your position was that it can't...
💁 If I can the balls you can have the books 🙋 How about I get 1 hat and 1 book?  Can’t take that deal.
Collaborative Negotiations (like legal cases) Distributive Negotiations 8 Benchmark Conversation Uncertainty Tasks
Will the petition be accepted? What are the chances of a deal?
2 Uncertainty Tuning and Inference Methods
Post Correction, Interpretable Eval, Open LMs
🔐
There is a 60% chance.P(token=“Yes”)
60% 80%
versations (Lewis et al., 2017; Zhou et al., 2019; Schluger et al., 2022; Argyle et al., 2023), or assessing trust/heterogeneity in a data source via metrics like entropy (Csáky et al., 2019; Kuhn et al., 2022).
Here, we focus on the case of negotiations; this type of conversation is not only particularly sensitive to social uncertainties, but also, outcomes are readily quantified post-hoc. We ask language models questions about the likelihood of deals, decisions, and emotional conflicts in settings like marketplaces, online forums, and courtrooms, totaling 8 tasks to test uncertainty modeling in negotiations. Our contributions include:
1. formalizing the conversation uncertainty modeling task, along with its metrics (§ 2.1); 2. introducing two methods for representing uncertainty about the outcome of conversations using language models (§ 2.2); 3. and proposing fine-tuning (§ 2.3, § 2.4) and inference-time strategies (§ 2.5) for improving these representations.
We call this task FortUne Dial.1 Experiments (§ 3) show GPT-4 and other large models can anticipate outcome certainty well, improving over prior knowledge by up to 9%. Moreover, results show the utility of our fine-tuning strategies: smaller (7B) models are tuned to outperform pre-trained, open-source models 10x their size. Indeed, metrics improve up to 11% on the tuning datasets and up to 3% out-of-distribution. Besides the performance of our model deliverables, experiments also communicate insight on the biases of pre-trained language models at this task, the ability of different models to make use of prior knowledge, and the generalization of different algorithmic strategies. Models and code are available on github.2

heading: 2 Modeling Uncertainty in Conversations

heading: 2.1 Problem, Notation, and Evaluation
text: Consider a natural language token set T . We observe partial multi-party dialogues D ∈ T ∗ consisting of K ∼ U{2, L} turns, with L + 1 being the eventual (random) length of the full dialogue.3 Speaker turns are delimited by special sequences of tokens; e.g., “Speaker 4: ...” These partial conver-
1Forecasting Uncertainty in Dialogue. 2https://github.com/anthonysicilia/fortune-dial 3Uncertainty may be higher at different points in the dialogue (e.g., the beginning). Uniform sampling ensures we capture all scenarios (both high and low uncertainty), evaluating the model on diverse contexts.
sations are unfinished, but have eventual outcome O ∈ O = {0, 1}.4 Nature picks a conversation distribution D over T ∗ ×O which governs our supervised observations: (D,O) ∼ D. A forecaster f maps D 7→ P̂ ∈ [0, 1] where P̂ estimates the probability O = 1.
Evaluation with Proper Scores A calibrated forecaster satisfies (Bröcker, 2009):
E[O | P̂ = p] = p ∀p ∈ {f(x) | x ∈ T ∗}, (1)
which intuitively means if we consider all conversations assigned p by the forecaster, the mean occurrence of the outcome should also be p. While commonly used to asses the verity of general probability estimates (Guo et al., 2017) the constraint in Eq. (1) is often too broad because calibration, by itself, fails to measure the variance of a forecast (Ovadia et al., 2019). For example, the constant forecast P̂ = E[O] is calibrated, but rarely captures the true outcome probability (conditioned on the conversation). The issue of variance is especially important in our setting, where social and temporal uncertainties make anticipation difficult; i.e., the basic, indiscernible prediction E[O] may be competitive. To accommodate calibration and variance, we consider the constraint
P̂ = P def = E[O | D] (2)
One way to achieve this is by optimizing a scoring function s : [0, 1]×O → R≥0:
minf E[s(P̂ , O)]. (3)
If the scoring function is strictly proper,5 Eq. (2) is satisfied by the minimizer of (3), so solving (3) recovers the true uncertainty as desired. Moreover, Eq. (3), indeed, optimizes variance and calibration equally, among other nice properties for ranking suboptimal forecasts (Bröcker, 2009).
Tangible Scores We use proper scores only, such as the Brier Score (BS; Brier, 1950), which is the mean squared error between forecast probabilities and true outcomes. While the use of proper scores is important (see previous), they do present some caveats: (1) they lack interpretable units and (2) for
4We only consider binary outcomes, but are flexible in application of this formulation, e.g., we can ask multiple questions to handle more outcomes in a one-vs-all fashion.
5Strict propriety requires that E[s(P,O)] ≤ E[s(P̂ , O)] for all P̂ with equality if and only if P̂ = P .
fixed tasks, they often vary on a small scale.6 To resolve these issues, we sometimes focus evaluation on the Brier skill score:
BSS = 1− BS/BSref (4)
where BS is the Brier score of the forecaster we are evaluating and BSref is the Brier score of some reference forecaster. One way to interpret the skill score is the percent improvement of the forecaster compared to the reference. A simple reference, first proposed by Brier (1950), is the constant prediction E[O], in which case BSref happens to be the variance of the outcome. Here, we may interpret the skill score as the percent of variance in outcome that is explained by the forecaster, like an R2-value. On the other hand, E[O] can also be viewed as prior knowledge, obtainable before observing D, implying skill conveys improvement over our prior knowledge. As desired, skill score tends to vary more than Brier score, while also having an interpretable unit (percentage).
Aleatoric and Epistemic Uncertainty There are traditionally two main types of uncertainty which are studied in machine learning (and other) literature: epistemic uncertainty, which relates to uncertainty caused by a person or model’s knowledge (lack thereof) and aleatoric uncertainty, which relates to inherent uncertainty in the data and is independent of the human/model (Hüllermeier and Waegeman, 2021). Importantly, this work only aims to improve quantification of the aleatoric uncertainty (inherent to the data) without consideration of the epistemic factors that impact specific interlocutors. We leave this for future research.

heading: 2.2 Language Models as General Forecasters
text: An (auto-regressive) language model LMθ is a function parameterized by θ ∈ Rd that returns a distribution over the next token t ∈ T conditional to any prefix x ∈ T ∗. We write T ∼ LMθ(x) for a single token sample and T ∼ LM∗θ(x) for the iterated sampling process, wherein we append a sampled token to x and re-sample until a stopping condition. We define a prompt Φ as a function Φ : T ∗ → T ∗ such that for any input x, it holds that x is substring of Φ(x). So, Φ takes an input text x and modifies it to a new text Φ(x), which contains the original text and (usually) adds important meta-information for solving the task; e.g., goal descriptors, expected
6This is perhaps the reason for common improper choices like calibration error, that do not account for forecast variance.
output, and other context. We consider two types of prompts, which can turn a language model into a probability forecaster:
1. Implicit Forecasts (IF): The prompt ΦO poses the question “Given the partial dialogue D, will the outcome represented by O occur?” Then, the language model forecasts as
P̂IF = P{T = yes}; T ∼ LMθ ◦ ΦO ◦D (5)
where yes ∈ T is an affirmation token and ◦ is function composition.
2. Direct Forecasts (DF): The prompt ΦO poses the modified question “Given the partial dialogue D, what is the probability the outcome represented by O will occur?” Then, the model forecasts as:
P̂DF = p ◦ T ; T ∼ LM∗θ ◦ ΦO ◦D (6)
where p : T ∗ → [0, 1] is a parser that extracts a “probability estimate” from sample T ; i.e., the model answers directly in natural language.
More details on prompts are in § 3. Abstractly, both prompts describe the uncertainty modeling task using language, but make different assumptions.

heading: 2.3 Uncertainty Tuning of Implicit Forecasts
text: We consider a language model with pre-trained parameters θinit, e.g., pre-tuned to follow instructions (Ouyang et al., 2022). The model computes a score vector Z|Φ(D) ∈ R|T | and uses Z to forecast:
P̂IF = P{T = yes} = exp(Zyes/τ)∑ t∈T exp(Zt/τ)
(7)
where temperature τ is a fixed hyper-parameter. A fine-tuning objective can then be written:
max θ : θinit→θ
E[O ln P̂IF +O lnP{T = no}] (8)
where no is a dis-affirmation token. In effect, Eq. (8) translates the objective in Eq. (3) to a finetuning objective by picking s to be the negative log score (a proper score, essentially equivalent to standard cross-entropy). Jiang et al. (2021) also consider calibration of pre-trained language models by direct supervision (as above), but focus on “factual” question-answering tasks where answers are more clearly right/wrong and the inherent social/temporal uncertainties of conversation are absent. In addition to a difference of setting, our
proposal also differs from Jiang et al. (2021) because we retain the language model’s whole token distribution during inference, instead of only a candidate set. In § A.1, we provide a first theoretical and empirical characterization of the impact of this choice when fine-tuning language models. Our main observation is these techniques are practically equivalent at inference-time with less than 1% average difference in forecast (for our corpora). Thus, we advocate to retain the whole token distribution, since it is more easily coupled with other language modeling tasks; e.g., it doesn’t require special machinery, like a loss with separate normalization.
Sampling Distribution In practice, we consider several negotiation datasets, defining distributions D1 . . .Dℓ and prompts Φ1 . . .Φℓ. At test-time, if dataset diversity is sufficient, we expect the forecasts will generalize to new, possibly unseen, environments Dℓ+1 and prompts Φℓ+1. Formally, this setup is called domain generalization (Blanchard et al., 2011; Muandet et al., 2013) and, while many approaches to this problem exist, simply training on the balanced aggregate of all available domains often performs best in practice (Gulrajani and LopezPaz, 2020); we take this approach in § 3.

heading: 2.4 Uncertainty Tuning of Direct Forecasts
text: Current pre-training strategies may prime models to express uncertainty best directly, via their output tokens; e.g. this is observed when models express uncertainty about factual correctness in questionanswering (Tian et al., 2023). Ideally, despite the different setting, fine-tuning can preserve and capitalize on this predisposition. One challenge is that direct forecasts make Eq. (3) non-differentiable, due to the parser. So, we formulate direct forecast tuning as a Markov Decision Process. We use reward R = −s(p ◦ T,O) with T ∼ LM∗θ ◦ Φ ◦ D and set s to the log score (see Eq. 8). In effect, the reward is the negative score of our forecaster. Then, the usual objective J(θ) of this Markov Decision Process is:
max θ : θinit→θ E[R] = − min θ : θinit→θ E[s(P̂DF, O)]. (9)
That is, we recover the original forecasting objective. While significant machinery has been developed for reward optimization (see Sutton and Barto, 2018) we apply policy gradient.

heading: 2.4.1 Policy Optimization
text: We focus on gradient-based policy optimization techniques, like REINFORCE (Williams, 1992)
and PPO (Schulman et al., 2017). In particular, we derive an off-policy version of the policy-gradient theorem, specific to our forecasting task, which uses Monte Carlo samples to produce unbiased estimates of the gradient-updates for our optimization problem. The off-policy aspect is an important one. It means we can iteratively sample any policy (distribution) over our token space T , and use these demonstrations to learn θ. Thus, while tuning, we can prioritize exploration vs. exploitation however we like, which can be an important factor for acting optimally in very general environments (Jiang et al., 2023), as is desired by our framework.
Off-Policy Policy Gradient For any random variable X , define µX as the mass function of X . Then, for any reference model Ref : T ∗ → ∆(T ):
∇θE[R] = E [ sT̃ · µT (T̃ )µ T̃ (T̃ ) · ∇θ logµT (T̃ ) ]
where T ∼ LM∗θ ◦ Φ ◦D, T̃ ∼ Ref∗ ◦ Φ ◦D, and sT̃ = −s(p ◦ T̃ , O).
(10)
We derive this in § A.3. While other off-policy policy gradient techniques exist (Degris et al., 2012; Imani et al., 2018; Kallus and Uehara, 2020), the specifics of our problem allow us to make simplifying assumptions and yield a “simpler” and unbiased estimate of ∇θE[R] as the above.
As a computational note, there may be instances where the ratio of mass functions uT /uT̃ becomes excessively large or small, leading to issues of gradient explosion or vanishing. To address this, we adopt a widely-used clipping strategy from Proximal Policy Optimization (Schulman et al., 2017). Specifically, for ϵ ∈ [0, 1], the update is:
sT̃ · ω · ∇θ logµT (T̃ ) where ω = min { max { µT (T̃ )
µ T̃ (T̃ )
, 1− ϵ } , 1 + ϵ } .
(11)
Besides the computational benefits, this has motivations related to on-policy trust-region optimization (Schulman et al., 2015), when Ref = LMθ.
Off-Policies We consider three main choices for the off-policy reference Ref in this work:
1. The Explorer takes random actions in the token space T ∗, restricted only in the sense that it must describe a probability; e.g., “53%.” Exploration can benefit generalization (Jiang et al., 2023), since it exposes the agent to more diverse state-action pairs at train-time.
2. The Exploiter takes the actions that are optimal based on experience. We use P̂IF has a proxy, since it does optimize a proper scoring function (Eq. 8). While optimal on the training data, this may not be true for new conversation domains or outcomes. 3. The Quantizer takes actions learned from ground truth data by binning. It is inspired by Lin et al. (2022), who calibrate model uncertainty to factual correctness by binning subtasks, computing average correctness in each bin, and training the model to predict these with routine supervision. Lacking clear “subtasks”, we propose a (new) more general strategy, using clustering to bin our data. In § 3, we also ablate our use of RL to optimize, instead of routine supervision. Greater detail on these policies, including precise definitions and implementation are in § A.4.

heading: 2.5 Post-Hoc, Inference-Time Corrections
text: A New View on Old Tricks If validation data is available, temperature (τ ) scaling (Guo et al., 2017) is common to correct the scale of implicit forecasts (Jiang et al., 2021; Kadavath et al., 2022). Yet, this helpful technique is not well-studied for direct forecasts because these are parsed from a discrete token sequence (they have no underlying latent scores to scale). To address this, we propose a unified correction strategy that is well-suited for both implicit and direct forecasts. We suggest estimation of the underlying latent scores to allow post-hoc scaling for any forecast style:
Ẑyes ← log P̂ /(1− P̂ ) Z̃yes ← Ẑyes/τ − β P̂new ← 1/(1 + exp(−Z̃yes))
(12)
where τ relates to temperature as before and β is a bias correction term. By estimating the latent score as above, we can effectively “simulate” traditional
(τ ) scaling in such a way that it works for both implicit and direct forecasts. We argue this theoretically and compare our proposal to other correction techniques in § A.2, finding it theoretically equivalent, practically equivalent, or better in general. We use Eq. (12) as the primary correction in § 3.
A Bayesian View Use of prior knowledge is considered an important qualifier for generalization in some theories of machine learning (McAllester, 1998). Motivated by this, we explore the use of natural language priors as a type of inference-time correction; e.g., , “On average, this type of conversation ends with {x} about {y}%...” Additional details on these priors are provided next.
Correction is Not Always Zero-shot Quality corrections require data in new domains. Data (n ≈ 250) is generally used to learn τ, β in Eq. (12), so scaling is not zero-shot. For priors, data can also be used or, in practice, a “guess” can be made. To simulate a “guessed” prior, we use averages outside a 95% confidence interval of the data average (n=50). Intuitively, this means we would rarely estimate this average using data (across repeated experiments), which is akin to, or worse than, human guesswork. § 3 considers data priors learned from data and bad priors using a high and low simulated guess.7 We consider only the second to be zero-shot, since it tests robustness to “guesses” we expect to rarely obtain via data.

heading: 3 Experiments
text: Data & Splits We consider 8 modeling tasks spanning both traditional (distributive) negotiations and collaborative negotiations (Chu-Carroll and Carberry, 1995); Table 1 summarizes the corpora. Tasks have diverse situations and outcomes, span multi-party/dyadic settings, and are both short/long.
7We also checked non-numeric priors, e.g., reminding all outcomes have non-zero likelihood; these were worse overall.
and data-dependent priors is not truly zero-shot. GPT-4 uses direct forecasting (DF), while Llama-2-chat 70B uses direct or implicit (IF). BSS = 0 corresponds to always forecasting the prior, e.g., for data prior, this is the corpus mean outcome. bad priors are defined in § 2.5. DF consistently improves upon prior knowledge (BSS > 0).
OOD doesn’t scale and uses “bad” priors. For 7B, we also tune on ×4 more data. Improvements are highlighted: Light green cells show improvement against corresponding pre-trained models (same setup, before uncertainty tuning), while darker cells (additionally) improve over larger 70B DF (LSS always compares to DF).
To simulate varying degrees of distribution shift, we group these datasets into different train/test splits, categorized as easy, medium, or hard. To make the forecasting task more difficult, each of these three splits hold out full datasets for testing. Conceptually, the splits are designed to create different degrees of train/test imbalance for important properties like the topic, the length of the conversation, the type of outcome, and the number of speakers; Tables 6+7 and § B.1 provide more detail on train/val/test splits. Sometimes, we restrict inference to data with affective conflict as outcome (neg) like, a personal attack or unhappy speaker. Here, to compute some metrics, we swap the positive and negative classes as needed, e.g., the positive class becomes 1 − O to study “both camps unhappy” instead of “both camps happy.”
Models & Prompts We use GPT-4 (0613, OpenAI, 2023), Llama-2-chat (Touvron et al., 2023), and Zephyr-β (Tunstall et al., 2023), which have all been pre-tuned for chat/instruction following (Zephyr is pre-tuned via distillation). We also use a “tiny” Llama-2 replicate trained in the style of Zephyr (Chat-v0.6, Zhang et al., 2023). Opensource model sizes range from 7B to 70B parameters. We use pre-trained to refer to these models before we tune uncertainty (§ 2.3, 2.4). Concretely, the prompts the models receive have: situational context specific to each dataset, like “the speakers are defending their opinions on an issue”; priors, as in § 2.5; and the main question that asks the model about the likelihood of outcome occurrence in the conversation, or just occurrence for implicit forecasts. Pre-trained models also receive system prompts to constrain output and clarify the task goals. We use QLoRA (Dettmers et al., 2023) for uncertainty tuning. Additional details are in § B.2.
Metrics We use the Brier Score (BS) and skill score (BSS) as discussed in § 2.1, macro-averaged across datasets and prompts. BSS refers to the original skill score (Brier, 1950) where the reference model in the skill score is the constant (average) outcome probability. When a prior is provided, we substitute this prior for the data mean in the reference score to account for how priors can implicitly bias forecasts. Indeed, variance around an (incorrect) prior is always higher than the true variance, so the prior-adjusted BSS reports the percent of this larger variance explained by the forecaster. Besides BSS, we also suggest a new skill score called the
Llama Skill Score (LSS). LSS is identical to usual skill scores, i.e., Eq. (4), but uses the Brier score of Llama-2-chat 70B (direct forecasts, same experimental setup) as the reference score BSref. This quantifies how smaller fine-tuned models compare to this large model by % improvement. Since tuned models are smaller, % improvement is sometimes negative: a less negative value is a smaller decrease (compared to Llama 70B) which means better performance. Finally, we report statistical bias (BI) to convey average over- or under-estimation of outcome probability (positive or negative values, respectively). In addition to discussion (§ 1, 2.1), empirical motivation for uncertainty-aware metrics is given in Table 8, comparing BSS to more typical classification metrics.

heading: 3.1 Results and Discussion
text: Forecasting is Better with Correction Table 2 shows Brier and skill scores of pre-trained models without uncertainty tuning. We modulate priors and ablate post-hoc scaling. GPT-4 has lower (better) Brier scores when granted access to validation data to make corrections, e.g., via data priors or scaling. Scaling appears to have the greatest impact on scores as they are lowest, even when excluding the data prior. However, priors are still useful. In zero-shot settings (i.e., no scaling or data prior), GPT-4 performs better when it has access to “bad” guesses of prior probability, rather than no guess at all. Trends are similar for Llama-2-chat 70B.
How Priors Help Curiously, “bad” priors can also improve model inferences. We hypothesize the benefit of priors may also come from eliciting “chain-of-thought” behavior at inference-time (Wei et al., 2023), compounding the improvement gained from increased information access. Manual analysis of GPT-4 on the corpora of Zhang et al. (2018) shows that GPT-4 explains its answers 60-85% more frequently, when provided a prior (depending on exact prior setting).

heading: Problems with Pre-trained Implicit Forecasts
text: For Llama-2-chat 70B, we have access to scores of every token, so we can compare implicit forecasting to direct forecasting. In Table 2, implicit forecasting is worse for this pre-trained model. Echoing Kadavath et al. (2022), we find post-hoc correction is vital to improve pre-trained implicit forecasts. Moreover, degradation of uncorrected implicit forecasts is high, suggesting amplification
of this effect in our unique (conversational) setting. Manual inspection suggests the model’s logit probabilities for “Yes” tends to be much smaller than 1%. When scaled, these probabilities range more appropriately (for example, 10% to 90%) and become adequately predictive.

heading: Have Data? Tuned Implicit Forecasts Are Best
text: Based on the previous results, we focus on uncertainty tuning with access to a prior (even a “bad” one) and compare uncertainty tuned models to direct forecasts of pre-trained versions. We consider an in-domain setting first, wherein test data follows the tuning distribution, post-hoc correction is used, and priors are data-dependent. Here, Table 3 shows tuned implicit forecasts can significantly improve over pre-trained Brier score, even compared to models 10x their size. For instance, a tuned Llama-2-chat 7B improves over the 70B model by about 11% (or, 5% with less data). With enough training data, out-of-distribution (OOD) scores are also about 4% better than Llama-2-chat 70B scores (if both have access to data for correction).

heading: Direct Tuning Generalizes Better (Sometimes)
text: Next, we consider a zero-shot OOD setting without data for correction. Here, performance of tuned implicit forecasts is still good, for Llama-2-chat 7B. With enough data, tuning brings Llama-2-chat 7B implicit forecasts to the skill of its 70B counterpart (+0.7%), but for Zephyr-style (distillationtuned) models, degradation is significant compared to (even) pre-tuning scores. In contrast, for all 7B models and different levels of data access, direct forecasts show consistent improvement of scores after uncertainty tuning (see light green cells). For Llama-2-chat 7B, tuned direct forecasts also handle difficult (hard) distribution shift up to 4% better than their 70B counterpart. As speculated earlier, direct forecast tuning may preserve some predispositions of pre-trained models to direct uncertainty signals; this may explain consistent generalization of these forecasts beyond the tuning distribution.
Qualitative Comparison of IF and DF Tuned implicit forecasts tend to have higher variance than tuned direct forecasts. Averaging over all tuning strategies in Table 3, implicit tuning of Llama-2chat 7B leads to 2× the standard deviation in forecasts compared to direct tuning (about 11% and 5% SD, respectively). As noted in § 2.1, all else equal, a higher variance is preferred by our metrics in order to capture the discernability of forecasts.
Potentially, directly tuned models tend towards distribution collapse due to insufficient regularization in our RL objective (Korbak et al., 2022). Methods to resolve this will be of interest in future work.
Impact of Scale Tuning of implicit forecasts allow in-domain scores of a 1B model to rival a model 70x it’s size (-0.5%). But, tuning methods show less improvement, or even degradation, when applied to the 1B model OOD. Possibly, and especially since we use QLoRA tuning, the reduction in trainable parameters is detrimental to these methods. This, and the observed benefits of increasing data (see previous discussion), suggest our tuning techniques may follow neural scaling laws (Kaplan et al., 2020), meaning generalization is strongly dependent on model, data, and compute scale.
Exploration & Exploitation Llama-2-chat 7B findings indicate pure exploitation is detrimental to generalization when compared with pre-trained models (see absence of green scores). On the other hand, exploration tends to offer some improvement, beating the pre-trained scores on easy and all as well as the 70B scores on hard. The best tuned direct forecasts use quantization , which is neither completely random nor optimal on the train set, offering a balance of exploration/exploitation.
Benefits of RL For our best performing direct forecast tuning mechanism, we also ablate the role of using RL to tune; i.e., we use a traditional supervised update rule, similar8 to Lin et al. (2022). We find improvements over pre-training are less consistent and worse overall.

heading: Human Preference Tuning May Induce Bias
text: We also postulate human preference tuning (RLHF; e.g., Ouyang et al., 2022) may bias pre-trained models to under-estimate negative affective conflicts, since these are presumably undesirable to human annotators. To study this, we report statistical bias (BI) of forecast probabilities in predicting negative (neg) emotional outcomes; i.e., personal attacks, unhappy interlocutors, or refusals to donate to charity. On average, direct forecasts from GPT-4 underestimate this probability by about 11%, while direct forecasts from Llama-2-chat 70B underestimate this probability by about 6%. For the same models, bias across all outcomes is not as staggering. Smaller language models do not necessarily exhibit this bias, but distillation-tuned models
8Ours is still more general, due to the clustering proposal.
may learn similar biases from their larger teachers (see 7B). Generally, uncertainty tuning does not appear to introduce as staggering bias against emotional conflict, but especially for distillation-tuned models, the overall bias can be elevated.

heading: 4 Related Works
text: Negotiation & Conversation Forecasting Negotiation and dispute modeling has a long history (Lambert and Carberry, 1992; Jameson et al., 1994; Traum et al., 2008; Lascarides and Asher, 2008) with early works hand-crafting models of interlocutor behavior by logical or discourse structures. Reinforcement learning in simulated environments offers improvement (Georgila and Traum, 2011; Efstathiou and Lemon, 2014) with most recent advances modeling opponents’ dialogue acts (Keizer et al., 2017), word choices (He et al., 2018), and mental states (Yang et al., 2021; Chawla et al., 2022). Instead of full simulation, we focus on efficient and interpretable outcome models (Sokolova et al., 2008; Nouri and Traum, 2014). Outcome models, or forecasts, are also common in broader dialogue for proactive moderation of social media (Zhang et al., 2018; Kementchedjhieva and Søgaard, 2021) as well as predicting task-success (Walker et al., 2000; Reitter and Moore, 2007), mental health codes (Cao et al., 2019), emotions (Wang et al., 2020; Matero and Schwartz, 2020), situated actions (Lei et al., 2020), and financials (Koval et al., 2023). Among these, ours is first to propose and evaluate probabilistic methodology, modeling dynamic uncertainty for the first time. Our proposal is also uniquely general, operating independent of setting or outcome of interest. Indeed, we evaluate on general negotiations, looking beyond distributive applications (zero-sum games in specific markets) to include common collaborative negotiations (Chu-Carroll and Carberry, 1995), like planning, where parties share some goals, but conflicts arise from competing sub-goals or beliefs.
Language Models & Uncertainty Modern large language models fine-tuned to human preferences (Ouyang et al., 2022) are increasingly general, “unsupervised” multi-taskers. When queried on factual information, these models also represent uncertainty about their solutions with little to no supervision (Kadavath et al., 2022). Uncertainty (about correctness) has also been studied in smaller models without preference tuning (Desai and Durrett, 2020; Jiang et al., 2021; Dan and Roth, 2021) with
many algorithms for improvement (Kong et al., 2020; Zhang et al., 2021; Li et al., 2022). Albeit similarly operationalized, modeling uncertainty about factual correctness is distinct from our focus in negotiations, which elicits modeling of social dynamics and mental states. Fewer works study uncertainty in social reasoning (Jiang et al., 2021; Hosseini and Caragea, 2022; Kumar, 2022). These look at smaller models without instruction-tuning, and lack focus on the interactive, temporal aspects of conversations that cultivate an inherent uncertainty about future outcomes. Ours is also one of few works that study how models communicate uncertainty directly via output tokens (Mielke et al., 2022; Tian et al., 2023), and fewer that propose tuning algorithms for this (Lin et al., 2022).

heading: 5 Conclusion
text: We show language models represent uncertainty about conversational outcomes quite well, depending on their size, inference strategy, training strategy, and access to prior knowledge. We design a task to evaluate this ability and show: • large (commercial-scale) models do this well,
provided limited data to pick hyper-parameters; • without data, these models still offer improve-
ment over low quality priors, like human guesses; • specialized fine-tuning can elevate small open-
source models to beat models 10x their size; • current pre-trained language models may be pre-
disposed to representing uncertainty in their textual outputs, instead of their logit distributions; • exploration at train-time can be more beneficial for generalization (compared to exploitation); • and finally, pre-trained models may be biased against forecasting emotional conflict. This work (and task) presents a first step towards understanding how language models can anticipate the certainty of outcomes in interactive social situations. We make our code, models, and data opensource to promote continued research.

heading: Limitations
text: While we explore a wide array of datasets and experimental setups, the generality of our conclusions are limited to what’s explored in this paper. Further study, e.g., replication study with different data, models, and settings, would provide evidence to confirm the generalization of our findings. One aspect of particular interest, is the application of these techniques to languages other than English.
Indeed, there is evidence that the uncertainty representations of language models may experience performance degradation when applied to other languages (Ahuja et al., 2022; Krause et al., 2023), especially those which are low-resource.
Additionally, while our task is motivated by a desire to probe a language model’s ability to anticipate social (un)certainties in conversation, there is no clear way for us to separate causation and correlation in this task. We cannot claim the language model actually “understands” the causes of social (un)certainties, like interlocutor mental states, since instead, it may be the case that language models capitalize on “superficial” or “spurious” statistical correlations associated with an outcome (Ho et al., 2022).
Finally, we quantify the quality of a forecast primarily through its improvement over prior knowledge. Effectively, this compares a forecaster with a performance lowerbound, demonstrating the forecaster is using information revealed through the dialogue to provide an improved prediction. In the future, a performance upperbound (such as human performance) would be useful to establish a ceiling for our goals. This is particularly important for proper scores, since it is exceedingly rare for forecasters to achieve a perfect value (e.g., 0 for the Brier score or 1 for the skill score).

heading: Ethics Statement
text: The models we use and train may exhibit, or even amplify, biases contained in the training data, such as societal biases. Robustness to adversaries and natural token perturbations is also not guaranteed. In any application, ethical and safety considerations should be made, such as bias mitigation methodologies and careful human moderation.
Moreover, even with mitigation strategies in place, using these models can introduce unexpected biases into the ecosystems where they are deployed. Users of these models (e.g., for decision making) can very well be lead astray by our models’ outputs, especially without appropriate scrutiny. The ramifications of such over-reliance can also be far reaching, impacting not only the model user, but many other downstream parties, at a scale which is amplified by the automation purposes our models serve. Deployment of these models should consider the potential impact on users, their potential over-reliance, and the far-reaching consequences un-checked use of our models can create.

heading: Acknowledgements
text: We thank Yejin Choi for her helpful feedback. Jack Hessel partially completed this work while at AI2. Anthony Sicilia completed this work while on internship at AI2.

heading: A Theory and Derivations

heading: A.1 Comparison of Implicit Forecasting Approaches in Inference and Uncertainty Tuning
text: When doing implicit forecasting with language models, we find two main approaches in the existing literature (i.e., on calibration to correctness). As in the main text, one can retain the whole token distribution during inference. Recall, this is written:
P̂IF = eZyes/τ∑ t∈T e Zt/τ . (13)
On the other hand, one can consider a normalized probability, restricted to a set of candidate answers. For example, Jiang et al. (2021) suggest this approach for smaller models, lacking appropriate instruction following capability. In our context, the approach is described:
P̂IFN = eZyes/τ
eZyes/τ + eZno/τ . (14)
Primarily, Jiang et al. (2021) argue for this strategy to cope with the spreading of probability across the many different ways to indicate “yes” or “no”, dependent on the question. In modern instruction-following models tuned with human feedback, Kadavath et al. (2022) consider both approaches. In their notation, “P(True)” takes the former approach (using the whole distribution), while “P(IK)” tunes a classification head on top of the model’s internal feature representation, making “P(IK)” equivalent to tuning in the fashion of Jiang et al. (2021). As we are aware, there has not been much exploration of tuning and inference in the fashion of Eq. (13). We provide a first, formal comparison of these approaches next.
Qualitative Pros & Cons We view the approach of Eq. (13) to be preferable for a few reasons.
1. Compatibility with Other Tasks: Most other language modeling tasks require retention of the full token distribution. While we do not experiment with this in our paper, uncertainty tuning of implicit forecasts, as in Eq. (13) and § 2.3, can theoretically be coupled with other tasks in a more broadly scoped fine-tuning pipeline. In contrast, use of normalized probabilities during uncertainty tuning, as in Eq.(14), would require distinct loss functions and inference protocols to be coupled with other tasks (due to differences in score normalization).
2. Generalized Extension: Extension beyond yes/no answers (and dialogue forecasting) is far easier when using Eq. (13). Indeed, we simply change the token in the numerator and add more data instances during training. In the alternative Eq. (14), we may require additional algorithms/compute to select candidate sets – as is done by Jiang et al. (2021). For one, these added protocols can inhibit inference due to compounding errors, and related to our first point, these added protocols also make uncertainty tuning on many different types of tasks and data prohibitive.
Theoretical and Empirical Characterization of Differences We can also consider the difference between these approaches more precisely. We begin with a theoretical result that is both conceptually informative and useful for empirical study. Primarily, we bound the absolute difference between the two types of implicit forecasts. Define X = T − {yes,no}, then:
|P̂IF − P̂IFN| = P̂IFN · |1− P̂IF/P̂IFN| = P̂IFN · ∣∣∣∣∣1−
1
P̂IFN + (1− P̂IFN) + ∑ x∈X e Zx/τ
eZyes/τ+eZno/τ
∣∣∣∣∣
= P̂IFN · ∣∣∣∣∣1− ( 1 + ∑ x∈X e Zx/τ eZyes/τ + eZno/τ )−1∣∣∣∣∣ = P̂IFN · ∣∣∣∣∣ ∑ x∈X e Zx/τ eZyes/τ + eZno/τ · ( 1 + ∑ x∈X e Zx/τ eZyes/τ + eZno/τ )−1∣∣∣∣∣ ≤ ∑ x∈X e Zx/τ eZyes/τ + eZno/τ ≤ ∑ x∈X e Zx/τ
eZno/τ = ∑
x
( eZx
eZno
)1/τ = ∑
x
ε1/τx
(15)
where we define εx = eZx/eZno as the excess score ratio for the token x. We estimate this value empirically for temperature τ = 1 across all different datasets, splits, and setups for our fine-tuned models, finding a small average of 1.1 (on a 100pt scale).
Interpretation So, for τ = 1, P̂IF and P̂IFN are practically equivalent forecasts. The main difference is the qualitative benefits of P̂IF we just discussed. Because xp decreases as a function of p > 1 (and x < 1), we also know this number will not get larger for smaller τ – so, our interpretation for τ < 1 remains the same. On the other hand, our observed difference does grow with τ , which can mar our interpretation if one uses post-hoc correction techniques like τ scaling. In our experimental study, this is largely unimportant, since our experiments on fine-tuned models actually observe the implicit signal at τ = 1 before doing our proposed (simulated) correction technique in § 2.5.
All this is to say, if one conducts correction as in § 2.5, the tuned forecasting approaches are equivalent with about 1% difference on average. Consequently, it makes sense to use Eq. (13), recouping the potential qualitative benefits discussed earlier. As for use of other correction techniques, we discuss these next.

heading: A.2 Comparison of Post-Hoc Correction Techniques
text: In this section, we discuss other choices of post-hoc correction, comparing them to our proposal in § 2.5. As we are the first to study language modeling of uncertainty in a conversational forecasting domain, we focus on studies calibrating uncertainty to model correctness. Namely, we consider approaches by
1. Kadavath et al. (2022), who infer an implicit signal and conduct temperature scaling using the language model’s entire predicted token distribution;9 2. Jiang et al. (2021), who infer implicit signals and scale temperature on only a set of candidate tokens; 3. and Tian et al. (2023), who study direct signals of model correctness and briefly suggest their own
temperature scaling approach for this their experiments. Our approach is mixed. We (a) infer implicit signals using the full token distribution like Kadavath et al. (2022), and then (b) conduct an “approximate” Platt scaling (Platt et al., 1999) on only a set of candidate tokens. Recall, the “approximation” (or simulation) in step (b) is what allows the method to be applicable to both implicit forecasts and direct forecasts.
As a standalone property, this unification is nice. It reduces the computational overhead to study many different methods, since we can use the same inference and post-processing code for all forecasts in our experiments. In addition, we point out our approach is computationally efficient. Indeed, using the approach we propose, correction is done using the probability of the yes token only, so we need only conduct one forward pass and save this single float per instance. In contrast, temperature scaling using the entire token distribution – for implicit forecasting, as done by Kadavath et al. (2022) – requires an increase in forward passes at least proportional to the number of temperatures we try, or otherwise, about 32000× more memory to avoid re-computing forward passes by remembering the token scores.
Next, we conduct some theoretical and empirical analyses comparing our correction technique to that of Jiang et al. (2021), showing it is practically equivalent. Later on, we also compare our post-processing with Kadavath et al. (2022) on implicit forecasts and Tian et al. (2023) on direct forecasts, using an empirical study. Again, we find ours to be practically equivalent (or better on average).
Comparison of Proposed Correction to that of Jiang et al. (2021) Recall our “estimated logit” post-processing technique described in Eq. (12). We have processed score:
Ẑyes = ln ( P̂ /(1− P̂ ) ) /τ − β (16)
and the post-processed forecast P̂PP = P̂new, expanded as below:
P̂PP = (1 + exp(−Ẑyes))−1 = ( 1 + eβ
(1− P̂ P̂
)1/τ )−1
= P̂ 1/τ
P̂ 1/τ + eβ(1− P̂ )1/τ
= eZyes/τ
eZyes/τ + eβ(eZno + ∑
x e Zx)1/τ
(17)
9Recall, this is their methodology for “P(True)”. Their methodology for “P(IK)” is most similar to Jiang et al. (2021).
Defining ε = ∑
x εx we have
|P̂PP − P̂IFN| = ∣∣∣ e Zyes/τ
eZyes/τ + eβ(1 + ε)1/τeZno/τ − e
Zyes/τ
eZyes/τ + eZno/τ
∣∣∣
= eZyes/τ · ∣∣∣ e
Zno/τ − eβ(1 + ε)1/τeZno/τ e2Zyes/τ + eZyes/T+Zno/τ + eβ(1 + ε)1/τeZyes/T+Zno/τ + eβ(1 + ε)1/τe2Zno/τ
∣∣∣
= ∣∣∣ 1− e β(1 + ε)1/τ
eZyes/T−Zno/τ + 1 + eβ(1 + ε)1/τ + eβ(1 + ε)1/τeZno/T−Zyes/τ
∣∣∣ ≤ |1− eβ(1 + ε)1/τ |
(18)
Since β is a free parameter, it can always be chosen to be 0 during post-processing (e.g., if we determine it is not helpful based on validation data). Thus, since any deviation due to β would be by choice to improve our forecasts, we consider estimation of this bound when β = 0. Indeed, we can easily collect statistics on (1 + ε) during the forward passes of our experiments, and do so, estimating this bound value with different selections of τ in Table 4. We find all differences to be practically negligible.
Comparison of Proposed Correction to that of Kadavath et al. (2022) and Tian et al. (2023) Lacking theoretical analysis, we compare our unified post-processing approach to the discussed techniques of Kadavath et al. (2022) and Tian et al. (2023), empirically. To keep all methods on equal footing, we use our fine-tuned Llama-2 7b model for implicit forecasts; i.e., since the method of Kadavath et al. (2022) only operates on implicit forecasts. We also limit study to one prompt setup (the data inferred prior) because the method of Kadavath et al. (2022) is more computationally expensive. To evaluate, in Table 5, we look at performance at minimizing Brier score on the validation set, reporting the percent of times we would have preferred the method of Tian et al. (2023) or Kadavath et al. (2022) due to a lower Brier score. We also report the average magnitude of preference; i.e., how much smaller the Brier score is. The logic here is to show the utility of each post-processing method in a practical setting, since we would never actually select a method that does worse on validation data in practice. Indeed, the results show that if we did use these other proposals, in conjunction with our own, and picked the best technique for each instance based on validation data, we would have still have used our own proposal most of the time.10 Moreover, the actual magnitude of preference for other methods is very small, so even when another method is preferred, we would ultimately expect similar performance when transferring to a test set.
Computational Details To select hyper-parameter τ and β, we consider two cases: β ̸= 0 and β = 0. For the first (β ̸= 0), we fit a 2 parameter logistic model of the outcome variable (i.e., this optimizes a proper scoring function – the log score). For the latter case (β = 0), we use either traditional temperature scaling (optimizing Brier score) or a newer scaling approach called Expectation Consistency, as described
10We used only our approach for experiments to avoid extra computational overhead, and because it was best overall.
by Clarté et al. (2023). To decide which of these cases (and subsequent optimization procedures) to use, we compare Brier scores, picking the method that yields the overall lowest on validation data. To re-implement the approach of Tian et al. (2023), we optimize the functional form exp(log(P )/τ), or more generally exp(log(P )/τ)/β when applying bias correction, picking parameters to minimize Brier score; this preserves the authors’ primary suggestion that scaling be done so the result is proportional to pα for some α.11 In general, we use the scipy optimization package or scikit-learn to implement the aforementioned parameter selection. When re-implementing the correction approach of Kadavath et al. (2022), it is too computationally costly to use the scipy optimization package, so we conduct a simple linear search for τ ∈ {0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2}.
11We inferred no other restrictions or implementation details from their description.

heading: A.3 An Off-Policy Policy Gradient Theorem
text: In this section, we show the claimed result from the main text:
∇θE[R] = E [ sT̃ · µT (T̃ )
µT̃ (T̃ ) · ∇θ logµT (T̃ )
]
where T ∼ LM∗θ ◦ Φ ◦D, T̃ ∼ Ref∗ ◦ Φ ◦D, and sT̃ = −s(p ◦ T̃ , O).
(19)
Let all random variables be as above and fix the mass functions µT and µT̃ . Then, we have
−∇θE[R] = ∇θE [ ∑
t∈T ∗ s(p ◦ t, O) · µT (t)
] = E [ ∑
t∈T ∗ s(p ◦ t, O) · ∇θ µT (t)
]
= E
[ ∑
t∈T ∗ s(p ◦ t, O) · µT (t) · ∇θ lnµT (t)
]
= E
[ ∑
t∈T ∗ s(p ◦ t, O) · µT (t) · µT̃ (t)µT̃ (t) · ∇θ lnµT (t)
]
= E
[ ∑
t∈T ∗ µT̃ (t)
( s(p ◦ t, O) · µT (t)µT̃ (t) · ∇θ lnµT (t)
)]
= E [ s(p ◦ T̃ , O) · µT (T̃ )
µT̃ (T̃ ) · ∇θ lnµT (T̃ )
] .
(20)
So, we have our desired result.

heading: A.4 Off-Policy Implementation Details
text: Next, we’ll discuss some choices for the reference policy Ref. The formal framework we’ve provided actually allows us to recover variants of some existing techniques for getting language models to output forecasts in token space.
The Quantizer Lin et al. (2022) fine-tune an LM to forecast the correctness of its answers in token space for a factual question-answering task. Primarily, they use the accuracy of different question types to assign confidence levels that the LM should predict for each type. We propose to extend this idea to more general settings via clustering. Instead of assuming pre-assigned partitions, we infer the partitions by clustering the data. The average outcome of a cluster is computed and assigned to each datum in the cluster, which defines a deterministic reference policy Ref∗ to be used in Eq. (10):
Ref∗C(X) = p † [ |C(X)|−1 ∑
N∈C(X) ON
] (21)
where C(X) is the neighborhood of X , ON is the outcome of neighbor N , and p† : [0, 1] → T ∗ is an inverse for p,12 mapping probabilities to tokens. In experiments, C is defined by k-means clustering over the internal feature representations of LMθ. These representations are the average (over time) of the last hidden layer of the model, ignoring masked inputs, and they are updated each epoch when clusters are re-assigned. In practice, we pick k by hyper-parameter tuning and run k-means individually for each dataset, re-aggregating the cluster assignments afterwards; our motivation for this is to prevent uninformative, imbalanced cluster assignments, which may occur if clusters correlate with dataset labels.
12p is not bijective in general, but we can consider a subset of T ∗ – e.g., strings like “72%” – for which p† does exist.
The Exploiter Given any pre-trained, fixed implicit signal forecaster, we can use it to train a direct forecaster via Eq. (10). For example, assuming we have trained a LM via supervised fine-tuning (§ 2.3) and fixed its forecasting function P̂IF, we define the deterministic reference policy:
Ref∗S(X) = p †[P̂IF ] . (22)
This provides a nice controlled view for the differences between implicit and direct forecasting, since the direct policy is actually learning from the implicit policy. Properties of the implicit policy that do not transfer to the direct policy will be of interest. As noted in the main text, this also represents a focus on exploitation, since the implicit forecasts P̂IF were designed to maximize the log-likelihood on the training data. Maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood (i.e., the log score) and it is known that the log score is a proper scoring function.
The Explorer Finally, it is interesting to consider that Eq. (10) indicates the language model LM∗θ can learn from any policy, even a bad one. To explore this, we suggest a context-less binomial reference policy which simply assigns random probability estimates to the dialogues. Presumably, by Eq. (10), LM∗θ can observe the rewards from these estimates and begin to make “sense” of them. For binomial parameters n (number of trials) and π (success ratio) we define the reference policy:
Ref∗B(X) = p †[B ] ; B ∼ Bin(n, p). (23)
In experiments, n = 20 and p is the average outcome in the training data (one for each dataset).
On Policy Alternatively, we can actually use LM∗θ as its own reference: Ref ∗ PPO = LM ∗ θ. As alluded by our notation, this makes Eq. (10) equivalent to on-policy policy gradient techniques, like Proximal Policy Optimization (Schulman et al., 2017). We leave investigation of on policy learning to future work.

heading: B Additional Experimental Details
text: Additional experimental details are provided next. In general, anything we have missed here will be available in the code, which will be made public.

heading: B.1 Additional Dataset Details
text: When available, we use default train/val/test splits from each dataset’s original proposal paper. When not available, we split each datatset according to a 70/15/15% split. All numbers reported in the main text are computed on the unseen test set for each individual dataset (i.e., even for the in domain setting).
Sampling of Training Data For every epoch of training, we sample 750 dialogues from each of the heldin training datasets of the current training split (i.e., easy, medium, or hard). We pick 750 because this ensures a balanced sample across the training data (all datasets have at least 750 training dialogues). Each dialogue is then randomly truncated to K ∼ U{2 . . . L} turns where L is the original dialogue’s turn length. As we generally train for 5 epochs, this means the model sees roughly 19-23K partial dialogues, with some dependency in examples across epochs (for the smaller datasets). We also tried using larger, imbalanced data samples for training. In this case, we sample 5K dialogues, or as many as are available. Accounting for the datasets that have less than 5K training examples, we estimate the model sees about 4× more data overall.
Sampling of Test and Validation Data For each dataset, we use the same validation and test sets across all experiments. We sample 250 dialogues from the validation split of each dataset, and randomly truncate each (in exactly the same way). We sample 550 dialogues from the test split of each dataset, again, randomly truncating each in the same way for all experiments. Some datasets have less than 550 dialogues total in their test split, in which case we use all of the available test dialogues.
Outcome Definitions Precise outcome definitions for each dataset are as follows. Most outcomes are directly annotated in their original dataset proposals. We point this out or explain how we use existing annotations to determine the outcome:
• Wikipedia editing (Attack): occurrence of personal attack is directly annotated
• Craigslist: buyer/seller goal prices are annotated. Best deal means sale price is closer to goal price
• Camp: satisfaction post-negotiation is directly annotated on a 5 point scale: 2 levels of unsatisfied, 1 level of neutral, 2 levels of satisfied. We use annotations directly, a camper is happy if they indicated either satisfied level.
• Reddit: occurrence of personal attack is directly annotated
• Charity: occurrence of donation is annotated
• Item allocation: occurrence of deal is annotated
• Wikipedia editing (Deletion): deletion of article is annotated
• Courtroom: petitioner winning is annotated

heading: B.2 Hyper-Parameters and Prompts
text: Hyper-Parameters and other Training Details Generally, we train for 5 epochs with a batch size of 12 using AdamW for optimization (Loshchilov and Hutter, 2017). We use 4bit QLoRA (Dettmers et al., 2023) with LoRA rank 32. On 4 NVIDIA RTX A6000 GPUs, single model training is an overnight process, so we only conduct full hyper-parameter selection (linear search) on the medium split using Llama-2-chat 7B to save time. We use the best hyper-parameters for Llama-2 7B on medium for all other train/test splits and models. For implicit forecast tuning, we pick the learning rate from the range {1e-4, 2e-5, 1e-5}. For direct forecast tuning, we pick the clipping constant ϵ from the range {0.2, 0.5, 0.8} and the learning rate from the smaller range {1e-4, 1e-5} to save time. Clustering for the Quantizer off-policy is also selected from the range {10, 20}. Log score on the in-domain validation data is used to pick the best parameters.13 Parameter selection is fairly consistent overall, with most tuning setups preferring the highest learning rate. ϵ was always 0.5 and the number clusters for the Quantizer off-policy was 10.
Inference Parameters As noted in our discussion, implicit forecasting uses τ = 1 in Eq. (13) conducting post-hoc correction using our “estimated logit” procedure, as in Eq. (12), after the fact. For sampling, to conduct direct forecasting, we typically use the default hyper-parameters indicated by the model parameters (e.g., in the API, Huggingface generation configuration, or Github repository). For GPT-4 this means temperature and top p are both set to 1. For Llama-2 models, this means temperature and top
13In domain generalization, its important to avoid picking parameters using the held out domains, since this can bias results (Gulrajani and Lopez-Paz, 2020).
p are set 0.6 and 0.9, respectively. The only exceptions are the pre-trained Llama-2 7b (no fine-tuning) and the Llama-2 7b with direct forecasts tuned on 4× more data, which both use greedy sampling due to clear (unsupervised) problems with the forecasts; e.g., output scores that prohibited usual sampling due to NaN values or extreme variance in token samples (200x more than greedy). For models tuned in the Zephyr style, temperature is 0.7 and top p is 1. For direct forecasting with pre-trained models, we sample up to 256 tokens to allow room for an explanation (frequently provided by some models). Tuned models generally followed the training format, so we only needed to sample up to 8 tokens in these cases. Upon sampling, our parser extracted a probability forecast by searching for percentages, ratios, and floats in [0,1] (in that order) using regex matching. Before parsing, some logic is also used to drop any non-answer explanations the model provides, i.e., this is based on keywords specified in the system prompts.
Prompts As noted in the main text, we use a standard formula for prompting all models. We provide an example of the of prompt format used for tuning Llama-2 7B below (with model tags):


 [Segment Start]
<partial dialogue inserted>
[Segment End]

heading: In the preceding conversation segment, a group of Wikipedia contributors are deciding whether an article should be deleted.
text: On average, this type of conversation ends with the article being deleted about 56% of the time. What is the percent probability that this specific conversation will end with the article being deleted? 
 

and another example:


 [Segment Start]
<partial dialogue inserted>
[Segment End]
In the preceding conversation segment, the speakers are defending their opinions on an issue. On average, this type of conversation ends with a personal attack about 50% of the time. What is the percent probability that this specific conversation will end with a personal attack? 
 
.
For pre-trained models, we also use a consistent system prompt to constrain output format and provide more task details. Below is the prompt used for direct forecasts:

heading: You are NegotiationPredictionGPT, an expert language
text: model at predicting the likelihood of outcomes in human
language negotiations. You will be given the first part of a conversation between several different speakers with potentially different goals. Use the first part of the conversation to put yourself in the mindset of the speakers and estimate the likelihood of the requested conversation outcome for these particular speakers. Use the keyword "OUTCOME" to report your predicted probability for the outcome of interest, requested by the user. Report the probability as a percentage using the format "OUTCOME = percent". For example, "OUTCOME = 70%
We focus on prompts for direct forecasts in these examples, but prompts for implicit forecasts are similar, changing only the main question asked (to evoke a yes/no response).

heading: C Examples
text: Here, we provide some examples of negotiations the model would see during training and testing. A wiki. editing example where the outcome of interest is the occurrence of a personal attack:
Speaker 3: Material moved from anon edit for discussion. I vaguely remember this or a similar incident from a TV news program. But I thought the kids were older. Notable? Verifiable?
”In 2005, approximately twenty sixth grade students at Reading Fleming Middle School (now Reading Fleming Intermediate School) in Flemington, New Jersey contracted syphilis after attendeding a "rainbow party".”
Speaker 3: Rainbow party
Speaker 2: I don’t really see much point in reporting every case of syphilis ever reported.... -
Speaker 1: Indeed. There is no source, it sounds rather urban-legendlike, and a rainbow party is a sure ingredient for those kind of tales. Sure enough, Googling for the string ”Flemington "rainbow party" syphilis” gives 0 hits.
Speaker 0: Source from Hunterdon Central Regional High School in Flemington, New Jersy http://central.hcrhs.k12.nj.us/bezsylko/discuss/msgReader$281?mode=day
I don’t think a teacher would assign that if it wasn’t true and, trust me, it is. One of my friend’s sister’s was one of the girls who contracted it. So, I’d appreciate it if you didn’t accuse it of being an "urban legend".
Speaker 1: This is ”absolutely” not a reliable source, apart from the fact that this received NO media coverage. Please stop reinserting this. When MMWR reports this, we can talk again.
An example from craigslist where the outcome of interest is whether the buyer will get the best deal:
Speaker 0: I am interested in this apartment! Can you tell me more about it?
Speaker 1: This apartment is located in San Pablo and close to everything! You will have a short commute to the office, the hottest stores, and the newest restaurants! The apartment has lots of closet space, two bedrooms, large windows that really brighten up the space, and an enclosed patio on the back.
Speaker 0: Great. I’m looking for a place in that area. Is a security deposit required?
Speaker 1: Right now we have a special . . . $99 security deposit! But you have to take advantage of the offer today!
Speaker 0: Would you be willing to go down to $800 for the first month’s rent?
Speaker 1: I am sorry, but the rent is $1725 . . . $800 is much too low.
Speaker 0: What about $1,200?
An example from the charity discussions where the outcome of interest is a occurence of a donation:
Speaker 1: Hi! Have you heard of an organization called Save the Children?
Speaker 0: I think I have once before, in a grocery store I believe
Speaker 1: Do you mind if i give you a little information about them?
Speaker 0: Sure, go ahead
Speaker 1: Just some ver basic info, Save the Children is an international non-governmental organization that promotes children’s rights, provides relief and helps support children in developing countries.
Speaker 0: Are they a non profit organization?
Speaker 1: Yes they are! They work 100% on donated funds. There is a lack of support for children in developing countries, especially in war zones. For instance, millions of Syrian children have grown up facing the daily threat of violence. In the first two months of 2018 alone, 1,000 children were reportedly killed or injured in intensifying violence. Your donation can address such problems.
Speaker 0: Oh wow, shocking news. Do you know how many children have been helped due to this organization?
Speaker 1: According t0 their yearend report they were able to reach 155 million children. Over 200k of those kids were in the US.
Speaker 0: Thats awesome! Are you apart of this organization or just support them?
Speaker 1: I am just a supporter but I would like to ask how much do you like to donate to the charity? Your donation will be directly deducted from your task payment. You can choose any amount from $0 to all your payment
An example from the courtroom discussions where the outcome of interest is whether the petitioner will have a favorable decision:
Speaker 2: I was reading, Your Honor, from the only place that I know of that the findings of fact of the district judge are reprinted. They’re in the petition – the appendix – no, Your Honor, that’s their brief. The petition for certiorari, page 45(a) –
Speaker 6: 45 (a).
Speaker 2: Which includes the district judge’s opinion and findings of fact and conclusions on this remand hearing. Thank you.
Speaker 5: Mr. Glasser.
Speaker 1: Thank you, Your Honor. The Kolod-Alderisio problem in our case would exist only in relation to the Florida bugging. We agree with the government. There’s no real issue on Florida, but there is a very severe issue, we say, in connection with an allegedly abortive additional bugging in Georgia. I haven’t spoken of that today. We’ve briefed it pretty completely, and I would ask the Court to watch for that item since there was some animation here at the end about the Kolod problem which I think it is currently before the Court. Now –
Speaker 3: Well, they didn’t get – they didn’t make any tapes at all or get any recordings, did they, in that second incident to which you refer?
Speaker 1: They – the agent who ran it said he didn’t get the tape, and I think one other agent who was in the car with him said they didn’t get any effective audible results. But, again, we had a very hard pushing hearing in which I, for one, can wait, feeling that I was entitled to make a strong appellate point against the credibility of those agents on that issue too. And, indeed, on that issue above all, they were crawling all over that part of Georgia. They were there about to score, and they were not hesitating to bug. They were bugging all over the country. We think we can’t prove that they were bugging in Europe. These fellows lived with bugs. It’s incredible to me that they didn’t have more than that one abortive car bug in Georgia. They must have bugged Desist’s room. I’m speaking of perhaps – well, all right, I’ll drop that point for now because it’s been thoroughly briefed. Our whole submission is sufficiently stated in the briefs. Now, on Fuller, again, may I say something that is – have been abrupt. We think this Court should withdraw its action in Fuller on the ground that certiorari there was improvidently granted and I’d like to say why. We’ve covered it thoroughly in our last brief. Fuller involved a telegram, we all know that, but back of that telegram was a subpoena. The police in Fuller were not defiant or willful towards existing law. The police in Fuller went to the Alaska Communications Body, whatever it’s called, got voluntary relinquishment of the telegram from that body pursuant to a federal regulation and they also got a subpoena. Now, the exact details of that whole subpoena picture, I don’t know for sure of myself because I haven’t seen the Fuller record but I’ve been guided through it in consultation in clause, consultation with one of the Fuller certiorari counsel. I have the page numbers. This is covered in our last brief. Now, if there was a subpoena in Fuller for that telegram, how can Your Honors reach the question in Fuller of a violation of 605 because the very first sentence of 605 provides for subpoenas nor, at least colorably and subject to a closer scrutiny of the record in Fuller than – which Your Honors may well wish to do because Fuller is a pretty drastic decision, and to render a drastic decision like Fuller on a record that may not stand up under scholarly criticism one of these days, I would think would be something that the Court that wish to hear about.
Speaker 7: What did the Alaska Court held in Fuller –
Speaker 1: The Alaska Court never –
Speaker 7: With respect to the 605 violation?
Speaker 1: Never touch this problem that I’m talking about now. Oh, well, they touched the 605 problem.
Speaker 7: What did it involve with respect to the 605 violation?
Speaker 1: Yes, they touched the 605, but they didn’t touch the problem of subpoena pursuant to 605.
Speaker 7: What did the Alaska Court hold with respect to the 605 violation of Fuller?
Speaker 1: They held that – let me think. Now, wait a minute.
Speaker 7: There’s a dissent, but the Court held that –
Speaker 1: They – oh, they held that 605 does not apply to states that they adopted the basic Schwartz line.
Speaker 7: Yes.

heading: Future Work
text: We leave investigation of on policy learning to future work