BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models
Authors: Yu Feng
Ben Zhou
Weidong Lin
Dan Roth
Venue: ICLR 2025 Oral (2025)
Keywords: Large language models
Reasoning
Planning
Trustworthiness
Interpretability
Probability Estimation
Bayesian Methods
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Predictive models often need to work with incomplete information in real-world tasks. Consequently, they must provide reliable probability or confidence estimation, especially in large-scale decision-making and planning tasks. Current large language models (LLMs) are insufficient for accurate estimations, but they can generate relevant factors that may affect the probabilities, produce coarse-grained probabilities when the information is more complete, and help determine which factors are relevant to specific downstream contexts. In this paper, we make use of these capabilities of LLMs to provide a significantly more accurate probabilistic estimation. We propose BIRD, a novel probabilistic inference framework that aligns a Bayesian network with LLM abductions and then estimates more accurate probabilities in a deduction step. We show BIRD provides reliable probability estimations that are 30% better than those provided directly by LLM baselines. These estimates further contribute to better and more trustworthy decision making.

Published as a conference paper at ICLR 2025
BIRD: A TRUSTWORTHY BAYESIAN INFERENCE FRAMEWORK FOR LARGE LANGUAGE MODELS
Yu Feng' Ben Zhou?
Weidong Lin' University of Pennsylvania AArizona State University
Dan Roth
ABSTRACT
Predictive models often need to work with incomplete information in real-world tasks. Consequently, they must provide reliable probability or confidence estima- tion, especially in large-scale decision-making and planning tasks. Current large language models (LLMs) are insufficient for accurate estimations, but they can generate relevant factors that may affect the probabilities, produce coarse-grained probabilities when the information is more complete, and help determine which factors relevant specific downstream contexts. In this paper, we make of these capabilities of LLMs to provide a significantly more accurate probabilistic estimation. We propose BIRD, novel probabilistic inference framework that aligns a Bayesian network with LLM abductions and then estimates more accurate probabilities in a deduction step. We show BIRD provides reliable probability estimations that are 30% better than those provided directly by LLM baselines. These estimates further contribute to better and more trustworthy decision making.
INTRODUCTION
Accurate, interpretable, and unbiased probabilistic estimation is crucial to artificial intelligence (AI) systems in mission-critical tasks that require trustworthy decision making. This is because, in many real-world applications, AI systems have to work with partial and incomplete observations that are insufficient for deductive and deterministic decisions (McCarthy & Hayes, 1981). Consider the charging station planning scenario in Fig. 1, where want to use large language models to decide where to build a certain number of charging stations over a region, subject to some constraints. Since cannot always observe all the necessary information regarding a potential location, we can only provide probability or confidence estimates on whether we should use the candidate location. If the number of planned stations is significant, we need to perform an optimization process such as ILP (Roth& & Yih, 2005) that maximizes the estimated probabilities of each individual station. A simple ranking of the candidates would not be sufficient in this scenario, as it may yield a suboptimal solution illustrated in Fig. 1. However, current LLMs are not ideal for such probability estimation tasks since 1) LLMs tend to be inaccurate and overconfident when estimating numerical confidences and probabilities (Xiong et al., 2024) and 2) they cannot provide an out-of-the-box interpretable and controllable process on how their estimates are derived (Li et al., 2024). This prevents us from using LLMs' eeciisoonmmaiing capabilities in large-scale automatic decision fecision-making tasks. Consequently, there is a need to find ways to infer better probabilities from these models.
In this work, we propose BIRD (Bayesian Inference from Abduction and Deduction). BIRD builds on three observations. The first observation is that although LLMS cannot produce convincing numerical probabilities directly, they can generate neutral and comprehensive well-world factors relevant to these probabilistic estimates. Moreover, we observe that they can make relatively correct coarse probability estimations when using these factors as contexts that provide more complete information. The last observation is that L.LMS can decide how specific contexts and conditions are related to these factors via entailment classifications. Together, these observations motivate BIRD, where we approximate a Bayesian network (Friedman et al., 1997) using LLM LLM-generated factors, optimize its internal conditional probabilities by sampling L.M coarse predictions on combinations of values of all the factors, and infer more accurate probabilities for specific downstream conditions by leveraging LLMs to map conditions to factors.
"fengyul @seas. upenn. edu. Code available https: //g thub. wwtp///////oo////////o/o/ RD..

---

Published as a conference paper at ICLR 2025
The government can build two charging stations: one in Area A (A1 or A2) and one in Area B (B1 or B2). They must choose eilner (A1 B2) (A2,81) eneuree balanced coverage and avoid overlap. Which option should they choose?
Location Characteristics
OpenAl o1 Confidence
OpenAi of Ranking
BIRD
Location A1 a susaee busy highway area with high concontration EVs cummntly lacks any existing charging stations.
90%
1
86.2 %
Location A2 on a busy highway with no existing charging stations.
90%
74.3%
support charging station.
Location B1 off main travel routes and has sufficient intrastructure
60%
3
65.5 %
Location B2 off main travel rounes and has numerous nearty amenities for rest and convenienoo noe.
60%
62.6
What's your Finel decision??
cannor dacida.
A1, B2
An example of mission-critical tasks. We first ask OpenAl ol to separately predict the probability of building a charging station each specific location. It estimates the same probabilities
Figure 1:
two different conditions twice, while BIRD (ours) successfully distinguishes the difference and can thus help the user make a more informed decision. This further demonstrates that while LLMs are capable of coarse estimations, they struggle to generate accurate probabilities. We further show through OpenAl ol ranking that although it can successfully rank all four locations based on how likely it should be used to build new charging station, it is still insufficient to complete the lask because ties, i.e., (1,4) and (2,3) are the same from ranking perspectives.
As illustrated in 2 we formulate the probability estimation problem as deciding PPO,|CC, where O, is a potential outcome (e.g., the government should/should not build a charging station here) and C' is the downstream condition or context (e.g.. the location is on a busy highway) under a Decision-making scenario S (e.g., deciding whether build charging station at the location). Given S, BIRD first generates a comprehensive set of real-worke factors and values f1, /2,..., Jn that may contribute to the probabilistic estimations. Then, approximates a Bayesian network structure using the generated factors and the outcomes {0). We propose a novel method for estimating the conditional probability table (CPT) for the outcome, i.c., IPO(,))). f F where F is the product space of value sets for all factors.' Specifically, we assume set of learnable parameters (fO,)f)) representing the conditional probability of each possible outcome given a single variable fz- We then sample f E F and acquire LLM's coarse probability estimations P FLLM (O,lf). Next, approximate the conditional probability Pestimatea(O:.1) in the CPT using the learnable parameters P(O:)J)) based on a derived approximation formula and optimize these parameters by minimizing the distributional distance between P Festimated((i)) (f.l) and the LLM's coarse predicted probability FLLM (O, f). From high level, this abduction step aligns language models' internal probability estimation with Bayesian constraints and has better error tolerance. The resulting Bayesian network generic to all possible conditions in the scenario S. In the following deduction step, we use the trained Bayesian parameters to estimate the probability of any possible conditions and observations P(O,CC). To do this, we employ an entailment process to determine P(){(C), as shown by "LLM Entailment" in Fig. 2, and the outcome we are interested in, IPO([CC = -EE P(O.I))(FICC
As we show in $4, the inferred probabilities from BIRD are more reliable. consistent with human judgments on 30% more instances than those directly predicted by LLMSs. We further show that BIRD probability predictions are good enough to be directly used for decision making (i.e., by computing an argmax on the predicted probabilities of each label for inference), outperforming standard chain-of- thought (CoT) (Wei et al., 2022) on several reasoning benchmarks. More importantly, BIRD achieves better probability estimations by optimizing an interpretable Bayesian network with natural- -lagua-- based variables. There two additional benefits of BIRD. First, we show that BIRD- BllD-aligned probabilities can serve as more accurate and fine- -ee-grained supervision signals for training smaller models. In addition, we demonstrate that BIRD'S 's factor generation process is neutral and unbiased to the specific conditions, so it can be used to generate better follow-up questions to improve decision- making confidence. Combining these findings, we show that BIRD improves trustworthiness crucial ability for mission-critical applications such as medical chatbots, where we need reliable decisions supported with enough evidence for high-confidence predictions.
IJ is the set of all possible combinations of different assignments of values to all factors (e.g., selecting value from each of the factors a/h/e/d/eff in Fig. 2). Refer $3.1.
2

---

Published as a conference paper at ICLR 2025
Scanario S: The governmnnt in plarning the locations for building cherging stations
demand and
Condition Ut:
High adlopian rale of olectric vatrclia (BV) in area
tocal
Low eiectic veliilee ownership ariong residents
high concentration on of EVs but currenty incks any evinting changing
Availabiiity info antrrsttaa
Suriicant arhauructree support chargin statin
The povertmien should
Outcomn 1:
2. Irauficant intrsmrature support changing station
Prosinity Existing Charging Stations
Dultome 1 115, U1) 86.2%
The local on buisy highway with no exlsting sharging statoone
Condition
neartry des
Dutcome 1 UZ) 74.3%
Routo to Popular Long Haull Destinadion
locol
Availability Amanitoss
Outoome 2:
The goverment shoold not charging
localion stuored buty highway within ontical habioon endangered species
Neartyy amendaas restaurants, and arsan for users widee
Luck amenddas restunnnts
Dutcome 215, U3) 95.6%
zoning regulations
Ares soredd commaniaal or retuatinal uow, allowing for chaiging staaas
Area zored for recidertie or protected use protecting a diarging andion
Extaliment: Conditon Mapping
Abtuciio Pactor Gererstion
Dayession Probebillty
Usoge of BIRD Follow-s Quostiin geneeaatoon
What feow question dhould ask to futher condim the canaiton the pention the
Is the localion zoned for oo commersai
of studkon?
Figure 2: Overview of BIRD. Given scenario 8, LLMS generate the factors F (a/b/...) with potential values (fi {a.L.a.2)) BIRD approximates Bayesian network parameterized by (G):)))), and optimizes by sampling LLM coarse predictions on Pt. [O. f E F (J is the set of all value combinations of F. e.g., / = (a.1, b.2, c. d.2, e. f.2)), and minimizing the distributional distance between an approximated P Pestimated (J) and [O. At inference time, each context C' (S+U1/S+U2/...) is mapped to some /, via entailment, and probability derived using Pestimated- BIRD can further generate follow-up questions.
2 RELATED WORK
Direct Inference in LLMs. Our paper first related to LLM direct inference methods (Wei et al., 2022; Wang et al., 2023), and decomposition-based reasoning (Wolfson et al., 2020; Tafjord et al., 2022; Zhou et al., 2022; Kassner et al., 2023). Compared to these works, BIRD considers using abduction and deduction to facilitate probabilistic induction, which aligns with the slow-thinking systems (McGlynn, 2014; Lin et al., 2023). Our work is also related to chain-of-though via latent- variable inference. Given question-answer pair (X, Y). existing methods (Hoffman al., 2023; Hu et al., 2024) aim to find latent chains of thought - token sequences Z that contribute the most the conditional likelihood. While BIRD has more structured Z and focuses more on the proper probability estimation instead of merely finding the optimal Z.
Uncertainty Estimations and Decision making under Uncertainty in LLMs. Most current works focus more sampling to access the model's uncertainty of factual knowledge (Kuhn al., 2023; Yadkori et al., 2024; Lin et al., 2024). Some works verbalize the uncertainty 2023; Xiong et al., 2024). However, Xiong et al. (2024) shows that LLMs are often overconfident when directly verbalizing their confidence. Hou et al. (2024) is most related to us, as it also incorporates the intuition of decomposition, but it focuses more addressing aleatoric uncertainty due to input ambiguity without rigorous probability calculation. This work is also related to decision making with uncertainty (Ren al., 2023; Han al., 2024; Liu et al., 2025) while all methods use LLM's direct uncertainty estimation such as token logits or verbalization. BIRD introduces external reliable Bayesian modeling to more accurately calibrate uncertainty LLMs for decision making.
LLMs (Tian et al.,
Probabilistic Inference with LMs. Chen et al. (2020) is one of the first papers to estimate probability out an NLI problem while with no explanation. Ozturkler et al. (2023) follows a similar intuition to ours, where the model first thinks by retrieving relevant associations, and then conducts probabilistic reasoning. Lew et al. (2020); Wong et al. (2023) discuss translating from natural language the probabilistic language of thought. However, none of the papers tackle the real-world complex settings under incomplete information as comprehensively as we do with the level of interpretability to users of any level.
3

---

Published as a conference paper at ICLR 2025
3 MODELING
3.1 BAYESIAN INFERENCE FROM ABDUCTION AND DEDUCTION
On a high level, given decision Y and partial information X, instead of direct inference using LLMs (induction: : X + Y). BIRD conceptualizes the general scenario behind X into intermediate factors (abduction: : X - 2), and then fits Bayesian model with Z. The model then estimates probabilities for Y (deduction X, Z + Y). Our problem setting is generic contextual inference problem of the type inspired by Feng et al. (2023). Given the context C. which includes a general scenario and corresponding downstream condition U. our task is to estimate P(O,|C',, where {Q} is denoted as O1 and O2. We decompose the overall context C' into the general scenario and downstream conditions U because want build neutral intermediate space Z based only S, so that the probability estimation is unbiased towards any specific conditions. This would mitigate models relying on incorrect signals, such as charging stations are normally built in higher-income neighhorhoods, but more generic factors like accessibility for trustworthy decision making.
1, C (S,U),
Instead of directly making prediction, we approximate controllable and interpretable inference process based on explicit Bayesian modeling (Gelman et al., 1995) containing natural-language- based variables. For scenario S, we conceptualize N factors (F; } based on S that relevant to the outcome O which can take on two possible values, O2, = 1,2, similar to the structure found in Bayesian networks (Koller & Friedman, 2009). Here the real-world factors and the outcome are equivalent to the variables in the network. Therefore, the choice between outcomes O₁ and O2 depends on the factors, with each discrete factor F, having a set of possible values as Fy. containing
all possible details of the factor required for deciding between O1 and O2. We use F F= = F3 to denote the product space of value sets for all factors. For an instance the space, each factor Fj should be attributed a specific value fj. We assume such derived factors are complete at this stage. That is, they include all the possible individual factors/angles that may affect the distribution of the outcomes. 2 As a result, JF is a complete information space for decisions. As in Fig. 2, (factor a value 1 01, factor b value 1 bj, factor с value 1 c1, factor d value 1 d1, factor e value 1 e1, factor f value I fi) is an element f' the product space F where there are 26 elements in total. We then calculate the conditional probability table (CPT) as (P(O VS E F,
f = (J1, ft....,,.)
We assume 1) {F) is complete, so O does not depend on any other intermediate factors, and 2) only one value can be assigned for each factor for given C. The predictive probability for the outcome is thus obtained by marginalizing over the product space FF for a context C (details in $A.
IPO(C))= = E(OANFF(FCC)
FEF
where i 1, (8.U), =, E F, = (f1, fN), E F,jj= F is the product space of value sets for all factors. f is a possible instance of complete information in the space.
Assumption on Conditional Independence of Variables. To provide relaxation for real-world applications, and since the factors should cover as many different aspects as possible, we assume factors are conditionally independent given the context. Therefore,
P(O EPO.D II P(SIC) FEF
In the following sections, we describe the missing implementation details of the framework: (1) the generation of the factors { ($3.2); (2) the assignment of the conditional probability table P(O..f) ($3.3), and (3) the computation of observations P(G]C)), = 1,... N ($3.4).
3.2 ABDUCTIVE FACTOR GENERATION
Large language models, although they cannot solve some complicated cases, are shown to possess parametric knowledge that enables them to solve more common cases that can be directly found
Note that our framework can later accommodate any additional factors by selectively recalculating specific parts as needed.
4

---

Published as a conference paper at ICLR 2025
in their memory (Kambhampati et al., 2024; Li et al., 2024; Zhou et al., 2024). Our method for generating the factors follows this similar intuition. For a scenario S, we first do abstractions with LLMs to derive N factors {_ The most straightforward approach involves utilizing LLMs to directly obtain a list of factors and their potential values. However, direct generation is effective but not sufficiently comprehensive, and we want to fully utilize past similar occurrences that the model memorizes in their parametric knowledge.
Therefore, encourage models to fully explore the range of alternatives, we divide the task into two stages: 1) generating sentences describing familiar situations that would increase the likelihood of an outcome, and 2) summarizing these sentences into factors with corresponding values. We show the prompts in Appendix Fig. 5 and Fig. 6. We further utilize LLMs to perform classification to assess how the value of each generated factor affect the decision, i.e., deciding the value supports which outcome, so that we can filter out factors that are unrelated to the outcomes. The prompt is in Appendix Fig. 7. In the example depicted in Fig. 2, a factor like "The color of the vehicle" will be eliminated since all potential values of this factor consistently support the same outcome, i.e., neutral.
3.3 ASSIGNMENT OF f) AS A CONSTRAINED OPTIMIZATION PROBLEM
E F in the conditional probability table of the Bayesian network. Since we do not have any observed data for approximation and LLMs alone cannot directly give an accurate assignment on IP(O, f). we adopt the formula proposed in Bordley (1982), which derived a version of the logarithmic opinion pool using axioms from the theory of additive conjoint measurement. This formula is proved to be the only continuous (over the open interval (0, 1)) group probability formula under the assumption detailed in $A.2 that the decision maker has an intuitive weak ordering on the set of factor odds ratios and a "noninteraction" property of factors. Our setting fits the assumption as the weak ordering of decisions is the most fundamental condition of consistency in decision making and we assume the conditional independence of factors. Therefore,
We now calculate each =
P(O.)f)) II
2O.) )m 1 (PO).)
PIO.IE) P(OO))) = 11,- F(O,) )us + II (- )m:
(3)
P(O,If,) T-R(O))
P(O,))
To determine the weights wy.² 3, we adopt the second case scenario mentioned in Bordley (1982), i.e. Σ,= W/ > 1. In this case, the decision maker with complete information more certain than the decision maker with only one piece of information through acquiring all different pieces of information from different factors. Since there isn't 't a priori information that suggests another choice exists, we use equal weight W/ = 1,j = I,.,, N and assume P(O,) 50%, 1,2. The final estimation formula can be rewritten as:
II P(O.f)) j=1 P(O)f) + II, (1 p(OAf)
P PPestimated/ (O.lf) = P eestmmated f2. SN) R
(4)
N
Constrained optimization for estimating P((O:))) From Eq. 4, it is not hard to see that each P(O,))), vf E can now be approximated using only P(O,))s), Vf, E F,.j 1,... N, which view learnable parameters. As discussed in $1, LLMs can perform relatively correct coarse estimation under complete information, which refers to contexts with all factors' values present (i.c. f F). Consequently, we prompt the LLMs to output verbalized probability estimations for outcome O2 given f. This approximates LLM's predictions of IP LLMS (O.lf). With Pestimated and P PLLM. we can now form constraint optimization problem that finds the values of parameters (P(O))))) that minimize the distributional distance between the Bayesian estimated probabilities P Prstimated (J) and LLM estimated probabilities (O, f). Vf E F, while conforming to the structure in Eq. and preliminary assessments about each IPOlff)) Vfy E Fj.j = 1, I,.., N provided by the LLM classification as mentioned in $3.2.
Learning algorithm for constrained optimization to estimate P(O,))) We propose gradient- descent algorithm for this optimization problem. The detailed algorithm can be found in $A.33, and
"Weights here refer to set of "hyperparameters that decides how highly the final decision regards each factor and how correlated the factors considered, not the learnable parameters in our Bayesian network.
5

---

Published as a conference paper at ICLR 2025
we give a general description here. As the parameter initialization, we utilize the LLM classification as mentioned in $3.2, where we direct the LLMs to ascertain which outcome is more likely supported by the value of each factor:
75%
f3 supports outcome i f, is neutral fj supports opposite outcome Ii
Pinit (O.15)) =
50%
(S)
25%
For the training data, given an instance of complete information f (f'. J2.... fxx. we prompt LLM to output verbalized probability score for each outcome E {very unlikely, unlikely, somewhat unlikely, neutral, somewhat likely, likely, very likely) which we use as IP "T..m((O,lJ')) The detailed prompt can be found in Appendix Fig. 10. We then assign a specific probability value for each coarse estimation, i.e., very unlikely: 0%, unlikely: 20%, somewhat unlikely: 40%, neutral: 50%, somewhat likely: 60%, likely: 80%, very likely: 100%, as supervision signals to learn the correct value of P(O,Iff), = I.... N for each fj in f using Eq.4. We randomly sample 128 instances in the space of complete information the training data for each scenario. Note that we manually assign probability values for LLM coarse decisions, but they can be changed as hyperparameters.
We adopt two loss functions: 1) we use the regular mean squared error loss for optimization and denote the loss as MSS (P (Postimated (O:]s'), (O, 2) we use the margin ranking loss to ensure the final trained individual probability preserve the original direction of its supported outcome and denote the loss as MMR
LMR = max(0,- -targer (aatargooOO:))) (IP (Ptrained trained (O:lfz) 0.5) +c)
Itarget (O:1f)) = sgn (Pint (O.1,5) 50%)
(6)
P Ptrained (O:lf)
Σ POO|f U ())(()))))))))))))))))))))))))))
Σ P(OIUU))))
card(F*)
P
E
ff, E F = I,..,, N. We apply probability marginalization to get the trained individual prob- ability and approximate it with the mean values since there are no explicit priors. We use F to denote the product space of the remaining factors except Fy, which is a subspace of J and f = (fi, Ji,., f5+1.... f E F*. Therefore, the final loss function we use for training is defined as follows:
L EASE + alMR
(7)
where a balances the two losses. We use SGD as the optimization function. As result, we can estimate any F(O:lf), E F with the learned (PO:)f)), Vf, E J, with Eq.4.
Customization to human preference. A benefit of our design is that we can explicitly ask for human preferences on how particular factor value will affect the outcome, i.e., manually setting P(O.)f)). ). Such preferences are desirable in certain applications with known constraints, and they can be easily integrated in Eq.4 by replacing P(O,lf)) with the human preferred value.
3.4 COMPUTATION OF P(JICC) WITH LLM ENTAILMENT
This section discusses how we compute P(fs(C). As noted in $3.1, C is the context containing the general scenario S, and a specific additional condition U. Given C, we employ an entailment task formulation to find which factors and their corresponding values are implied or mapped by C. Specifically, we adopt the prompt in Appendix Fig. 8 where we directly ask if the context entails value from a factor. We conduct another round of self-correction using the prompt in Appendix Fig. 9 to check if the model believes all the previously identified values are actually entailed.
We define the notion of observed factors, which refers to subset of factors among all factors that are being implied by the context. For example, as shown in Fig. 2, context S + U, implies aj, C₁ and d1, S + U2 implies C₁ and d1, while the additional condition S + U3 implies dn and fz. We assume only one value can be chosen per factor. We write the product value space of the implied factors as M F = -IIM= F,,, which a subspace of F. We define the conditional probabilities of the values of these observed factors to be 1 if the value is implied by C, a 0 if not:
P(J mm (C) = {
Jim Sim E F 8mm + Sim E FF,m
(8)

---

Published as a conference paper at ICLR 2025
Where the implied f7.,, E EF Vm = -,...,.M. M. For the unobserved factors, we have no information regarding which value to be selected. In order to be neutral and unbiased, we assume that, in an unob-
served factor, each value has an equal probability of being selected. We use Frt = -IIm M+ Fim to denote the space of unobserved factors, which is the orthogonal complement space of F in F.
1
P(f (C) card(F f3m E F.
(9)
Where Vm - M + I,.,, N. We can therefore calculate the final probability as presented in Eq utilizing the formulations provided in Eqs.2 through 9.
EXPERIMENTS
4.1 DATASETS AND SETTINGS
We convert three datasets focusing on reasoning and planning to our problem setting including O S, and U, as described in $3.1. We detail this process in $A.4. COMMON2SENSE (Singh al.. 2021) focuses on commonsense reasoning, and we gather 216 scenarios and 3822 instances, where 9 additional conditions on average support each outcome. PLASMA (Brahman et al., 2023) focuses on planning, and we gather 279 scenarios and 1395 instances, where 5 additional conditions support the less common outcome. TODAY (Feng et al., 2023) focuses on temporal reasoning, where there are 1000 instances. Among the three datasets, the most challenging task is TODAY and the easiest COMMON2SENSIA judged by baseline model performances. The complexity of the scenario and conditions is highest in TODAY. The conditions support the less common outcome in PLASMA and COMMON2SENSE presents general decision making with common sense.
We conduct experiments with Llama-2- 2-700--instrrcc and Llama-3. 1--00-Inntrrun For the factor generation task in $3.2, we generate 10 sentences for each outcome before summarization. For the LLM classification task for outtome mapping, we generate times and conduct majority vote. For the LLM entailment task in $3.4, we sample 3 times and conduct a majority vote with I round of self-reflection. We set learning rate le-02, epoch 20, batch size 4, = 0, a 10 for the learning algorithm mentioned in $3.3.
4.2 INTRINSIC EVALCATIONS OF THE BIRD FRAMEWORK
Reliability of BIRD'S Estimated Probabilities. We first propose an evaluation method that directly benchmarks the quality of probability estimations. We adopt similar setting mentioned in $3.1. Given scenario S and two potential outcomes O and O2, we find two specific additional conditions Uj and U2, both supporting O, over Oz.² S U, and S + U2 will form two contexts C₁ and Cz, respectively. The goal of this task is to evaluate if models can assign fine-grained probability estimations, that is, sufficient to distinguish which context between C1 and C2 better supports O₂ over Og or if both C1 and C2 equally support O₃ over O2, by predicting probability values of P(O,(C1) for C1 and PCO) for C2 individually." That is, assuming C1 better support On over O2, we expect P(O, > P(O,|C2) > P() > P(Oz|C1). and vice versa. If C1 and C2 equally support O₂ over O2, we should see P() = (C)) > C() = P(O)|C2). This task is significantly more difficult than providing two contexts that support different outcomes, systems will have to understand the nuances in probability estimation.
We collect human gold labels for which context better supports the gold outcome with rigorous quality control using MTurk. We explain our choice of this relative evaluation over asking human annotators to provide absolute probabilities from a psychological perspective, along with the details of human annotation in $A.5. We evaluate on 350 randomly sampled instances from COMMON2SENSE when the conditions can be mapped to at least one factor and therefore BIRD does not produce "unknown""
and https://
huggingface, co/neta- woo/metaallama//aee
Instr-cn
Herr O₂ will be the gold outcome. During evaluation, the two conditions can also support O2 over Oz. 6 For proper evaluation, systems predict prohabilities one at a time, without seeing other contexts. AAthooghh the setting may seem to favor proposed method, it is fair comparison, hecause "unknown" predictions are label-agnostic, and can use the same direct inference approach, such as CoT, for such cases.
7

---

Published as a conference paper at ICLR 2025
Model
Different,
Different
Same
Average
Random Guessing
0.333
0.333
0.333
0.333
GPT3,5 CoT
0.306
0.306
0.242
0.283
GPT4 CoT
0.312
0.357
0.216
0,289
Llama2-700 Instruct Logit
0.263
0.228
0,205
0,228
Llama2-70b Instruct Vanilla
0.375
0.333
0.243
0.311
LJama2-70h Instruct CoT
0.315
0.323
0.254
0.294
IJama3. 1-70h Instruct Logit
0.300
0.282
0.242
0.269
Llama3. 1-7b Instruct Vanilla
0.365
0.301
0,251
0.303
Llama3.1 1-70b Instruct CoT
0.373
0.351
0.187
0.309
Llama2-70b Instruct EC*
0.530
0.529
0.207
0,503
Llama3. 1-70b Instruct EC*
0.535
0.538
0,286
0,511
GPT4 EC*
0,588
0,533
0,300
0,540
Llama3, 1 BIRD (ablation w 1/2 assumption)
0.527
0,532
0.196
0,480
1 BIRD (ablation w 1/h assumption)
0,572
0,584
0.272
0,532
Llama3.1 1 BIRD (ablation w fixed initial prob)
0.614
0,597
0.337
0,568
Llama2 BIRD (ours w optimized prob)
0.614
0.624
0,450
0,592
1 1 BIRD (ours w optimized prob)
0.612
0.625
0.382
0.588
Table 1: Preference-based pairwise evaluation using Fl score for predicted probabilities across various model settings. Different, indicates the Fl score for category where C1 supports the gold outcome better than C2. and vice versa for Differentz. Same refers to the F1 score of the category where C1 and C2 equally support the outcome. Average refers to calculating Fl metries globally by counting the total true positives, false negatives, and false positives. EC* settings see both C₁ and C2 at the same time instead of predicting individual probabilities, hence not directly comparable.
We report the F1 score the evaluation metric. We further explain how the F1 score is computed in $A.5.
We compare the following methods that directly acquire probability estimations from LLMs as baselines. 1) Vanilla (Wang et al., 2023): models directly verbalize the estimated probabilities. 2) Logit: select the normalized token probability of the decision. 3) CoT (Wei et al., 2022): models output reasoming process followed by verbalized estimated probabilities. 4) EC: models select which additional condition better supports the outcome without outputting any probabilities. Note that this formulation is unfair to BIRD and other baselines that individually estimate the probabilities. To demonstrate the effectiveness of our learning algorithm in $3.3, we propose two ablation baselines that use other methods to find the conditional probabilities in the Bayesian network: 1) 1/2 and 1/N Assumptions: We adopt two neutral assumptions without LLM classifications for CPT calculation. 1/2: (P(Osl)) is 1 only if all factors' values, except for any neutral ones, support O;; if not, O. 1/N: P(O.))) is calculated as the percentage of factors whose values support outcome O,, 2) Fixed Initial Prob: We utilize the initial probability of Eq.5 to directly estimate [fO)))) with Eq.4. The details of the baselines are presented also in $A.6.
Table lists the results. Our framework exhibits superior alignment with human preference, with both models achieving an Fl of 59%, more than 30% higher than GPT-4 Such results demonstrate that BIRD produces reliable probability estimation without additional human supervision. Notably, such probability estimation is done by judging each condition separately (i.e., BIRD assigns one probability condition 1, and later another probability condition 2). We find that vanilla language models do much better when both conditions are provided at the same time (i.e.. the EC setting) where the model can perform argmax operations. However, even under this setting that unfair to our framework, BIRD still outperforms GPT-4 by 5%. All results suggest that vanilla LLMs cannot properly estimate probabilities, and BIRD effectively improves them. addition, by outperforming all the ablation settings, we demonstrate that our constrained optimization algorithm proposed in $3.3 effectively approximates better Bayesian network's conditional probabilities.
Applying BIRD'S Probabilities in Decision Making. In this section, evaluate the performance of applying the predicted probabilities from BIRD decision-making tasks. In the general decision- making setting, we similarly assume context C S + U, two possible outcomes O and Oz,
8

---

Published as a conference paper at ICLR 2025
Dataset
BIRD_1/2
B/RD_1/n
BIRD BiRD_fxxc_prroh
BIRD_optimized (ours)
CoT
Llama-2-70b-inne 2-70b-instruct
TODAY
73.7
72.8
73.4
73,9
71.5
PLASMA
72.8
72.3
72.7
74.0
76.8
COMMON2SENSE
86.9
86.8
87.5
89.0
93.8
Llama-3. 1-70b-Instruct
TODAY
65.5
68.9
65.5
74.3
72.6
PLASMA
71.3
65.7
73.0
71.5
CONMON2SENSE
78.1
85.4
86.7
92.3
90.8
Table 2:
Performance comparisons of BIRD and baselines on decision-making benchmarks. BIRD opttmizee is our final model; chain-of-thought with self-consistency (CoT) is our main base- line; others are ablation baselines.
and the task is to decide which outcome is more likely given C, similar to Fig. 2. The inference process we use for BIRD simply taking the argmax between P(O, C)) and ((O2]C). We remove all instances where they cannot be mapped to any factor values(i.e., BIRD'S decision is "unknown"). This does not undermine our experiment setting because such removal label-agnostic, and we can always fall back to other methods such as CoT when BIRD predicts "unknown". We compare our framework with thain-of-thoughl inference with self-consistency on the same instances where BIRD predictions are not "unknown". We present the prompt for CoT in Appendix Fig.14.
The results are shown in Table 2. Our first observation is that BIRD'S probability estimations are accurate enough for direct decision making. Our proposed method outperforms the chain-of-thought baseline with Llama-3.1 on all benchmarks and performs comparably using Llama-2 as the base model. This is partially because Llama-3.1 generates more diverse and accurate factors, and analyze this in $A.7. At the same time, we see that BIRD performs better on the most challenging ToDAY dataset on both models. This demonstrates that BIRD is desirable solution for more complex scenarios where CoT tends to fail more. We also observe the inferior performance of our method compared to CoT may be attributed to the fundamental differences between probability estimation and hard label classification. Table 6 in the appendix shows an example that CoT sometimes does correctly due the spurious signals and biases in the conditions, which highlights the importance of our controllable inference approach. We can conclude that BIRD outperforms direct inference methods like CoT in most cases and is least comparable in decision making while offering significantly better interpretability and controllability throughout the process.
Ablation Studies on BIRD'S Components. We conduct an ablation study regarding factor generation. proposed abductive sampling method
We compare the final decision-male making accuracy based and the direct factor generation method mentioned $3.2. Appendix Table 7 shows the results: we that our proposed method, on average, surpasses the direct method by 4.4% in accuracy for Llama-2/ Llama-3.1. I, indicating the effectiveness of our proposed factor generation that collects more comprehensive signals from models' parametric knowledge and memorization. We also conduct ablation studies on different ways of estimating the conditional probabilities of our Bayesian network (i.e., 1/2, 1/n, and fixed initial prob). Results Tables and show that our proposed constrained optimization learning algorithm effectively finds more accurate estimations by outperforming all ablation baselines. We demonstrate that both losses Eq.7 are crucial in $A.9.
4.3 EXTRINSIC APPLICATIONS OF THE BIRD FRAMEWORK
The Usage of the Reliably Estimated Probability. As shown in $4.2, BIRD can produce reliable probabilities that contribute to better decision making. Such findings motivate us to investigate whether these probabilities can be used as supervision signals, as models may benefit from seeing more than the discrete classification labels (Zhou et al., 2020). Specifically, we use BIRD to estimate probabilities of some eerision-making training instances and finetune a T5-large model using these generated probabilities. We then evaluate its performance the experimental settings described in Feng et al. (2023) and the details can be referred to in $A.10 10. We instances from COMMON2SENSE, where the argmax of the estimated probabilities from
cross-domain datasets. We fully replicate

---

Published as a conference paper at ICLR 2025
framework using Llama-2 supports the correct outcome, additional training data, besides the originally proposed training data in that paper. We test on two general temporal reasoning datasets, MATRES (Ning et al., 2018) and TRACIE (Zhou al., 2021), as well as two commonsense reasoning and planning datasets, TODAY, and PLASMM. The results are shown in Table 3. We see that models achieve an average performance increase of 1.3% compared with BiRD-generated probabilities as additional signals. We use cross-entropy loss with class probabilities as the target for our soft probability setting and cross-entropy loss with class labels as the target for the hard-label setting. Such results suggest another effective application of the BIRD framework, where smaller models can benefit from soft probability alignment and make better overall predictions.
Model (Train Data)
TODAY (exp)
TODAY
TRACIE
MATRES
PLASMA
Average
T5 large
77.8
57.9
73.0
83.5
48.1
65.6
hard label
83.6
62.9
68.2
75.2
65.2
67.9
+ BIRD prob (ours)
84.3
63.4
71.4
77.6
63.3
68.9
PatternTime
82.2
60.9
79.8
85.8
50.5
69.3
+ hard label
83,9
61.8
76.2
83.7
56.9
69.7
+ BIRD prob (ours)
85.1
62.6
75.7
85.1
61.5
71.2
Table 3: System performances under different supervision data across three binary temporal bench- marks and one binary planning benchmark. For simplicity, we use "hard label" representing that we use COMMON2SENSE supervision data with explicit binary labels, and "BIRD prob" representing that we use COMMON2SENSE supervision data with estimated probabilities. TODAY (exp) uses gold explanations during evaluation.
Using BIRD Generate Better Follow-up Questions. Generating accurate and informative follow-up questions is an important goal for LLMs to be applied in mission-critiert tasks that require unambiguous decision making. A natural benefit of our BIRD framework is that it analyzes what needs to be known through the comprehensive and unbiased abduction step (i.e., factor generation) and interprets how new information will change the probability or confidence of a desired decision during the deduction step (i.e., probability estimation). We conduct follow-up question generation experiment to demonstrate this benefit. For a given decision- decision-making problem, we ask both BIRD and LLMs to generate binary (yes/no) follow-up questions. For BIRD, identify implied factors, choose the most important unmentioned one, adopt the most important value aligning with the current outcome, and rephrase it question. We directly ask LLM generate questions for better decision making. The details can be referred to in $A.1 We then recruit crowd-workers on MTTurk to provide pairwise preference between the BIRD generated question and the question generated directly by the LLM. We do this on 250 examples from COMMON2SENSE, An example can be seen in Appendix Fig. 4 and the details can also be seen in $A.11. Appendix Table 9 shows that the follow-up questions generated by BIRD are more often preferred: questions generated by BIRD are preferred 52.8% of the time, while those directly generated by LLM are preferred 32.8 8% of the time; in the remaining 14.4%, both are considered equally useful. This result motivates future work on applying BIRD in an interactive agent system for more efficient and controlled decision making.
5 CONCLUSION
In this paper, we propose BIRD, Bayesian inference framework for large language models. BIRD utilizes LLMs' existing reliable capabilities such as abductive thinking and coarse-grained classifica- tion to optimize an explicit Bayesian network for more reliable probability estimation. We show that BIRD can output accurate probabilities, outperforming LLM vanilla probabilties by 30% on direct es- timation tasks, and outperform chain-of-thouge outputs are explainable, because its natural- Langull---gggee-aaa based variables in the ially-bnterrretable Bayesian network. We also show that BIRD be used in several other settings, such generating fine-grained supervision signals and asking informative follow-up questions, to further improve the trustworthiness and controllability of L.LMs when applied to real-world applications.
decision-making tasks. At the same time, BIRD'S
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTS
This work was sponsored by the Army Research Office and was accomplished under Grant Number W91 INF-20-1 1-00000 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was also supported by the Office of Naval Research [ONR 000014-23-1-23644 We thank Kyurac Kim, Ollie Liu, Vivek Gupta and colleagues at CogComp UPenn for valuable discussions.
REFERENCES
Robert F Bordley. A multiplicative formula for aggregating probability assessments. Management science, 28( 28(10):133--144) 1982.
Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona J. Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. Plasma: Making small language models better procedural knowledge models for (counterfactual) planning, 2023.
Tongfei Chen, Zhengping Jiang. Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. Uncertain natural language inference. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8772-8779, Online, July 2020. Association for Computational Linguisties. doi: N0.18653/11/2220..u main. 774. URL https: / aclaatooootoooyoyoocoooo org/2020 acl-main. 774.
Yu Feng, Ben Zhou, Haoyu Wang, Helen Jin, and Dan Roth. Generic temporal reasoning with differential analysis and explanation. Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 12013 Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: //aclanthology org/2023 acl acl-long. 671.
acl-long.671. URL https:
Nir Friedman, Dan Geiger, and Moises Goldszmidt. Bayesian network classifiers. Machine learning, 29:131-163, 1997.
Andrew B. Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. Chapman and Hal//CRC, Boca Ratan, Florida, 1995.
Jiuzhou Han, Wray Buntine, and Ehsan Shareghi. Towards uncertainty- awaae language agent, 2024.
Matthew Douglas Hoffman, Du Phan, david dohan, Sholto Douglas, Tuan Anh Le, Aaron T Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A. Saurous. Training chain-of-thought latent-variable inference. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023. URL httpst::/// http:///ppneeeeeeeeeeeee net werrrrmmiidaal. 41ppaa1........
Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. Decompos- ing uncertainty for large language models through input clarification ensembling. In Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 19023-19042. PMLR, 21-27 Jul 2024. URL https: pttpp/ppoceeng...c/ee///po mlr press 2t///22355///// howu 2houu44 4b. html.
Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ /ttp////ppeeee.ee/////e/// /
net / Eorum? fi d 1ul
Daniel Kahneman. A perspective on judgment and choice: mapping bounded rationality. In The American psychologist vol. 58,9 (2003): 697-720, 2003. doi: 10. 00.103370003-0666.5888

---

Published as a conference paper at ICLR 2025
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. Position: LLMs can't t plan, but can help planning in LLM-modulo frameworks. In Forty-first International Conference on Machine Learning, 2024.
URL httptttt///ppeeeee/////// ew. net / forum? i d=ThhdJPmmE
Nora Kassner, Oyvind Tafjord, Ashish Sabharwal, Kyle Richardson, Hinrich Schuetze, and Pe- ter Clark. Language models with rationality. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 14190-14200 Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.100....011 101118861122233 /ttf://///2223...mmmmmaa.a....m 877. http::/ /gtp:////l./.aattooooooooooooo ocl.eemtooooogg.oo/222223333 org/20233. emn.p-mainne 877.
Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques Adaptive Computation and Machine Learning. The MIT Press, 2009. ISBN 0262013193.
Lorenz. Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Confer- ence on Learning Representations, 2023. URL http::: http:///ppeeeiiiiee net/ forum?in ?id VD-AYtPOdve.
Alexander K Lew, Michael Henry Tessler, Vikash K Mansinghka, and Joshua B Tenenbaum. Lever- aging unstructured statistical knowledge in a probabilistic language of thought. In Proceedings of the annual conference of the cognitive science society, 2020.
Bangzheng Li, Ben Zhou, Fei Wang, Xingyu Fu, Dan Roth, and Muhao Chen. Deceptive semantic shortcuts on reasoning chains: How far can models go without hallucination? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7675-7688, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 0018186/1/2000 http:/222...aanoonnnaaa naacl-long 424. URL https: //aps//aaaaaaaaaa http://aaaaanttooooo.oeoe. org/ 2024. www.naannnnaa... llngg 424.
Bill Yuchen Lin, Yicheng Fu, Karina Yang. Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. ArXiv, abs/2305.17390 2023.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-box large language models, 2024. URL hotphtsts///rxii..rrgg /2305.
19187.
Ollie Liu, Deqing Fu, Dani Yogatama, and Willie Neiswanger. DeLLMa: Decision making under uncertainty with large language models. The Thirteenth International Conference on Learning Representations, 2025. URL https: openreview htteeeewwwweeeeeeeeeeeeeeeeeeeee m 1d=Acvo2RGSCy.
John McCarthy and Patrick J Hayes. Some philosophical problems from the standpoint of artificial intelligence. In Readings in artificial intelligence, pp. 431 431-450. Elsevier, 1981.
N. F. McGlynn. Thinking fast and slow. Australian veterinary journal, 92 12:N21, 2014.
Qiang Ning, Hao Wu, and Dan Roth. A multi-axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1328- Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/118-. 1122 URL https::p:.... http://aaaaanntoo...oo/.. org/ wwtp//1-1/1/111///11111 18-1122.
Batu Ozturkler, Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. ThinkSum: Probabilistic reason- ing over sets using large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto. Canada, July 2023. Association for Computa- tional Linguistics. doi: 1/28633/1/22233.. 2023. acl 1ll-loogg long. 68.
l/n6533w///233alllooooooooooooo acl-long.68. URL https: / aalantholooge org/
12

---

Published as a conference paper at ICLR 2025
Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar. Robots that ask for help: Uncertainty aligmment for large language model planners. In 7th Annual Conference on Robot Learning, 2023. URL https: / /oppneeeeevvece ew net/ forum?id= 10=22000D2222
Dan Roth and Wen-tau Yih. Integer linear programming inference for conditional random fields. In Proceedingss of the 22nd international conference Machine learning, pp. 736-743, 2005.
Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormolabashi. Te-lin Wu, Xuezhe Ma, and Nanyun Peng. COM2SENSE: A commonsense reasoning benchmark with complementary sentences. In Findings of the Association for Computational Linguisties: ACL-IJCNIP 2021, pp. 883- 898, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021. indings-acl.77. URL https: httpa//ooaaatoooo...ooooooooooo. org/2021 findings- acl. 78.
Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Entailer: Answering questions with faithful and truthful chains of reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 2078-2093 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 11/8886331/2222..2/ 1/20222.mmllpmain 134. URL https: / / anthologyy org/ 2022. emw.p-manppmm.memm 14..
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 5433-5442, Singapore, December 2023, Association for Computational Linguistics. doi: 10. 18653/66//v11 00/553/11120222 2023. . ewnlp-main.nn..... 330.
URL https: h/tla///hollaaaoaooogooooooo org/
AMOS TVERSKY and DANIEL KAHNEMAN. Judgment under uncertainty: Heuristics and biases. In Science, 185(4)57), 1124-1131, 1974. URL https: /toi. wttp.//...../..iii.roooeooo 185. 1...415...22 1124.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https: http:////ppeeeeee.neemee inppenrrrreee..nef/ooomiiiiiid
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi. Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: oppnreveeeeew net/ forum? id= <d=_VioMMeBB_J.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguisties. 8:183-198, 2020. doi: a..106. URL https: // //aclantholooyy....oooo org/202000000 tacl-aalll. 11.13.
Lionel Wong, Gabriel Grand, Alexander K. Lew, Noah D. Goodman, Vikash K. Mansinghka, Jacob Andreas, and Joshua B. Tenenbaum. From word models to world models: Translating from natural language to the probabilistic language of thought, 2023. URL https: //aa://///aii//////a//aa/aaamamr haa/aaxivvvoo///./////aa org/ abs 2306 12672.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https: / //oopreeview... net/ forum? i d=gjeekFxFPP.
Yasin Abbasi Yadkori, Ilja Kuzborskij, András György, and Csaba Szepesvári. To believe or not to believe your 1mm, 2024. URL httpt:tpp//// /ttp///xvvvv/v.////v////////.aal wwwpii..wrg/b///oo 24466 02543.
13

---

Published as a conference paper at ICLR 2025
Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. Temporal common sense acquisition with minimal supervision. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7579-7589, 2020.
Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. Temporal reasoning on implicit events from distant supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, pp. 1361-1371, Online, June 2021. Association for Computational Linguis- tics. doi: 10. 18653/v1/20222 .naacl-main. 107. URL /tattps:/aacaaathoooooooooo ogy o org/2021. neacl-ma In. 107.
Ben Zhou, Kyle Richardson, Xiaodong Yu, and Dan Roth. Learning to decompose: Hypotheti- cal question decomposition based on comparable texts. In Yoav Goldberg. Zoriissa Kozareva, Empirical Methods in Natu-
and Yue Zhang (eds.), Proceedings of the 2022 Conference ral Language Processing, pp. 2223-2235, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 1tt33//112222e/r// www.:022.emlppmaii.....m 142. URL
https://ae hatp://///claahoooogyyy
http:/2222.mmm.... www.p-mainnnnnnnneeeeeemmm main. 142.
Ben Zhou, Hongming Zhang, Sihao Chen, Dian Yu, Hongwei Wang, Baolin Peng. Dan Roth, and Dong Yu. Conceptual and unbiased reasoning in language models. arXiv preprint arXiv:2404.00205, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DERIVATION OF THE EQUATION FOR UNCHRTAINTY ESTIMATION
We derive the outcome probability for the given context C of partial information, i.e., any additional condition U under the scenario S, based on the complete information space F of scenario S:
P(O,IC) = Σ
P(OUI,C)F((CC) JEF
Σ
JEF
E
P(O.IF.CBF(IIC)) +
E
P(OLIIC)P((ICC)
|EFN[/CC=/}
TEFN(mC**)
(10)
E P(O)))((IC)))
E P(O(IICC)P(FICC)
rEFn(JnC=
(EFOU
E
P(ONUPU()))
E
P(ONT))(UCC))
rEFNUnC-=1)
(EFn(nOC*1)
Σ P(ONS)T(C)) JEF
Where the fourth line follows from that P(O,),, PPO..f) since fnc== f. And the second last line is derived from that (P([IC) = 0f nc # as for a given C, only one value can be chosen for each factor in Fig. 2.
A.2 ASSUMPTIONS FOR ESTIMATING P(O,If)
Weak ordering of the set of factor odds ratios. Suppose given factors f2. SNN we assess odds P(O.If))
of 01, 02, ... ON, respectively in favor of the outcome O,, i 1,2 occurring. Oj I-F(O,1,)) 1. I..., N, the intuitive ordering 2 g is defined as:
(01,02...,,N)) ZE (0i, o5...,
(11)
if and only if the decision maker feels least sure about the outcome O,, occurring when given factors fi, f2,, fN we assess odds 01, 02 .., ON -as when given factors fi, f2..,.. fN assess odds 07, 02..... O'.- The ordering ZE is Weak Ordering if and only if for any
(01, 02. ..., oN ).(( og.... )o oz ...., oN we have:
(a) either (01, 02 ON) ZE (of og.....oh: or (oi oz.... 0yy) Zg (01. 02,..., ON);
(b) (ot, 02, ..., ON) ZE (0f 0z ...,..,, owy) and if (oj, of ..., 0w) 2 (ot oz ..... 0 then
(0), ...... ON) ZE (of 02 ..... )
"Noninteraction" property of factors. The non interaction condition says that the decision maker's interpretation of how one factor affects the outcome does not depend on how other factors affect the outcome.
A.3 LEARNING ALGORITHM FOR ESTIMATING TP((.).)
We present the complete algorithm in Appendix Algorithm Since PP(O)) and TP(O2) are com- plementary, we only need learn (f))))))) fja Fj.j I...,,N: a 1,.., card(FF),, and ((O2lffa)) = - (o()))fj.) As a result, for given scenario S, the number of learnable parameters the total number of values for all the generated factors, i.e., Σ cFrd(FF)-
15

---

Published as a conference paper at ICLR 2025
All training instances for a scenario are drawn from the complete information space for the same scenario and possible outcomes {O} and will share the same factors {F) We randomly sample 128 instances in the space of complete information as the training data for each scenario. This represents the minimum number of instances required for effective training. Ideally, increasing the number of sampled instances would improve the model's alignment with the underlying L.M.M For scenario with fewer than 128 instances in the space of complete information, we employ a resampling technique, selecting instances repeatedly until we reach total of 128 samples.
The value of Pimit (O) in Eq.5 is assigned as it represents random initialization. To determine the mappings of rankings to probabilities in the algorithm, we consulted two psychology experts and adopted the Likert scale theory. The finalized mapping between verbalized probabilities and numerical probabilities reflects typical human behavior and effectively distinguishes between verbal- ized probabilities in an unbiased manner. Additionally, slight adjustments to the mappings do not noticeably affect overall performance. All training hyperparameters for the algorithm are optimized using a grid search with hold-out validation set from PLASMA.
A.4 EXPERIMENT DATASET DESCRIPTION
We convert three datasets focusing on reasoning and planning to our problem setting.
COMMON2SENSE (Singh et al., 2021): focuses on commonsense reasoning and plan- ning. The original task constitutes a multi-domain complementary commonsense reasoning benchmark and comprises natural language truc/false statements. We exclusively focus instances of comparative reasoning where pre-trained smaller model exhibits low confidence in its prediction logits. Using GPT-4 we rewrite each instance statement into its opposite, creating distinct outcomes. We then outline a general scenario based the two statements and generate 10 additional sentences for each statement, with detailed condi- tions that would support the statement. Additionally, we implement reverse verification step to eliminate any incorrect additional sentences to ensure dataset quality. We gather 216 scenarios and 3822 instances. 9 additional conditions on average support each outcome.
TODAY (Feng et al., 2023): focuses on temporal reasoning. The dataset is designed to study how introducing an extra sentence specific scenario affects the temporal relation, aligning well with the context of our task. There are 1000 instances.
PLASMA (Brahman al., 2023): focuses on planning. The original task is to generate or revise a plan based on a given goal while adhering to an additional condition. We've ve observed that the key variations between the initial and the revised plans given the additional condition typically focus on a specific step within the plan. We then utilize GPT-4 to identify the step and adapt the description of the step in the original and revised plans into two potential outcomes. We manually identify the less common outcome in the 279 scenarios and use GPT-4 to generate additional sentences for the outcome, with detailed conditions that would support this outcome instead of the other. We gather 279 scenarios and 1395 instances with 5 additional conditions for each scenario.
For the two GPT-4 generated datasets, we recruited the authors and their colleagues conduct a human quality check on 100 randomly selected instances from each dataset, with three people reviewing each instance. All three reviewers agreed on the label for 91% of the generated data, For 94% of the data, at least two reviewers agreed on the label.
A.5 HUMAN EVALUATION ON BIRD ESTIMATED PROBABILITIES
We first explain our choice of relative evaluation over asking human annotators to provide absolute probabilities from a psychological perspective. TVERSKY & KAHNEMAN (1974) shows that humans use heuristic principles to make probability judgments, which often suffer from systematic and predictable errors due to cognitive biases. Kahneman (2003) further discusses how human judgment in probabilistic situations is generally better in relative than absolute terms due to cognitive biases and limitations in processing complex information. We conducted pilot study with human
We use GPT-4-0613 when utilizing GPT-4 for annotations.
16

---

Published as a conference paper at ICLR 2025
Algorithm 1:
Training single probabilities to estimate P(O.
input :Training set of complete information {f). E F, outcome O,, 1,2, LLM M, learning rate 7), number of epochs E, batch size B.
output: :Trained probabilities P() E Fj.j 1 ,,,,,,,,,
N,a ala=1,..... 1,.., card (Fz), 1,2.
Get verbalized probability scores from the LLM.
for f' E (ʃ) do FLLM(Oiff) - end
Initialize probabilities and calculate target for the margin ranking loss.
for j to do for = 1 to card(FF) do
75%
fja supports outcome 1
P Pinit (O2lffa)
50%
fja is neutral
25%
fja supports opposite outcome 2
Pinit (init(O1 (0]] > 50%
Vtarget (Oilffa)
0
Pinit(O [j]] 50%
Pmii (nie(fiffja < 50%
end end
Training. epoch = to
10
for each batch of B samples from the training set do for b 1.... B(in parallel) do
12
T P(OL)5))
13
PP Pestimated (O,15)) = PO. HIO
# Forward
propagation.
c MSE (PLLM(O))f**) - (estimated(O1If Compute MSE loss for each sample, for Laa E Fj. f3, aF J,0) 0, =,,.,, N.
15
end
MSE 1
B Σt Øb MSE # Compute MSE loss.
16
17
for j 1 to NN do
18
for a = 1 to card (F)) do Pirained (0 (u) Exerr- Pestimated (Oilf" U (ff
card(FF) Epeer. Pe Pestimmtted ( U fza) gi MR = max(O, Utarget (Persiieee (Oilfsa) - 0.5) +c)
20
21
end end
22
(Ty) pja MR
LMR N E card
2==1
# Compute MR loss.
L MMSE
Em Compute final loss.
for j to N do for a = 1 to card(F,) do ((O = P(Or)fça) - ( Update parameters.
26
oc
27
((O2)faa) = 11- 1-P(Oi end end
30
31
end
12 end
17

---

Published as a conference paper at ICLR 2025
Requaseer
Rewand: 50
Trkks aualattee
Dacatin:: 3 Hoow
Gualiications Required: HIT Approval Rate 7N) for Raqumbeer Iifh goater dan Lecaton io
Welcome! given additional conditions based scenario outcome.
Please choose which condition better supports the
ALL QUESTIONS must be anewered carefully, otherwise you are not paid. This only a pilot teet, so we kindly ask each annotator to take assignments at most. The 6th assignment you answered will not be paid.
Scenario: ((scenario)
((outcome)
Condition S(additional conpriiionnontioo__111
Condition 2: sjaddtional.condiee conditionnndiiii
Evalutte 1. Consider the key elements of the scenario. Evaluste the two given conditions and select the one that better supports the outcome. Condition better supports outcome than Condition Condition better supports the outcome than Condition After careful consideration, cannot decide and think both conditions equally support the outcome.
Figure 3: The interface for human evaluation on preference- based pairwise evaluation of the estimated probabilities.
annotators MTurk, asking them to provide absolute probabilities for each additional condition separately. However, we were unable to reach a consensus among the annotators.
We then introduce the human annotation process for pairwise comparison. We collect human gold labels with rigorous quality control for the preference-basee pairwise evaluation of the estimated probabilities. We restrict each annotator take 5 samples most and there are 386 distinct annotators. We only allow participants who have 98% or more HITs acceptance rate and are located in the US. We also require annotators to spend least minute for each instance (the hourly salary is $15). We show the experiment interfaces in Appendix Fig. 3. We first carry out the human annotation process five times on 450 randomly sampled instances from COMMON2SENSE when the condition can be mapped to at least one factor and therefore BIRD does not produce "unknown" during our experiments, and both conditions support one target outcome. If the annotations have less than 80% agreement, i.c., less than 4 out of the 5 annotators agrec the same condition label, invite additional annotators to reassess this instance. the agreement of the 7 annotators is more than 70%, i.e., least 5 out of the 7 annotators agree the same condition label, accept the instance. Other than the previously accepted instances, we also accept an instance as a neutral label if all three labels have been chosen during the annotations and the agreement for the neutral label more than or equal 40% and the agreement for any of the remaining two labels is less than or equal 40%. We get 350 instances in total, where 12.3% of the instances have a neutral label where condition I and condition 2 equally support the outcome, 44.0% of the instances have label towards condition where condition 1 supports the corresponding outcome better than condition 2, 43.7% of the instances have a label towards condition 2 where condition 2 supports the corresponding outcome better than condition
We now introduce how precision and recall are calculated. Given testing instance, assuming the gold outcome is O* and the opposite outcome is O there will be 3 model output labels: 1) P(O'(CI) > P(O*IC2), i.e., the model thinks CI better support the gold outcome; 2) P(O > P(O))CI). ie., the model thinks C2 better support the gold outcome; 3) P(O"(C2) = C(O ie., the model thinks both C1 and C2 equally support the gold outcome. We want to check if the model's label is the same as the human annotator's ' label. Therefore, this can further be seen as a 3-label classification problem.
18

---

Published as a conference paper at ICLR 2025
A.6 BASELINE DESCRIPTION
We adopt the following baselines to evaluate our BIRD framework.
Vanilla Verbalization with Self-Consistency (Vanilla), (Wang et al., 2023) Given the scenario, additional condition, and two corresponding outcomes, the model is asked to directly verbalize an estimated probability for each outcome. We show the Prompt in Appendix Fig. 11. The above process is carried out independently for each of the two additional conditions. The additional conditions with higher output probabilities for the gold outcome are the predicted answers. We use temperature 0.7 to generate 3 responses and take the majority vote answer as the final answer.
Logit- Probability. We directly use the output token probability for the decision. We use greedy decoding here to set the temperature 0.
Chain-of- Thought Verbalization with Self-consistency (CoT)), (Wei et al., 2022) We ask for reasoning process to think step by step before verbalizing an estimated probability. The remaining process is the same as vanilla verbalization with SC. We show the Prompt in Appendix Fig. 12.
Explicit Comparison (EC). Instead of evaluating each additional condition separately, both conditions are presented simultaneously to the model. Given the scenario and the gold outcome, the model asked to select which additional condition better supports the outcome without outputting any probabilities. Note that the task is less challenging for this particular baseline than the others due to the explicit formatting of inference this baseline. As a result, performances under the EC setting not directly comparable with model and other baselines' performances since they only see one condition at a time and cannot perform argmax operations. We show the Prompt in Appendix Fig. 13.
1/2 Assumption. Instead of utilizing the learning algorithm mentioned in $3.3 to estimate PPO,)) f). We adopt a neutral assumption. An outcome is determined based on the factors only if, except the values of the factors that are neutral, the values of all remaining factors uniformly support that particular outcome. Conversely, if the values of the factors indicate different outcomes and there is no prior information, we cannot conclusively determine the outcome.
FOID = P(OI)) 2 }. Vj except P(O))))))))))))>))))))))))))))() ビ.j I,..,,N otherwise
(12)
The intuition behind this assumption is that we focus solely on the present context, if the current information supports both outcomes and no additional information is available finalize the decision, it becomes impossible predict which outcome is more likely to occur.
1/N Assumption. Instead of utilizing the learning algorithm mentioned $3.3 to esti- mate ff),,,,, we adopt another commonly used neutral assumption. The outcome O,'s probability is determined by the aggregate count of factors whose values support outcome O,
P(O.))) EM M(P(O.1)) J
(13)
N
Fixed Initial Prob. Instead of training with the algorithm to get an estimated prob for each P(O()))) ff) in $3.3, we utilize the initial probability of Eq.5 to directly estimate P(O,,.f) with Eq.4.
A.7 ANALYSIS OF DIRECT DECISION MAKING
We first conduct error analysis for Llama- 1lama-2-70b-aa 2-70b-instruct on direct decision making where BIRD gets the wrong outcomes while CoT makes the correct decisions. We detail the type of errors with its frequency percentage Appendix Table We observe that our proposed method outperforms the chain-of-thouuhh baseline with Llama-3. 3. rather than using Llama-2. This is partially because Llama-3.1 generates more diverse and accurate factors, and we present an example in Appendix Table 5.
19

---

Published as a conference paper at ICLR 2025
Error Type
Frequency Percentage
The generated factors and their values are not comprehensive enough to cover all elements of the conditions. The selected values only imply partial information in the additional condition.
56.7%
The additional condition is mapped to incorrect values of the factors that it does not necessarily imply.
30%
The calculation of the final probability is inaccurate.
13.3%
Table Error analysis for Llama-2-70b-instee for direct decision making.
the ease of carrying cup is being compared between one person six people. Outcome would be easier one person to carry cup. would six people carry cup. Outcome 2: would be casier for six people to carry the cup, than it would be one person carry the cup.
Scenario:
Llama-2-70h Instruct Generated Factors
Weight and Size of the Cup:
The cup is light and small
The cup is heavy and large
The cup is of medium weight and size
Material of the Cup:
The is fragile
The is sturdy
The cup is made of moderately durable material
Strength of the People:
The person able-bodied and strong enough carry the cup The combined strength of six people greater than that of person Experience the Person People Involved:
The person more experience carrying objects
six people have more experience in lifting objects
Formation the People:
The six people carry the cup a specific formation designed to distribute the weight more evenly The six people do not carry the cup in specific formation
Llama-3, 3.1- Instruct Generated Factors
Cup Size and Weight: The is small and light The cup is large and heavy
The is small and heavy
The cup is large and light
Cup Material:
The cup is fragile
The cup is sturdy
The cup is constructed from material of moderate durahility
Cup Design and Features:
The cup has comfortable grip and balanced ergonomics
cup has uncomfortable grip unbalanced ergonomics cup has a neutral grip and average ergonomics
Carrier's Physical Abilities:
The one carrier has more strength and endurance than the six carries The combined strength of the six carries is stronger than the one carrier Group Dynamics and Coordination:
The group is well-coordinated and works together seamlessly
The group is disorganized and has difficulty working together
The group has average coordination and works together moderately well Physical Demands of the Task:
The task requires exceptional strength and endurance The task requires minimal strength and endurance The task requires moderate strength and endurance
The Size and Shape of the Carrying Area: narrow and confined The carrying wide and open
The carrying
Table An example where Llama-3.1 generates better factors than Llama-2.
20

---

Published as a conference paper at ICLR 2025
We then give insights into LLM direct performance. We present example from the COMMON2SENSE dataset in Appendix Table where the LLM accurately infers the correct outcome based predominantly inductive biases, ignoring the other primary information provided in the condition, i.e., the peppers are used in large quantities, which will be captured in our framework.
Example
Scenario: The spiciness of hot being discussed relation the choice hetween using habanero
peppers or jalapeno peppers.
Condition:
The sauce recipe uses a significunt amount habanero peppers relative the quantity
other ingredients.
Outcome I: A hot sauce going to be hotter if uses habanero peppers rather than jalapeno. Outcome 2: A hot sauce is going to be hotter if uses jalapeno peppers rather than habanero.
Llama-2 Generated Deicision Making Process
Habanero peppers are generally much hotter than jalapeno peppers. This is measured on the Scovillé scale, which is measure of the pungency (spiciness or heat) of chili peppers and other spicy foods. So the condition better supports Outcome I: A hot sauce is going to be hotter if it uses habanero peppers rather than jalapeno.
Table example where the LLM correctly makes the decision but with inadequate emphasis on the context.
A.8 ABLATION STUDY TOWARDS ABDUCTIVE FACTOR GENERATION
We compare the final decision-making accuracy based on two different factor generation methods and present the results in Appendix Table We see that our proposed method performs much better than the direct generation method, indicating the effectiveness of utilizing the model's inductive bias inherited from training data.
Dataset
Method
BIRDuame2
%Unknown
BIRDiiamas.I
%Unknown
TODAY
Direct
70.9
68.9
46.3
Abduction
73.9
54.0
74.3
34.0
PLASMA
Direct
68.6
69.0
24.1
Abduction
74.0
31.4
73.0
19.0
C2S
Direct
84.2
54.9
89.9
29.0
Abduction 89.0
34.6
92.3
27.1
Table 7: An ablation study towards different factor generation methods. Direct refers to the direct factor generation method. Abduction refers to our abductive factor generation.
A.9 ABLATION STUDY TOWARDS THE EFFECT OF THE TWO LOSS FUNCTIONS IN THE LEARNING ALGORITHM
We conduct ablation study the loss function in Eq.7 for decision making, detailed in Appendix Table 8, and demonstrate that both components of the loss function are crucial for final performance.
Dataset
w/o MR Loss
w/o MSE Loss
BIRD
TODAY
65.5
65.2
74.3
PLASMA
60.0
65.7
73.0
C2S
77.2
86.8
92.3
Table An ablation study towards the effect of the two loss functions Eq.7 for decision making with Llama-3. 1-mta-...-70b-nmeee I-70b--nstrruc.
21

---

Published as a conference paper at ICLR 2025
A.10 EXPERIMENT SETTINGS FOR USING ESTIMATED PROBABILITIES ADDITIONAL TRAINING DATA
The T5-large model and PatternTime (Zhou al., 2021) specialized temporal reasoning models, as outlined in Feng et al. (2023). Both models were specifically fine-tuned on temporal reasoning datasets using the original temporal training data from Feng al. (2023). In contrast, the additional training data from COMMON2SENSE in $4.3 focuses on commonsense reasoning.
A.11 YES-OR-NO FOLLOWUP QUESTION GENERATION
We experiment to see if BIRD can further facilitate decision making in a controllable way, where use BIRD to generate a yes-or-no follow-up question about an unobserved factor in the context. Specifically, given a scenario and an additional condition, we first identify which factors and their corresponding values and is most important for the decision in the remaining unmentioned factors, as the target question area. We adopt the factor value that better aligns with the outcome direction suggested by the current implied factor values and has the largest condition probability. We then rewrite the statement of the factor value into its corresponding question. Simultaneously. we ask Llama2- 70----700---ssree directly generate a yes-or-no follow-up question. The prompt is in Appendix Fig. 15.
implied by the condition, we then select factor that is not mentioned
We use MTurk and ask annotators to provide pairwise preference between the BIRD generated question and LLM directly generated question. We adopt the same Mturk setting as mentioned in $A.5. we carry out the human annotation process three times, using the majority vote to find the preferred generated question. We show the experiment interfaces and example in Appendix Fig. 4. Appendix Table shows that the follow-up questions generated by BIRD are more often preferred.
Fepp setar:
Ranasc: 30d5 per SAR
Taaks avallail St
Dusstio:: 21bar
you presse which one has the potentaal
valuable decition Make doatsion based
information question could
ALL QUESTIONS must answered carcfully otherwise you are not paid. This is only piaor test, so we kindly ask cash annorator take assignments of most, The 6th anaignment you ancwerred will not be pard.
You poss le yeu- ar-na follow-up questtons are sadved based the scenario, lea., the user can only respord to each mecion either 'yea' 'no' without pr provide ding any further explanation or details.
Scenario: Johnn has good physical stringth and d'duranie hou.
Question Dove, Lohnn have more wxper winc and skill Laing a 10 pound skedgshammer?
Quastion 2 Dose, John have
using mwweeet
Outcome John found it eseier to break of conowte with pound sledgenamme a pound sledgeeermmee
Outcome John found It
break
concrete with eledgehammer
stedgetammes
1. Which yea-crno qu eueeton, whose answer can only be "yes" or "no", would have the potential to provide more usefu Info mattottio not mentioned in the scenario help you choose between two outcomes?
Question has pokentasl whist useru information decason making than Question a.
Question hon potenrbal wksst more uswhd inforration for dedielo making Cuestion
Ol cannot decide which queetion could provide more information think both questions ank for the same additional information for owctsion making.
Figure 4: The interface for human evaluation on preference- -hased pairwise evaluation of the generated questions. Question is generated by BIRD and question 2 is generated by LLM directly. The three annotators all prefer question 1.
Dataset
BIRD Win Direct
Tie
BIRD Lose Direct
COMMON2SENSE
132
36
82
Table 9: Preference judgments for yes-or-no follow-up questions generated by BIRD versus questions directly generated by LLMss. We randomly sample 250 instances from COMMON2SENS5
A. 12
PROMPT
22

---

Published as a conference paper at ICLR 2025
EXAMPLE ADDITIONAL SENTENCE SAMPLING PROMPT
SYSTEM
You are given scenario and an accompanying hypothesis. Generate 5 sentences covering different conditions that would add ibjectivv information relevant to the hypothesis such that the hypothesis is more likely to hold true. The information should not definitively imply the hypothesis. You must follow the below structure to just generate sentences with no explanations.
USER
Scenario: You want to move around with your cell phone when it is being charged.
Hypothesis: You can move around more freely with your cell phone if it is being charged with a one-foct cord rather than a six-foot cord.
ASSISTANT
The cell phone being charged with a portable power bank located in your pocket, allowing you to move around wich being tethered to a fixed outlet.
The user is working in compact space where longer cords could easily snag minimize this risk.
furniture
equipment, thus one-foot cord could
# The phone is needed for tasks that require frequent handling and close proximity to the user, making a shorter cord more practical to avoid excessive dangling, The charging setup includes small desktop charger that keeps the phone elevated and stable, limiting the practicality of longer cord.
The user is in busy environment like kitchen workshop, where sherter cords reduce the hazard of tripping catching on moving objects.
Figure Example Additional Sentence Sampling Prompt
23

---

Published as a conference paper at ICLR 2025
EXAMPLE FACTOR SUMMARIZATION FROM ADDITIONAL SENTENCES PROMPT
SYSTEM
From the given sentences each outcome, identify and list distinct and concrete factors, ensuring each bread yet specific and focuses on unique aspect.
Your respense should strictly adhere to the JSON format provided, without additional explanations.
For example: ("distinet factor" <ENSURE each factor focuses on unique aspect>: "factor values" <Each factor MUST cover at least one condition to support the Statement and one condition to support the Opposite statement.>/
Ensure that each factor's value MUST directly reference specific elements mentioned in the statements, avoiding vague terms like 'the object'.
1.
2. Knsure the factor values are not too concrete.
3. Do not only mention the common situations.
USER
Scenario: You are charging your cell phone and wish to move around with your cell phone.
Outcome 1: You can move around more freely with your cell phone if it is being charged with six feet cord rather than ene feet cord.
Sentences:
#1 A longer cord provides more flexibility and allows for a greater range of movement while using the cell phone. This is because the additional length the six-foot cord gives the user larger radius of movement, enabling them to cemfortably use their phone while it is charging without feeling restricted or confined to a specific location.
Outcome 2: You can move around more freely with your cell phone if being charged with a one-foct cord rather than six-foot cord.
Sentences: #2 If the cell phone is plugged into portable power bank or a USB port on computer, one-foot cord provides greater mobility because it shorter and less likely to get tangled or caught on objects while moving.
#3 If the cell phone is constantly being used while charging and the user prefers to keep the phone clase the charger at all times, one-foot cord allows for easier mobility and reduces the risk of tripping over longer cord."
ASSISTANT
"The cell phene's charging method":
"The charger is portable", "The charger is unmovable"),
"The user's s mevement range":
["The user stays very close to the charger", "The user has a large radius of movement"], "The location of the phone charger":
["The user leaves the charger somewhere", "The carries the charger"]]
Figure 6: Example Factor Summarization from Additional Sentences Prompt
24

---

Published as a conference paper at ICLR 2025
EXAMPLE PROMPT FOR DECIDING THE VALUE BETTER SUPPORTS WHICH OUTCOME
SYSTEM
Decide whether each of the given values supports Statement 1 or Statement 2. The statement and opposite statement are provided, along with list values. Determine which statement each value supports. You must follow the belcw structure to just generate json with an explanation. For example:
Explanation: <explanations for how each value supports which statement>
List: { "value <ottuttt only Statement 1, Statement 2, Both, or Neither>, "value 2": <output only Statement 1, Statement 2, Hoth, or Neither>}
USER
Here is a scenario:
I'd like you to determine which statement
each value supports.
Values:
Statement 1:
Statement
Ploase analyze each value and determine which statement it supports.
ASSISTANT
Explanation: by analysis here]
List: "value"; "Statement 1"
Figure 7: Example Prompt for Deciding the Value Better Supports Which Outcome
EXAMPLE PROMPT FOR DECIDING THE SCENARIO WITH THE CONDITION IMPLIES THE VALUE OF THE FACTOR
SYSTEM
Given scenario, read sentence and determine whether value from given 1ist is implied by the sentence. You must strict on considering value as impliea and return one value or "None" with an explanation.
Here are rules that you must follow:
1. A value is implied only if the sentence implies it in most situations, meaning the sentence provides clear, direct infermation relevant to the value.
2. Do not select a value based on indirect associations, context, related hints unless they strongly imply the value in most situations.
3. Do not assume causality or infer connection if not clearly stated in the sentence.
4. If the sentence implies more than one value, choose the most implied one. If two values are both implied, choose the one that is positive (non-negative).
5. If the sentence does not imply any value di rirectly in most situations, say "None". USER
Here is scenario:
Sentence:
List: "factor"; [ "value 1",
ASSISTANT
("explanation":
"answer": value/None)
Figure 8: Example Prompt for Deciding the Scenario with the Condition Implies the Value of the Factor
25

---

Published as a conference paper at ICLR 2025
EXAMPLE PROMPT FOR REEVALUATING IMPLIED VALUE
SYSTEM
Based on a scenario, delete value the list the key does not imply value. You should output an explanation and then à jeon. You must generate double quetes. Here rules that you must follow:
Ensure that retained values align with key's implications, do not assume causality or infer a connection if the value is not clearly stated in the key. Avoid over-generalizatt ionaiions and ensure logical consistency in your decisions.
Be lenient with deletions. Only
value if the key clearly
does not imply the value.
Provide a clear explanation for each deleticn or retention. USER
Scenario: The government is planning the locations for building charging stations.
("The locatien is on busy highway with no existing charging stations.":
["No nearby charging stations",
"Location is major travel route, serving Long-distance EV travelers",
"Nearøy amenities like restaurants, shops, and rest areas for users while charging"]]
ASSISTANT Explanation:
nearby charging stations" was kept because the key explicitly mentions the absence of existing charging stations in the area. Similarly, "Location is on a major travel route, serving long-distance EV travelers" was retained because being busy highway strongly implies that it serves for such travelers. However, "Nearby amenities like restaurants, shops, and rest areas for users while charging" was removed since the key does not directly mention or suggest the presence of these amenities.
key travel route
List: {"The location is busy hi highway with existing charging stations.":
'NNo nearby charging stations", "Location is en major travel route, serving Long-distance EV travelers"]}
Figure Example Prompt for Reevaluating Implied Value. Explanation optional.
26

---

Published as a conference paper at ICLR 2025
EXAMPLE PROMPT FOR OUTPUTTING A VEREALIZED PROBABILITY SCORE
SYSTEM
As an AI assistant, your zole is to respond accurately to user queries. While answering think step-by-step and justify your answer. Analyze the given scenaric and condition to determine the likelihood
outcomes. Use only the information provided, without relying on prior knowledge. Assess the probability using the specified terma: 'very likely', 'likely', 'somewhat likely', 'neutral', 'scmewhat unlikely', 'unlikely', 'very unlikely'. Ensure that your assessments "likely', the other must be 'unlikely', and so on. You should first give your reasons and then format your final answer.
complementary: if outcome is deemed
USER
Scenario: You want to move around with your cell phone when it is being charged.
Condition:
The charger is portable. The user stays very close to the charger. The user carries the charger.
Outcome 1: You can move around more freely with your cell phone with a ene-foot cord rather than six-foot cord. Outcome 2: You can move around more freely with your cell phone with a six-foot cord rather than a one-foot cord.
ASSISTANT
Given that the user is carrying pertable charger, shorter cord like one foot would indeed be far more manageable, making it easier for the user to move around freely.
Given the same conditions, a Longer cord like six feet might become an impediment, making it more challenging for the user whe is carrying the charger to move around freely due to the possibility of tangling or managing the extra length. Therefore, Cutcome 1: Likely
Outcome 2: Unlikely
Figure 10: Example Prompt for Outputting a Verbalized Probability Score
EXAMPLE PROMPT FOR DIRECT PROBABILITY VERBALIZATION BASELINE
SYSTEM
You are an AI assistant that helps people make decisions. The user will give you question. Choose which outcome the scenario and its corresponding additional condition better support. Provide your confidence level as a percentage to indicate how certain you are about your answer.
USER
Scenario:
Additional Condition:
Outccmme 1:
Outcome 2:
ASSISTANT
Answer and Confidence (0-100): Cutcome i, X
Figure 11: Example Prompt for Direct Probability Verbalization Baseline
27

---

Published as a conference paper at ICLR 2025
EXAMPLE PROMPT FOR DIRECT PROBABILITY VEREALIZATION WITH COT BASE- LINE
SYSTEM
You are an AI assistant that helps people make decisions. The user will give you question. Your task is to answer as faithfully you can. While answering, think step-by-step and justify your answer. You are asked to provide a probability for each outcome being correct (0% to 1008) based on the scenaric and its corresponding additional condition. Ensure that (outcome 1) + (outcoom 2)
USER
Scenario:
Additicnal Condition:
Outcome 1:
Outcome 2:
ASSISTANT
Explanation ion:
[step-by-step analysis here]
Confidence:
P (outcome 1) x3, P (outcome 2) Ys.
Figure 12: Example Prompt for Direct Probability Verbalization with CoT Baseline
EXAMPLE PROMPT FOR EXPLICIT COMPARISON BASELINE
SYSTEM
assistant that helps people nake decisions. The user will give you a question. Your task to answer as faithfully as you can. While answering, think step-by-step and justify your answer. You are asked to decide which condition makes the cutcome more likely to be true given the scenaric.
You are
USER
Sconario:
Outcome:
Condition 1:
Condition
ASSISTANT
Explanation: (step-by-step analysis herel
Answer: Condition i/Both.
Figure 13: Example Prompt for Explicit Comparison Baseline
28

---

Published as a conference paper at ICLR 2025
EXAMPLE COT FOR DECISION MAKING PROMPT
SYSTEM
You are a helpful assistant. While answering think step-by-step and justify your answer. Given scenario and additicnal condition, decide which of the two cutcomes is supported.
USER
Scenario: The government is planning the locations for building charging stations.
Condition: The location is on busy highway with no existing charging stations.
Outcome 1: The government should build a charging staticn here.
Outcome 2: The government should not build charging station here.
ASSISTANT
The scenario presents busy highway with no existing charging stations, and the government is planning where to build new Given the high traffic volume and the absence of charging infrastructure, building station here would fill critical gap, support electric vehicle drivers, and encourage the transition to sustainable transportation.
Based on the available information, Outcome 1: The government should build charging station here is supported.
Figure 14: Example COT for Decision Making Prompt
EXAMPLE DIRECT YES-OR-NO FOLLOW-UP QUESTION GENERATION PROMPT
SYSTEM
You are a helpful assistant. while answering think step-by-step and justify your answer. Ask follow-up yes questien that you can better decide which cutcome better supported by the scenaric and the condition. You can only ask for specific situation.
USER
Scenario: You want to move around with your cell phone when it is being charged.
Condition: The charger is portable and the user carries the charger.
Outcome 1: You can around more freely with your cell phone with cne-foot cord rather than six feet cord. Outcome 2: You can move arounc more freely with your cell phene with a six-foot cord rather than one feet cord.
ASSISTANT
In this scenario, the primary factor consider is cord length, in most situations, a greater cord length allows more movement freedom, as you can be further from the outlet and still keep your device charged. However, considering that user carries the charger, it might bring about the confusion between the two outcomes.
Thus, further ascertain which outcome is supported better, I'd ask the following question:
Does the portable charger need to be plugged into an outlet to charge the phone?
Figure 15: Example Direct Yes-or-no Follow-up Question Generation Prompt
29


Reviews:
[{"review_id": "0Tmg1y66mD", "reviewer": "Reviewer_gTw6", "summary": "This work proposes an inference framework that aims to produce more reliable and accurate probability/confidence estimations using LLMs for large-scale decision-making and planning tasks. The proposed method, BIRD, uses abductive factor generation to factorize the inference task, and estimates the outcome by accumulating the predictions made on the specific factors. The experimental results on three datasets show that BIRD is able to produce more accurate probability estimations in decision-making and planning, which can be used to (1) improve the accuracy of decision-making; (2) supervise the training of small models; (3) propose follow-up questions for decision-making. For all these applications, BIRD shows superior performance than the baseline methods.", "questions": "In Section 4.3, \"The Usage of the Reliably Estimated Probability\", it seems that the original T5 large model is better on some datasets compared to the fine-tuned models. What would be the reasons for this? Besides, what is \"PatternTime\" in Table 3? It is not mentioned anywhere else in the manuscript.\n\nPlease see \"Weaknesses\" for a few additional questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Bayesian inference framework is conceptually interesting. Factoring the overall decision-making into multiple fine-grained factors is intuitive and useful for mitigating LLMs' limitations in providing accurate fine-grained probability/confidence estimation.\n\n2. The experiments performed are carefully designed, and the overall writing is clear and with enough details.\n\n3. The experiment results demonstrate the effectiveness of BIRD, showing its value in multiple application scenarios.", "weaknesses": "1. The training setting for \"Learning algorithm for constrained optimization to estimate $P(O_i|f)$\" (Line 266) should be more clearly described. The current description does not clarify (1) the number of learnable parameters ($P(O_i|f_j)$); (2) whether different instances share the same factors $f_j$; (3) whether there are $f_j$ in the test data that are unseen in the training data.\n\n2. In the experiment for \"Applying BIRD’s Probabilities in Decision-Making\" (Line 430), the test instances on which BIRD's decision is unknown are filtered out. While it is claimed that \"this does not undermine our experiment setting because such removal is label-agnostic\" (Line 450), it may still introduce biases since this filtering process depends on the output of BIRD. It might be the case that BIRD predicts \"unknown\" when it is uncertain internally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "At7Jzlno66", "reviewer": "Reviewer_ZTvP", "summary": "This paper presents a method allowing LLMs to reason more soundly about complex problems with uncertainty, by combining LLM introspection and verbal estimates of uncertainty into Bayesian reasoning about outcomes. In many ways it resembles chain-of-thought, in that the LLM is being asked to provide high level reasoning about an uncertain scenario, and is required to brainstorm to solve the problem. The incorporation of a Bayes net sets the method apart from prior work, and experiments show that this leads to much better estimates in pairwise evaluation of propositions, but less compelling evidence for the work when evaluated on decision making benchmarks.", "questions": "There several magic numbers in 3.3 and 3.4 (mappings of rankings to probabilities), as well as key design decisions in model formulation. Can you comment on sensitivity of the method to these values/decisions?\n\nWill the evaluation data - and the code - be released?\n\n372, 449: What instances lead to 'unknown' predictions for all factors? Is this a shortcoming with the factor generation method, or is it that the problem itself is riddled with uncertainty.\n\n452: Can you elaborate on the spurious biases that affect CoT, giving it such strong performance? The same criticism could be applied to the LLM prompting as part of Bird, and the probabilities produced.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper presented a very novel idea, a detailed method for achieving this, and a convincing evaluation of the utility of the technique. Specific strengths include:\n - mechanism for prompting a LLM to extract factors & potential outcomes of a complex decision problem\n - means to convert the factors into a binary Bayes network\n - incorporation of model uncertainty estimates into the parameterisation of the Bayes network\n - evaluation against human judgements of probability\n - strong baselines", "weaknesses": "The paper was quite dense to start, and took several pages to make the application scenario clear. On my first pass it wasn't apparent to me what elements of the data were part of the problem definition, what parts were computed by the LLM, and what was computed by inference in a Bayes net. Fig 1 is reasonably easy to follow, but Fig 2 is less clear - why have the conditions changed between Fig 1 and Fig2, and what parts of the central box are given as part of the data versus inferred from LLM interactions? As I understand it, the whole of the inner box is auto-generated, as well as the edges linking the given conditions with coarse probability estimates.  \n\nThe paper also takes a long and detailed dive into Bayes net formality, which is more than needed given the setting is relatively simple and could be communicated effectively with more of a focus on the running example (Fig 2 could be unpacked). As it stands, the paper takes several pages to get going, but ends up deferring many critical details to the Appendix. The method for prompting the LLM is of fundamental importance to the technique, for instance, but none of this is in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cJKejkZgHM", "reviewer": "Reviewer_VqFc", "summary": "This paper proposes a BIRD framework that enables more reliable probability estimation based on incomplete instances from LLMs. It leverages factors generated by LLMs to make a Bayesian network and estimates more precise probabilities. BIRD with optimized prob presents a better performance in several experiments than existing methods. BIRD also demonstrates improved reasoning performance across various datasets, particularly excelling in more complex scenarios.", "questions": "1) Page 1: LLMs’s → LLMs’\n2) The LaTeX expression for P in the introduction differs from its expression in the later sections.\n3) Where is the example of the factor, \"The color of the vehicle,\" mentioned by the authors in Line 228, depicted in Figure 2? I couldn't find it in the figure as described in the text.\n4) Appendix A.2: Week →Weak\n5) This may be a minor issue, but there could be confusion between Line 318 'S + U₃ implies d₁ and f₂' and f, which is a possible instance of complete information due to the notation.\n6) An additional question is, have the authors considered any experiments involving more than two outcomes (i.e., beyond binary classification)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) The proposed framework and optimization settings, which account for weak ordering and non-interaction in the set of factors, are valid.\n2) The authors' assumptions about enhancing the validity of the proposed Bayesian Inference Framework are reasonable.\n3) Including unobserved factors in the calculations strengthens the effectiveness and utility of the authors' proposed method.\n4) Besides precise probability estimation, the presence of follow-up generation and cross-domain experiments demonstrates the proposed method's reliability and practicality.", "weaknesses": "1) In Line 147 and Line 155, the explanation of the outcomes is separated. It seems possible to consolidate these explanations into one part. Additionally, the explanation of 𝐹 in the same paragraph could be made more concise. The space saved from these revisions could then be used to more clearly elaborate on the connection between Equation (5) and Equation (6) and the content in Algorithm 1.\n\n2) Ablation settings could be expanded to cover more aspects. For example, experiments could be conducted by excluding the MR loss from the unified loss function or by assuming that in an unobserved factor, each value does not have an equal probability of being selected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)