DeLLMa: Decision Making Under Uncertainty with Large Language Models

## Abstract

Abstract The potential of large language models (LLMs) as decision support tools is increasingly being explored in fields such as business, engineering, and medicine, which often face challenging tasks of decision-making under uncertainty .
In this paper, we show that directly prompting LLMs on these types of decision-making problems can yield poor results, especially as the problem complexity increases.
To aid in these tasks, we propose DeLLMa (Decision-making Large Language Model assistant), a framework designed to enhance decision-making accuracy in uncertain environments.
DeLLMa involves a multi-step reasoning procedure that integrates recent best practices in scaling inference-time reasoning , drawing upon principles from decision theory and utility theory, to provide an accurate and human-auditable decision-making process.
We validate our procedure on multiple realistic decision-making environments, demonstrating that DeLLMa can consistently enhance the decision-making performance of leading language models, and achieve up to a 40% increase in accuracy over competing methods. Additionally, we show how performance improves when scaling compute at test time, and carry out human evaluations to benchmark components of DeLLMa.

## 1 Introduction

Large language models (LLMs) are rapidly gaining traction across many domains due to their potential for automating and enhancing a broad spectrum of tasks [, ].
One important potential use is in decision making under uncertainty, i.e., deciding which action to take given some set of possibilities, properly factoring in user goals and uncertainty about the world.
The ability to make good decisions under uncertainty holds broad relevance across high-stakes tasks in fields such as business, marketing, medicine, aeronautics, and logistics [, ]—and its value is not limited to organization-level decisions but extends to aiding individuals in making informed choices as well.
The ability of LLMs to analyze large quantities of data makes them potentially well-suited for sophisticated decision support tools, and ensuring that these models give accurate, context-aware recommendations could significantly augment human decision-making capabilities.

However, optimal decision making under uncertainty is often challenging. For humans, there exist frameworks from decision theory and utility theory (developed in fields such as economics, statistics, and philosophy) to provide a structured approach for better decision-making [, , ].
Research has consistently demonstrated that without these frameworks, human decision-making can often be highly irrational, swayed by biases and incomplete information [].
Similarly, making decisions with LLMs faces its own set of challenges.
Issues include the tendency to fixate on specific explanations or information without adequately balancing all evidence, and the inability to effectively handle uncertainty, manage biases, or align with a user’s goals and utilities [, ].
Our paper presents experiments that exemplify these issues.

Furthermore, beyond merely making rational decisions, it is crucial to understand why an LLM made a particular decision.
This aids in building trust in the decision, assessing its quality, and improving any components that may lead to suboptimal outcomes.
The ability to explain decisions and verify decision-making quality—which we refer to as human auditability—is essential for the practical application of LLMs to aid decision making in many real problems [].

In this paper, our goal is to develop a framework that enables LLMs to make better decisions under uncertainty.
Our aim is not only to enhance decision-making accuracy but also to allow human users to understand the rationale behind each decision. Drawing inspiration from prior work on multi-step reasoning like Chain-of-Thought (CoT) [] and Tree-of-Thoughts (ToT) [], in which compute is scaled at inference time, we
design a procedure
based on
classical decision theory, originally designed for rational decision making under uncertainty by humans.
Our approach involves three key steps: first, identify and forecast pertinent unknown variables given in-context information; second, elicit a utility function that aligns with the user’s goals; and finally, use this utility function to identify the decision that maximizes expected utility.
We call our proposed framework DeLLMa, short for
Decision-making Large Language Model assistant.

We evaluate DeLLMa on real decision-making scenarios in agriculture and finance, and compare it against existing strategies for LLM decision-making, including zero-shot (direct) prompting, self-consistency [], and CoT approaches.
We find that DeLLMa significantly enhances decision-making accuracy, with improvements of up to a 40% increase in accuracy, particularly as the complexity and number of potential actions increases.
Additionally, DeLLMa can consistently enhance performance across a variety of leading language models, and its structure allows us to understand the rationale behind each decision.
In full, our contributions are:

- •
We introduce DeLLMa, a method for human-auditable LLM decision making under uncertainty, employing a multi-step reasoning process at inference time based on classical decision theory.
- •
We evaluate components of DeLLMa, including the calibration of its state forecasting approach and a human agreement study to assess its utility elicitation method.
- •
On realistic decision-making environments, we show that DeLLMa gives up to a 40% improvement in decision-making accuracy over competing methods, and yields consistent improvements when deployed across multiple leading LLMs.

## 2 Related Work

Exemplified by OpenAI o1 [], a combination of recent advances in scaling inference-time reasoning—such as task decomposition and structured search—has brought forth superhuman performances on deterministic reasoning tasks [, ]; but we nonetheless find that they are insufficient for decision-making under uncertainty. DeLLMa offers a specialized inference-time solution that scales favorbly with parallel sampling [], a key ingredient for performant utility elicitation and decision optimization. Below, we summarize prior works pertaining to the DeLLMa framework.

#### LLMs for Decision Making.

Prior works have leveraged LLMs for optimizing blackbox functions [, , ]. These settings involve methods that make a substantial number of low-cost decisions (which do not incur a high price for suboptimality). Instead, we focus on single-step expensive decisions, particularly in the prescence of uncertainty, with a focus on the optimality of decisions.
Additionally, a number of applied domains that involve decision making have started to explore LLM-based methods, such as in supply chain optimization [], medicine and health [], and automated driving [].

#### Uncertainty in LLMs.

LLMs, without proper calibration, can be overly confident in their responses []. Such pitfalls make them unlikely to make reliable decisions under uncertainty. Prior work has aimed to solve this issue; one line of research involves asking LLMs for their own confidence, with or without additional finetuning [, , , , , , ].
Referring to [] for a detailed survey, many recent advances [, , , ] adopt a Bayesian inference framework to quantify and reason with uncertainties in LLMs. Other works have shown that tool usage [], retrieval augmentation [], and model ensemble [] can improve calibration and forecasting capabilities of LLMs. Our framework can take advantage of these advances in LLM-based forecasting for improved decision making.

#### Inference-time Reasoning in LLMs.

Many recent works have leveraged inference time compute to extend the computational power of language models [, ]. These can be categorized into three main approaches: (1) instructing the model to generate intermediate reasoning traces [, , , , , , ], (2) decomposing a complex reasoning problem into tangible components [, , , ], and most recently (3) scaling the number of parallel samples to be postprocessed into a final solution [, , , ].

## 3 Methods

#### Preliminaries.

Suppose that a decision maker needs to make a choice between a set of options to achieve some goal—i.e., has a decision problem.
We begin by formalizing such a decision problem, and afterwards describe how we approach decision making with LLMs.
There are three main components to the decision problems that we will describe: actions, states, and utilities.

First, the actions are the possible options that a decision maker wishes to choose between. We use $\mathcal{A}$ to denote the space of actions, and $a\in\mathcal{A}$ for a single action.
Second,
the set of unknown states of nature are denoted $\Theta$.
In our formulation, we define a state $\theta\in\Theta$ to be any latent variable whose true value is unknown, yet affects outcomes relevant to the decision maker’s goals.
To perform optimal decision making, one must act while accounting for uncertainty over these unknown states.

The third component involves
the decision maker’s preferences for different possible outcomes.
We formalize our framework for decision making under uncertainty using utility theory, which can be viewed as “modeling the preferences of an agent as a real-valued function over uncertain outcomes” [, , ].
A key element is the utility function (in some formulations, this is instead given in terms of a loss function $L$).
The utility function, denoted $U:\Theta\times\mathcal{A}\rightarrow\mathbb{R}$, assigns a scalar value to any state and action $(\theta,a)\in\Theta\times\mathcal{A}$.
Intuitively, a higher utility means that the state-action pair yields a more-preferable outcome for the user.

The goal of the decision maker will be to choose a final decision $a^{*}\in\mathcal{A}$, which yields the highest possible utility, while accounting for uncertainty in the unknown states $\theta$.

Figure: Figure 1: Given a decision problem and contextual information as a prompt, DeLLMa (decision-making LLM assistant) maximizes an expected utility to select an available action. We illustrate the key steps of DeLLMa on decision-making tasks in agriculture planning (top) and finance (bottom).
Refer to caption: x1.png

#### Decision Making with LLMs: Setup and Current Approaches.

We first describe the setting in which we intend our framework to operate. Suppose a human wishes to use this LLM assistant to help make a decision. They begin by describing a decision problem via a user prompt $\mathcal{P}$.
We formalize a user prompt as a triplet $\mathcal{P}=(\mathcal{G},\mathcal{A},\mathcal{C})$, which includes: a natural language description of the user’s goal $\mathcal{G}$, a list of $n$ actions $\mathcal{A}=(a_{1},\ldots,a_{n})$, and a passage of contextual information $\mathcal{C}$, which might be, e.g., pages from a report, or a text-based representation of historical data.

Referring to the agriculture planning decision problem in Figure 1 as a running example, the goal $\mathcal{G}$ is for a farmer to maximize their revenue in the forthcoming year; the action set $\mathcal{A}$ lists the possible produce the farmer is considering planting (e.g., apples, avocados, pears); and the context $\mathcal{C}$ consists of historical summaries of agricultural yields or information about the climate around the farm.

It is tempting to delegate such decision-making to LLMs with direct prompting.
However, we observe that responses from conventional approaches, such as Self-Consistency and CoT, do not adequately balance available evidence, handle uncertain information, or align with user preferences; we show in Sec. 4 that these methods perform poorly, especially with increasing numbers of actions.

#### DeLLMa: Decision-Making LLM Assistant.

To help encourage improved decisions under uncertainty, we propose a framework that guides an LLM to follow the scaffolding of classical decision theory.
By restricting LLMs to this scaffold we can also explicitly see components of the decision-making process—e.g., predictions of unknown states and utility function values—which provides human-auditability, allowing a user to identify why a given decision was made by the model.

In our initial formalization of this framework, we restrict ourselves to a slightly curtailed class of problems, and thus make a few simplifying assumptions.
For example, we have assumed above that there are a discrete, enumerable set of $n$ possible actions, i.e., $\mathcal{A}=(a_{1},\ldots,a_{n})$. We will also assume there is a discrete set of $m$ possible states, $\Theta=(\theta_{1},\ldots,\theta_{m})$, though $m$ may be quite large.

Our framework will use an LLM to produce a belief distribution over the unknown states, given the input context $\mathcal{C}$.
We view this as a posterior belief distribution over the states, which we denote by $\pi(\theta\mid\mathcal{C})$.
Implicitly, we are assuming that the LLM implies a prior belief distribution $\pi(\theta)$, given only the model weights or training data.

Our framework will also elicit a utility function, based in part on the description of the user’s goals $\mathcal{G}\in\mathcal{P}$.
This utility function assigns a scalar value to any state-action pair $(\theta,a)$. We denote this utility function as $U(\theta,a)$.
Given these, the expected utility under our LLM of taking an action $a$, given some additional context $\mathcal{C}$, can be written

$$ $\displaystyle U_{\mathcal{C}}(a)=\mathbb{E}_{\pi(\theta\mid\mathcal{C})}\left[ U(\theta,a)\right]=\sum_{\theta\in\Theta}\pi(\theta\mid\mathcal{C})U(\theta,a).$ (1) $$

Then, following the expected utility principle for rational decision making [, ], we select the Bayes-optimal decision $a^{*}$, which maximizes the expected utility, written

$$ $\displaystyle a^{*}=\operatorname{arg\mathrm{max}}_{a\in\mathcal{A}}U_{c}(a).$ (2) $$

We call our framework DeLLMa, short for Decision-making Large Language Model assistant.
DeLLMa carries out this sequence of four steps—state enumeration, state forecasting, utility elicitation, and expected utility maximization. A full description of DeLLMa is shown in the box below.
In the following sections we give details on our specific implementation of each of these four steps.

### 3.1 State Enumeration

As an initial demonstration of DeLLMa, our goal is to develop a simple implementation that performs well empirically; however, each component in the framework can be extended or made more sophisticated.
We first describe the strategy that we adopt for enumerating a space of relevant latent states $\Theta=(\theta_{1},\ldots,\theta_{m})$.
Given $\mathcal{P}$ as context, we prompt an LLM to identify $k$ latent factors that are predicted to influence the user’s goal $\mathcal{G}$ (see §C.2 for details on this prompt).
Each latent factor is a string (a word or phrase), which can be viewed as describing a dimension of our state space $\Theta$.
We denote these $k$ latent factors as $(f_{1},\ldots,f_{k})$.

For each latent factor, we prompt our LLM to generate $\ell$ plausible values of the latent factor (empirically we find that it is sufficient to set $\ell$ to be small, e.g., $<$$5$).
For a latent factor $f_{j}$, we denote its plausible values as $\tilde{f}_{j}^{1:\ell}$, where each plausible value is also a string (a word or phrase). This process discretizes the state space, where each of the $k$ dimensions has $\ell$ bins.
We find that this strategy, while simple, yields an empirically effective method for forecasting states (described in §3.2).

A single state $\theta_{j}$ in this state space consists of one plausible value from each of the $k$ latent factors, which we can denote by $\theta_{j}=\theta_{j}^{1:k}\in\Theta$.
In total, this produces a discretized state space of size $|\Theta|=m=\ell^{k}$.
While this state space is too large to enumerate explicitly, we develop a procedure to forecast probabilities for these states in a scalable manner.

### 3.2 State Forecasting

In the next step of DeLLMa, we form a probabilistic forecast of the unknown states, given information contained in the context $\mathcal{C}\in\mathcal{P}$.
A number of recent works have shown that LLMs are capable of returning well-calibrated forecasts with respect to some provided information [, ] (see §2). Here, we develop a relatively simple forecasting method that we find works well empirically; though DeLLMa allows us to flexibly use other forecasting methods in the future (potentially leveraging search and retrieval of information).
This step yields a distribution over the state space, which we can sample from to get a Monte Carlo estimate of the expected utility.

For each of the $k$ latent factors, and each of their $\ell$ possible values $\tilde{f}_{j}^{1:\ell}$, we prompt our LLM to assign a verbalized probability score $\in$ $\{$very likely, likely, somewhat likely, somewhat unlikely, unlikely, very unlikely$\}$. In total, we must assign $k\times\ell$ scores. We provide all prompts for this probability score procedure in Figure 11.
We then define a dictionary $\mathcal{V}$ that maps each verbalized probability score to a numerical value. Similar strategies converting from verbalized to numeric scores have been used with success in prior work [, ].
After normalization, this
yields a distribution over the state space $\Theta$, assuming independence between the $k$ latent factors, which we posit for computational simplicity.
We sample states from this distribution by iterating through each of the latent factors, sampling according to its approximate marginal probability,
and concatenating the samples.
In Sec. 4.1 we directly evaluate this state forecasting procedure to show that it yields well-calibrated forecasts in real-data scenarios, and we conduct an ablation study in Sec. 4.3.

The full procedure is shown in Algorithm 1.
Here, the $\operatorname{Normalize}$ function simply scales the weights instantiated in the marginal distribution to a well-defined probability mass function (PMF).
We consider the sampled states to be from an LLM-defined proposal distribution $\pi^{\text{LLM}}(\theta\mid\mathcal{C})$, returned as output from Algorithm 1, which approximates the posterior belief distribution $\pi(\theta\mid\mathcal{C})$.

Figure: Algorithm 1 StateForecast

### 3.3 Utility Function Elicitation

Next, we need a method to elicit (which is to say: construct) a utility function $U:\Theta\times\mathcal{A}\rightarrow\mathbb{R}$, which maps a state-action pair to a real value.
An accurate utility function, which balances the preferences of a human user with respect to the goal that they describe, is difficult to define directly in a general-purpose manner.
There is a long history of work on utility elicitation methods [], which aim to construct a utility function from, e.g., pairwise preference data. Here, we combine these methods with large language models to try and automatically elicit a utility function.

We conduct the following procedure. We first sample states from the forecast state distribution $\pi^{\text{LLM}}(\theta\mid\mathcal{C})$, and from these form a set of state-action pairs.
We group these pairs into minibatches, and prompt our LLM to rank the elements of each minibatch, given the user’s goal $\mathcal{G}\in\mathcal{P}$.
This LLM-based ranking of items—where each item consists of an action and a particular instantation of states—is a procedure that can be broadly applied, and LLMs have a history of being successfully used for similar comparisons [, ].
Based on these rankings, we are able to extract pairwise preferences, which we can use in classic utility elicitation algorithms.

We show this procedure in Algorithm 2 and discuss two implementations of the $\operatorname{FormatRank}$ step: $\operatorname{Rank2Pairs}$ and $\operatorname{One-vs-All}$. Denoting $(\theta,a)_{(i)}$ as the $i$-th preferred state-action pair of the minibatch, $\operatorname{Rank2Pairs}$ converts a ranking of decreasing preference $\mathcal{R}=\left((\theta,a)_{(1)},\cdots,(\theta,a)_{(b)}\right)$ to a list of pairwise comparisons by adding $(\theta,a)_{(i)}\succ(\theta,a)_{(j)}$ whenever $i<j$. In contrast, $\operatorname{One-vs-All}$ assumes that the LLM is indifferent towards all but the top-ranked state-action pair, i.e., $\left\{(\theta,a)_{(1)}\succ(\theta,a)_{(i)}\mid\forall~{}2\leq i\leq b\right\}$. This implementation may be desirable when accurate comparisons of certain suboptimal state-actions is challenging.
We then make use of these preferences as training data for a Bradley-Terry model [] to elicit an approximate utility function $U:\Theta\times\mathcal{A}\rightarrow\mathbb{R}$ with respect to the sampled state-action pairs.
Finally, we find two additional ingredients are beneficial for scaling our inference-time reasoning, which help improve the accuracy and computational efficiency of utility elicitation: batching and variance reduction.

Figure: Algorithm 2 UtilityElicitation

#### Batching.

We implement a batched inference procedure that slices state-action samples $S_{A}=\{(\theta,a)\}$ into overlapping minibatches for ranking. We ensure that $q$% of samples are shared between two consecutive minibatches drawn from $S_{A}$, where $q$ is a hyperparameter that modulates a minibatch’s degree of exposure to the preference of the previous minibatch. Larger $q$ results in finer-grained preference at the cost of more queries. Effects of $q$ are ablated in Figure 3.

#### Variance Reduction.

Directly sampling $|S_{A}|$ state values from the proposal distribution $\pi^{\text{LLM}}$ may lead to high-variance estimates of utilities. We instead sample $|S_{A}|/|\mathcal{A}|$ independent state values from $\pi^{\text{LLM}}$, create $|\mathcal{A}|$ duplicates, and pair them with each action $a$ (see Figure 5 in Appendix A).

In § 4.3, we conduct ablation studies to validate our utility elicitation procedure, which shows high agreement rate between DeLLMa rankings and human preferences. We also study scaling laws of sample size and overlap percentage. We find that this type of specialized inference-time solution scales favorably against general-purpose systems such as OpenAI o1 [].

### 3.4 Expected Utility Maximization

In the final step of DeLLMa, we compute the expected utility for each action, and then return the action that maximizes the expected utility.
In particular, for each action, we compute a Monte Carlo estimate of the expected utility using state-action samples (drawn from the state forecast distribution $\pi^{\text{LLM}}(\theta\mid\mathcal{C})$), as well as the elicited utility function.
Note that these calculations are all performed analytically (not via an LLM).
We can then approximate the expected utility $U_{\mathcal{C}}(a)$ as

$$ $\displaystyle U_{\mathcal{C}}(a)=\mathbb{E}_{\pi(\theta\mid\mathcal{C})}\left[ U(\theta,a)\right]\approx\frac{1}{|S|}\sum_{\theta\in S}U(\theta,a),$ (3) $$

given a set of state samples $S\subseteq\Theta$ drawn from our LLM-defined state forecast distribution, which is an approximation of the LLM’s posterior belief distribution about states given context $\mathcal{C}$,
i.e., $S\overset{\textit{i.i.d.}}{\sim}\pi^{\text{LLM}}(\theta\mid\mathcal{C})\approx
\pi(\theta\mid\mathcal{C})$.
After computing the expected utility $U_{\mathcal{C}}(a)$ for each action, DeLLMa returns the final decision:
$a^{*}=\operatorname{arg\mathrm{max}}_{a\in\mathcal{A}}U_{\mathcal{C}}(a)$.

## 4 Experiments

We evaluate the performance of DeLLMa on two decision making under uncertainty environments sourced from different domains: agricultural planning (Agriculture) and finance investing (Stocks). Both involve sizable degrees of uncertainty from diverse sources, and are representative of distinct data modalities (natural language and tabular) involved in decision making.
We propose the following DeLLMa variants designed to assess algorithmic improvements, as outlined in Section 3:

- •
DeLLMa-Pairs is the
method using all techniques in §3.3 and $\operatorname{Rank2Pairs}$ for utility elicitation.
- •
DeLLMa-Top1 is identical to DeLLMa-Pairs, but replaces $\operatorname{Rank2Pairs}$ with $\operatorname{One-vs-All}$.
- •
DeLLMa-Naive is a base version of DeLLMa-Pairs, where we sample multiple states per action and construct pairwise comparisons from a single batch (i.e., no batching or variance reduction).

For DeLLMa-Pairs and Top1, we allocate a per action sample size of 64 and a minibatch size of 32. We set the overlap proportion $q$ to 25% for the Agriculture dataset and 50% for the Stocks dataset due to budget constraints. For DeLLMa-Naive, we fix a total sample size of 50.
As a default LLM in experiments (except for comparisons across different LLMs), we use GPT-4 [](^1^11We use GPT-4 checkpoint gpt4-1106-preview.).

We compare DeLLMa against three baselines—zero-shot, self-consistency, and Chain-of-Thought (where example prompts for these baselines are given in Figures 9, 10, 19, and 20):

- •
Zero-Shot. Only the goal $\mathcal{G}$, the action space $\mathcal{A}$, and the context $\mathcal{C}$ is provided. We adopt a greedy decoding process by setting temperature = 0.
- •
Self-Consistency (SC) []. We use the same prompt as in zero-shot, but with temperature = 0.5 to generate a set of $K$ responses. We take the majority vote of the $K$ responses.
- •
Chain-of-Thought (CoT) []. For decision-making tasks, there is no standard CoT pipeline. Inspired by workflows from decision theory, we create a prompting chain consisting of three steps: (1) ask for unknown factors that impact the decision; (2) given these, ask for their possibility of occurence; (3) then ask for a final decision. Such a mechanism is similar to the DeLLMa pipeline (see §3) but only consists of prompting.

#### Evaluation Metrics.

For both datasets, our action spaces consist of a set of items, and we evaluate the performance of both DeLLMa and baseline methods by comparing the accuracy of their prediction from this set against the ground-truth optimal action (i.e., the action that maximizes ground-truth utility). We also report normalized utility—i.e., the ground-truth utility of the action chosen by a given method, normalized by the optimal ground-truth utility—in Appendix B.

In what follows, we report the performance of DeLLMa against baseline approaches, while controlling for the LLM backbone. In § 4.3, we also evaluate OpenAI o1 [] to study the difference between general inference-time reasoning and our specialized approach for decision making under uncertainty. This experiment is not included in our main result as we are unable to control the LLM backbone. We defer more involved decision-making under uncertainty problems, such as constructing a weighted combination of actions (i.e., a portfolio), to future works.

Figure: Figure 2: Results on the Agriculture environment. Left: DeLLMa variants outperform baseline methods for varying numbers of actions. Right: We see that DeLLMa yields a consistent improvement in decision-making accuracy across three families of leading LLMs.
Refer to caption: x2.png

### 4.1 Agriculture

#### Data Acquisition.

We collect bi-annual reports published by the United States Department of Agriculture (USDA) that provide analysis of supply-and-demand conditions in the U.S. fruit markets(^2^22[www.ers.usda.gov/publications/pub-details/?pubid=107539](www.ers.usda.gov/publications/pub-details/?pubid=107539)). To emulate real-life farming timelines, we use the report published in September 2021 as context for planning the forthcoming agricultural year. We additionally supplement these natural language contexts with USDA-issued price and yield statistics in California(^3^33[www.nass.usda.gov/Quick_Stats](www.nass.usda.gov/Quick_Stats)).

We define the utility of planting a fruit as its price $\times$ yield reported in the forthcoming year. We identify 7 fruits as our action set $\mathcal{A}$—apple, avocado, grape, grapefruit, lemon, peach, and pear—that are both studied in the September 2021 report, and endowed with these statistics in 2021 and 2022. We create decision making problems by enumerating all possible combinations of availble fruits, resulting in 120 decision problem instances. For each decision-making instance, we use related sections of the USDA report and current-year price and yield statistics as context. See § C.1 and Figure 8 for additional details on preprocessing, and Figure 12 for DeLLMa prompts.

#### Main Result.

From Figure 2, we observe that all DeLLMa strategies consistently outperform baseline comparison methods, especially for larger action sets.
We plot decision accuracy here, but include equivalent plots showing the utility of each strategy in Appendix B.
Among DeLLMa variants, the performance of Pairs and Top1 are consistent across the board; both are significantly better than Naive. This observation shows the benefits of the algorithms outlined in §3.3. We give more detailed analyses in the ablation study below.
We also show a comparison of DeLLMa deployed on multiple leading LLMs (GPT-4 [], Claude-3 [], Gemini 1.5 []), with consistent performance improvements across multiple model families.
We refer readers to Appendix C and supplementary material for details on data, prompts, and responses.

**Table 1: An evaluation of state forecasting, showing ECE and NLL. Lower is better.**
|  | ECE $\downarrow$ | NLL $\downarrow$ |
| --- | --- | --- |
| GPT-4 | 0.062 | 1.11 |
| Claude 3 | 0.142 | 1.20 |
| Gemini 1.5 | 0.064 | 1.09 |

#### How Good Are State Forecasts?

We first evaluate the quality of forecast distributions.
Towards this, we manually annotate a set of ground truth values for states that the LLMs forecast within DeLLMa, and then compare this with the forecast state distribution. We compare the average calibration (ECE), and the negative log-likelihood (NLL), two popular metrics for assessing the quality of a probabilistic forecast.
As shown in Table 1, all models achieve reasonable calibration performances, which correlate with the overall DeLLMa prediction accuracy. We conduct additional ablation studies to assess each DeLLMa module contributions in § 4.3.

Figure: Figure 3: Left: Study on sample size and overlap percentage used by DeLLMa. Scaling the compute at test time produces better average accuracy. When ablating overlap percentage, we fix sample size at 16; when ablating sample size, we fix overlap percentage at 25%. Right: Illustration of the DeLLMa decision tree for the Agriculture dataset, showing two of the actions, and two of the sampled states per action. Each weight $w$ denotes the posterior probability $\pi(\theta_{i}^{(j)}\mid\mathcal{C})$.
Refer to caption: x4.png

#### Failure Modes of Baseline Methods.

One surprising observation is that, in the case of larger action sets, our baseline methods often underperform random guessing. A failure mode is that these methods elicit decisions that echo the sentiments presented in context, and they lack the ability to reason what-if scenarios that lead to utility changes. Our experiments on SC and CoT indicate that neither sampling reasoning paths, prompting the model to imagine the alternatives, nor augmenting an LLM with a posterior belief substantially enhance the model’s ability to reason with uncertainty. By conditioning on sampled states, DeLLMa can avoid this pitfall while leveraging in-context learning to decide the preferred state-action pair. See § C.4 and Figures 13, 17 and 18 for discussions on failure cases of baseline methods that are addressed by DeLLMa.

In addition to performance improvements, our structured approach is endowed with human auditability. In Figure 3 (right), we show an abbreviated decision network, with actions, states, sampled latent factors, and derived utilities constructed from outputs of a DeLLMa agent. This modular approach to decision making can facilitate transparency and trust of LLMs in high-stake scenarios.

### 4.2 Stocks

Making decisions involving financial investing requires handling a variety of uncertainties; however, it is fundamentally different from the agriculture decision problems.
Most evidently is the difference in input format—contexts for agriculture rely more on textual information (summarizations from USDA reports) whereas contexts for stocks involve tabular data (historical prices per share). Stocks decision problems are well suited to test LLMs capability in reasoning on dynamic tabular data.

#### Data Acquisition.

Similar to the setup in §4.1, the action space $\mathcal{A}$ consists of individual stocks and we curate a total of 120 decision problem instances of varying sizes.
In our experiments, we choose popular stocks whose symbols are AMD, DIS, GME, GOOGL, META, NVDA and SPY. Unlike agriculture data where the context $\mathcal{C}$ are collected through USDA reports, we collect historical stock prices as the context for this problem. As illustrated in Figure 1, each stock is presented with 24 monthly price in history. In preventing possible data leakage and promoting LLMs to use their common-sense knowledge in making decisions, when using gpt4-1106-preview as the LLM checkpoint, historical price between December 2021 to November 2023 are provided as the context $\mathcal{C}$. These historical monthly prices are collected via Yahoo Finance(^4^44[finance.yahoo.com](finance.yahoo.com)) manually by the authors.

The goal of the LLM agent is to choose which stock to invest on December 1st, 2023, and sell on the last trading day of that month (December 29, 2023) so that the return is maximized. Detailed prompts are presented in § D.1. We note that we only consider the simplistic setting—choosing one stock from a set of options $\{a_{1},\cdots,a_{n}\}$.

Figure: Figure 4: Results on the Stocks environment. Left: We see that on average, DeLLMa-Top1 outperforms all baselines. Right: Illustration of the DeLLMa decision tree for the Stocks dataset, showing two of the actions and two of the sampled states per action.
Refer to caption: x6.png

#### Main Results.

Similarly, we compare three variants of DeLLMa with the three popular baselines. Shown in Figure 4, most of the observations are consistent with those in the agriculture experiments—DeLLMa’s variants outperform the baseline candidates. Unlike the agriculture setting, here DeLLMa-Naive only slightly improves over the baselines. We hypothesis this is due to the high volatility of stocks data, where inefficient sample size without variance reduction produces highly volatile predictors as well. This also validates the need for the design of the DeLLMa-Pairs and DeLLMa-Top1 methods.

Additionally, DeLLMa-Top1 performs better than DeLLMa-Pairs in stocks data. This is surprising at first glance since DeLLMa-Pairs have more observations in terms of pairwise preferences and ranking of sampled action-state pairs. However, upon further consideration, we hypothesize potential explanations: LLM hallucination is still an unsolved problem. For high-volatity data like stocks, LLMs can easily hallucinate with internal rankings if asked to perform difficult tasks (such as to rank a batch of options where a ground-truth ranking may not exist). On the other hand, the model may still have high confidence about its prediction of the top choice in the batch. In such scenarios, using the hallucinated rankings may only provide extra noise that hinders the model performance.

### 4.3 Ablation Studies

#### Scaling Laws for Inference-Time Compute.

Referring back to Figure 2, a potential explanation for the performance difference between DeLLMa-Naive and Pairs/Top1 is the discrepancy in sample size: for Naive, we allocate a fixed sample size (50) for all decision-making instances, and we allocate a fixed per action sample size (64) for Pairs and Top1. We eliminate this confounder in our ablation study, by noting that with per action sample size 8 (middle subfigure in Figure 3), DeLLMa-Pairs/Top1 achieve comparable performance to Naive, despite only receiving 16-48 samples in total for each problem instance.

In Figure 3 (left), we observe linear performance trends when scaling up overlap percentage and sample size. Intuitively, both higher overlap percentages (i.e., more exposure between minibatches) and larger sample size lead to construction of finer-grained pairwise comparisons and thus high quality approximate utilities. Furthermore, DeLLMa-Pairs consistently outperforms Top1, implying that a nontrivial portion of the pairwise comparisons are meaningful. However, we note that the number of required API queries scales linearly with both parameters, and users are advised to choose these parameters that balance performance and cost. We report statistics for API queries and prompt lengths for our methods in Table 5 of Appendix B.

#### State Forecasting Ablation.

We now assess the quality of the state forecasting procedure by evaluating DeLLMa in the presence of low-quality or extraneous state information. On the agriculture dataset, we report in § 4.3 the ablation results from 3 variants for the propoal distribution $\pi^{\text{LLM}}$: uniform, underspecified, and overspecified. We fix a per action sample size of 16 and an overlap percentage of 25%, and refer readers to Appendix E for additional details.

We observe that the performance of the modified forecasting methods in GPT-4 and Gemini 1.5 are similar to full DeLLMa performance, and are better than our baseline methods. With these models, DeLLMa appears to be robust against misspecified, insufficiently representative, and extraneous forecast states. However, with Claude 3, ablation performance is more akin to zero-shot baselines. This discrepancy may be due to Claude’s weaker performance in DeLLMa (c.f. Figure 2). Overall, our state forecasting procedure appears to be beneficial for DeLLMa performance.

#### Evaluation Against OpenAI o1.

In § 4, we have compared different prompting approaches while fixing the LLM backbone. We now evaluate OpenAI o1—a class of more advanced models adept in inference-time reasoning—against DeLLMa. § 4.3 shows that our best performances (attained with a per action sample size of 64 and overlap percentage of 25%) can outperform o1 with the zero-shot prompt (c.f. Figures 9 and 22) by a wide margin, while attaining similar costs (DeLLMa: $0.09 to $0.37 vs. o1: $0.21 per instance)(^5^55DeLLMa costs can be further reduced by 50% with the OpenAI batch API.). This study indicates the benefits of specialized inference-time reasoning for decision making under uncertainty.

#### Human Evaluation for Preference Ranking.

Lastly, we perform a study to evaluate the quality of utility elicitation in DeLLMa. We curate a large set of state-action samples;
human annotators (the paper authors) are then shown pairs of these state-action samples and asked to annotate a preference, based on the decision prompt $\mathcal{P}$.
We then compute the agreement rate between the pairwise annotations given by DeLLMa
and those given by the annotators.
Although this step is noisy, we see a strong agreement between LLM and human annotations, as shown in § 4.3, across multiple LLMs. We also find that our inter-annotator agreement rate is 77.5%, which is on par with the human-LLM agreements. Additional details of our annotation procedure are reported in Appendix F.

Figure: Table 2: Performance comparison across variations of our state forecasting procedure.

## 5 Conclusion

We propose DeLLMa, a framework designed to harness LLMs for decision making under uncertainty in high-stakes settings.
This is a structured approach in which we provide a process for inference-time reasoning in LLMs that follows the principles of classical decision theory.
We then develop a feasible implementation with LLMs. Through experiments on real datasets, we highlight a systematic failure of popular prompting strategies when applied to this type of decision making task, and demonstrate the benefits of our approach to address these issues. The modularity of our framework avails many possibilities, most notably auditability and using LLMs for a broader spectrum of probablistic reasoning tasks.

## Appendix

## Appendix A Utility Elicitation Details

In Figure 5, we show an illustration of the overlapped batching and variance reduction strategies that DeLLMa uses in its utility elicitation procedure (described in detail in Section 3.3).

Figure: Figure 5: Schematic diagram of overlapped batching and variance reduction.
Refer to caption: x8.png

## Appendix B Additional Results

In Figures 6 and 7 we show the normalized utility (i.e., ground-truth utility of the chosen action, normalized by the maximum ground-truth utility) of each method.
These results can be contrasted against the accuracy of each method’s prediction of the optimal action in Figures 2 and 4.

Figure: Figure 6: Normalized utility for the Agriculture dataset. Across all action sizes, DeLLMa-Pairs/Top1 outperforms other methods, including DeLLMa-Naive. Similar to accuracy, DeLLMa-Pairs slightly outperforms Top1, which is consistent with our hypothesis that the complete ranking generated from GPT-4 is meaningful.
Refer to caption: x9.png

Figure: Figure 7: Normalized utility for the Stock dataset. DeLLMa-Top1 is competitive againt Pairs, suggesting that financial decision making may be a challenging scenario for GPT-4 to elicit preference unequivocally.
Refer to caption: x10.png

Aside from performance improvements, we also compare the cost measured in terms of prompt length and API calls in Table 5. For the Agriculture dataset, DeLLMa-Naive is comparable to SC in terms of prompt length and significantly outperform the latter, while only requiring 20% of API calls. DeLLMa-Pairs/Top1 can push the performance further, but incurs a higher cost, especially for large sample sizes.

**Table 5: Number of GPT-4 API calls and word counts per decision-making instance (with action set size 4 for the Agriculture dataset) across all methods discussed in § 4.1. For DeLLMa-Naive, we set the total sample size to 50. For DeLLMa-Pairs, we fix the overlap percentage to 25% and vary the per action sample size from 16 to 64. DeLLMa-Top1 has the same statistics as DeLLMa-Pairs since they only differ in post processing.**
|  | Zero-Shot | CoT | SC | D-Naive | D-Pairs (16) | D-Pairs (64) |
| --- | --- | --- | --- | --- | --- | --- |
| API Calls | 1 | 3 | 5 | 1 | 3 | 10 |
| Token Counts | 495 | 1946 | 2477 | 2629 | 5181 | 20639 |

## Appendix C Additional Details for Agriculture

### C.1 Dataset Curation

To reduce context length, we first extract executive and per-fruit summaries of the report, reducing the context length from 8,721 words to around $<700$ words. These summaries are constructed on a per-model basis: each LLM (GPT-4, Claude-3, and Gemini 1.5) is prompted to generate its own summaries. We proof-read these summaries to ensure that they do not contain any factual errors. For each decision making instance, we use the executive summary, the summaries relevant to the fruits in consideration, and their current-year price and yield statistics as context. We provide these summaries in our supplementary material, and refer readers to the prompt for summarizing the report in § C.2, and report concrete summaries and statistics in supplementary materials.

### C.2 Prompt Used by Agriculture

#### Summary Prompt

In Figure 8 we present the prompt used for summarizing the context in our Agriculture experiments. This prompt is shared across all models tested.

#### Zero-Shot Prompt

In Figure 9 we present an example prompt for zero-shot experiments consisting of $\mathcal{P}=(\mathcal{G},\mathcal{C},\mathcal{A})$, with GPT-4 summaries generated from Figure 8 as context. We fix the prompt format for all LLMs and select the corresponding summaries for each LLM.

#### Self-Consistency Prompt

We adopt the same zero-shot prompt as the one shown in Figure 9, but take a majority vote from $K$ reasoning paths as the final prediction.
Our preliminary study finds that LLMs tend to be very confident in their decisions, and even with much higher temperature than 0.5, often all $K$ independent runs yield consistent decisions. We thus set $K=5$ to balance cost and performance.

#### Chain-of-Thought Prompt

We design a multi-prompt CoT procedure that closely emulates our DeLLMa agent, consisting of three steps: (1) ask for unknown factors that impact the decision; (2) given these, ask for their possibility of occurence; (3) then ask for a final decision. This procedure differs from DeLLMa agents as it does not use an external program for utility elicitation and expected utility maximization, but delegates each step of the process to an LLM. Example prompt can be found in Figure 10.

#### DeLLMa Prompt

Similar to the CoT Prompt, DeLLMa is a multi-prompt procedure (1) ask for unknown factors that impact the decision; (2) given these, ask for their possibility of occurence (i.e. a belief distribution). Instead of directly deciding on an action, DeLLMa samples state-action pairs (via an external program) from this belief distribution, and leverage an LLM to elicit a utility function $U:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ from pairwise comparisons. From here, we use another external program to search for an action that maximizes the expected utility.

In Figs. 14 and 11 we present the prompt for state enumeration and state forecasting (i.e. step (1) and (2)) for the Agriculture dataset. Our implementation combines these two modules, but in the future they should be separate as our framework continues to progress. For example, state enumeration may leverage retrieval-augmented generation [] for generating high-quality unknown factors, and state forecasting may resort to tool usage to delibrate calibrated belief distributions from quanitative analysis.

In Figure 12, we provide a DeLLMa prompt to generate ranked comparisons from sampled state-action pairs. These state action pairs are sampled from the belief distribution via an external program, written in Python. We provide all our implementations in the supplementary material.

### C.3 Agriculture Dataset Responses

#### Summaries

See Figure 14 for formatted GPT-4 summaries from the USDA agricultural report and statistics. Claude-3 and Gemini summaries follow the exact same format, but differ in LLM generated contents.

#### Zero-Shot Responses

See Figures 13, 17 and 18 for sample GPT-4 responses to zero-shot prompts.

#### Self-Consistency Responses

SC responses follow the exact same format as those in Figures 13, 17 and 18.

#### Chain-of-Thought Responses

See Figure 15 for a curtailed version of GPT-4 CoT response.

#### DeLLMa Responses

See Figure 16 for a curtailed version of GPT-4 DeLLMa response.

### C.4 Failure Cases on the Agriculture Dataset

In § 4.1, we postulate that a failure mode of our baseline approaches is that they lack the ability to perform probablistic reasoning, but are rather following the sentiment presented in context. Here, we qualitatively test this hypothesis, by showcasing prompts and responses from our Zero-Shot experiments. We present instances in Figures 13, 17 and 18 that are not able to make optimal prediction with baseline methods, but are solved by DeLLMa agents.

Within each figure, we first present $\mathcal{P}$, followed by the (incorrect) decision and explanation generated from the Zero-Shot baseline, and then present the top-ranked state action pair generated by DeLLMa along with its explanation.

In the case of deciding between apple and grapefruit in Figure 13, GPT-4 presumes that a high price for grapefruit is sustainable due to the presence of extreme weather conditions in the previous year. With CoT, the model reasons with some counterfactual scenarios, such as more suitable weather for the forthcoming year, but not in a systematic way. With DeLLMa, we provide these a list of counterfactual scenarios as context, and leverage the model’s ability for situational thinking and ranking to elicit an informative preference. We make similar observations for Figures 17 and 18.

## Appendix D Additional Details for Stocks

### D.1 Prompt Used by Stocks

#### Zero-Shot Prompt

Similar to the agriculture setup, we first present a sample zero-shot prompt in Figure 19. The zero-shot prompt consists of three main parts (see Figure 1 for visual illustrations): (1) an enumeration of action spaces (here AMD or GME), (2) a provided context (here, historical stock prices between December 2021 to November 2023), and (3) the goal of the user (choose only one stock to maximize their profit via investing in stocks).

#### Self-Consistency Prompt

SC prompts are identical to zero-shot prompts.

#### Chain-of-Thought Prompt

Similarly, we examplify a prompt for CoT. Notably, we break the chain into three parts in Figure 20: (1) Enumerating unknown factors, (2) Enumerating $\pi^{\text{LLM}}(\cdot\mid\mathcal{C})$, and (3) making the final decision given the previous responses and context.

#### DeLLMa Prompt

Finally, we showcase the DeLLMa ranking prompts where the LLM is asked to provide comprehensive rankings of given state-action pairs.

### D.2 Stock Dataset Responses

#### Zero-Shot Responses

See Figure 22 for sample GPT-4 responses to zero-shot prompts.

#### Self-Consistency Responses

SC responses follow the exact same format as those in Figure 22.

#### Chain-of-Thought Responses

See Figure 23 for a curtailed version of GPT-4 CoT response.

#### DeLLMa Responses

See Figure 24 for a curtailed version of GPT-4 DeLLMa response.

## Appendix E Additional Details for Ablation Studies

- •
Uniform: we maintain the same set of states, but replace the forecast distribution
$\pi^{\text{LLM}}$ with a uniform distribution.
- •
Underspecified: we reduce the diversity of our set of states by removing a latent factor $f_{k}$, chosen at random, and then perform state forecasting as usual.
- •
Overspecified: in addition to the original set of latent factors, we add a small set of additional factors, which are intentionally unrepresentative or redundant. This represents a case where the latent factors contain extraneous information, which may confuse the LLM backbones.

## Appendix F Details on Annotation Pipeline

For each pair of state-action values, we ask human evaluators (paper authors) to annotate which state-action tuple is more preferred to the other, and compare our annotations against preferences elicited by LLMs. To reduce bias in this procedure, we present these pairs in shuffled orders, i.e. it is not always true that $(s,a)_{1}\succ(s,a)_{2}$ if $\{(s,a)_{1},(s,a)_{2}\}$ is presented to the annotators. For each pair, we ask the annotators to either label them as 1 (state-action tuple 1 is preferred), 2 (state-action tuple 2 is preferred), or 0 (uncertain). We then evaluate annotator-LLM agreement after evaluation is concluded.

For inter-annotator agreement, we present to two annotators a fixed list of state-action pairs (also in shuffled order), while holding out LLM preferences. We then evaluate annotator agreement with exact match.

## Appendix G License for Existing Assets

#### Large Language Models

DeLLMa is built on top of frontier class LLMs such as GPT-4 [](^6^66[https://chatgpt.com/c/fe00ee45-e34c-4b64-812d-295abaedd20f](https://chatgpt.com/c/fe00ee45-e34c-4b64-812d-295abaedd20f)), Claude-3 [](^7^77[https://www-cdn.anthropic.com/files/4zrzovbb/website/e2d538c84610b7cc8cb1c640767fa4ba73f30190.pdf](https://www-cdn.anthropic.com/files/4zrzovbb/website/e2d538c84610b7cc8cb1c640767fa4ba73f30190.pdf)), and Gemini 1.5 [](^8^88[https://ai.google.dev/gemini-api/terms](https://ai.google.dev/gemini-api/terms)). We are not aware of their licenses, but ascertain that we abide to their respective terms of service (ToS) for the usage of these models.

#### Agriculture Dataset

The Agriculture dataset is curated from statistics and reports published by the U.S. Department of Agriculture. These assets are freely accessible under the Freedom of Information Act. We ascertain that we abide to the USDA ToS(^9^99[https://www.usda.gov/policies-and-links](https://www.usda.gov/policies-and-links)).

#### Stock Dataset

The Stock dataset is curated with the Yahoo Finance API(^10^1010[finance.yahoo.com](finance.yahoo.com)). We ascertain that we abide to the Yahoo Developer API ToS(^11^1111[https://legal.yahoo.com/us/en/yahoo/terms/product-atos/apiforydn/index.html](https://legal.yahoo.com/us/en/yahoo/terms/product-atos/apiforydn/index.html)).

## Appendix H Societal Impact and Limitations

Rational decision making under uncertainty has been extensively studied across many disciplines, but remains elusive even for humans. Our work bears significant societal consequences as we live in a world where humans increasingly rely on intelligent assistants for diverse problems. Unfortunately, existing foundation models and LLMs are not capable of acting optimally under uncertainties yet, which may induce harm if the general public delegate these models for decision making. These risks are exacerbated when intelligent assistants operate as a black box without adequate transparency. DeLLMa serves as a first step towards an approach that builds trust for humans to rely on these systems to make important decisions under uncertainty.

While our approach sensibly improves the performance of LLMs for decision making under uncertainty, deploying a prototype system in the wild without additional guardrail could incur significant financial losses, in case of a suboptimal decision. We expect users to conduct extensive field tests for the feasibility of such systems, or leverage a hybrid approach that integrates analyst judgements for optimal decision making.

Finally, a limitation of our approach stems from constrained state and action spaces. Our framework currently operates on a small number of discrete state and action spaces, due to the context window size of state-of-the-art LLMs. This drawback may potentially limit the applicability of our system in more complex use cases, or scenarios that require sequential decision making.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See the points above as "(question)". Happy to read your take.
soundness: 3
strengths: - Exposition and writing  in general is in  good condition. (There are small rooms for improvement in organisation and clarification.)

- The whole framework is written with careful formal details, thanks to taking decision-theoretic frame, which is a great idea to start with.

-  The results on improvements are powerful; also has high impact potential  due to popularity of prompt engineering in LLMs. The idea is novel and justifiable (with some inherent difficulties and pitfalls ).

- The depth of the analysis and various ablation studies is a strength. 

- The framework also provides (to some extent) an explainable medium which is a strength.
summary: The authors  propose  framework to work with LLMs for decision making under uncertainty; in doing so, they take classical utility/decision theoretic framework as a design guideline: (1) they enumerate first all the states. (2) estimate the probability of those states (3) elicit the utility function from the user, and (4) choose the one that maximises expected utility.   They show their results working with various LLM and in various settings; different preference ranking strategies (pair by pair vs. top1) against benchmarks of zero shot and CoT in two real world  decision problems (one in agriculture the other in stocks investment) and show that they their framework helps with the results significantly. Also considered are ablation studies (on  things such as overlap percentage (of minitibatches in drawing from state-action samples) and sample size, human agreement etc ). Approach also comes with the decision trees naturally, hence making the whole process more explainable.
weaknesses: - Content-wise: Unfortunately, too many reference to appendix in many crucial points. This is of course due to  tight page limitation, but authors should seriously think about re-arranging the text more complete. For instance explanations for Table 2 , 3 and 4 really not sufficient (page 10 only call their names without telling what they are).  It seems like the paper would serve much better as a journal publication or other venues with more page (e.g., ICML).  Related: Say also that Figure 11  is in appendix actually.  They try to provide too much (for instance human evaluation at best a study on its own, or should be at least extended to present something substantial)

- On the technical side there are strong assumptions and unclarities:  the assumption of independence between k-latent factors is very strong, and  actually hard to maintain,  also due to natural language part of things.  As a result, uncertainty in the estimation step is taken very crisp, it is pretty difficult to ensure whether these scores are reflecting reality. There is a section called: "How good are state forecasts" (which is a good to have section), however I am not sure it reaches its goal:  This is inherently difficult. They show table 1: claiming two metrics they showing "good results" is hard to justify since we don't know reference to what (maybe its my misunderstanding.) Why are these reasonable? Also the same section, manually annotating a set of ground truth values, how to get it, etc, not very clear to me. 

- Less problematic is the issue of  the number of alternatives are known (coming from the context or imposed to LLM) is a strong working assumption, and undermines  real world settings applicability. 

-Another major issue is the part with Table 4 (human evaluation): this is a nice to have I agree, however, if you go for it then it should  be done in a state that does not weaken the paper. Currently, as explain this is just done by authors, of which we don't know how many, say 4 or 5 on average. Is this result than solid? It is problematic for two reasons: 1)  It should be done not by authors  but by a handful of people (who has nothing to do with paper acceptance) 2) not clear how does it compare to sole human comparison 3) amount of people matters: what if it was a single author paper (I assume it is not due to their expression). What is the standard deviation, significance level  or other statistics on these values? Without all these things , it would be pretty difficult to publish this in even a low-rank psychology journal. And yes, this venue is on rather AI but we still should maintain some sensible quality on experiments if we are to show that. Currently, it just serves only as good as an anecdote.  (Question)


- It is shown to have much better performance against OpenAI's o1 model, but which DeLLma is used is not clear. (Question) 

-  Organisation wise  the DELLMa main picture (there is no name unfortunately for the figure btw to refer to it, but perhaps understandable) comes before the explanations in 3.1 and other sections and create questions immediately. This is a very minor issue. 

-Unclarities in  Page 9, Main results, second paragraph is not satisfying:  The vagueness of explanation is not a weakness per se since authors are careful to call these as hypotheses. Yet, the difference on phenomenon is very small, and now we enforce ourselves to explain the phenomenon, and likely we hallucinate ourselves.  The result can be much more simple: the amount of data considered, and the processing capacity of the model "can be" pretty much be  a reason that top 1, simplifies it much better compared to other.   Also see that when number of alternatives is 5 it already performs better, which might be due to a particular alternative. Relatedly the wayt the average is taken matters: 1) some difficulties might be due to assessing particular actions (are these scores are evaluated permuting through all the alternatives? (E.g., which 2 actions matters)?) 2) averaging across the accuracy score of  different size of scores matters. There I would go for higher weight to the higher number of actions.  These subtleties are unfortunately not clear, (question)

-Mini typo: availble fruits.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - How will DeLLMa perform on other tasks related to chain-of-thought or tree-of-thought methods discussed in related work?
- If scalability is not a significant issue in certain cases, will DeLLMa outperform other methods?
- In the experiments, one task involves investing on December 1, 2023, and selling on the last trading day of that month (December 29, 2023) to maximize returns. Why were these specific dates chosen? Why not other dates?
soundness: 2
strengths: - Clearly explain the main concept of DeLLMa as illustrated in Figure 1.
- Capture LLM in decision making with a triplet P=(G, A, C) and four steps, where G is the user goal inferred from the description, A is a list of actions, and C is the contextual information. 
- Include two ranking types: pairwise and top-1.
- Provide clear prompts and code in the appendix.
summary: This paper proposes DeLLMa (decision-making LLM assistant) to maximise the utility in decision problems, i.e., agriculture planning and finance. DeLLMa consists three steps, 1) based on in-context information, infer and forecast related unknown variables, 2) elicit the utility function which aligns with the user's goals, 3) use the elicited utility function to make the best decision which maximise the expected utility.   Authors claim that DeLLMa is motivated from chain of thoughts and tree of thoughts, but is faster at the inference time especially in the scalability.
weaknesses: - Inferring the user’s goal from the context is done by the LLM, which requires the user to have a clear goal in mind. For example, if the farmer has not decided to plant only one type of fruit, DeLLMa may not know how to handle an implicit multiple-fruit situation.
- The applicable scope of DeLLMa is limited. The experiments include only agriculture and stocks, which may mislead readers into thinking DeLLMa generalises across all decision-making tasks.
- There is little information about human agreement; the paper states that the authors served as human annotators, but details are missing.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How can DeLLMa be adapted to handle continuous state and action spaces? 
2. The state forecasting method assumes independence bw latent factors. How does this impact total performance?
3. Is DeLLMa able to handle cases where the user task cannot be distilled into a hard utility function?
soundness: 4
strengths: Originality - the paper is quite original, and the proposed pipeline is creative. I have not seen this multistep approach used with LLMs anywhere before.

Quality - I found the method to be quite well motivated and well presented. The paper is reproducible, and the experiments seem cohesive and well-motivated.

Clarity - The paper is well articulated and clearly structured.

Significance - The paper is definitely an important step in the right direction. Uncertainity quantification in ML has been an active area of research and I am curious to see how this is furthered with LLMs. However I feel that the method has limited applicability to real world scenarios.
summary: The paper introduces DeLLMa for better decision making by LLMs under uncertainty. It uses a multistep process to achieve this, based on decision theory - state enumeration, forecasting, then utility elicitation and expectation maximization. Then the paper applies this to agro and fin scenarios - showing good improvement over baselines (zero shot, CoT etc). Finally the paper shows that this method generalizes across LLMs and is human auditable.
weaknesses: 1. The context window of LLMs clearly limits the state and action spaces, which makes this method of limited real world applicability.
2. It appears that the method is very sensitive to the first state forecasting step. While the method is robust in general, I would like to understand if this is indeed a big bottleneck.
3. The utility elicitaiton method proposed (pairwise ranking) seems a bit too simple for real world scenarios. I would like to know if other methods were tried for this.

Meta-review: The reviewers were generally positive about the paper, specifically with the framework and the rigor in the writing. There are suggestions toc make the paper self-contained and not refer so much to the appendix, increase the details on the experiments and justification of the inherent assumptions in the framework.  We hope the authors consider the feedback in improving the paper for the final version.

Decision: Accept (Spotlight)