Venue: NAACL 2025

# DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learning

Xinyu Tang, Xiaolei Wang, Wayne Xin Zhao, Ji-Rong Wen Gaoling School of Artificial Intelligence, Renmin University of China

txy20010310@163.com, wxl1999@FOXmail.com, batmanfly@gmail.com

 Equal contribution.Corresponding author.

###### Abstract

Zero-shot in-context learning (ZS-ICL) aims to conduct in-context learning (ICL) without using human-annotated demonstrations. Existing ZS-ICL methods either use large language models (LLMs) to generate (input, label) pairs as _pseudo-demonstrations_ or leverage historical pseudo-demonstrations to help solve the current problem. They assume that all problems are from the same task and traverse them in a random order. However, in real-world scenarios, problems usually come from diverse tasks, and only a few belong to the same task. The random traversing order may generate _unreliable_ pseudo-demonstrations and lead to _error accumulation_. To address this problem, we reformulate ZS-**ICL** as a _planning_ problem and propose a **D**emonstration-**A**W**are MoNe Carlo Tree Search (MCTS) approach (**DAWN-ICL**), which leverages MCTS to strategically plan the problem-solving trajectories for ZS-ICL. In addition, to achieve effective and efficient \(Q\) value estimation, we propose a demonstration-aware \(Q\)-value function and use it to enhance the _selection_ phase and accelerate the _expansion_ and _simulation_ phases in MCTS. Extensive experiments demonstrate the effectiveness and efficiency of **DAWN-ICL** on in-domain and cross-domain scenarios, and it even outperforms ICL using human-annotated demonstrations. The code is available at [https://github.com/txy77/MCTS42STCL](https://github.com/txy77/MCTS42STCL).

## 1 Introduction

In-context learning (ICL) Brown et al. (2020); Dong et al. (2023); Wang et al. (2024) represents a significant advancement in the capabilities of large language models (LLMs). It allows LLMs to rapidly adapt to new tasks without updating the parameters by adding only a few examples as demonstrations to the input.

Although no training is required, most ICL work assumes access to large-scale external resources (_e.g._, training dataset Liu et al. (2022); Ye et al. (2023) or relevant corpus Tanwar et al. (2023); Chatterjee et al. (2024)) for demonstration selection, which is usually not available in real-world scenarios. To eliminate this dependency, _zero-shot context learning_ (ZS-ICL) Lyu et al. (2023); Chen et al. (2023) is proposed, which uses LLMs to generate (input, label) pairs as _pseudo-demonstrations_ for ICL. However, since LLMs are limited in data

Figure 1: Comparison of our method with previous methods. Although both predictions at \(i\)-th step are correct, in previous work, the example is randomly selected and helpless. In contrast, in our method, the example is selected with planning and helpful.



synthesis (Seddik et al., 2024; Longpre et al., 2024; Dohmatob et al., 2024), the performance of ZS-ICL usually falls behind ICL with human-annotated demonstrations. To remedy this deficiency, recent work (Su et al., 2024) employs previously predicted examples as the source of demonstrations, eliminating the need for input synthesis and reusing the predicted labels. This method assumes that test examples are from the same task and traverse them in a random order. However, in real-world scenarios, examples usually come from diverse tasks, and only a few belong to the same task. The random traversing order may cause LLMs to generate _unreliable_ pseudo-demonstrations and lead to _error accumulation_, as illustrated in Figure 1.

To address the above problems, we aim to optimize the traversing order of examples in ZS-ICL and formulate it as a _planning_ problem. To search the traversing order, we take inspiration from Monte Carlo Tree Search (MCTS) (Coulom, 2006), which can conduct a strategic tree search and strike a balance between exploration and exploitation. In the algorithm, MCTS maintains a state-action value function \(Q(s,a)\) to estimate the expected future reward of taking action \(a\) in state \(s\), which is updated by the simulation and back-propagation step in each iteration. However, for ZS-ICL, such an updating method is too costly to achieve accurate estimation since the state space is very large (\(n!\) for \(n\) examples), and each state requires the LLM to perform one inference for reward calculation.

To this end, in this paper, we propose a novel **D**emonstration-**AW**are MoNte Carlo Tree Search for **ZS-ICL**, namely **DAWN-ICL**. Our core idea is to leverage MCTS for planning the problem-solving trajectories in ZS-ICL. To achieve effective and efficient \(Q\) value estimation in MCTS, we propose to integrate the information of the pseudo-demonstration set into the \(Q\)-value function. With this demonstration-aware \(Q\)-value function, we can enhance the _selection_ phase and accelerate the _expansion_ and _simulation_ phases of MCTS for more effective and efficient search. Furthermore, we design a calibration-enhanced aggregation method to derive the final prediction from MCTS, which aggregates results from multiple iterations and debiases the prediction with pre-trained priors. To validate the effectiveness of our approach, we conduct experiments on the in-domain and cross-domain scenarios of BBH and MMLU across various LLMs. The experimental results show that **DAWN-ICL** consistently surpasses the best ZS-ICL baseline method and even outperforms ICL using human-annotated demonstrations.

Our contributions can be summarized as follows:

\(\bullet\) To the best of our knowledge, we are the first to formalize ZS-ICL as a planning problem, which is closer to real-world scenarios.

\(\bullet\) We propose a demonstration-aware MCTS for ZS-ICL to achieve a more effective and efficient search for the problem-solving trajectories.

\(\bullet\) Extensive experiments demonstrate the effectiveness of our approach on in-domain and cross-domain scenarios, and it even outperforms ICL using human-annotated demonstrations.

## 2 Related Work

Zero-shot In-context Learning.Zero-shot in-context learning (ZS-ICL) (Lyu et al., 2023; Chen et al., 2023; Su et al., 2024) aims to conduct in-context learning (ICL) using model-generated pseudo-demonstrations. Most ZS-ICL work separately generates pseudo-demonstrations for each example (Lyu et al., 2023; Chen et al., 2023). There is also some work that employs previously predicted examples as demonstrations and stores them in memory for future usage (Su et al., 2024). However, these methods traverse examples in a random order, which may lead to error accumulation. In this paper, we reformulate ZS-ICL as a planning problem and search for the optimal problem-solving order.

Enhancing LLMs with Planning.Recent advancements in enhancing LLMs through planning have shown promising results (Yao et al., 2023; Hao et al., 2023; Wan et al., 2024; Wang et al., 2024). They often engage in deliberate reasoning processes by utilizing strategic planning algorithms like MCTS (Coulomb, 2006) to explore intermediate steps. In this work, we aim to better utilize the in-context learning ability of LLMs by performing a novel demonstration-aware MCTS for ZS-ICL over the demonstration space.

## 3 Methodology

In this section, we present our demonstration-aware MCTS for ZS-ICL, namely **DAWN-ICL**. We first give an overview of our approach, then discuss how to integrate the information of demonstrations into the \(Q\)-value function, and finally present the planning approach to ZS-ICL. The overall architecture of **DAWN-ICL** is illustrated in Figure 2.

 



### Overview of Our Approach

**Problem formulation.** Zero-shot in-context learning (ZS-ICL) enables task adaptation of LLMs at test time without using human-labeled examples as the demonstration. Formally, given an LLM \({\mathcal{M}}\) and a set of \(n\) test examples \({\cal E}=\left\{(x_{k},y_{k})\right\}^{n}_{k=1}\), at \(i\)-th step, an example \(x_{i}\) is first selected from \({\cal E}\), and then the pseudo-demonstration \(d_{i}=f_{c}(x_{i},{\cal D}_{i-1})\) for \(x_{i}\) is constructed based on the existing pseudo-demonstration set \({\cal D}_{i-1}\). Based on this, the LLM makes the prediction \(\hat{y}_{i}=f_{p}(x_{i},d_{i};{\mathcal{M}})\) and obtains a new pseudo-demonstration \(\hat{d}_{i}=(x_{i},\hat{y}_{i})\). Then, the pseudo-demonstration set is updated as \({\cal D}_{i}=f_{u}({\cal D}_{i-1},\hat{d}_{i})\). The above process is repeated until all the problems in \({\cal E}\) are solved.

**The general planning framework.** In existing work Su et al. (2024), the test examples are assumed to belong to the same task, and the example to solve at each step (\(x_{i}\)) is usually randomly selected. However, in real-world scenarios, test examples come from diverse tasks, and only a few belong to the same task, limiting the performance. To solve this, one feasible way is to optimize the traversing order of test examples. This is because ICL is sensitive to the selection of demonstrations Liu et al. (2022). Hence, the order of traversing is important for effectively leveraging historical examples as pseudo-demonstrations to help solve the current one. To this end, our idea is to formalize ZS-ICL as a planning problem using a Markov Decision Process (MDP), represented by the tuple \(({\cal S},{\cal A},T,r)\). In this planning framework, we define the state \(s_{i}\in{\cal S}\) as the set of test examples that have been solved at \(i\)-th step, along with the pseudo-demonstration set \({\cal D}_{i}\). The action \(a_{i}\in{\cal A}\) is to select the next problem \(x_{i+1}\) to solve. The transition function \(T\) from the current state \(s_{i}\) to the next state \(s_{i+1}\) first performs the pseudo-demonstration construction function \(f_{c}\), then the prediction function \(f_{p}\) to solve \(x_{i+1}\), and finally the pseudo-demonstration set updating function \(f_{u}\) to obtain the new state \(s_{i+1}\). The reward function \(r_{i}=r(s_{i},a_{i})\) measures the quality of the action \(a_{i}\) applied to the state \(s_{i}\). Since ground-truth labels are not available at test time, we take the confidence of model prediction in \(f_{p}\) as the reward, which has been shown to be aligned with the model performance Xiong et al. (2024); Zhang et al. (2024).

**Monte Carlo Tree Search for zero-shot ICL.** Our approach is inspired by the powerful planning algorithm Monte Carlo Tree Search (MCTS), which can be used to strategically conduct tree search for problem-solving trajectories in ZS-ICL and strike a balance between exploration and exploitation to find high-reward trajectories. To perform an

Figure 2: The overview of **DAWN-ICL**. (a) An illustration of the four phases in MCTS. We select nodes using our proposed DUCT (Eq. 3), perform expansion using our proposed DQ function (Eq. 1), accelerate simulation with an action cache supported by the DQ function, and finally back-propagate the rewards. (b) We improve the \(Q\)-value function with pseudo-demonstration information. We retrieve \(k\) pseudo-demonstrations and add the score of confidence and similarity as the initial value of the \(Q\) function.



effective search, MCTS maintains a state-action value function \(Q:{\cal S}\times{\cal A}\rightarrow\mathbb{R}\), where \(Q(s,a)\) estimates the expected future reward of taking action \(a\) in state \(s\). For ZS-ICL, the quality of the pseudo-demonstration set in the current state is also an important signal for \(Q\) values since ICL is known to be sensitive to demonstrations Yoo et al. (2022); Liu et al. (2022). Taking this into consideration, we design specific metrics to integrate this information and propose a _demonstration-aware_\(Q\)-value function (Section 3.2). Based on this, we conduct MCTS for ZS-ICL (Section 3.3), where the proposed \(Q\)-value function is used to enhance the _selection_ phase and accelerate the _expansion_ and _simulation_ phases. Furthermore, we design a calibration-enhanced aggregation method to derive the final prediction from MCTS, which aggregates results from multiple iterations and debiases the prediction with pre-trained priors. In what follows, we introduce these two parts in detail.

### Demonstration-aware \(Q\)-value Function

The MCTS algorithm maintains a state-action value function \(Q(s,a)\) to estimate the expected reward of taking the action \(a\) in the state \(s\). Originally, this \(Q\)-value function is updated by simulating the future states (_i.e.,_ simulation) and aggregating their rewards (_i.e.,_ back-propagation). However, for ZS-ICL, such an updating method is too costly to achieve accurate estimation since the state space is very large, and the reward calculation of each state requires the LLM to perform one inference.

To perform effective \(Q\) value estimation, we propose to leverage the contextual information of the current state and action to initialize the \(Q\) value. The performance of ICL is known to be highly dependent on the selection of demonstrations. Inspired by this, we propose to initialize the \(Q\) value by evaluating the quality of the pseudo-demonstration set \(D_{i}\) in the current state \(s_{i}\) with respect to the problem \(x_{i+1}\) (_i.e.,_ action \(a_{i}\)). Specifically, we first retrieve \(k_{d}\) demonstrations that are most semantically similar to the problem, and then evaluate their quality by aggregating their confidence and similarity scores. The demonstration-aware \(Q\)-value function DQ can be represented as follows:

\[\text{DQ}(s,a)=Q_{0}(s,a)+w_{Q}\cdot Q(s,a),\ \tag{1}\] \[Q_{0}(s,a)=\frac{1}{k_{d}}\sum_{i=1}^{k_{d}}\left(C(d_{i})+S(d_{ i},x_{i+1})\right), \tag{2}\]

where \(d_{i}\) is the retrieved demonstration, \(C(d_{i})\) is the confidence score of the demonstration from the prediction function \(f_{p}\), \(S(d_{i},x_{i+1})\) is the similarity score between the demonstration and the problem \(x_{i+1}\) chosen by the action measured with the BGE model Xiao et al. (2024), and \(w_{Q}\) is a constant to balance the initial value (\(Q_{0}(s,a)\)) and updated value (\(Q(s,a)\)).

With this demonstration-aware \(Q\)-value function, the estimation of \(Q\) values can be more accurate with a limited computational budget. Based on this, we conduct MCTS for ZS-ICL, where the demonstration-aware \(Q\)-value function is used to enhance the _selection_ step and accelerate the _expansion_ and _simulation_ step. In the next section, we will introduce this in detail.

### Strategic Planning for Zero-shot In-context Learning

In this section, we first introduce the demonstration-aware MCTS to plan problem-solving trajectories for ZS-ICL, then detail the calibration-enhanced aggregation method to derive the final prediction from the searched trajectories.

#### 3.3.1 Demonstration-aware MCTS

The reformulation of ZS-ICL as a planning problem (Section 3.1) enables us to leverage principle planning algorithms, notably the Monte Carlo Tree Search (MCTS). Specifically, MCTS iteratively constructs a tree for search, where each node represents a state and each edge represents an action and the transition from the current state to the next one by applying the action. Each iteration consists of four phases: selection, expansion, simulation, and back-propagation. For ZS-ICL, we randomly select a problem with the empty pseudodemonstration set as the initial state and execute the algorithm until a predefined number of iterations is reached. In each iteration, we use the proposed demonstration-aware \(Q\)-value function to enhance the _selection_ phase and accelerate the _expansion_ and _simulation_ phases. Next, we detail the four phases of MCTS for planning problem-solving trajectories in ZS-ICL. The pseudo-code is presented in Algorithm 1.

**Selection.** The first phase of MCTS is to select the most promising part of the existing tree for expansion. A well-known selection strategy is the Upper Confidence bounds applied to Trees (UCT) algorithm Kocsis and Szepesvari (2006), which can 

effectively balance exploration (less visited times) and exploitation (high values). However, as mentioned in Section3.2, the estimation of values is too costly to be accurate in ZS-ICL. Therefore, we propose to integrate our DQ function (Eq.1) into UCT. This demonstration-aware UCT function DUCT can be represented as follows:

\[\text{DUCT}(s,a)=\text{DQ}(s,a)+w_{a}\sqrt{\frac{\ln N(s)}{N(\text{c}(s,a))}}, \tag{3}\]

where \(N(s)\) is the number of times node \(s\) has been visited, \(\text{c}(s,a)\) is the child node for \(s\) after applying action \(a\), and \(w_{a}\) is a constant to balance exploration (\(U(s,a)\)) and exploitation (\(\text{DQ}(s,a)\)). We start from the root node (_i.e.,_ initial state \(s_{0}\)) and repeatedly select a child node with the maximum DUCT value until reaching a leaf node.

**Expansion.** This phase expands the tree by generating child nodes for the leaf node selected above. Since the action space of a state (_i.e.,_ remaining problems to solve) can be large, we use our proposed DQ function for efficient action selection. Specifically, we first calculate the value of each action using Eq.1, and then choose the top-\(k_{a}\) actions with the highest values for expansion. For each selected action \(a_{i}\), we need to predict the corresponding state \(s_{i+1}\), which is the role of the transition function \(T\) described in Section3.1. In \(T\), the first step is to construct the pseudo-demonstrations for the problem \(x_{i+1}\) selected by the action (_i.e.,_ function \(f_{c}\)). To make the pseudo-demonstrations relevant and diverse, we first retrieve \(k\) most semantically similar ones with the problem \(x_{i+1}\) from the pseudo-demonstration set to increase the relevance, and then randomly select samples with different pseudo-labels from them to enhance the diversity. With the pseudo-demonstrations \(d_{i+1}\), the second step in \(T\) is to predict the label for the problem \(x_{i+1}\) using the LLM (_i.e.,_ function \(f_{p}\)). Here, we use greedy decoding to generate the prediction. Finally, the predicted label \(\hat{y}_{i+1}\) paired with the problem \(x_{i+1}\) is added to the pseudo-demonstration set (_i.e.,_ function \(f_{u}\)), and the new state \(s_{i+1}\) is obtained. For the expanded nodes, we choose the one with the largest reward for the next simulation phase.

**Simulation.** This phase simulates the future trajectories for the node selected from the previous expansion step. The simulation process typically involves a roll-out policy to reach the terminal state and calculate the future rewards. For simplicity, we follow the same procedure as the expansion phase, _i.e.,_ selecting \(k_{a}\) candidate actions with the highest DQ values and picking the one with the largest reward. To further accelerate the simulation process, we propose a cache mechanism based on the DQ function. Specifically, we maintain the maximum DQ value for each action \(a_{i}\) as \(\text{DQ}^{(i)}_{\max}\) and record the corresponding pseudo-demonstration \(\hat{d}_{i}=(x_{i+1},\hat{y}_{i+1})\). If \(\text{DQ}^{(i)}_{\max}\) breaks through the threshold \(\epsilon\), we add the (\(a_{i}\), \(\hat{d}_{i}\)) pair into the cache. In the simulation process, if we take an action that exists in the cache, the pseudo-demonstration is read from the cache, and we skip the transition function \(T\) to directly obtain the new state.

**Back-propagation.** This phase is executed when we reach a terminal node. We back-propagate the rewards along the path from the terminal node to the root node by updating the \(Q\)-value function. Specifically, we update \(Q(s_{i},a_{i})\) by calculating the mean rewards in all the future trajectories starting from \(s_{i}\).

#### 3.3.2 Calibration-Enhanced Aggregation

The above MCTS algorithm could produce multiple trajectories and predictions for each test example through multiple iterations. In this part, we introduce a calibration-enhanced aggregation method to produce the final answer while debiasing the answer with pre-trained priors.

**Aggregation.** Considering that in ICL, the unique correct answer can be derived from multiple different demonstrations, we collect the predictions from each iteration of MCTS to make the final prediction. Specifically, we calculate the average probabilities for each label and select the one with the highest probability as the final answer, which can be represented as follows:

\[y^{*}=\operatorname*{arg\,max}_{y_{i}\in\mathcal{Y}}\frac{1}{N_{i}}\sum_{j=1 }^{N_{i}}\text{Pr}(y_{i}|x,d_{j};\mathcal{M}), \tag{4}\]

where \(\mathcal{Y}\) is the label space, \(N_{i}\) is the number of predictions for label \(y_{i}\), and \(\text{Pr}(y_{i}|x,d_{j};\mathcal{M})\) is the probability for \(y_{i}\) given by the LLM \(\mathcal{M}\) with problem \(x\) and pseudo-demonstration \(d_{j}\) as the input.

**Calibration.** LLMs are known to suffer from common token bias Zhao et al. (2021), which means they are biased towards tokens common in their pretraining data. To debias the prediction of LLMs, we adopt a calibration strategy based on prior probability. Specifically, we first obtain the prior probability 

of each label by calculating its average probability predicted by the LLM across all the test examples. Then, we derive the calibrated probability of each prediction by dividing the prior probability. We can integrate this strategy with aggregation, which is represented as follows:

\[y^{*}=\operatornamewithlimits{arg\,max}_{y_{i}\in\mathcal{Y}} \frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\frac{\text{Pr}(y_{i}|x,d_{j};\mathcal{M})}{ \text{Pr}(y_{i}|\mathcal{M})}, \tag{5}\] \[\text{Pr}(y_{i}|\mathcal{M})=\frac{1}{|\mathcal{E}|}\sum_{j=1}^{| \mathcal{E}|}\text{Pr}(y_{i}|x_{j};\mathcal{M}), \tag{6}\]

where \(\text{Pr}(y_{i}|\mathcal{M})\) is the prior probability of the LLM \(\mathcal{M}\) for \(y_{i}\) and \(\text{Pr}(y_{i}|x_{j};\mathcal{M})\) is the zero-shot probability for \(y_{l}\) given by the LLM \(\mathcal{M}\) with only the problem \(x_{j}\) as the input.

## 4 Experiments

In this section, we first set up the experiments, then report the results and conduct a detailed analysis.

### Experimental Setup

**Datasets.** To evaluate the effectiveness of our method, following Su et al. (2024), we conduct experiments on the BIG-Bench Hard (BBH) (Suzgun et al., 2023) and Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2021) benchmarks. Specifically, we consider two scenarios: in-domain and cross-domain. For the in-domain scenario, we evaluate each task of BBH and MMLU separately. For the cross-domain scenario, we randomly select 8 samples from each task of BBH to construct a dataset called BBH-mini.

**Baselines.** To facilitate a systematic comparison, we select several representative methods:

\(\bullet\)**Zero-shot**: The model directly makes predictions without any demonstration.

\(\bullet\)**Few-shot**(Brown et al., 2020): The model makes predictions with human-annotated demonstrations. It is not entirely fair to compare it with other methods, as it uses external information.

\(\bullet\)**Self-ICL**(Chen et al., 2023): The model makes predictions with self-generated pseudo-demonstrations.

\(\bullet\)**DAIL**(Su et al., 2024): The model makes predictions with pseudo-demonstrations retrieved from previously predicted examples.

**Models.** We use representative open-source LLMs (_i.e.,_ Llama3.1-8B (Dubey et al., 2024), Qwen2.5-7B (Yang et al., 2024), and Mistral-7B-v0.3 (Jiang et al., 2023) and a close-source LLM (_i.e.,_ GPT-4o-mini (OpenAI, 2024)) for experiment.

**Implementation Details.** In MCTS, we set the number of iterations as 5. For the \(Q^{\prime}\) function, we set \(k_{d}\) as 30 and \(w_{Q}\) as 1. For the selection phase, we set \(w_{a}\) in Eq. 3 as 5. For the expansion phase, we set \(k_{a}\) as 3. For the simulation phase, we set \(\epsilon\) as 1.5. Following Su et al. (2024), we set the number of demonstrations in \(f_{p}\) to 3 for BBH and 4 for MMLU. Notably, we can not obtain the logits of GPT-4o-mini, so we do not use calibration for it. To test the full potential of our approach, we also consider removing the cache mechanism in the simulation phase, named "DAWN-ICL w/o cache".

### Experimental Results

**In-domain scenario.** Table 1 presents the results across various LLMs on the in-domain scenario. As we can see, Self-ICL performs poorly and even worse than zero-shot prompting for some LLMs. The main reason is that LLMs are limited in data synthesis (Seddik et al., 2024) and may even have insufficient domain knowledge for specific tasks. Thus, they struggle to generate high-quality pseudo-demonstrations. In contrast, DAIL shows decent improvements and consistently outperforms zero-shot prompting. DAIL employs previously predicted examples as the source of demonstrations, eliminating the need for input synthesis to improve the quality of pseudo-demonstrations. However, DAIL still underperforms few-shot prompting sometimes, as it randomly selects examples at each step, and the historical examples may not be beneficial for the current one. Finally, **DAWN-ICL** surpasses all the ZS-ICL baselines by a large margin and even consistently outperforms few-shot prompting. We reformulate ZS-ICL as a planning problem and use MCTS to search for the optimal traversing order. Thus, the historical examples can better help the current one, as they are selected based on \(Q^{\prime}\) values rather than chosen at random. In addition, we observe that integrating a cache mechanism has minimal impact on performance but can significantly speed up the expansion and simulation processes of MCTS. More detailed analysis of search efficiency are presented in Section 4.3.2.

**Cross-domain scenario.** For the cross-domain scenario, the results are shown in Table 2. Similar to the in-domain scenario, Self-ICL and DAIL (ran 

dom) with the random traversing order perform poorly. In contrast, if DAIL sequentially deals with each task (_i.e.,_ DAIL (sequential)), the performance improves greatly, which means that traversing order is important, especially in the cross-domain scenario. Finally, **DAWN-ICL** achieves the best performance and even surpasses the few-shot prompting. It demonstrates that our approach is generally applicable to various scenarios in ZS-ICL.

### Detailed Analysis

In this part, we construct a detailed analysis of the effectiveness of our approach.

#### 4.3.1 Ablation Study

Our approach incorporates several important components to improve search quality and performance. To validate the effectiveness of each component, we conduct the ablation study by removing demonstration-aware initial \(Q_{0}\) value function, aggregation, and calibration strategies, respectively. The results are shown in Table 3. We can see that removing any component would lead to performance

\begin{table}
\begin{tabular}{l l|c c c|c c c c c} \hline \hline  & & \multicolumn{5}{c|}{**BBH**} & \multicolumn{5}{c}{**MMLU**} \\ \cline{3-9}  & **Task** & \multicolumn{2}{c}{**Binary****Choice****Choice****} & \multicolumn{2}{c|}{**Multiple****Average**} & \multicolumn{2}{c}{**STEM****} & \multicolumn{2}{c}{**Human****anities****Sociences****Other**} & \multicolumn{1}{c}{**Average**} \\ \hline \multirow{6}{*}{**LIama3.1-SB**} & Zero-shot & 52.26 & 38.81 & 42.32 & 48.91 & 52.22 & 68.28 & 65.82 & 58.00 \\  & Few-shot & 53.93 & 42.42 & 45.42 & 51.92 & 56.77 & 73.94 & 69.13 & 62.18 \\  & Self-ICL & 50.87 & 31.64 & 36.65 & 43.51 & 47.57 & 61.94 & 58.77 & 52.29 \\  & DAIL & 52.82 & 39.08 & 42.66 & 52.65 & 56.30 & 75.14 & 70.04 & 62.65 \\ \cline{2-9}  & **DAWN-ICL** & 60.26 & 43.69 & 48.01 & 54.04 & 58.11 & 75.24 & 70.94 & 63.79 \\  & **DAWN-ICL w/o cache** & **61.86** & **43.86** & **48.56** & **54.65** & **58.94** & **76.34** & **71.58** & **64.58** \\ \hline \multirow{6}{*}{**Qwen2.5-TB**} & Zero-shot & 59.71 & 46.56 & 49.99 & 64.03 & 59.64 & 80.18 & 74.90 & 68.50 \\  & Few-shot & 55.74 & **50.76** & 52.06 & 67.68 & 64.31 & 82.03 & 75.99 & 71.54 \\  & Self-ICL & 52.75 & 45.83 & 47.63 & 62.92 & 56.77 & 75.98 & 71.29 & 65.57 \\  & DAIL & 55.74 & 47.25 & 49.46 & 67.71 & 64.59 & 82.74 & 76.29 & 71.86 \\ \cline{2-9}  & **DAWN-ICL** & 64.51 & 49.21 & 53.20 & **68.41** & 64.91 & 83.26 & 75.47 & 72.06 \\  & **DAWN-ICL** w/o cache & **65.90** & 50.17 & **54.27** & 68.03 & **65.27** & **83.82** & **76.44** & **72.43** \\ \hline \multirow{6}{*}{**Mistral-7B**} & Zero-shot & 53.79 & 33.46 & 38.76 & 46.69 & 52.75 & 66.17 & 64.85 & 57.01 \\  & Few-shot & 60.26 & 40.03 & 45.31 & 51.74 & 54.98 & 71.40 & 68.10 & 60.70 \\  & Self-ICL & 56.44 & 33.51 & 39.48 & 40.63 & 48.20 & 58.21 & 58.80 & 51.04 \\  & DAIL & 58.39 & 36.43 & 42.15 & 50.62 & 54.54 & 72.44 & 68.59 & 60.69 \\ \cline{2-9}  & **DAWN-ICL** & 61.10 & 41.80 & 46.83 & 51.51 & 57.32 & 72.77 & **68.97** & 61.98 \\  & **DAWN-ICL** w/o cache & **62.14** & **42.24** & **47.43** & **52.14** & **58.51** & **73.06** & 68.78 & **62.54** \\ \hline \multirow{6}{*}{**GPT-40-mini**} & Zero-shot & 52.26 & 37.43 & 41.30 & 40.37 & 52.77 & 59.05 & 56.90 & 52.28 \\  & Few-shot & **65.97** & 49.56 & 53.84 & 45.26 & 64.34 & 71.17 & 69.07 & 62.60 \\  & Self-ICL & 63.47 & 47.91 & 51.97 & 48.37 & 59.17 & 71.56 & 65.59 & 60.88 \\  & DAIL & 64.44 & 50.00 & 53.77 & 44.08 & 61.91 & 61.46 & 59.25 & 57.22 \\ \cline{2-9}  & **DAWN-ICL** & 62.98 & 54.10 & 56.41 & 51.95 & 65.65 & 70.75 & 68.20 & 64.26 \\ \cline{2-9}  & **DAWN-ICL** w/o cache & 64.93 & **54.25** & **57.03** & **53.38** & **66.18** & **73.12** & **69.18** & **65.62** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance comparison across various LLMs on the in-domain scenario using BBH and MMLU. The best method in each group is marked in **bold**, and the second-best method is marked with an underline.

\begin{table}
\begin{tabular}{l|c c c|c c} \hline \hline
**Task** & \multicolumn{2}{c|}{**BBH**} & \multicolumn{2}{c}{**MMLU**} \\ \hline
**Model** & **Liamas3.1** & **Owen2.5** & **Liamas3.1** & **Owen2.5** \\  & **-SB** & **-7B** & **-SB** & **-7B** \\ \hline
**DAWN-ICL** & 48.01 & 53.20 & 63.79 & 72.06 \\ \hline
**w/o \(Q_{0}\) function & 47.12 & 52.53 & 63.20 & 71.88 \\  & w/o aggregation & 45.94 & 53.08 & 63.15 & 71.98 \\  & w/o calibration & 44.96 & 51.12 & 63.11 & 71.91 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Ablation study on BBH and MMLU.

 

degradation. It indicates that all the components in our model are helpful. Among them, performance decreases the most after removing the calibration strategy. This indicates the importance of calibration in our approach since it can effectively mitigate the inherent biases of LLMs.

#### 4.3.2 The Impact of Search Strategy

To systematically investigate the MCTS-based planning method in our approach, we conduct a comprehensive ablation study by comparing it with several alternative search methods on Llama3.1-8B, including a single Monte Carlo (MC) search, greedy search, and beam search. To ensure a fair comparison, we replace MCTS with other search methods while keeping other factors unchanged. Specifically, MC search is a directionless method, which randomly selects one action at each step. Greedy search selects the action with the highest DUCT score at each step. Both methods perform a single pass through the problem space, meaning that LLMs generate one answer for each sample, which we use as the final answer. Beam search maintains multiple promising paths by selecting the top beam-size paths with the maximum mean DUCT values at each step. To keep a similar exploration space with MCTS, we set the expansion number for each node (beam size) to 3 and keep the best 5 nodes for the next iteration to obtain the same number of trajectories. In addition, we use the same calibration-enhanced aggregation method in Section 3.3.2 to obtain the prediction results.

The results are shown in Table 4. Greedy search is better than MC, which suggests the importance of a structured search strategy. Beam search further improves the performance by maintaining multiple paths and exploring them in parallel. However, these methods cannot strategically explore the problem space since they cannot look ahead and backtrack. Thus, they are short in either performance or efficiency. In contrast, our demonstration-aware MCTS can perform better with less inference budget. Our approach explores the problem space with DCUT and efficiently looks ahead through memory-augmented simulation, making it more powerful for the complex search space in ZS-ICL.

#### 4.3.3 Exploration Efficiency Analysis

In this section, we conduct an exploration efficiency analysis of our approach. Due to limited computational resources, we run MCTS with up to 15 iterations and conduct experiments on BBH using Llama3.1-8B and Qwen2.5-7B. To keep the same number of inference times across different strategies, we do not use the cache method to ensure a fair comparison. The results are presented in Figure 3. As the number of iterations increases, the performance of our approach (DUCT) quickly improves and reaches the plateau with about 9 iterations, showing its effectiveness and efficiency. Compared with the original selection strategy UCT in MCTS, our proposed DUCT strategy achieves faster convergence and better performance. The main reason is that it uses the demonstration information to initialize \(Q\) values, achieving a more reliable estimation of the expected future reward.

### The Effect of Demonstration Selection Strategy

In this part, we explore the effect of various demonstration selection methods (_i.e.,_ random selection, BM25 Robertson and Zaragoza, 2009), TopK Liu et al. (2022), and TopK+DPP Ye et al. (2023)). Specifically, the random selection method refers to randomly selecting the demonstrations for each sample. BM25 selects demonstrations by computing the BM25 relevance score, which is a popular ranking function in information retrieval. TopK se

Figure 3: Accuracy on BBH with increasing numbers of iterations using the selection strategy of random, UCT, and our proposed DUCT.

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline
**Task** & MC & Greedy & Beam & MCTS & 
\begin{tabular}{c} MCTS \\ w/o cache \\ \end{tabular} \\ \hline Boolean & 75.20 & 79.20 & 82.00 & 80.40 & **83.20** \\ Formal & 50.40 & 52.40 & 52.40 & 52.00 & **54.00** \\ Geometric & 28.80 & 33.60 & 40.80 & **43.60** & 42.40 \\ Hyperbaton & 70.40 & 71.60 & 78.00 & 78.80 & **80.40** \\ Movie & 72.40 & 78.80 & 81.60 & 76.00 & **83.60** \\ Reasoning & 37.60 & 41.60 & 45.60 & **50.00** & 48.40 \\ Snarks & 57.30 & 57.30 & 59.55 & 58.43 & **61.80** \\ \hline Accuracy & 56.01 & 59.21 & 62.85 & 62.75 & **64.83** \\ Time & \(\times 1.0\) & \(\times 1.0\) & \(\times 14.8\) & \(\times 4.8\) & \(\times 14.8\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Average performance and inference time of different search methods on the selected tasks of BBH.

 

lects demonstrations that are semantically relevant to each sample. TopK+DPP employs a two-stage demonstration selection strategy. In the first stage, this method retrieves \(k_{d}\) candidates that are semantically similar to the input sample. In the second stage, Determinantal Point Processes (DPP) are applied to simulate interactions between the query and the candidate samples to select a diverse set of demonstrations.

The experimental results are illustrated in Figure 4. It can be observed that the similarity-based demonstration selection methods (_i.e.,_ BM25 and TopK) perform worse than the random selection method. This can be attributed to the fact that these methods tend to use samples with the same label as demonstrations, which can lead to incorrect predictions and subsequent error accumulation. As shown in Figure 5, selecting semantically similar samples typically results in demonstrations that have the same label. Due to the copying phenomenon of ICL Olsson et al. (2022), LLMs can exhibit majority bias Zhao et al. (2021), which results in generating answers that are frequent in the demonstrations. Furthermore, these incorrectly predicted demonstrations can mislead subsequent predictions made by the LLMs. To address this issue, we select samples with more diverse labels, allowing LLMs to make predictions without being influenced by the labels of the demonstrations. Experimental results indicate that our proposed TopK+Diverse demonstration selection method consistently achieves the best performance across different LLMs, confirming the effectiveness of incorporating label diversity into the demonstration selection process for ZS-ICL.

## 5 Conclusion

In this paper, we introduce **DAWN-ICL**, a strategic _planning_ approach for ZS-ICL that utilizes MCTS to search for the optimal problem-solving sequence. To achieve effective and efficient \(Q\) value estimation, we propose a novel demonstration-aware \(Q\)-value function that aims to enhance the _selection_ phase and accelerate the _expansion_ and _simulation_ phases in MCTS. Experimental results demonstrate that our approach consistently outperforms existing ZS-ICL methods and even performs better than ICL with human-annotated demonstrations. Overall, our work highlights the importance of planning for ZS-ICAL in real-world scenarios, paving the way for more effective deployment of LLMs.

## 6 Limitations

In this work, we employ the MCTS algorithm for planning the problem-solving path in ZS-ICL. More advanced planning algorithms remain to be explored. We estimate the expected future rewards by simulation and back-propagation. This is quite time-consuming for ZS-ICL since each simulated state requires the LLM to perform one inference for reward calculation. A promising direction for future work is to train a value model for efficient evaluation Wang et al. (2024a). In addition, due to limitations in computational resources, we only conduct experiments on several representative LLMs.

## Acknowledgements

This work was partially supported by National Natural Science Foundation of China under Grant No. 92470205 and 62222215. Xin Zhao is the corresponding author.

## References

* Brown et al. (2018) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind

Figure 4: The Accuracy (%) of BBH with different demonstration selection methods.

Figure 5: The error accumulation phenomenon of the similarity-based demonstration selection method.

 

Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _NeurIPS_. Chatterjee et al. (2024) Anwoy Chatterjee, Eshaan Tanwar, Subhabrata Dutta, and Tanmoy Chakraborty. 2024. Language models can exploit cross-task in-context learning for data-scarce novel tasks. In _ACL (1)_, pages 11568-11587. Association for Computational Linguistics. Chen et al. (2023) Wei-Lin Chen, Cheng-Kuang Wu, Yun-Nung Chen, and Hsin-Hsi Chen. 2023. Self-icl: Zero-shot in context learning with self-generated demonstrations. In _EMNLP_, pages 15651-15662. Association for Computational Linguistics. Coulom (2006) Remi Coulom. 2006. Efficient selectivity and backup operators in monte-carlo tree search. In _Computers and Games_, volume 4630 of _Lecture Notes in Computer Science_, pages 72-83. Springer. Dohmatob et al. (2024) Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. 2024. Strong model collapse. _Preprint_, arXiv:2410.04840. Dong et al. (2023) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey for in-context learning. _CoRR_, abs/2301.00234. Dubey et al. (2021) Abhimanyub Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Scheleten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Himsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethamj Brion, Binh Tang, Bobbie Chern, Charlotte Cauchetcux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinine Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hypkes, Egor Lakcomkin, Ehab ABlabaway, Elina Lobanova, Emily Dinhan, Eric Michael Smith, Filip Radenoviac, Frank Zhang, Gabriel Synnaeeve, Gabrieille Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Malion, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloundum, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. _CoRR_, abs/2407.21783. Hao et al. (2023) Shibio Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. _CoRR_, abs/2305.14992. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _ICLR_. OpenReview.net.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lelio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothe Lacroix, and William El Sayed. 2023. Mistral 7b. _CoRR_, abs/2310.06825. Kocsis and Szepesvari (2006) Levente Kocsis and Csaba Szepesvari. 2006. Bandit based monte-carlo planning. In _ECML_, volume 4212 of _Lecture Notes in Computer Science_, pages 282-293. Springer. Liu et al. (2022) Jiaching Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In _DeC-LIO@ACL_, pages 100-114. Association for Computational Linguistics. Longpre et al. (2022) Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinnvalae, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhi Singh, Manuel Cherp, Ahmad Anis, an Dinh, Caroline Chitongo, Da Yin, Damien Siloe, Deividas Mataciunas, Diganta Misra, Emad A. Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kush Tiwary, Lester James V. Miranda, Manan Dey, Minnie Liang, Mohammed Handyk, Niklas Muenninghoff, Seonghyeon Ye, Seungome Kim, Shrestha Mohanty, Virbi Gupta, Vieek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, and Sandy Pentland. 2024. Consent in crisis: The rapid decline of the AI data commons. _CoRR_, abs/2407.14933. Lyu et al. (2023) Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Z-ICL: zero-shot in-context learning with pseudo-demonstrations. In _ACL (1)_, pages 2304-2317. Association for Computational Linguistics. Olsson et al. (2023) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conferly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, 

Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022. In-context learning and induction heads. _CoRR_, abs/2209.11895.
* OpenAI (2024) OpenAI. 2024. Gpt-4o mini: advancing cost-efficient intelligence.
* Robertson and Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. _Found. Trends Inf. Retr._, 3(4):333-389.
* Roy and Roth (2015) Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 1743-1752. The Association for Computational Linguistics.
* Seddik et al. (2024) Mohamed El Amine Seddik, Suei-Wen Chen, Soufinae Hayou, Pierre Youssef, and Merouane Debbah. 2024. How bad is training on synthetic data? A statistical analysis of language model collapse. _CoRR_, abs/2404.05090.
* Su et al. (2024) Yi Su, Yunpeng Tai, Yixin Ji, Juntao Li, Yan Bowen, and Min Zhang. 2024. Demonstration augmentation for zero-shot in-context learning. In _ACL (Findings)_, pages 14232-14244. Association for Computational Linguistics.
* Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In _ACL (Findings)_, pages 13003-13051. Association for Computational Linguistics.
* Tanwar et al. (2023) Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tannoy Chakraborty. 2023. Multilingual lms are better cross-lingual in-context learners with alignment. In _ACL (1)_, pages 6292-6307. Association for Computational Linguistics.
* Wan et al. (2024) Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2024. Alphazero-like tree-search can guide large language model decoding and training. In _ICML_. OpenReview.net.
* Wang et al. (2024a) Chaojie Wang, Yanchen Deng, Zhiyi Lv, Zeng Liang, Juijie He, Shuicheng Yan, and Bo An. 2024a. Q*: Improving multi-step reasoning for llms with deliberative planning. _CoRR_, abs/2406.14283.
* Wang et al. (2024b) Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, and Jiang Wen. 2024b. Investigating the pre-training dynamics of in-context learning: Task recognition vs. task learning. _CoRR_, abs/2406.14022.
* December 9, 2022_.
* Xiao et al. (2024) Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In _SIGIR_, pages 641-649. ACM.
* Xiong et al. (2024) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In _ICLR_. OpenReview.net.
* Yang et al. (2024) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. _arXiv preprint arXiv:2407.10671_.
* Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In _NeurIPS_.
* Ye et al. (2023) Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In _ICML_, volume 202 of _Proceedings of Machine Learning Research_, pages 39818-39833. PMLR.
* Yoo et al. (2022) Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Huygeoli Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022. Ground-truth labels matter: A deeper look into input-label demonstrations. In _EMNLP_, pages 2422-2437. Association for Computational Linguistics.
* Zhang et al. (2024) Mozhi Zhang, Mianoiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, and Xiepeng Qiu. 2024. Calibrating the confidence of large language models by eliciting fidelity. _CoRR_, abs/2404.02655.
* Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In _ICML_, volume 139 of _Proceedings of Machine Learning Research_, pages 12697-12706. PMLR.

 



## Appendix A More experiments

### The Effect of Larger Models

In this part, we conduct experiments on larger open-source LLMs (_i.e.,_ Gemma2-9B, Qwen2.5-14B, and Qwen2.5-32B) on BBH. As illustrated in Table 5, **DAWN-ICL** can outperform other competitive ZS-ICL baselines and even surpass few-shot prompting with human-labeled demonstrations on larger LLMs. This further demonstrates the effectiveness of our proposed method.

### Experiments on Generation Tasks

To confirm the effectiveness of our proposed method on generation tasks, we conduct experiments on two types of generation tasks (_i.e.,_ math reasoning and symbolic reasoning) using GPT-4o-mini. Specifically, for the math reasoning problem, we use Multiarith Roy and Roth (2015) for evaluation. For the symbolic reasoning problem, we use Last Letter Concatenation Wei et al. (2022) for evaluation. As shown in Table 6, **DAWN-ICL** consistently outperforms other methods on these generation tasks. This further highlights the importance of strategically planning problem-solving trajectories and demonstrates the effectiveness of **DAWN-ICL**.

### Hyper-parameters Analysis

**DAWN-ICL** includes a few hyper-parameters to tune. In this section, we report the tuning results of four hyper-parameters on BBH: the expansion size (_i.e.,_\(k_{a}\)), the number of retrieved candidates (_i.e.,_\(k_{d}\)), the balance constant between exploration and exploitation (_i.e.,_\(w_{a}\)), and the balance constant between the initial and updated \(Q\) value (_i.e.,_\(w_{Q}\)). The results are presented in Figure 6. We observe that **DAWN-ICL** achieves the best performance when the expansion size is set to 3. If the expansion size is too large, it may introduce suboptimal states which can lead to a shift in the exploration direction. Conversely, if the expansion size is too small, the tree will not expand sufficiently due to the absence of some important nodes. On the other hand, we find that the number of retrieved candidates cannot be too small or too large. If the number is too small, some important examples can be overlooked, which results in performance degradation. In contrast, if the number is too large, irrelevant examples can be introduced during the calculation of \(Q_{0}\), leading to a deviation in the exploration direction. For \(w_{a}\) and \(w_{Q}\), we

Figure 6: Performance comparison _w.r.t._ the number of expansion nodes, retrieved candidates, balance constant between exploration and exploitation, and balance constant of the initial and updated \(Q\)-value of **DAWN-ICL**.

\begin{table}
\begin{tabular}{l|c c} \hline \hline
**Method** & MultiArith & Last Letter \\ \hline Zero-shot & 94.44 & 86.00 \\ Few-shot & 94.44 & 88.00 \\ Self-ICL & 96.67 & 86.67 \\ DAIL & 96.11 & 89.33 \\ \hline
**DAWN-ICL** & 97.22 & 91.33 \\
**DAWN-ICL** w/o cache & **98.33** & **92.00** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance comparison on generation tasks using GPT-4o-mini. The best method in each group is marked in **bold**, and the second-best method is marked with an underline.



find that they have minimal effect on performance, with the best results achieved when \(w_{a}\) is set to 5 and \(w_{Q}\) is set to 1.

## Appendix B Details of Experimental Cost

In this section, we present detailed information on the experimental costs of GPT-4o-mini in Table 7.

## Appendix C Detailed Results

In this part, we report the detailed experimental results on BBH and MMLU across four LLMs in Table 8, 9, 10, and 11.

## Appendix D Example Prompts

In this part, we present the example prompt for BBH and MMLU.

**Pseudo demonstrations:**

 Q: Which statement is sarcastic?

 Options:

 (A) But his eyes were on the ball, shouldn't be a red  (B) But his cleats were on the ball, shouldn't be a red   A: (B)

 Q: Is the following sentence plausible? "John Carlson scored in the third period."

 A: yes

 Q: Is the following sentence plausible? "Elias Lindholm beat the buzer."

 A: no

 **Test example:**

 Q: Is the following sentence plausible? "Marcelo got on the end of a through ball."

 A: yes

 **Example Prampton for MMLU**

 **Pseudo demonstrations:**

 Question: Objects that absorb light appear A.black

 B.white C.dark D.bright

 Answer: A

 Question: Of the following, most children will develop which skill first?

 A.write with a pencil B.cut with a knife C.say a sentence D.clap their hands

 Answer: D

 Question: Which of the following most accurately explains why a pool with water temperature of 82 degrees may feel cool to a person who has been sunbathing, yet warm to a person who has been inside in the air conditioning?

 A.Sensory restriction B.Perceptual constancy

 C.Relative clarity D.Sensory adaptation

 Answer: B

 Question: What part of the human body does a gastroenterologist examine?

 A.Brain B.Skeleton C.Stomach D.Nose

 Answer: A

 **Test example:**

 Question: Paper will burn at approximately what temperature in Fahrenheit?

 A.986 degrees B.2125 degrees C.3985 degrees D.451 degrees

 Answer: D
 



**Algorithm 1****DAWN-ICL\({}_{\blacksquare}\)**

```
Inputs: Problem set \(\mathcal{X}=\{x_{i}\}_{i=1}^{n}\), Initial state \(s_{0}\), deterministic state transition function \(t_{\theta}\), reward function \(r_{\theta}\), action expansion number \(k_{a}\), iteration number \(\tau\), action cache threshold \(\epsilon\) Initialize: State to action mapping \(A:\mathcal{S}\mapsto\mathcal{A}\), children mapping \(\mathrm{ch}:\mathcal{S}\times\mathcal{A}\mapsto\mathcal{S}\), rewards \(r:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}\), visited counter \(\mathcal{N}:\mathcal{S}\mapsto\mathbb{N}\), action cache ac \(:\varnothing\), State-action value function \(Q:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}\), Demonstration-aware state-action value function DQ \(:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}\) for\(n<0,\ldots,\tau-1\)do \(t\gets 0\) while\(\mathcal{N}(s_{t})>0\)do\(\triangleright\) selection Select \(a_{t}\) from \(A(s_{t})\) with the highest DUCT score based on Equation 3 \(s_{t+1}\leftarrow\text{ch}(s_{t},a_{t})\), \(\mathcal{N}(s_{t})\leftarrow\mathcal{N}(s_{t})+1\) \(t\gets t+1\) endwhilewhile\(s_{t}\) is not a terminal state do\(\triangleright\) expansion and simulation Select \(a_{t}\) from \(A(s_{t})\) with the top-\(k_{a}\) DQ score based on Equation 1 for\(i\gets 1,\ldots,k_{a}\)do if\(a_{i}^{c}\) in Ac then Read \((x_{i},\hat{y}_{i})\) from Ac else Perform zero-shot in-context learning for \(x_{i}\) to generate \(\hat{y}_{i}\) \(r_{i}^{i}\gets r_{\theta}(s_{i}^{i},a_{t}^{i})\) ifDQ\({}_{i}^{G}>\)then Write \((a_{i},(x_{i},\hat{y}_{i}))\) to Ac endif endif \(s_{t+1}^{i}\gets t_{\theta}(s_{t},a_{t}^{i})\) endfor \(a_{t}^{*}\leftarrow\arg\max_{a_{t}\in A(s_{t})}r_{i}^{i}(s_{t}^{i},a_{t}^{i})\) \(s_{t+1}\leftarrow\text{ch}(s_{t},a_{t}^{*})\), \(\mathcal{N}(s_{t})\leftarrow\mathcal{N}(s_{t})+1\) \(t\gets t+1\) endwhile T \(\leftarrow\) the actual number of simulation steps for\(t\gets T-1,\dots,0\)do\(\triangleright\) back-propagation Update \(Q(s_{t},a_{t})\) with \(\{r_{t},r_{t+1},\dots,r_{T}\}\). endfor  endfor
```

**Algorithm 2****DAWN-ICL\({}_{\blacksquare}\)**

The algorithm can be extended to a single-layer LSTM model using a convolutional neural network that learns a sequence of features from a given input sequence. The output of this model is shown in Figure 7. In this case, we have two layers: 1) the first layer of the LSTM model, which is trained by a CNN-based RNN encoder. Each layer has a hidden layer and outputs an output sequence of features from a given input sequence. We use the same architecture as the one used in the previous section.

## 5 Conclusion

In this paper, we proposed a novel method for constructing a class of recurrent neural networks that are capable of producing accurate predictions of the parameters of the inputs. Our approach is based on the idea of generating a class of recurrent neural



\begin{table}
\begin{tabular}{l|c c c c c|c c c c c c c} \hline \hline  & \multicolumn{8}{c|}{**Mistral-7B**} & \multicolumn{8}{c}{**GPT-4o-mini**} \\ \cline{3-12}
**BBH Tasks** & ZS & FS & Self-ICL & DAIL & **DAWN-ICL** & \multirow{2}{*}{_Dawn-ICL_ w/o cache_} & \multirow{2}{*}{_ZS_} & \multirow{2}{*}{FS} & \multirow{2}{*}{Self-ICL} & \multirow{2}{*}{_DAIL_} & \multirow{2}{*}{_DAWN-ICL_ w/o cache_} & \multirow{2}{*}{_DAWN-ICL_ w/o cache_} \\ \cline{5-9} \cline{6-12} Boolean Expressions & 76.00 & 82.40 & 73.20 & 74.40 & 79.20 & 84.80 & 51.20 & 63.20 & 59.20 & 57.20 & 49.20 & 57.20 \\ Causal Judgment & 50.80 & 55.08 & 52.94 & 53.48 & 53.48 & 52.41 & 59.89 & 66.31 & 62.57 & 60.96 & 64.17 & 64.71 \\ Date Understanding & 50.00 & 52.00 & 44.80 & 54.00 & 57.60 & 60.80 & 22.00 & 56.00 & 48.40 & 62.40 & 61.60 & 61.20 \\ Disambiguation QA & 49.20 & 60.80 & 57.60 & 54.40 & 62.00 & 68.40 & 37.60 & 50.00 & 51.20 & 50.80 & 52.00 & 53.20 \\ Formal Fallacies & 53.60 & 47.20 & 48.00 & 50.00 & 48.80 & 52.40 & 52.80 & 63.20 & 58.00 & 60.80 & 60.00 & 60.80 \\ Geometric Shapes & 9.20 & 38.40 & 10.80 & 27.60 & 41.20 & 40.00 & 31.20 & 34.00 & 30.80 & 38.00 & 40.80 & 40.00 \\ Hyperdendum & 74.80 & 76.40 & 70.00 & 54.40 & 74.80 & 76.40 & 51.60 & 61.60 & 68.80 & 69.60 & 86.00 & 88.00 \\ Logical Deduction(five objects) & 27.60 & 34.40 & 27.60 & 31.20 & 36.00 & 37.20 & 18.40 & 24.80 & 22.40 & 29.60 & 34.00 & 32.40 \\ Logical Deduction(seven objects) & 20.40 & 26.00 & 21.60 & 28.00 & 36.00 & 34.80 & 15.20 & 17.20 & 18.00 & 17.20 & 19.60 & 20.00 \\ Logical Deduction(three objects) & 43.20 & 46.00 & 47.60 & 42.80 & 52.80 & 52.80 & 35.20 & 76.00 & 69.20 & 67.20 & 75.60 & 74.40 \\ Movie Recommendation & 39.20 & 67.60 & 46.40 & 66.40 & 78.80 & 80.80 & 31.20 & 70.40 & 56.00 & 69.20 & 77.20 & 77.20 \\ Navigate & 45.60 & 50.40 & 52.00 & 48.80 & 55.60 & 54.00 & 50.80 & 68.80 & 69.60 & 68.40 & 65.60 & 67.60 \\ Penguins in a Table & 32.88 & 34.25 & 32.88 & 41.10 & 38.36 & 40.41 & 48.63 & 58.22 & 54.79 & 58.22 & 53.42 & 54.79 \\ Reasoning about Colored Objects & 35.20 & 31.60 & 27.20 & 28.00 & 28.00 & 27.60 & 62.80 & 68.00 & 69.60 & 67.20 & 71.60 & 72.80 \\ Ruin Names & 40.00 & 43.60 & 35.20 & 31.20 & 54.80 & 53.60 & 81.60 & 85.20 & 84.80 & 83.20 & 87.20 & 87.20 \\ Salient Translation Error Detection & 14.80 & 32.40 & 21.20 & 32.80 & 34.80 & 30.40 & 59.20 & 58.40 & 55.20 & 56.00 & 58.40 & 59.60 \\ Snarks & 50.00 & 55.62 & 43.26 & 47.75 & 53.93 & 52.81 & 64.04 & 59.55 & 53.93 & 67.42 & 71.91 & 70.79 \\ Sports Understanding & 47.20 & 72.80 & 60.40 & 66.00 & 72.40 & 75.20 & 51.60 & 54.00 & 78.00 & 82.00 & 86.40 & 87.20 \\ Temporal Sequences & 21.20 & 18.80 & 19.20 & 14.40 & 8.40 & 8.40 & 27.60 & 77.60 & 74.00 & 71.20 & 83.20 & 83.60 \\ Tracking Shuffled Objects(five objs) & 19.20 & 19.20 & 22.00 & 23.20 & 16.00 & 12.00 & 16.00 & 14.00 & 17.20 & 12.80 & 14.40 & 14.00 \\ Tracking Shuffled Objects(seven objs) & 13.20 & 13.20 & 11.60 & 11.60 & 8.00 & 9.60 & 10.80 & 9.20 & 11.60 & 11.20 & 8.00 & 7.60 \\ Tracking Shuffled Objects(three objs) & 33.20 & 32.40 & 33.20 & 35.60 & 31.20 & 34.40 & 35.60 & 28.80 & 33.20 & 27.20 & 29.60 & 30.40 \\ Web of Lies & 48.80 & 52.40 & 51.20 & 56.40 & 55.20 & 51.60 & 49.20 & 50.40 & 53.20 & 56.40 & 52.80 & 52.00 \\ \hline
**All Tasks (avg)** & 38.76 & 45.31 & 39.48 & 42.15 & 46.83 & **47.43** & 41.30 & 53.84 & 51.97 & 53.77 & 56.41 & **57.03** \\ \hline \hline \end{tabular}
\end{table}
Table 9: Detailed results on BBH tasks for Mistral-7B and GPT-4o-mini. “ZS” and ”FS” denote zero-shot and few-shot prompting, respectively.









\begin{table}
\begin{tabular}{l|c c c c c c|c c c c c c c} \hline \multirow{2}{*}{**MMLU Tracks**} & \multicolumn{8}{c|}{**Mistral-7B**} & \multicolumn{9}{c|}{**GPT-to-mini**} \\ \cline{3-14}  & ZS & FS & Self-ICL & DAIL & **DAWN-ICL** & \multicolumn{1}{c|}{**DAWN-ICL**} & \multicolumn{1}{c|}{ZS} & FS & Self-ICL & DAIL & **Dawn-ICL** & \multicolumn{1}{c|}{**DAWN-ICL**} & \multicolumn{1}{c|}{**w/o cache**} \\ \hline abstract algebra & 26.00 & 31.00 & 26.00 & 31.00 & 30.00 & 24.00 & 30.00 & 24.00 & 30.00 & 26.00 & 35.00 & 35.00 & 24.00 & 35.00 & 24.00 & 35.00 & 24.00 & 24.00 & 35.00 & 24.00 & 35.00 & 24.00 & 24.00 & 35.00 & 24.00 & 24.00 & 35.00 & 24.