Joint Prompt Optimization of Stacked LLMs using Variational Inference
Authors: Alessandro Sordoni
Xingdi Yuan
Marc-Alexandre Côté
Matheus Pereira
Adam Trischler
Ziang Xiao
Arian Hosseini
Friederike Niedtner
Nicolas Le Roux
Venue: NeurIPS 2023 poster (2023)
Keywords: deep prompt optimization
llm
variational inference
graphical model
chaining
TL;DR: By stacking smaller LLMs together and training them jointly using variational inference, we are able to match the performance of much larger LLMs.

Abstract: Large language models (LLMs) can be seen as atomic units of computation mapping sequences to a distribution over sequences. Thus, they can be seen as stochastic language layers in a language network, where the learnable parameters are the natural language prompts at each layer. By stacking two such layers and feeding the output of one layer to the next, we obtain a Deep Language Network (DLN). We first show how to effectively perform prompt optimization for a 1-Layer language network (DLN-1). Then, we present an extension that applies to 2-layer DLNs (DLN-2), where two prompts must be learned. The key idea is to consider the output of the first layer as a latent variable, which requires inference, and prompts to be learned as the parameters of the generative distribution. We first test the effectiveness of DLN-1 in multiple reasoning and natural language understanding tasks. Then, we show that DLN-2 can reach higher performance than a single layer, showing promise that we might reach comparable performance to GPT-4, even when each LLM in the network is smaller and less powerful.

Joint Prompt Optimization of Stacked LLMs using Variational Inference
Alessandro Sordonith+
Xingdi Yuan Mare-Alexandre Côté Matheus Pereira
Adam Trischler Ziang Xiao
Arian Hosseini
Friederike Niedtner Nicolas Le Roux
Microsoft Research Montréal
MILA 5
Abstract
Large language models (LLMs) be atomic units of computation map- ping sequences to distribution over sequences. Thus, they can be seen as stochastic language layers in language network, where the learnable parameters natural language prompts at each layer. By stacking two such layers and feeding the output of one layer to the next, we obtain a Deep Language Network (DLN). We first show how to effectively perform prompt optimization for a I-Layer language network (DLN-1). Then, we present an extension that applies 2-layer DLNs (DLN-2), where two prompts must be learned. The key idea is to consider the output of the first layer latent variable, which requires inference, and prompts to be learned as the parameters of the generative distribution. We first test the effectiveness of DLN-1 in multiple reasoning and natural language understanding tasks. Then, we show that DLN-2 can reach higher performance than single layer, showing promise that we might reach comparable performance to GPT-44, even when each LLM in the network is smaller and less powerful. The DLN code is open source
the
Introduction
The size of large language models (LLMs) has grown significantly over the last few years, mainly because of emerging capabilities [1] but considerable technical and societal costs 49 2
Recent efforts have focused either on learning smaller models matching the abilities of larger ones on some tasks using distillation
43 36 3 offloading part the computation to other dedicated In the latter case, this is done through carefully crafted instructions to retrieve the necessary information from these additional modules
components [2 22
25
18
24].
Instruction-tunet LLMs map an input sequence to a distribution over output sequences conditioned on an instruction, or prompt. In this paper, we view such LLMs as stochastic language layers, whose learnable parameters are the prompts. Multiple layers can be stacked to form a Deep Language Network (DLN) whose learnable parameters the prompts associated each layer. Specifically, each layer uses template to organize both its prompt and the inputs coming from the layer below into a single sequence before producing the output (see Figure []. This layering induces a learnable decomposition of the task into series of smaller sub-tasks, each of which might be more easily solvable by LLM. This view shares similarities recent works that chain LLM calls 44 (4].. In this work, we move towards integrating learnable components in the pipeline: each prompt can be learned to maximize the final objective.
"Corresponding author: alsordon woiiommotiiiiiccoocccoooomm
'Code: http://ghhhcconmncoooroccco/ggoe
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(A)
trainable
LM
frozen
[( promot 1>
templete
negative
prompt 7
[( inout }}
Your thoughts ware: {( intdden })
Additionaaly, considar the fact that some connextss the come before month (e.g. UK),
Arover:
LM
examplo, tha givan contoxt is "It is 4/18/19EB today, What the date a manth ago MM/DD/YMY"? you
hidden
The current date is May 2001. One yeur
{ prompt
ago from lodlyy May 30,
prompt π
clessiicatton template
[ inout }?
For the sentence "supported" choose "positive", The sentonoa is "derall" chouse "negalive"
Answer:
LM
( input }}
Similerty if the sentenoe and stabiity and prospenit" cchoose
prompt TTg
E proept }) Let's think step by stop.
Nidden tamplate
Decompoee the problem staps. First, find the current date from the given context. Then, calculate the number of days months adid subtnact the get the resut. Anditionaaly....
"postive" Note words like "artificial" tend to have nagativa sentiment.
input
into
decade dramatic deciine
input.x
The deadline Jun 2021, which 2 days away from now. one year ago from today
the MWDDDMYYY 'n OptionsinsA)
05/30/2000 in (F) 04/02/2020...
Figure 1: Left: An illustration of a DLN-1 performing a sentiment analysis task: input and the trainable prompt are merged using a template and fed to the LM for answer generation. Right: a DLN-2 with a residual connection, performing the date understanding task: two prompts need be learned. In this example, the hidden template extends Chain Mhuin-O--ie Th-Thtttt [48] with learnable prefix; we consider the output of the first layer, hidden, latent variable h. We use variational inference to learn To. T2. Templates can be considered as an hyperparameter of the network.
We first show how to perform prompt optimization in a shallow I-layer language network (DLN-1) which parametrizes distribution Pix (y] x) where z and y are string input and output respectively. and π the learnable prompt (Figure I left). Our prompt optimization techniques extend the Automatic Prompt Engineer (APE) procedure from Zhou et al. [57]. We show how our prompts can include a verbalization difficult examples from the task: the final prompts are a combination of instruction directives, akin to zero-shot learning (17), and task examples, akin to in-context learning 20 This significantly improves downstream performance, surpassing APE on several tasks.
Then, we show how to train 2-layer DLN (DLN-2), which parametrizes a probability distribution:
PoLN-2(y(x) Σ Pix h, x, x1 PL(()) mo).
where h is the string output of the first LLM layer (Figure right). We consider à as a latent variable: to maximize the marginal log- liliiiiiihodd we formulate variational inference algorithm that uses an approximate posterior over h. Note that this formalism easily encompasses more than two layers.
Considering outputs of the hidden language layers as latent variables allows us to encompass various established prompting methods, such as Chain-Of-Thooe Thought (CoT) [48] and self-consistency (SC- CoT) (46). Particularly, CoT can be seen as particular DLN-2 with the first layer prompt set to "Let's think step by step" and the second layer prompt set to 'The answer is"; we can either learn such prompts learn supplement to those as in Figure [ SC-CoT can be seen as marginalizing over CoT strings sampled from a task-agnostic prior: when using the template in Figure D (right), our method generalizes this perspective by learning task-specific prior distribution over successful CoTs.
The rest of the paper is organized as follows. First, we provide an interpretation of LLMSs shallow networks, drawing a number of analogies with standard parametric and non- cor-marmetrric models and explaining how best to train them. After exploring their limitations, we propose to stack two such

---

LLMs to form a DLN-2. We show how they can be trained using form of variational inference, then demonstrate their performance on series of reasoning and language understanding tasks.
2 One-Layer Language Networks
pre-trained LLM with frozen weights might be thought of as a complete function class indexed by prompts. The output y of an LLM for an input can be modulated through prompt by feeding combination of т and to the LLM. Hence, from a low-level perspective, the function class of an LLM is defined by its architecture, i... its depth, number of heads, context size, etc., and training happens at the parameter level. It is data and compute intensive and should be done rarely. From high-level perspective, the function class of an LLM is defined by the pre-trained model chosen (LLAMA [44]. text-davinci-003, GPT-4, etc.), and training happens by fine-tuning the model or by choosing the prompt.
There are two ways of optimizing an L.MM the prompt level. The first one is prompt engineering, parametric optimization, where the optimization space is independent of the size of the dataset. Because this optimization usually happens in discrete space, gradient- bassed techniques do not apply and most efforts rely a combination of random local search and human heuristics 21 55). The second one is in-context learning (ICL), a non-parametric optimization technique where the solution is direct function of a subset of examples [5] 20]. This approach works well for few-shot learning but scaling it to larger datasets has both performance and computational issues. We shall now generalize previous work in discrete prompt optimization [57 [1] with the ultimate goal of learning a set of prompts in a language network.
2.1 Language Layers
We use language layer to refer to a (stochastic) computation that takes as input a string a and outputs a string y. This computation is modulated by another string, generally called prompt or instruction [57]. The string transduction is performed by an operator LM, by feeding æ and n as context and generating continuation y. Templates describe the way and being fed to the LM operator. These are functions that accept strings as variables and output a string. We will denote such templates with this font T. A simple forward template F is the concatenation, iee. F(z, π) = " (z)" We also explore more complex ones, examples of which can be seen in Figure
combined prior to
Given an input æ, a prompt т,, and template F, a language layer defines probability distribution Prx(v) (ar, T)) over output strings y as computed by the LM. In the next section, we describe generic framework for optimizing the weights π for a language layer.
2.2 Prompt Optimization: Improved APE
Because the search for the best prompt happens over discrete space, will rely local search method, using an LLM to implement a distance measure between prompts. The procedure can be seen as extension of Automatic Prompt Engineer (APE), recently proposed by Zhou al. [57), and will serve as stepping stone towards introducing our algorithm for training deep language networks. Our prompt optimization algorithm can be structured as follows:
1. Given the current prompt and current batch of examples {z, (z,y). generate N "local" candidates 7² " N using a prompt proposal distribution;
2. Score each candidate using a (potentially stochastic) scoring function 8, then choose π = arg max," s(n²).
Prompt Proposal Local search algorithms assume distance measure between inputs to crawl the search space. In this setting, we rely on LLMs to generate local modifications to the prompts. Our prompt proposal distribution takes as conditioning information i) the batch given as input to the layer, ii) its corresponding output (x,, y. ). and iii) the current prompt æ. The proposal distribution Pix(""") [Br, y. ú}. TT) wraps this information using a particular "backward" template B,, which can be found in Appendix [ This approach is similar to the instruction template used by Zhang et al. [55), with the exception that we also integrate information about the model's own predictions, which
3

---

Algorithm One-Layer Language Network (DLN-I) Training Algorithm
Require: ÿ ~ ())(y)c)
generates a completion of prefix e with temperature return log-prob of h following
Require: log (og(c)))))
Require: N: prompt samples, iterations, D: dataset
Require: F: template for the inferencefforward pass
Require: B₃: template for prompt proposa//backward pass.
1: Initialize ı with task description or empty
2: for in [1 do
3:
I. 1/ D
Sample minibatch b. Do inference pass
4:
<
Pix F(x
5:
1....,
Pi (m Br ((z.g.)).
Sample N candidate prompts
6:
s" s'....,,''
N + log Pex F x()
b Score all prompts
7:
774and ara maxx" (s ,...,
Select prompt with best score
8:
end for
found empirically help performance given that the model tends to propose prompts that correct its own errors. We sample from the prompt proposal distribution to generate a set of prompts. A particularly important aspect is ensuring the diversity of the candidate pool n I'...., n. We devise several strategies to improve the diversity and the usefulness of the candidate samples in Section 4
Prompt Selection Once set of N prompts has been generated, we use scoring function to select the updated prompt.
assume access to the log-likelisoods of the LM operator and we rank candidate prompts to maximize data log-like likeliiieood we normalize this log-proo probbabbbiity by the length of the output string. While we focus on that metric in this work, there is restriction the scoring function that can be used. We use backtracking to increase the robustness our selection mechanism, as well as a memory well-performing prompts for efficiency. We present both strategies in Section 4 The sketch of 1-layer prompt optimization algorithm is described in Algorithm ignoring backtracking and memory for simplicity.
arg max,, log Pix (y/ F(2; r²)). In practice,
The results of our prompt optimization may be found in Table Tanle will be discussed in detail in Section 5.2 We now turn to extending prompt optimization to architectures with two layers.
3 Two-Layer Deep Language Networks (DLN-2)
The natural extension of DLN-1 is DLN-2, which language layers are stacked, i.e. the output of the first language layer is the input to the second one. A 2-layer network induces distribution over outputs of the form:
POLu-2 (y)n)) =
Σ (h) m)))
(I)
where h is latent variable that potentially makes it easier to explain the target y. The output layer also conditioned
through F,, forming residual connection (Figure This formulation is
reminiscent of past work using latent language representations to guide document summarization [26]. In our case, however, the encoding/decodiin distributions are parameterized by natural language prompts II { m and we do not assume access to the LLM parameters.
While this architecture has more expressive power than shallow language network, the prompt optimization problem becomes harder now that we have to jointly search over both To and TT Doing random search in this space is impractical [8] and manual tuning of the weights is also exponentially harder than with a single prompt. We turn to variational inference to address this issue.
3.1 Variational Inference Objective
The layerwise decomposition of our system allows us to leverage tools from approximate inference in probabilistic models to learn TT. In particular, we propose to use variational inference learn Il 3 I6). We posit approximate posterior q(h) over the latent variable h, and bound the marginal

---

log-likelihood of y given x by computing the ELBO:
log 2 N Σ (h))
pix Pix(( (l,, ()))px((([((): mn))] + H [g(4)]
(2)
which allows us to decompose the optimization over II in two independent optimizationn problems, over both Ty and IT:
To = arg max
Σ wh log (((((((I,
.ं arg max
Σ (2.g),
wn log llo
TTD
The search over Tı is identical to the prompt optimization described in Section 2 with the difference that the inputs now depend on the approximate posterior samples h in addition to the inputs æ. The search over uses a similar strategy but uses the posterior samples h as targets, instead of
Although this bound allows to decompose the optimization w.r.t. II, it is only useful if it is close to the true value. Since its looseness the KI. divergence between g(h) and the true posterior, (L(g(h))IP(hIg)) x)): In what follows, we specify how we parametrize the approximate posterior and how we tighten the approximation via posterior sharpening.
need to find approximate posterior closely matching the true posterior.
Hidden Proposal We will also be using an LLM to sample candidate hidden states from q(h). Unless specified, for simplicity, we use the same LM operator used in our language layers. The approximate posterior can condition on arbitrary amount of information but especially useful might be to condition on the true target If it conditions on the hidden state coming from the "forward pass", h ~ Pun Pum(h[F(z, To)), then Godit (h) = PIn Ba (h, y, π1)). B,, is specifically tailored hidden proposal template (Appendix) D Gedit performs a sort of edit operation, where the LM is tasked to rewrite the hidden variable h given some extra knowledge of the ground-truth label and of mi Alternatively, can set the posterior to be equal to the prior, i.e. Opra (h) Pix Pux(N(F(xx mo)), the prior with additional information about the label This amounts to re-computing the hidden state knowing privileged information about the label. We found most effective to sample hidden states from a mixture of 9pri and qprit-
(h) PLn By (2,, wo, y)) (Appendix DD.
Posterior Sharpening Given the absence of learnable parameters in g(h), the induced approximate posterior might still be far from the true posterior. To bridge this gap, we reweigh each sample h' based on its probability under the true posterior distribution. More precisely, we compute 20 = log (y (h', x, T1)) + log (ix(h' [a To)), then assign to each h; the probability wi = exp(cv0:)) E, exp(crū;)) where is a tunable temperature parameter that controls the entropy of the posterior weights. The full algorithm for training DLN-2 is presented in Algorithm
4 Practical Instantiation
Although method aim to learning prompts in stacked LLM architectures, we do rely on good amount of prompt engineering for our templates. Hereafter, we detail some choices that were fundamental to make approach work in practice.
Proposal Diversity To ensure diversity of the samples for both the prompt proposal distribution, we found helpful to use two strategies. The first is to modify the backward templates B,, before drawing sample from the proposal distribution Pix (T). To achieve so, we parametrize the basic templates with a "[message)" variable that we instantiate from pool of hand-written instructions, that describe different behaviors the model should follow to propose new т, c.g. "shorten the previous instruction', "give useful examples', etc. These can be interpreted meta-instructionss, i.c. high-level directives that inform the model on how to create a better instruction for the task, and extend
templates used in 12 55). These can be found in Appendix D In the future, we could envision to extend learning to these instructions. In the case of B they could function parameters for prior over the weights of the DLN. The second strategy ensure more diversity is that we instantiate B, with a different random subset of examples in the current batch, before drawing each sample n, This effectively modifies the generation context for each sample n
Learning In-Context Learning One strategy we found particularly effective to integrate in the pool of ineta-instructions additional instruction that asks the LM to give useful examples to improve
5

---

Algorithm 2 Two-Layer Deep Language Network (DLN-2) Training Algorithm
ý ~ Pin(c)
generates a completion of prefix e with temperature
2: Pog(ee
return log-prob of h following 3: N: prompt samples, K: posterior samples, I: iterations, D: dataset
4: F. F,: templates for the inferencefforward pass without and with residual connection
5: Ba, B₁; templates for prompt and hidden proposa//backward pass.
6: Initialize To: TT1 with generic sentence task description
7: for in [ do
8: y. D
Sample minibatch Do inference pass, first layer b. Do inference pass, second layer D Sample K posterior proposals for Compute prior log-probs for all h samples Compute log-likelihoods for all h samples Compute normalized posterior weights Select best posterior sample > Sample N candidate prompts for To b. Sample N candidate prompts for mi > Compute ELBO for all prompts Ti
9:
h (F(r, m)))
- Pix(F, (h, m1))
11:
h h" ~ Pin (Bn v, TI
a k Pog F(x, mol)
B 3K + log FF (hh x1))
9ºt expla* k + BB² (2x exp(a + Bb))
15:
ha arg maxx 1q- o²}
T m
0.7 (Br (x, h h² T)))
17:
a ai
~ ply (Br 0. y). x1))
18:
- -e gh log PLn (1* F(x, mb))
19:
81
Σn gh log Pax (y) (1*
, I. xf))
D. Compute ELBO for all prompts Tī Select prompt To with best score b Select prompt with best score
20:
TT f arg maxap
o (
5
21:
IT < arg maxop 81 }
22:
end for
its current prompt т. Empirically, observed that this allows the model to sample candidate prompts n" that contain synthetic examples for the task, embedded in natural language. Examples of this interesting behavior be found in Appendix F We found that this behavior particularly interesting as the resulting prompts often perform better than standard ICL. We hypothesize this is due to both i) the "verbalization" of the example in the prompt, which modifies the dataset syntax into a more suitable one, and ii) the fact that the model can dynamically select which examples are most important to integrate in the prompts, given the errors made during training. Therefore, suspect that DLN achieves a similar effect to recent techniques that select important examples for with the improvement of naturally conditioning the selection on the end task
ICL [20 35 performance via end-to-end training.
40 18 47)
Backtracking and Memory Optimization of both DLN-1 and DLN-2 is challenging due to the fact that we do not have gradient information and we sample a restricted set candidates Tº at each optimization step due computational reasons. We deploy multiple strategies to allow the network to be robust to sampling/selection errors. First, we include the current prompt æ into the set of candidate prompts be scored the current iteration π" This allows the model not take the step if the previous prompt performed better. Second, we keep a memory of M = 5 best prompts found by tracking validation set performance.
Exploration Reward When training a DLN-2, we empirically observed that the first layer prompt To was updating very slowly. Due to the fact that the approximate posterior shares templates with the prior used in the forward pass, the posterior samples h' are close to h and the maximizer of Equation (3) remains ma- To address this issue, we add to the scores of each candidate prompt an exploration reward that is proportional to the negative log-probabilit of those that led to an incorrect prediction: = -A log PIx (ir(((,"")) x")), if + y. This encourages the model to both find prompts that maximize the log-probability of high-probability posterior samples and at the same time minimize the oog-robbbbility of prior samples that led to incorrect predictions. We anneal À to 0 during training with constant schedule and we select the initial À by monitoring validation performance for each task.

---

5
Experiments and Results
We design and conduct set of experiments to help answer two main research questions:
Q1: Can we outperform APE and In-Context Learning (ICL) with a DLN-1?
Q2: Does network depth provide further improvement upon DLN-1?
5.1
Experimental Setup
Datasets and Tasks We adopt a set of nine NLP and reasoning tasks commonly used in prior work studying zero- few-shot learning capabilities of LLMs 23 10
42 1. We focus (Hyper., Nav., Date. and Logic. ..) we use the 250 data points provided by BBH as test set. We take the remaining data points from BigBench (39] that were not included in BBH, and randomly split them (evenly) into training randomly sample 400 and 250 data points from their training and test sets, respectively. We use the original validation sets. For tasks adopted from Leopard [ (Disaster and Airline), we randomly sample 400, 250, and 250 data points as training, valid, and test. We list all tasks and their statistics in Table B the Appendix.
39
classification tasks. For tasks adopted from BigBench- Haad (BBH)
and validation sets. For tasks adopted from 23 (Mpqa, Tree, and Subj),
We accuracy the evaluation metric. Specifically, given input, we compare system's output string against the ground-truth output string provided by the dataset. We score if the two strings identical and 0 otherwise. Before the comparison, to deal with issues like tokenization and capitalization. In all our DLN
process the strings from both the model
output and the ground-trum experiments, we perform hyperparameter search and the same hyperparameter setting with three random seeds. We report the test accuracy averaged over three seeds corresponding to the hyperparameter setting that achieves the highest average validation accuracy. We report details of the hyperparameter search in the Appendix ![
Throughout this paper, the backbone to our proposed systems unless otherwise specified. For DLNs, we use a batch size of 20 and train for 20 iterations by early-stopping validation performance evaluated every 2 iterations. We then report test scores. We sample
OpenAl's models, specifically GPT-3 ((exx-davinci-003) and GPT-4, as
20 prompt proposals and K = 5 hidden samples.
Baselines We compare the DLN against two classes of baseline systems. First, we test set of systems equipped with the same backbone (i.e., GPT-3):
O-shot: Given an input, the LLM is required to generate the answer in a zero-shot manner.
5-shot (ICL): Given an input as well as five data points as in-context examples, the LLM is queried to generate answer. The five examples
randomly sampled from the training set.
KATE 200 Given an input, we retrieve the five most similar data points from the training set using an the-the-shelf sentence encoder, and them as in-context examples.
APE [57] The LLM is queried to generate pool of candidate prompts for the task given few input-output pair examples. The candidate prompts are evaluated on a validation set to find the best performing instruction prompt. The best instruction is then used for O-shot evaluation. We optimize the prompt over both 15 and 400 examples (APE-15 and APE-400 respectively).
CoT [ Given an input, the LLM is first queried generate a reasoning path with the prompt "Let's think step by step"" Then, conditioned on the input and its first output, the LLM is queried to generate an answer. This is the zero-shot version of CoT and is a natural baseline for DLN-2 2: it performs two LLM calls and can be seen as DLN-2 without optimization. We will report performance of this baseline when comparing to DLN-2.
Additionally, we compare against one of the most advanced LLMs to date, GPT-4, We test (-shot and ICL settings with GPT-4.
5.2 DLN-1
Our first set of experiments evaluates the 1-layer language network (DLN-I) described in Section 2 presents results on the full suite of test tasks. We see that it matches the performance of
Table I
We only use the variant with seven objects.
7

---

Table 1: Test accuracy averaged over three random seeds of a shallow, 1-layer language network (DLN-1) compared to baselines both on GPT-3 and GPT-4 For trainable systems (i.e., APE and DLN-I) or systems relying on GPT-4, we report the 95% confidence interval.
BigBench Hard
NLU
Leopard
Method
Hyper.
Nav.
Date.
Lugic.?
Mpqu
Tree
Subj
Disaster
Airtine
GPT-3
0-shot
60.8
64.1
56,4
45,9
88.0
61.9
61.7
81.6
75.6
5-shot
55.6
56.5
62.1
36,7
87.2
80.0
76.4
81.2
82.7
KATE
71.1
56.9
61.1
44.4
88.4
77.6
71.1
76.0
81.6
APE-15
68.5155.
67.31777
32.112266
45.51:4.7
85.51:4.
71.3:::55
61.3:1222
83.5::3.5 14,81:4.6
APE-100
63,5+44.7
56.9-432.9
23.5++14.1
45.6+:124
84.9499.7
72.0
63.7+99.2
60,3+:37,4
12,3++10.0
DLN-1
91.443.0
68.5444.7
55.71445
47.5422.1
88.5422.5
89.74:3.2
83.2:26.5
81.7465
83.24555
GPT-
0-shot
64.021.
7..0::10
79.2422
68.5
86.320
6..8=21.7
1.
47.7
84.5
5-shot
88,4126
75.7111
79.3111
62.81:1.7
88 08.0:::3.
82.52:38
94.7:1355
63.61885
1..0:1:0
16-sbot
93.3+22.
75.5445.1
80.94:5.00
66.4-1:3.6
91.3+:1.5
83.7
96.5:225
67.1
88.3-42.11
DLN-1
95.24150
27.1447
76.74:3.
69.1
91.1433
89.5
93.1=:5.0
82.1+3388
85,9
DLN-1 prompt on Hyperbaton (GPT-3)
When constructing sentence with mulliple adjectives, the order should be opinion, size, age, shape, color, origin, material, and purpose. Adjectives of the same type should be listed in descending order from largest to smallest. When adjectives of different lypes are used, the order should be opinion, size, age, shape, color, origin, material, and purpose. For example, in the phrase "massive ancient chair" size (massive) should come before age (ancient). Examples: little old fashioned Russian silver rectangular ship; silly large old leather hiking chair; brand-new spherical Mexican sweater; enormous old spherical green Nigerian exercise car: medium-size triangular wool eating ship: good square brown Egyptian ship: lovely massive drinking monkey; archaic dircular white plastic shoe. In each of the following examples, the adjective order is wrong. Identily the correct adjoclive order:
DIN-1 prompt on Hyperbaton (GPT-4)
To determine the correct adjective order, follow this sequence: opinion, size, shape, age, color, origin. material, and purpose. For example. choose "large red plastic ball" over "red large plastic ball" since it follows the order: size (large). color (red). and material (plastic). Not adjectives may be present, but the order should still be maintained. the options are "ancient prismlike white leather whitting match" and 'leather white ancient prismlike whittling match", choose the first option. as it follows the order: age (ancient). shape (prismlike), color (white), material (leather), and purpose (whittiing. Remember that opinion always comes before age, "abnoxious old fashioned typing shoe" is correct over "old-fash oid-fashhonne obnoxious typing shoe," Ensure opinion adjectives come before other adjectives in the sequence. When comparing options, follow the order of adjectives for each category: before color, color before origin, and on. In are switched, like "paper walking monkey" vs "walking paper monkay". choose the option where material comes before the purpose. Additionally, always prioritize the given sequence over the position of adjectives the sentences. For example, choose "midsize brand-new gray Chinese wood sweater' over "Chinese brand-new gray midsize wood sweater" it follows the order: size (midsize), age (brand-new), color (gray), origin (Chinese), and material (wood).
where purpose and material adjectives
The final prompt of the DLN-1 on Hyperbaton includes not only instructions but also examples from the training set. These samples were automatically chosen by the prompt optimization. In way, this approach combines in-context learning and prompt optimization.
Figure 2:
the best GPT-3-based method on Disaster, Mpqa and Airline and narrowly beats the best GPT-3 baseline on Logic.7 and Nav.. On Hyper., Trec, and Subj, DLN-1 significantly outperforms the best GPT-3 baseline (by about 20, 10, and percentage points, respectively). On Hyper., Trec, and Disaster, it even surpasses GPT-4 baselines, unsurprisingly underperforming GPT-4 on all other tasks. DLN-I's excellent performance on Hyper., BBH task about ordering adjectives according to linguistic convention, is surprise To better understand this result, we show the final prompt in Figure 2 We see that the prompt contains both instructions and a list of examples from the training set. These examples were automatically chosen by the optimizer based on their impact on the performance. This can be seen as a combination of KATE, which selects training examples to put in context based on their similarity with the test example, and APE, which selects the prompt based on its performance. On Date., DLN-1 tends to systematically under-perform the 0-shot baseline both for GPT-3 and GPT-4. We observed that DLN-1 overfits due to paucity of examples in the validation set.
8

---

Table 2: DLN-2 test accuracy using GPT-3 as LLM.
Method
Nav.
Date.
Logic.7
Disaster
Subj
O-shot
64.1
56,4
45,9
81.6
61.7
CoT
69.3
72.4
41.1
54.4
59.3
APE
67.3
32. 12.15
45.5
54.8
61.3a7a
APE-400
+3.9
23.5+11.11
45.6+124
60.3=37.4
63.7+92
DLN-1
68.5247
53.7
47.5+21
81.7=65
83.2
DLN-2
83.1 +247
75.2+148
45.7
82.8=25
85.9
5.3 DLN-2
We investigate the effectiveness of depth through experiments with 2-layer language networks (DLN- 2) on tasks where we expect depth to be most useful, and on which DLN-1 significantly underperforms the GPT-4 0-shot baseline, i.e., Nav., Date., and Logic.7 [42] Since the Nav., Date. and Logic.7 tasks from BBH require more complex spatial and temporal reasoning. they are the ones where we most expect decomposition into subtasks to be helpful. We also include Subj and Disaster
example where DLN-1 performs well (even outperforming the GPT-4 O-shot baseline), since we are interested to see to what extent DLN-2 can further push performance.
Results for DLN-2 can be found in Table 2 Compared to DLN-1, DLN-2 provides an average boost of 7.2% absolute score. On Nav. and Date.. DLN-2 largely improves the performance of DLN-1, outperforming all single layer networks. On Logie.7, all methods appear to perform similarly. This could point to the fact that the task might be too hard for the base LLM and thus highlights the limits of prompt optimization of weak base model. On Subj and Disaster, DL.-2 achieves further improvement over DI Compared to 0-shot GPT-4 results in Table | on Subj and Disaster, DLN-2 on average provides more than 20% in absolute improvement. We encourage readers find additional experimental results in Appendix C
Related Work
Prompt-Based Machine Learning GPT-3 S launched a new paradigm in NLP called in-context learning (ICL), applied beyond traditional NLP tasks [1 The discovery of chain-of-though prompts (CoT) marked major advance in prompting: LLM performance improves markedly when the prompt includes examples of intermediate reasoning steps
(48) (few-shot CoT), simply instructs the model to "think step by step" 17 (zero-shot CoTT) Like COT, DL.Ns break a problem down into intermediate steps but they operationalize these steps as separate LLM calls, each defined by its own learned prompt. Since the introduction of CoT, prompting techniques have evolved to be more dynamic and iterative. Recent methods often operate recursively. Examples include RECITE (41), Self-ask [33], and related methods for question-answeriin
Creswell et al. [6], Zhou et al. [56].
similar class of methods relies on "introspection" [4, where an LLM is prompted to ingest, evaluate then possibly act on its own previous output. Self-critique 46). ReAct [54], Reflexion [38
Self-refine 22 fit this mould along with Hao et al. [II, Du al. 99. Yao et al. (53]
Prompt Optimization Techniques based on notions of self-talk and self-evaluation align naturally function in DLNs. Early work in this category includes Autoprompt (37] and GRIPS (32] Deng et al. 7 argue that mnuuerraion--tee-sslleeree
with automatic prompt optimization-a
heuristics for optimizing discrete prompts do not explore the prompt space systematically. They take an RL approach to overcome this problem, training policy network, via soft Q-lcarning with suitably designed and stabilized reward function, to generate effective prompts. Through Gibbs sampling, Reprompting (52) iteratively searches CoT recipes to improve prompt performance automatically. Most relevant to DLNs, Zhou al. [57] present Automatic prompt engineer (APE). APE optimizes an initial prompt by searching over a pool of candidates to maximize score function. We use an APE-inspired approach in DLNs and we east the proposal/scoring functions as elements of variational inference. In a concurrent work, Pryzant et al. prompt optimization. This algorithm uses L.MM's nonparametric feedback guide prompt generation and sclection.
34 proposed using textual gradients in automatic

---

Multi-Layer LLM systems Several recent works compose LLMs as nodes in computational graph, which is the core idea of DLNs. Some work cited above can be seen as instances of this idea. Similarly, Khot et al. 15] induce an L.M to generate a basic "control flow" that calls distinet 50 propose AI chains, an interactive system of chained L.LMs based on
LLM modules. Wu et al. a set of "LLM primitive" operations. They conduct 20-person user study in which participants modify chains, and find this process to improve task performance, transparency, and controllability. Dohan et al. [8] unify LLMs and graphical models as "language model cascades", Specifically, they cast LLM compositions as graphical models with string-valued random variables They show how scratchpad (3), maiin-of-thoughe [48], tool use (25). and several other prompting strategies fit their formalism. DLNs can likewise be considered an instance of language model cascade, because of that framework's 's generality. However, going beyond the conceptual work of Dohan et al. [8], present an effective technique for doing inference in an LLM--bsed graphical model and we apply learned networks of LLMSs several downstream tasks.
7 Conclusion and Future Work
In this paper we introduced an algorithm for joint prompt optimization in deep networks where each layer an LLM. To do so, we consider outputs of each hidden LLM layer latent variable we need to do inference over. From conceptual perspective, we demonstrated how CoT can be seen as a DLN-2 with a residual connection. Similarly, Generated Knowledge Prompting [19] could be considered as a fixed forward-only DLN-2 where, in the first layer, an LLM generates related knowledge, and in the second layer, another L.M takes the generated knowledge as input and generates the final answer. Other prompting techniques like ReAct (54), Reflexicon (38] and
Self- Cofsistesyt [46] could all be ensembles of DL with different prompt initializations.
Although we only tested I-layer and 2-layer LNs so far, we already show that the performance of smaller LLMs can be boosted when stacked and prompted properly. We believe the modularity of these architectures will make them more adaptable and reusable to new use cases. While accuracy on downstream tasks is an appealing metric, we argue that other considerations are just as important, for example the ease of adapting model to one's 's's own use case, or the ability to leverage multiple existing models.
We noticed that GPT-3 has a tendency to always produce an answer given an example: this could be due to the particular 0-shot fine-tuning procedure, which biases the model towards generating useful responses. This raises the question of whether we can fine-tune "stackable" LLMs and whether DL.Ns can be used as a framework to generate training data for that purpose. Second, we engineered our backward and forward templates; in the future, we wish to expand our work to learn parts of such templates: we expect this to make the variational bound tighter and thus easing DLN's optimization. Additionally, while we only proposed 2-layer DLNs, the framework accommodates arbitrary directed acyclic graphs
Impact statement While we are fully aware of the limitations of addressing societal issues through technical work, we hope that modular approaches like ours will alleviate some of the issues associated with LLMs, like the concentration of power associated with the difficulty to train them. We also hope that, by facilitating the reusability and adaptivity of such models, we shall make them more amenable to wider variety of use cases. However, while we discuss the performance of these models on artificial benchmarks, we do not address the question of when and how such models should be deployed, nor do we offer additional guarantees against their misuse. We also emphasize that performance on artificial tasks, even if realistic, is neither representative of performance in uncontrolled environments, nor enough to justify the deployment of these models in high stakes situations.
We would like to acknowledge Silviu Pitis for the useful feedback on the draft, Nikolay Malkin and Tong Wang for their advice during the first steps of this project.
Acknowledgements
Earlier work by Miao and Blunsom
27 also treated strings random variables.
10

---

References
[1] Bansal, T., Jha, R., and McCallum, A. (2020). Learning to few-shot learn across diverse natural language classification tasks. In Proceedings of the 28th International Conference on Computational Linguistics, pages on Computational Linguistics.
Barcelona, Spain (Online). International Committee
[2] Bender, E. M., Gebru, T., McMillan-Majoo A., and Mitchell, M. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on faimess, accountability, and transparency, pages 610-623.
[3] Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. (2017). Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859-8777
[4] Blodgett, S. L., Liao, Q V., Olteanu, A., Mihalcea, R., Muller, M., Scheuerman, M. K., Tan, C., and Yang, Q. (2022). Responsible language technologies: Foreseeing and mitigating harms. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA '22, New York, NY, USA. Association for Computing Machinery.
[5] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances neural information processing systems, 17:187711
[6] Creswell, A., Shanahan, M., and Higgins, I. (2022). Selection-inferencc:: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09711.
[7] Deng, M., Wang, J., Hsieh, C., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E. P., and Hu, Z. (2022). Riprompt: Optimizing discrete text prompts with reinforcement learning. In Goldberg, Y., Kozareva, Z.,, and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3369-3391. Association for Computational Linguistics.
[8] Dohan, D., Xu, W., Lewkowyzz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski. H., Saurous, R. A., Sohl-Dickstein, J., et al. (2022). Language model cascades. arXiv preprint
[9] Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. (2023). Improving factuality and reasoning in language models through multiagent debate.
[10] Gao, T., Fisch, A., and Chen, D. (2021). Making pre-trained language models better few- shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computational Linguistics.
[11] Hao, S., Gu, Y., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z., and Hu, Z. (2023). Reasoning with language model is planning with world model. arXiv preprint 499.
[12] Honovich, O., Shaham, U., Bowman, S. R., and Levy, O. (2022). Instruction induction: From few examples to natural language task descriptions.
[13] Hosseini, A., Reddy, S., Bahdanau, D.,, Hjelm, R. D., Sordoni, A., and Courville, A. C. (2021). Understanding by understanding not: Modeling negation in language models. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tür, D.,, Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y., editors, Proceedings of the 2021 Conference of the North Ameri- Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT 2021, Online, June 6-11, 2021, pages 1301-1312. Association for Computational Linguisties.
[14] Huang, W., Xia, F., Xiao, T., Chan, H., Liang. J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., al. (2022). Inner monologue: Embodied reasoning through planning with language models. arXiv preprint rxXiv:2007.5600

---

[15] Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., and Sabharwal, A. (2023). Decomposed prompting: modular approach for solving complex tasks. International Conference on Learning Representations.
[16] Kingma, D. P. and Welling, M. (2022). Auto-encoding variational bayes.
[17] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho. K., and Oh,
A., editors, Advances in Neural Information Processing Systems, volume 35, pages 22199-22213. Curran Associates, Inc.
[18] Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. (2022). Internt-auugmented language models through few-shot prompting for open-domain question answering. arXiv preprint 25X1v:2203.05115.
[19] Liu, J., Liu, A., Lu, X., Welleck, S., West, Le Bras, R., Choi, Y., and Hajishirzi, H. (2022a). Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 3154- 3169.
[20] Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L.,, and Chen, (2021). What makes good in-context examples for gpt-3?
[21] Liu, P., Yuan, W., Fu, J., Jiang. Z., Hayashi, H., and Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 1--35.
[22] Liu. R., Wei, J.. Gu, S. S., Wu, T.-Y., Vosoughi. S., Cui, C., Zhou, D., and Dai, A. M. (2022b). Mind's eye: Grounded language model reasoning through simulation. arXiv preprint arXiv:2210.05359.
[23] Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. (2022). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8088 8098, Dublin, Ireland. Association for Computational Linguistics.
[24] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe. S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y.. Welleck, S., Majumder, B. P.. Gupta, S., Yazdanbakhsh, A., and Clark, P. (2023). Self-refine: Iterative refinement with oofffceedback.
[25] Mialon, G., Dessi, R., Lomeli. M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi- Yu, J., Celikyilmaz, A., Grave, E., LeCun, Y., and Scialom, T. (2023). Augmented language models: survey.
[26] Miao, Y. and Blunsom, P. (2016a). Language as a latent variable: Discrete generative models for sentence compression. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 319-328, Austin, Texas. Association for Computational Linguistics.
[27] Miao, Y. and Blunsom, P. (2016b). Language as a latent variable: Discrete generative models for sentence compression. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 319-328.
[28] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1 11048-11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
[29] Mukherjee, S. and Awadallah, multilingual models. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 2221 22344,.4 Association for Computational Linguistics.
(2020). Xtremedistil: Multi-stage distillation for massive
12

---

[30] Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz. A.. Bosma, M., Luan, D., al. (2021). Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21
[31] Ouyang, L., Wu, J., Jiang. X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
[32] Prasad, A., Hase, P., Zhou, X., and Bansal, M. (2022). Grips: Gradient-free adit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07228
[33] Press, O., Zhang, M., Min, S., Schmidt, L.., Smith, N. A., and Lewis, M. (2022). Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350.
[34] Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., and Zeng, M. (2023). Automatic prompt optimization with" gradient descent" and beam search, arXiv preprini arKiv:2305.034:
[35] Rubin, O., Herzig, J., and Berant, J. (2022). Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, pages 2655-2671.
[36] Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.011088.
[37] Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. (2020). Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235.
[38] Shinn, N., Labash, B., and Gopinath, A. (2023). Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366.
[39] Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alnsso, A., et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.044
[40] Su, H., Kasai, J., Wu, C. H., Shi, W., Wang. T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N. A., et al. (2023). Selective annotation makes language models better few-shot learners. International Conference on Learning Representations.
[41] Sun, 7.,, Wang, X., Tay, Y., Yang, Y., and Zhou, D. (2022). Recitation-augmented language models. arXiv preprint 2rXiv:2210.012000
[42] Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D.,, and Wei, J. (2022). Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint 1rXiv:22100.9261.
[43] Tang, R., Lu, Y., Liu, L., Mou, L., Vechtomova, O., and Lin, J. (2019). Distilling task-specific knowledge from BERT into simple neural networks. CoRR,
[44] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. (2023a). Llama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.13971.
[45] Touvron, H., Martin, L.. Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra. S. Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A.. Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S.. Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov. T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodrigue7, A., Stojnic, R., Edunoy, S., and Scialom, T. (2023b). Llama 2: Open foundation and fine-tuned chat models.
13

---

[46] Wang, X., Wei, J., Schuurmans, D., Le, Q.. Chi, E., and Zhou, D. (2023a). Self-consisteney improves chain of thought reasoning language models. International Conference on Learning Representations.
[47] Wang, X., Zhu, w., Saxon, M., Steyvers, M., and Wang, W. Y. (2023b). Large language models are implicitly topie models: Explaining and finding good demonstrations for in-context learning.
[48] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q- V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS.
[49] Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-.,, Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., al. (2022). Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214-229
[50] Wu, T., Terry, M., and Cai, C. J. (2022). Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-22.
[51] Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P.. Feng, J., Tao, C., and Jiang, D. (2023a). WizardIm: Empowering large language models to follow complex instructions.
[52] Xu, W., Banburski-Fahey, A., and Jojic, N. (2023b). Reprompting: Automated chain-of-thought prompt inference through gibbs sampling. arXiv preprint arXiv:2305.0999)
[53] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, L., Cao, Y., and Narasimhan, (2023a). Tree of thoughts: Deliberate problem solving with large language models.
[54] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023b). React: Synergizing reasoning and acting in language models. International Conference on Learning Representations.
[55] Zhang, Z., Zhang, A., Li, M., and Smola, A. (2023). Automatic chain of thought prompting in large language models. International Conference on Learning Representations.
[56] Zhou, D., Schärli, N., Hou, L.., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le,
Q. and Chi, F. (2023a). Least-to-most prompting enables complex reasoning in large language models. International Conference on Learning Representations.
[57] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. (2023b). Large language models are human-leve prompt engineers. International Conference on Learning Representations.
14

---

Contents in Appendices:
Appendix we list the contribution of each author to this work.
provide additional experimental details including task statistics and the prompt strings we used to initialize DLN.
In Appendix
B
In Appendix we provide additional experiments and baselines we compare to.
Appendix we provide forward and backward templates being used in DLN.
In Appendix we provide algorithm that generalizes DLN training in multiple layer setting.
In Appendix
we show examples of learned weights that exhibit behavior similar to in-context learning.
In Appendix G
we show examples of learned weights by 2-Layer DLNs.
In Appendix
we show an example of the hidden states produced by a 2-Layer DLN.
In Appendix we provide implementation details, including hyperparameter information.
In Appendix
we discuss resource used in DLN development and their pricing.
15

---

Table 3: Tasks used in this work.
Task
train]
|valid]
[tet]]
[class]
Description
Mpqa
400
256
250
2
Sentiment analysis.
Tree
400
256
250
6
Question type classification.
Subj
400
256
250
2
Determine whether sentence is subjective objective.
Disaster
400
250
250
2
Determine whether a sentence is relevant to disaster. Airline tweet sentiment analysis.
Airline
400
250
250
3
Hyper.
400
1000
250
2
Order adjectives correctly in English sentences. Spatial reasoning given navigation instructions.
Nav.
375
375
250
2
Date.
250
Infer date from context.
Logic.7
225
225
250
7
Deduce the order seven objects given instruction.
A
Contributions
Alessandro Sordoni proposed the general idea of DLN, where multiple prompts are learnt at each layer through backward natural language operations; they proposed to generate synthetic in-context examples and the exploration reward for DLN-2; they wrote the code and ran the experiments; they focused on Sections 2, 3, 4 and contribute writing the rest of the sections.
Xingdi Yuan co-developed the basic idea of DLN and wrote part of the code, they also co-designed and helped conducting experiments. They contributed the writing of the paper, mainly Sections and 6.
Marc-Alexandre Côté helped with the experiments and the infrastructure make calls to OpenAl models. They also built a demo to visualize the evolution of DLN's prompts during training and contributed to the writing of the paper, mainly focusing on the algorithms and the appendix.
Matheus Pereira co-coded an earlier, non-variational backwards operator with AT, helped with the APE and DLN-2 layers experiments, implemented the method for estimating the total cost of experiments, build a demo to visualize the evolution DLN's prompts during training, and contributed to the release of the DLN code.
Adam Trischler helped with template conception and iteration, co-coded an earlier, non-variational backwards operator with MP, and contributed to paper writing, mainly the literature review.
Ziang Xiao helped with the model evaluation and experiment setup and contributed to the paper writing, mainly the literature review and discussion.
Arian Hosseini participated in the development discussions throughout the project and contributed to writing the literature review the paper.
Friederike Niedtner organized and managed the project, helping the team focus on the right priorities.
Nicolas Le Roux proposed the variational inference formulation and the posterior sharpening. They offered guidance and mentorship for the project. They also contributed to the writing of the paper, mainly sections 2, and 3.
B Additional Experimental Details
B.1 Additional Task Information
In Table we provide short descriptions for all tasks and their statistics.
B.2 Prompt Initialization
We initialize the "classification" layer of the DLNs, i.e. the first layer of the 1-Layer LN and the second layer the 2-layer DLN, with question or task description as reported in Table We use these same initializations to compute the 0-shot performance. Therefore, at initialization, 1-layer LNN is equivalent to the (-shoo baseline. For the hidden layer of the 2-layer DIN, we initialize the prompt to 'Decompose the problem to make it simpler:" for Nav. and Subj. and
16

---

Table 4: Prompt initializations.
Task
Initialization
Mpqa
Read the following review, then choose whether negative or positive.
Trec
Read the following question, then choose whether it is about deseription, entity, expres-
sion, human, location number.
Subj
Read the following sentence, then choose whether it is subjective objective
Disaster
Read the following sentence, then choose whether it relevant a disaster.
Airline
Read the following sentence, then choose whether it is positive, negative, or neutral. Which sentence has the correct adjective order.
Hyper.
Nav.
Read the following sentence, then determine whether you return to the starting point. Infer the date from context.
Date.
Logic.7
The following paragraphs each describe a set seven objects arranged in fixed order.
The statements are logically consistent within each paragraph.
Table Test accuracy averaged over three random seeds. We compare the layerwise and the end-to- end trainings for DLN-2.
Nav.
Date.
Logic.7
Subj
DLN-2 2 (fix 2nd)
73.1
61.6
43.3
80.2
DLK-2 (fine-tume 2nd)
76.4
62.8
40.7
84.5
DLN 2 (end-to-end)
83.1
75.2
45.7
85.9
(empty string) for Date. and Logic.7. We didn't itdn't try other initializations for this hidden layer, we leave this for future explorations.
C Additional Experiments
In addition the prompt engineering and few-shot baselines from the main paper, include here comparisons to more of those on a subset of the datasets. Particularly we added results for:
Table O shows DLN-1 and DLN-2 outperforming CoT+APE (implemented described in Section 4.3 of Zhou et al. [57);
Table 7 shows that even increasing the number of examples for ICL and KATE cannot match DLN-T and DLN-2 performance;
Table 8 and Table 9 show results for DLN-1 using open-source language models such as Wi ..2 [5] and LLaMA2-70B-Chr [45] respectively.
Layerwise training of DIN-2 We also explored different learning strategy for DLN-2: layer-wise pre-training. We start the learning of single layer DLNN using the technique from Section 2 We call the prompt obtained at the end of this optimization x². Then, we train DLN-2 by initializing иi = n², and then learn To, the parameters of the bottom layer, using variational inference. We explore two variants: one keeps the last layer fixed and one it fine-tunes the last layer. We report their results in 5
Table Test accuracy averaged over three random seeds with 95% confidence interval. All models use GPT-3. DLN-1 and DLN-2 outperform CoT+APE.
Method
Hyper.
Nav.
Date.
Logic.7
APE-15
68.5 s.5
17.3
32.14 1+ 286
45.5
CoT+APE
50.9
61.5 1s
58.64 22
38.9
DIN-1
91.9*
48.5
55.7+
47.5 2.1
DIN-22
83.1
75.2+ 148
45.723. 3.5
17

---

Table 7: Test accuracy averaged over three random seeds with 95% confidence interval (where applicable). All models use GPT-3. Increasing the number of ICL examples helps performance, but cannot match DLN-1 and DLN-2 in general. Context length limit is an issue for ICL and KATE 32-shot.
Method
Nav.
Date.
Logic.7
Subj
S-shot
56.5
62.1
36.7
76.4
ICL - 10-shot
62.9
38.9
72.0
ICL 32-shot
66.0
63.5
83.2
KATE 5-shot
56,9
61.1
44.4
71.1
KATE 10-shot
59.5
62.0
41.6
73.9
KATE 32-shot
67.5
62.8
80.4
DLN-1
68.5
55.7
47.5+
83.2
DLN-2
83.1 24.7
75.2= 14.8
45.7
85.9487
Table Test accuracy using WizardLM-v1.22 13B as LLM. This open source model seems signifi- cantly less able to capture few-shot examples from the context. DLN-1 outperforms ICL on all tasks here.
Method
Nav.
Logic.7
Subj
0-shot
58.0
0.0
65.8
5-shot
56.0
28.0
50.8
DLN-1
61.1
31.0
Table Test accuracy averaged over three random seeds with 95% confidence interval. All methods use LLaMA2-70B-Chat as LLM, with DLN + GPT3 employing text-davinci-003 as the backward L.M for prompt and hidden proposals.
Method
Nav.
Date.
Logic.7
Subj
0-shot
42.0a..
25.2
14.4a00
62.4-00
5-shot
43.2
21.1197
16.4126
67.7=155
DIN-1 GPT3
43.6+4a
21.9457
33.1
80.9-115
DLN-1
44.94-6.
31,64105
38.4433
76.1
DLN-2 GPT3
43.7+3a
51.1144
21.9444
59.1-167
DLN-2
68.9
61.74176
20.0
63.1=254
18

---

D Templates
D.1 Forward Templates
"Classification" Template F for 1-Layer LN In the 1-layer LN, the following template to elicit the output given the input æ. prompt is substituted with the value of the current prompt.
Classification Template F
template: {| prompt
K imput)) ))
Answer:
Residual Classification Template F, for 2-Layer DLN In the 2-layer LN, the last layer just concatenates the input to the output of the first layer, h, before eliciting an answer.
Residual classification template F,
template: {1 prompt
{[ input }] Your thoughts were: { h }}
Answer:
Hidden Layer F The variable prompt is substituted with the value of the current prompt To- This has the effect of providing additional information about how "Let's think step by step" should behave.
Hidden Layer F
template: [| (npptt) ))
(1 prompt Let's step step.
For 2-Layer DLN on Subj, we use the following hidden template. We couldn't 't run with the previous template due to lack of time, observed that the step by step trigger tended to generate lengthy hidden states.
Hidden Layer
template: || prompt
[ Input]] })
Brief Analysis:
D.2 Backward Templates (Prompt and Hidden Proposals)
Prompt Proposal Template B, B, is used to propose new candidate prompts. The template takes as input the current prompt, prompt, and mini-batch of examples stored in backward_ backward lerarrdiinfoooo input stores the input to the layer (z if we are proposing prompts for the first http:aaddddfooodaaaaaaaeeeaa layer or h if it is the second layer); backward. info. target stores the target to the layer (h* if it is
the first layer or otherwise); backward. info. output stores the predictions of the model during the forward pass (h if is the first layer, and ý if it is the second layer). message is substituted with one of the message_alternase alternatives, sampled random during the DLN training, This induces diversity in the generated prompts and allows emergence of learning to ICL behaviors, where the prompts contain synthetic examples that can help solve the task.
19

---

Prompt Proposal Template B.,
template:
A student completing task requires producing a output input. The student instruction that describes how lo produce the oulpul given each input.
The student has made some errors. Your task is improve the instruction such student can errors.
This was the instruction. Instruction
([ prompt !) [END]
Student successes
for backward info backward_ infos %) (% www.baadddiioooooo info.loss %)
## Input:
{{ backward info.input 3/
##* Correct Output:
([
target X)
(% endif %) (% endfor %)
Sludent
(% for backward info in backward_ infos %6 1% if backward info.loss 0.0 %)
Input:
([ backward info inppt
Student Output:
([ backward info.output )
Correct Output:
(( backward info.target !)
0% endif %) (% endfor %)
Improve the instruction to fix the student errors. message ]] Instruction
message alternatives:
Clarify the instruction by adding few words or a short sentence. Be concise.
the Be concise. Shorten the instruction by removing superflous words or sentences.
Improve
providing examples
Rewrite the instruction by providing detailed information to avoid ambiguity. concise.
Backward Hidden Templates B, We experiment with multiple backward templates to sample hid- den states from the approximate posterior distribution (h). Each vary in the amount of conditioning information. The simpler way of sampling hidden states is to use the same hidden template as in the forward pass. This corresponds using posterior distribution ((h) which is equivalent the prior distribution p(h). The other alternative is to condition the posterior template on the answer illustrated below:
Backward Hidden Template B, (y conditioning)
template: [ input]] ))
Given that answer is: (1 yy))
( prompt Let's step step.
The next alternative we experiment with is more verbose template that takes as input the prompt for the input æ in input, the hidden state from the forward pass h in
the final layer Ti in and the ground-truth output y. We use a similar strategy of sampling different message alternatives to substitute with message to increase diversity of the hidden samples:
20

---

Backward Hidden Template B
template:
This is the context needed to solve the problem: { next__ prompt ]]
This the problem:
[f input i]
These were your thoughts:
W h II
Given that this is the answer:
H ky)) ))
{ message }]
Thoughts:
alternatives:
- Rellect and refine your thoughts for this problem by adding detailed explanations. the your reasoning. Add examples illustrate your thoughts. Be concise.
In practice, we found that sampling from hidden states from a mixture of forward template F, i.e. Pix (h]ar, mo), and backward template with y conditioning, i.e. q(h|ar, y, To) works well. We suspect that this has the effect of capping the KI. divergence term between posterior and prior distribution, i.e. the (L(((h)) (L)(())) principled way by learning prompt for the posterior proposal.
mo)) appearing in the ELBO. In the future, this could be addressed in more
E Generalized VI to Multiple Layers
We report the generalized training algorithm for multiple layers in Algorithm
F Learning to In-Context Learn: Additional Examples
In Figure 2 we report examples of prompts found by the I-layer DLN on the Hyperbaton task, which exhibit either integration or verbalization of task examples. Here, provide additional examples of prompts found by DLN-1 on the MPQA task.
DLN-1 prompt on MPQA (GPT-3)
Read each sentence, then decide if the sentence expressing positive negative sentiment, For example, if the sentence is "supported", choose "positive", and the sentence "derail". choose "negative". Similarly, the sentence is "victorious", choose "positive", and the sentence is "would multiple words, consider the overall sentiment of the sentence and choose the appropriate option. For example, the sentence sentence "peace stability prosperity". choose "positive". words like "artificial" tend to have negalive sentiment.
bad idea", choose "positive". Additionally,
contains
"counting choose "negative"
DLN-1 prompt MPQA (GPT-4)
Determine whether given input has a positive negative connotation by analyzing the meaning of the words and phrases context. the expresses favorable, desirable, pleasant meaning, choose "positive. the expresses an unfavorable, undesirable, or unpleasant meaning, choose "negative." Consider the overall sentiment expressed by the input rather than focusing individual words phrases. example, "calling for" generally advocating or supporting something. while 'a true Muslim fighter' can be seen as positive, since it refers to someone dedicated their beliefs. Keep mind some phrases may have positive connotation when they imply improvement or resolution, like end to." Additionally, phrases like "to the contrary" can have positive connotation when they suggest differingg yet valid perspective or opinion. When analyzing the input, consider the context in which is used, as the connotation of word phrase change depending the situation. For example, "extra vigii" have posilive connotation when it implies increased awareness and preparedness.
positive connotation
implies
21

---

G Examples of 2-Layer Best Weights (GPT-3)
G.1 Navigate (81.6% Dev Acc)
DLN-2 Prompt: T
Decompose the problem breaking down into and describing the position after each Note that the direction you are facing stays the same unless specified. Start by specilying the direction you are facing and the coordinates the starting point. For each step, describe the number steps taken and the direction movement (e.g. North, South, East, West). Specity the new coordinates and direction after each step.
DLN-2 Prompt: I
Start facing north. Take the specified number of steps in the indicated dirrctionn and turn when specified. Make sure to keep track of your direction and number of steps taken you the starting point.
G.2 Subj (89.9% Dev Acc)
DLN-2 Prompt: To
Reflect on the inpul to produce an oulpul that eilher objective (lactual) or subjective (involving opinion or value judgment). Objective statements describe events facts that be verified, while subjective statements express opinion personal feelings about a fact, event, or situation that cannot be confirmed as an accurate statement of fact. For example. the input "the film that can be verilied." "I the input is "the film was an amazing experience; the output would be "This statement is subjective, expressing personal opinion about the accurate statement." If the input "neither Juan Antonio nor Sehor Maximiliano know what they are for when the tables turned by the sly Carmen," the outpul would be "This statement an objective facl, as it describes events that can be verilied regarding the actions input is "humorless, self-conscious art drivel, made without glimmer of intelligence or invention, the output would be 'This statement subjective, expressing negative opinion
success the box office,"
output would
statement an objective
describes events
question that reflects approval and judgement, and cannot be confirmed as an
Antonio, Señor, Maximiliano. and the change circumstances caused Carmen." If the
film question that reflects disapproval judgement, cannol be confirmed as an accurate statement: " If the inpul is hypothetical situation, such as "how would you feel when woke, the nightmare had just begun?", ?", output would be "This statement is subjective, expressing personal opinion about hypothetical siluation that cannot be verified."
DLN-2 Prompt: I
Read the sentence. Determine is expressing fact or opinion. A fact is accurate statement that can be confirmed, while opinion is a personal viewpoint that reflects someone's ' beliefs. Facts are typically statements about something that happened, such events, actions, or conditions, statements that describe state of being, such someone's personality or a physical object. Opinions are lypically statements that express judgement, approval, or disapproval. Examples of facts include statements about as 'the weather is sunny," or statements that describe state of being, such as "the detective is strong and independent. Examples opinions include statements about good beautiful." Examples questions that are not facts opinions include hypothetical questions, such as "how would feel if when you woke, the nightmare had just begun?". Select "objective" for fact and "subjective" for an opinion.
that occurred, such 'the film released 2012, statements about conditions, such
somethiing
"the film terrile'' "the weather
H Examples of Hidden States
We report the forward pass produced by 2-Layer DLN in Section G.I on Navigate below.
Input
If you follow these instructions, do you return to the starting point? Take 3 steps. Take 10 steps. Take 4 steps. Take 1 step. Options:
Yes
Hidden h
Take 3 steps: (3. 0) facing east
steps: (13. 0) facing east
3. Take 4 steps: (17, 0) facing east
Take step: (18, 0) facing east
Answer: No, you do not return to the starting point.
22

---

Table 10: Hyperparameter search space.
hyperparam
search space
L-Layer LN
bh_tpl
9 actioon prompt:v3.0. 1_aatioonpropptvt_
q action prompt:v3.5
tolerance
-1,0,2
uemomy
0,2
held_out_ppre weld_out_prompt__t ptwpproppt_rakkingggggggg
True. False
2-Layer DLN PT fix 2nd layer
bh_tpl
q action prompt:v3.0, q action prompt:v3.5
logp- log-ppenalty
0., 0.5, 2.
2-Layer DLN PT + fine-tune 2nd layer
th_tpl
q
g__action_ppottte protppvvpp
logp logp_penalte penalty
0., 0.5.
2-Layer DIN end-to-end
num num_h_ h_spples
5,
Output y
No
Implementation Details
We report hyperparameter search space in Table 10 A brief description of the hyperparameters is as follows:
bh bh_tpl is the type of backward prompt template use B,, v3.5 is equal to the B, report in Section D In v3.0, we remove "Be concise." at the end of each message alternat ives We noticed that in general v3.5 works better it implements sort of regularization the length of the found prompts. Future work could address length regularization a more principled manner.
logp_penalty is the coefficient for the exploration reward mentioned in the paper.
num_h_ hasmmeles is the number of samples to generate from the approximate posterior num
distribution.
use_memory is whether or not we use the backtracking mechanism. Usually works well across tasks.
describes whether use only half of the mini-batch exam- ples for each prompt proposal, as described in the main paper.
cout_prompt_rrree
tolerance describes after how many iterations we reload the best weights found during the last validation the current validation score is lower than the best score obtained far.
For the 2-Layer experiments, we have to restrict this search space due to computational costs. We use bh_tpl "v3.5", tolerance 2, use_memory = 2, uutdd__ut_promp____ ty
True,
Pricing
We keep track the number of tokens we interact with GPT-3 via its online API. According to OpenAl's pricing policy, user pays for both the input tokens (prompts) and the output tokens. Using the Hyperbaton task as example, while training 1- l-layer LN, the total number of tokens use is
23

---

Table 11: Test accuracy along with inference cost expressed in tokens (in gray) averaged over three random seeds of a shallow, 1-layer language network (DLN-1) compared baselines on GPT-4. We also report the 95% confidence interval on the test accuracy. We emphasize the cost at the testing time because it more relevant in real-world deployment and the training cost one-off.
BigBench Hard
NLU
Leopard
Method
Hyper.
Nav.
Date.
Logic.7
Mpga
Tree
Subj
Disaster
Airline
GPT4
O-shor
64.021.0
74,0=10
79.2::26
68.5
86.32:0
64.82:1.7
72.5
47.7=206
84.52:0
(7.6k)
(12.9)
(23.6k)
(46.2k)
(3.7k)
(7.6k)
(10.22)
(10.1K)
(9.9k)
5-shor
88.422.6
75.721.
79.31
62.811.7
88.01:30
82.5
94.74335
63.62285
88.01:10
(48.0k)
(79.20)
(143,30)
(287.56 .k)
(24,5k)
(52
(62.6k)
(63
(61.7k)
16-shot
93.34+2.3
75.5+5.1
91.34-15
83.74-06
96.54225
88,3422.11
(136.8))
(229.99)
(405.1k)
(817.6k)
(70.3k)
(149100)
(177.99)
(179.4k)
(175.2k)
DIN-1
95.2+500
47.1=47
1.5
69.1
91.1432
89.5
93.145.0
82.1
15.9
(77.2k)
(29.9k)
(52.3k)
5k)
(63.4k)
(120.7%)
(46.5k)
(47
(38.2k)
Algorithm 3 Deep Language Network Training Algorithm
1: h ~ Pby(1): generates completion of prefix with temperature L.
2: log PIN (1)(: return log prob of h following x.
3: N: prompt samples, K: # posterior samples, 1: # iterations, L: # layers. D: dataset
4: F: template for the inference (forward pass).
5: B,, B₁: templates for prompt and hidden proposal (backward pass).
6: Initialize all layers Ty with generic sentence or task description.
7: for in [] do
8: T, D
Sample minibatch
9: ho
10:
hu,.... hı < 0 (F(((u11,,
b Inference pass for all layers O < <1 2 L
11:
bi 6-2 y
12:
for
l in [L - -1,1] 1,1,do
13:
h1
bik p (fu, h441 m))
Sample K posterior proposals for hu at + log log (h/ [[ TT- b Compute prior log-probs for all A samples
14:
a)
15:
, 3K + MoogPnnligg FF m)) >> Compute log-likelihoods for all hf samples
16:
of + explaf 3/) 1Exexplaf explat + +)))
Compute normalized posterior weights
17:
hj +- arg max {4t . .
b Select best posterior sample for layer /
18:
end for
19:
for / in [L 1, do
20:
mi
Pin 0.7 (B,r (he, hets, 1i m))
Sample N candidate prompts for m > Coopute ELBO for all
21:
st,.... s + -x Ex of dht log Pun 1F
prompts o
22:
Ti +- arg max,; {8} , xN
Select prompt n; with best score
23:
end for
end for
2,941,360, For 2-laaer DLN, the total number of tokens we use is 13,654,962. According to the current price for GPT-3 ($0.02/1k tokens), a single run of a 1-layer and 2-layer DLN cost roughly 59 USD and 273 USD, respectively.
In Table II we report the cost (lower is better) in terms of total number of tokens for the test set (prompts included). We emphasize the cost the testing time because it more relevant in real-world deployment and the training cost is one-off. We can see DLN-1 improves over ICL on 5 out of 9 tasks on GPT-4 at comparable token cost. Some tasks do not benefit from ICL (i.e. reasoning tasks) while other tasks like Subj, Trec, and Hyper. benefit significantly.
24


Reviews:
[{"review_id": "eqM7ojWfEu", "reviewer": "Reviewer_2Jdr", "summary": "This paper introduces a novel discrete prompt tuning method. The paper is discussed under a scenario that regarding discrete prompt as the only tunable parameters while freezing all other model parameters. In this sense, the author focuses on getting optimized prompt from one LLM and feeding which as the input prompt to next LLM. Such kind of stacking LLMs workflow is regarded as the main contribution as the authors claimed. Moreover, the authors introduce a new latent variable to further improve this pipeline. \n\nIn the experiment, the authors verify the effectiveness of the proposed method via comparing with a bunch of baselines including several in-context learning method and one instruction-tuning method APE. The authors highlight the outperformance of their method over 0-shot GPT-4 is achieved via using text-davinci-003 backbone on Hyper, Tree, Disaster dataset. ", "questions": "1. I kind of curious why you are adopting a subset of test set for evaluation. Any reason for that?\n2. I kind of doubt the performance of 0-shot GPT-4. For example, on Subj dataset, FT RoBERTa and prompt tuning RoBERTa can achieve ~97% score, while 0-shot GPT-4 can only get 65.8. I am little shocked by GPT-4's bad performance on this task. It would be better for the authors to release the prompt of 0-shot gpt-4 they use for reproducing purpose. \n3. Why not reporting the 95% confidence interval on APR & 1-Layer LN & 2-Layer LN in Table 2?\n4. Why you split the main results into two tables?", "limitations": "typo: \nAlgorithm 2 Line 13 \\beta_k -> \\beta_1\nAlgorithm 2 Line 18 \\pi_0^n -> \\pi_0^I\nAlgorithm 2 Line 19 \\pi_1^n -> \\pi_1^I\n", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "originality: This paper shows a novel discrete engineering method from a new perspective that interprets the process of getting optimal prompts as the stack of two LLMs. This angle is novel and interesting.\nFrom the methodology perspective, authors develop their method based on improved APE and generalize which to 2-layer case. \n\nquality: overall quality is mediocre, though the idea is novel, the authors do not provide enough evidence to support their claim. \n\nclarity: the authors attach the codes for 1-layer & 2-layer case, which makes it easier to interpret their ideas. However, the intuition for some settings need to be improved. \n\nsignificance: authors evaluate the proposed 1-layer method on 9 tasks and 2-layer method on 4. They re-run the experiment 3 times for verify the significance.\n\n", "weaknesses": "1. The paper is not comparing with enough correct baselines. The authors are mainly comparing with 0-shot GPT3&4 and APE method in two main tables. However, if you regard the number of training example that proposed method can touch with as the x-shot, then the comparison objective should be x-shot GPT3&4 instead of 0-shot GPT3&4 (especially in Table 2). It's widely acknowledged that increasing the number of few-shot examples (in a range) would enhance model's performance. Is it still fair to compare with 0-shot model with your proposed method? Then it means, the only baseline authors are correctly comparing with is APE. I'd encourage authors to compare more baselines in both discrete prompt engineering and continual-space prompt tuning area. \n\n2. The generalization of the proposed method is questionable. The author claims their proposed method can generalize to more number of layers. But the comparison between 2-layer and 1-layer model is insufficient. Authors only compare them on 4 datasets and on some of them the 1-layer results are even better than 2-layer's. It's not convincing. \n\n3. Even the authors have mentioned in conclusion&future work that testing with other LM is needed. I want to highlight this as a major weakness of the paper. The value of this paper should be reevaluated is it only works on GPT-3 model. Why not stacking two GPT-4 models which should not be hard to do (just change slightly in API calling)? Or maybe on BERT/RoBERTa models. Without corresponding evidence, it's hard to justify the effectiveness of the proposed method. \n\n4. The intuition and some setting of the proposed method is unclear. For example, in Line 137, the authors suddenly define the a weight without illustrating why the sum of these two terms is a good fit for their purpose. I am little confused to interpret. \n\n5. The authors also mentioned that they hope to match the performance of the largest LLMs without incurring the large computational and data cost. However, one should also be careful that GPT-3 API calling is also a considerable computational cost. Without disclosing the number of times they call the GPT-3 API, it's hard to support their claim as incurring low computational cost. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\nPlease find hereafter our answers:\n\n---\n***“1. The paper is not comparing with correct enough baselines. [...] it means, the only baseline authors are correctly comparing with is APE.”***\n\nIn the paper, in addition to APE, we compared to other non 0-shot baselines: ICL (5-shot) uses 5 examples from the training set, so we have one additional baseline which is *not* zero-shot. KATE retrieves from 400 examples, therefore the same number as we use for optimization, and uses 5 of those to perform few-shot learning.\n\nFor completeness, we performed additional experiments with 10-shots and 32-shots on the tasks we compared with DLN-2 and the results are as follows. Please, take a look at the shared response (***Table A***) for these results.\n\nAs reported in the above table, for some tasks (e.g., Logic 7), the length of extra in-context demonstrations could quickly use up the context length limit (4096 in our case of text-davinci-003). This suggests an advantage of prompt optimization in comparison to in-context learning, despite both methods leverage information from the training set.\n\n---\n\n***“2. Scale beyond two layers; two layers is not always better than one layer”***\n\nScaling beyond two layers might be a hard problem. We think that prompt optimization can benefit from human engineered initializations, which can make optimization easier. We hope that our algorithm can serve as a starting point to fine-tune human crafted chains of prompts.\nSome tasks might not benefit from CoT or reasoning paths. For this paper, we ultimately consider the 1 vs 2 layers as a model selection problem for each task.\n\n---\n\n***“3. Testing with other LM is needed. I want to highlight this as a major weakness of the paper.”***\n\nWe tested our prompt optimization algorithm both with GPT-4 and an open source model WizardLM. We cannot unfortunately stack two GPT-4 models, as we need to have access to the log probabilities, which are not available at this time. We couldn’t run 2 layers on the open source model given that we require “echo” functionality of the log_probs and VLLM (the service we use for inference) doesn’t have it yet. This requires more time. We will report the numbers in the camera ready.\n\nWe report mean accuracy across 3 seeds and cost for inference over the test set as the number of processed tokens (less is better). These results were obtained using the latest GPT-4 openai endpoint. These results can be found in ***Table C*** and ***Table D*** the shared response.\n\n---\n\n***“5. the authors suddenly define the a weight without illustrating why the sum of these two terms is a good fit for their purpose.”***\n\nThe sum of these two terms corresponds to weighting each sampled hidden state by their probability under the posterior distribution. This can be viewed as a biased but more robust importance weighting scheme where the prior distribution is assumed to be uniform. We will make this clearer in revision. \n\n---\n\n***“6. Without disclosing the number of times they call the GPT-3 API, it's hard to support their claim as incurring low computational cost.”***\n\nFor reference, the costs for calling GPT-3 are in the appendix. While DLN-2 requires twice as much compute as a single language layer using the same base LLM, our hope is that each language layer in a DLN-2 could be much smaller than the language layer in DLN-1 for the same performance, thus limiting or even negating the additional inference cost of using multiple layers.\n\n---\n\n***“Why you are adopting a subset of test set”***\n\nAdopting a subset of the test set is also a common practice shared by other works due to computational costs associated with running large LMs. For example, this paper also uses a subset of the test set for evaluation: https://arxiv.org/abs/2104.08786\n\n---\n\n***“Doubt the performance of 0-shot GPT-4... FT RoBERTa and prompt tuning RoBERTa can achieve ~97% score”***\n\nFor the 0-shot prompt, we use the instruction in the Appendix. FT and prompt tuning use gradient-based optimization and thus achieve a higher score. The following paper shows the superiority of gradient-based optimization wrt 0-shot and in-context learning for NLP tasks https://arxiv.org/abs/2205.05638. Our new results use the latest GPT-4 version available on the API, the score improved to 74.4% with this new model but still significantly far from ~97% score that you suggest.\n\n---\n\n***“Typo in Algorithm 2”***\n\nWe fixed the typo in Algorithm 2, line 17-18 $n$ should be $N$, thanks for noticing!"}, {"review_id": "J3UPaoAcbI", "reviewer": "Reviewer_Ttvx", "summary": "The authors provide an interpretation of LLMs as shallow language networks. They explained how One-layer language networks can be used for joint prompt training, and then moved to stacked (two) language networks. The authors propose to use variational inference for the training, and figured out a few practical instantiation that makes the optimization work well.\n\n", "questions": "Can the authors explain the results in Table 2 with richer information. Why does the approach seem to work very well on Nav and Subj, but worked clearly worse than other strong baselines? The 2-layer DLN end-to-end worked badly presumably due to the difficulty of optimization mentioned in section 4?", "limitations": "The authors present some bullet-points on potential limitations in the draft, I did not identify significant negative societal impacts. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Viewing LLMs as stochastic layers and prompts as tunable parameters, the authors provide a framework for joint prompt training of stacked LLMs using variational inference. The overall idea looks sound, and the framework and variational inference solution look elegant. The authors also experimentally verify their Deep Language Network on a few artificial datasets, and achieve comparable performance. The ablation study aligns with the authors' claim.", "weaknesses": "-- As also claimed by the authors, it is difficult to make Eq1 (the bound) tight, and the optimization is also not straight-forward. In order to make the whole system work, a lot of practical instantiation is needed, making the whole system more cumbersome. In order to get diverse prompts and identify the near-optimal prompts, I expect the value of K and N in algorithm 2 will not be too small. An ablation study looking into that, to study the computational cost and performance effect given different K and N could be interesting.\n\n-- Also, this work only evaluates on small scale datasets, and works on some artificial tasks. This may prevent the audience from understanding the benefit of the proposed methods. Is it because the training/evaluation is not so efficient, so the authors decided to work on a small scale?\n\n-- I wonder if the proposed methods can be easily extended to three layers or beyond. Technically, it seems possible, but the optimization could be very challenging. \n\n-- In line 186, the authors mentioned that they selected lambda for each task. This could be cumbersome when working on a new task/dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and insights.\n\n---\n\n***“An ablation study to study the computational cost and performance effect given different K and N could be interesting.”***\n\nAn ablation study would be absolutely interesting and we will add this to the camera ready. Empirically, we find that setting the number of prompt samples `N` 10/20 is enough. We found this number by optimizing dev. Hyperbaton performance. We suspect that more prompts won’t help due to diversity issues in the prompt proposal. Augmenting the hidden samples `K` is costly especially if every hidden layer generation gets long. We found `K` ~5/10 to work well across tasks. We found this number by optimizing Navigate performance.\n\n---\n\n***“Can the authors explain the results in Table 2 with richer information. Why does the approach seem to work very well on Nav and Subj, but worked clearly worse than other strong baselines? The 2-layer DLN end-to-end worked badly presumably due to the difficulty of optimization mentioned in section 4?”***\n\nThank you for noticing this. For Date, we noticed a dataset problem, where the dev set is very small. Sometimes DLN-2 would get 90% on dev and 40% on test. For Logic.7, we think that the base model (dv003) is simply not strong enough to perform such reasoning and therefore we would need a \"deeper\" DLN where each step does a simpler reasoning step. Note that Logic.7 is the hardest of the logic datasets in BBH.\n\n---\n\n***“I wonder if the proposed methods can be easily extended to three layers or beyond. Technically, it seems possible, but the optimization could be very challenging.”***\n\nWe agree. It can be very challenging. Notably, one the most challenging things is ensuring enough diversity in the prompt proposals. However, we think that in industrial applications, the optimization won’t start de novo, but from a set of carefully human designed prompts. In that setting, DLN can be considered as a way to fine-tune this well-thought initialization, in a principled way. Future work should be concerned in finding such scenarios which are pervasive in industry but less so in academia."}, {"review_id": "dVuKjSTmZw", "reviewer": "Reviewer_SaT7", "summary": "This paper suggests stacking multiple large language models (LLMs) together, with tunable parameters represented as prompts at each layer. Given that these prompts are discrete natural language elements, their direct optimization using a gradient-based method is challenging. To address this, the authors propose a prompt optimization framework: (1) generating N local candidates using a proposal distribution, and (2) scoring each candidate to select the one that maximizes the scoring function. To apply this framework to a 2-layer setting, they propose a variational inference objective introducing an additional hidden proposal. The effectiveness of this method, tested on multiple datasets, is demonstrated in both 1-network and 2-network settings.", "questions": "In the context of variational inference, is the LLM that parameterizes q(h) separate?\n\nA straightforward baseline would be COT+APE. How does the performance of this simple baseline compare?\n\nStacking multiple LLMs increases computational complexity. An alternative is to directly use an ensemble of multiple LLMs. How does the proposed method compare with a simple ensemble of multiple LLMs? This could offer insights into whether LLMs should be combined horizontally or vertically.", "limitations": "Please refer to the Weaknesses section for a detailed discussion of the paper's limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed prompt optimization framework is both general and plausible, with its extension to the 2-layer setting using variational inference technically sound. The exploration of stacking multiple LLMs together is an intriguing and under-researched area. The paper's comprehensive experimentation on various datasets effectively showcases the method's applicability in both 1-layer and 2-layer settings.", "weaknesses": "The paper's title, \"Deep Language Networks\", is misleading, as the main content only employs 2-layer LLMs. The term 'deep' should be avoided to prevent any misconceptions and exaggeration.\n\nWhile the introduction claims that existing modular approaches are heavily dependent on prompt engineering to break a task into smaller tasks, the proposed method also relies on an initial given prompt. This initial prompt prescribes the subtask for each layer. For instance, in the 2-layer setting experiments, the first layer always serves as the COT step, and the second layer provides the final answer. The roles of the two layers are determined by the initial human-provided prompt.\n\nThough the paper suggests that it's easy to generalize from a 2-layer network to multiple layers, it's not straightforward to establish the initial prompts for each layer. While it's manageable in a 2-layer setting, mirroring the COT steps where the first layer handles reasoning and the second layer provides the final answer, it becomes substantially more challenging as the layer size increases. Therefore, the main motivation of this paper may not be as practically applicable as suggested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n***“The paper's title, \"Deep Language Networks\", is misleading given that it’s only 2 layers”***\n\nOur title expressed the conceptual framework that originated the idea of the variational inference prompt optimization algorithm, which is our main contribution. While we acknowledge that our models are not “deep”, the algorithm can be readily extended to multiple layers, at least theoretically, as you can find in the Appendix. If the reviewer thinks it’s useful, we could either remove the mention of “Deep Language Networks” in the title and write “Two-Layer Language Networks”, or specify “Towards Deep Language Networks..”.\n\n---\n\n***“While the paper argues that modular methods are heavily dependent on prompt engineering to break a task into smaller tasks, the proposed method also relies on an initial given prompt.”***\n\nWe agree! We rewrote our introduction and pointed out that our method is also relying on initial prompts and thus can be seen as a step towards **“integrating learnable components in a human-designed pipeline of prompts”**: in fact, we see our algorithm as a principled way to fine-tune initial prompts that were originally human generated.\n\n---\n\n***“The paper claims that it's easy to generalize from a 2-layer network to multiple layers”***\n\nWe apologize for the confusion: we did not mean to claim that it is practically easy to optimize more than 2 layers: indeed, we are open in discussing the optimization difficulties encountered in the two layer case. The algorithm is readily extendable to multiple layers as we point out in the Appendix. We hope that our paper can be considered as a starting point from which we can think about training more than two layers in a principled way. We have rewritten our intro to make sure the training challenges with longer chains are clear.\n\n---\n\n***“It's not straightforward to establish the initial prompts for each layer.”***\n\nWe added a discussion about this point in the paper. Engineering prompts is arguably easier than choosing initializations of random neural network weights: this has long remained a difficult question. When working with commercial applications and engineering libraries such as, e.g. LangChain, designers manually craft pipelines of prompts. One potential straightforward application of our prompt optimization algorithm is to fine-tune an already carefully human engineered chain of prompts given some task data.\n\n---\n\n***“COT+APE / Ensemble of multiple LLMs”***\n\nThank you for suggesting the baselines. For CoT+APE, we followed the procedure described in Section 4.3 of [Zhou et al., 2023], that is we use APE to find a prompt starting with “Let’s” that maximizes the likelihood of correct chain-of-thought reasoning steps. We ran that baseline on the four BBH datasets and observed lower performance except for the date dataset. Looking at the generated prompts from CoT+APE, they are much simpler than what we can obtain with DLN. For instance, on hyperbaton: “Let’s review the correct adjective order in English” vs the example shown in appendix D. Additionally, we also report the scores obtained by a system ensembling three DLN-1, we train the three DLN-1 individually and ensemble by majority voting during test. These results can be found in ***Table A/B*** in the shared response.\n\n---\n\n***“In the context of variational inference, is the LLM that parameterizes q(h) separate?”***\n\nWe use the same LLM (davinci-003) but in principle it can be separate, q(h) could either be a bigger LLM or a smaller LLM. Understanding the implication of this is definitely an interesting direction for future work."}, {"review_id": "TEhZQSAzUn", "reviewer": "Reviewer_Y4LZ", "summary": "The paper introduces a framework called Deep Language Network (DLN) that involves stacking multiple large language models (LLMs) as stochastic layers in a deep network. The prompts at each layer serve as tunable parameters, and the output of one layer is fed as input to the next layer. The LLMs are trained jointly using variational inference. The DLN architecture achieves higher performance than a single layer and can sometimes match the performance of larger and more powerful models like GPT-4. The authors discuss the analogy between LLMs and traditional parametric and nonparametric models, explore the limitations of LLMs, propose the use of variational inference for training DLN, and demonstrate the performance of DLN on various datasets. They suggest that DLN can serve as a framework for characterizing and optimizing prompt learning techniques. The paper also discusses future directions such as testing with different language models, fine-tuning stackable LLMs, and expanding DLN to accommodate arbitrarily directed acyclic graphs. The authors hope that DLN and modular approaches like theirs will address the challenges associated with LLMs and make them more adaptable to different use cases. However, they acknowledge the limitations of technical solutions and the need to consider deployment and ethical considerations when using such models in real-world applications.", "questions": "Can you provide more details on the computational and memory requirements of the proposed Deep Language Networks (DLNs) compared to single-layer models?\n\nAre there any specific challenges or limitations associated with implementing and training DLNs that were not mentioned in the paper?\n\nCould you include a more comprehensive comparison of DLNs against other state-of-the-art language models or architectures? This would help understand the relative performance and advantages of DLNs in a broader context.\n\nAre there any plans to explore alternative metrics in future work?", "limitations": "Yes, the content you provided appears to adequately address the requirement of addressing the limitations and potential negative social impact of the authors' work. The authors acknowledge the limitations of addressing societal issues through technical work and emphasize that their modular approach aims to alleviate some of the issues associated with large language models (LLMs), such as concentration of power and difficulty in training. They also express the hope that their approach will make LLMs more adaptable and suitable for a wider range of use cases. However, the authors explicitly state that they do not address the deployment of such models, when and how they should be used, or provide additional guarantees against their misuse. They highlight the importance of considering the performance of these models in uncontrolled environments and the need for justification before deploying them in high-stakes situations. By acknowledging these limitations and potential negative societal impacts, the authors demonstrate transparency and responsibility. They recognize the need for further consideration of ethical and societal implications beyond the technical performance of their work. The content provided covers the requirement and fulfills the guidelines for addressing limitations and broader societal impacts as specified in the checklist.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper introduces the concept of Deep Language Networks (DLNs), which stack multiple Large Language Models (LLMs) and train them jointly using variational inference. This approach offers a new perspective on leveraging the power of LLMs and demonstrates improved performance compared to a single-layer model. It presents a layered architecture that decomposes the task into a series of smaller sub-tasks, each of which is more easily solvable by an LLM. This decomposition allows for more efficient training and better performance. It details prompt engineering and in-context learning techniques for optimizing the prompts associated with each layer of the DLN. These techniques enable fine-tuning of the DLN's performance and allow for better adaptation to different tasks and datasets. The paper utilizes variational inference to learn the prompts in the DLN. This approach enables the joint search over the prompts, addressing the challenge of optimizing multiple prompts in a deep architecture. The paper provides experimental results on various datasets, demonstrating the effectiveness of DLNs compared to single-layer models. The results show that DLNs can achieve performance comparable to higher-capacity models like GPT-4, even when each LLM in the network is smaller and less powerful. It discusses practical aspects of DLN implementation, such as proposal diversity, learning in-context learning, and backtracking and memory optimization. These considerations make the DLN approach more feasible and effective in real-world scenarios. The strengths of the paper lie in its innovative approach, theoretical framework, practical insights, and experimental validation, showcasing the potential of Deep Language Networks for improving language modeling tasks.", "weaknesses": "The proposed Deep Language Networks (DLNs) introduce a more complex architecture compared to single-layer models. This complexity may make the implementation and training of DLNs more challenging, requiring significant computational resources and expertise. Although the paper compares DLNs to single-layer models, it does not provide a comprehensive comparison to other state-of-the-art language models or architectures. Without such comparisons, it is difficult to assess the relative performance and advantages of DLNs against other advanced models. Additional evaluation metrics would provide a more comprehensive assessment of DLNs' strengths and weaknesses. The experiments and results presented in the paper may not generalize to all types of language modeling tasks and datasets. The performance of DLNs could vary depending on the specific domain, language, or task requirements. The paper should provide a detailed analysis of the limitations and potential failure cases of DLNs in different scenarios. DLNs consist of multiple layers of Large Language Models (LLMs), which can increase the computational overhead during training and inference. It does not extensively discuss the computational efficiency of DLNs, including the training time, memory requirements, and inference latency. These factors could limit the scalability and applicability of DLNs in resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! We are glad that you appreciated our work.\n\n---\n\n***“comparison to other state-of-the-art language models or architectures”***\n\nWe provide additional experimental results as listed in the general response. Please refer to ***Table A/B/C/D*** in the general response for these comparisons.\n\n---\n\n***“analysis of the limitations and potential failure cases of DLNs in different scenarios”***\n\nWe will discuss limitations of DLNs more in depth in the camera ready. Notably, one of the biggest limitation is that the base LLM cannot be *too small*. This can be seen in Logic.7 for example, which is the hardest task in our benchmark, where every layer must do a notable amount of computation. So there is a tradeoff between capacity of the base LLM and depth of the DLN that is just not discussed enough in the paper right now and that we will address in the camera ready.\n\n---\n\n***“Computational efficiency, including the training time, memory requirements, and inference latency”***\n\nMemory: We emphasize that DLN does not have neural net components per se, so there is no GPU needed on a local machine. We require access to a endpoint that can be either local or remote. If the endpoint is local, this needs to be powered by local GPU resources of course.\n\nTraining time / inference latency: we are strongly dependent on traffic of OpenAI API. Our DLN-2 results roughly take 1 hour to execute (when without too much traffic). \n\nWe are working to relax DLN's dependence from online APIs, specifically, we are looking into applying DLN on open sourced LLMs, such as WizardLM. Please refer to ***Table D*** in the general response for some preliminary results. \n\n---\n\n***“Alternative metrics”***\n\nTo our understanding, the reviewer refers to the scoring functions DLN uses to obtain training signals. Indeed, assuming access to output log probabilities somewhat makes DLN more difficult to use blackbox LLMs (e.g., GPT-4) as backbone. We agree alternative training signals could be a useful future direction. In fact, we are actively exploring alternative ways to help DLNs learn. In our additional experimental results (***Table C***), we include a variant of DLN-1 using GPT-4 as backbone. Specifically, we use accuracy as the final scoring function in that setting. Devising a scoring function for the hidden layer in the case where the log-probabilities are not available (i.e. for GPT-4) is a future direction."}]

Author rebuttal:
We thank the reviewers for their comments. In addition to the replies to the specific questions raised by each reviewer, we want to address points that were raised multiple times.

---

***About the discrepancy between the ambition of our proposed framework and its technical instantiation***

While the framework can technically accommodate more than two layers, we acknowledge that we limit its instantiation to 1 and 2-layer networks. Some of the barriers to extending this framework to more general architectures, like the quality of the variational inference, will likely be challenging. We will rephrase our claims and be more explicit about the difference between the theoretical approach and what has currently been tested. While we believe this framework is both conceptually interesting and has already delivered some interesting results, we do not want to downplay the challenges to overcome to instantiate it more generally, nor the limitations of our current analysis, mainly due to the large computational costs of working with models of this scale.

---

***Regarding the lack of comparison with more models and baselines***

In addition to reemphasizing that we did compare with several prompt engineering and few-shot techniques in the original submission, we added comparisons to more models, also using different LLMs as language layers, as allowed by our framework. Particularly we added results for:
- APE+CoT
- ICL (in-context learning) and KATE with more in-context examples
- DLN-1 ensemble
- DLN-1 on top of GPT-4
- DLN-1 for open-source models such as [WizardLM-13b-v1.2](https://github.com/nlpxucan/WizardLM), an open sourced LLM developed based on Llama-2.

We cannot run DLN-2 on top of GPT-4 due to the lack of available log-probabilities. We couldn’t compute DLN-2 for open-source models in time for the rebuttal, but we will report the results in the camera ready version. We also report the cost (lower is better) in terms of total number of tokens for the test set (prompts included, #tok). We emphasize the cost at the testing time because it is more relevant in real-world deployment and the training cost is one-off. Note that: 1/ We used the newest GPT-4 version (0613) in Table C; 2/ we re-optimized DLN-2 E2E hyper-params on the datasets tested and got improvements, especially for navigate.

Here are all the results:

---

| ***Table A*** | nav | date | logic 7 | subj |
| - | - | - | - | - |
| ICL - 5-shot | 56.5 | 62.1 | 36.7 | 76.4 |
| ICL - 10-shot | 61.3 | 62.9 | 38.9 | 72.0 | 
| ICL - 32-shot | 66.0 | 63.5 | exceeds ctx len | 83.2 |
| KATE - 5-shot | 56.9 | 61.1 | 44.4 | 71.1 |
| KATE - 10-shot | 59.5 | 62.0 | 41.6 | 73.9 |
| KATE - 32-shot | 67.5 | 62.8 | exceeds ctx len | 80.4 |
| DLN-1 | 67.1 | 55.7 | 47.5 | 83.2 |
| DLN-2 | 82.9 | 63.9 | 45.2 | 85.9 | 
| DLN-1 Ensemble-3-seeds | 65.6 | 51.2 | 46.8 | 83.2 |

***TLDR***: For Text-Davinci-003, increasing the number of ICL examples helps performance, but cannot match DLN-1, DLN-2 performance. More in-context examples could quickly use up the context length limit.

---

|  ***Table B*** | hyper | nav | date | logic 7 |
| - | - | - | - | - |
| APE-15 |  68.5 ± 5.5 | 67.3 ± 7.7 | 32.1 ± 28.6 | 45.5 ± 4.7 |
| CoT+APE | 50.9 ± 0.8 | 61.5 ± 1.5 | 58.6 ± 2.2 | 38.9 ± 1.6 | 
| DLN-1 | 91.9 ± 3.0 | 67.1 ± 7.6 | 55.7 ± 4.5 | 47.5 ± 2.1 | 
| DLN-2 | - | 82.9 | 63.9 | 45.2 |

***TLDR***: DLN-1 and DLN-2 outperform CoT+APE.

---

| ***Table C*** | nav / #eval tok | logic 7 / #tok | subj / #tok |
| - | - | - | - |
| 0-shot GPT-4 | 73.8 / 16k | 62.5 / 50k | 74.4 / 13k |
| ICL-5-shot GPT-4| 72.6 / 83k | 60.0 / 291k | 90.4 / 66k |
| ICL-16-shot GPT-4| 77.1 / 233k | 62.2 / 822k | 93.8 / 181k |
| DLN-1 GPT-4| 78.5 / 86k | 64.0 / 80k | 90.6 / 57k |

***TLDR***: DLN-1 improves over ICL on 2 out of 3 tasks on GPT-4 at a comparable token cost. Some tasks do not benefit from ICL (i.e. reasoning tasks) while other tasks like SUBJ benefit significantly. In reasoning tasks, DLN-1 outperforms ICL even for GPT-4 by learning strategies to solve the problem. We combined DLN-1 and 5-shot ICL for SUBJ and matched 16-shot performance at a lower token cost, i.e. DLN-1 GPT-4 + 5-shot: 93.8 / 116k

---

| ***Table D*** | nav / #eval tok | logic 7 / #tok | subj / #tok |
| - | - | - | - |
| 0-shot WizardLM-v1.2 13B | 58.0 / 32k | 0.0 / 54k | 65.8 / 14k |
| ICL-5-shot WizardLM-v1.2 13B| 56.0 / 240k | 28.0 / 318k | 50.8 / 74k |
| DLN-1 WizardLM-v1.2 13B | 61.1 / 48k | 31.0 / 69k | 79.8 / 17k |

***TLDR***: Open models seem significantly less able to capture few-shot examples in the context. DLN-1 outperforms ICL on all tasks here.

Meta-review: The paper proposes to do prompt engineering for a single language model and a stack of two language models. The former is just done through a Monte Carlo sampling (sample prompts which are modifications of other prompts using the language model itself, and then chose the one with the best scores) and search type strategy; the latter is done by a variational technique. In both cases the search is over the discrete prompts.

They show results that are sometimes better than the baselines; but they admit that a couple of the datasets have issues (For Date, we noticed a dataset problem, where the dev set is very small. Sometimes DLN-2 would get 90% on dev and 40% on test. For Logic.7, we think that the base model (dv003) is simply not strong enough to perform such reasoning and therefore we would need a "deeper").

The reviewers didn't like it that they called it Deep Language Networks because it wasn't really deep. Others didn't like it because they thought the paper didn't do enough ablation experiments, in their opinion. They also thought that one of the reasons this method did better was that it saw a lot more data in the samples. The authors ended up doing a lot of the experiments that the authors asked for. But it complicates the story and how good the paper turns out on how the presentation of all these new experiments sits with the rest of the paper. Nevertheless the method presented is novel, and gives some rigor to the prompt-engineering process. And the results, while mixed some times, were indicative that the method could indeed work. Authors did perform a lot of the ablations that the reviewers asked for, and I hope the authors will do a decent job of incorporating them.
Decision: Accept (poster)