SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Prompt initializations.
Task	Initialization
Mpqa	Read the following review, then choose whether it is negative or positive.
Trec	Read the following question, then choose whether it is about a description, entity, expression, human, location or number.
Subj	Read the following sentence, then choose whether it is subjective or objective
Disaster	Read the following sentence, then choose whether it is relevant to a disaster.
Airline	Read the following sentence, then choose whether it is positive, negative, or neutral.
Hyper.	Which sentence has the correct adjective order.
Nav.	Read the following sentence, then determine whether you return to the starting point.
Date.	Infer the date from context.
Logic.7	The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph.
Table 5: Test accuracy averaged over three random seeds. We compare the layerwise and the end-to-end trainings for DLN-2.
Nav.	Date.	Logic.7	Subj
DLN-2 (fix 2nd)	73.1	61.6	43.3	80.2
DLN-2 (fine-tune 2nd)	76.4	62.8	40.7	84.5
DLN-2 (end-to-end)	83.1	75.2	45.7	85.9
(empty string) for Date. and Logic.7. We didn't try other initializations for this hidden layer, we leave this for future explorations.
In addition to the prompt engineering and few-shot baselines from the main paper, we include here comparisons to more of those on a subset of the datasets. Particularly we added results for:
- Table 6 shows DLN-1 and DLN-2 outperforming CoT+APE (implemented as described in Section 4.3 of Zhou et al. [57]);
- Table 7 shows that even increasing the number of examples for ICL and KATE cannot match DLN-1 and DLN-2 performance;
- Table 8 and Table 9 show results for DLN-1 using open-source language models such as WizardLM-13b-v1.2 [51] and LLaMA2-70B-Chat [45] respectively.
Layerwise training of DLN-2 We also explored a different learning strategy for DLN-2: layer-wise pre-training. We start the learning of a single layer DLN using the technique from Section 2. We call the prompt obtained at the end of this optimization  $\pi^{*}$ . Then, we train a DLN-2 by initializing  $\pi_{1} = \pi^{*}$ , and then learn  $\pi_{0}$ , the parameters of the bottom layer, using variational inference. We explore two variants: one keeps the last layer fixed and one it fine-tunes the last layer. We report their results in Table 5.
Table 6: Test accuracy averaged over three random seeds with  $95\%$  confidence interval. All models use GPT-3. DLN-1 and DLN-2 outperform CoT+APE.
Method	Hyper.	Nav.	Date.	Logic.7
APE-15	68.5±5.5	67.3±7.7	32.1±28.6	45.5±4.7
CoT+APE	50.9±0.8	61.5±1.5	58.6±2.2	38.9±1.6
DLN-1	91.9±3.0	68.5±4.7	55.7±4.5	47.5±2.1
DLN-2	-	83.1±24.7	75.2±14.8	45.7±3.5
Table 7: Test accuracy averaged over three random seeds with  $95\%$  confidence interval (where applicable). All models use GPT-3. Increasing the number of ICL examples helps performance, but cannot match DLN-1 and DLN-2 in general. Context length limit is an issue for ICL and KATE 32-shot.
Method	Nav.	Date.	Logic.7	Subj
ICL - 5-shot	56.5	62.1	36.7	76.4
ICL - 10-shot	61.3	62.9	38.9	72.0
ICL - 32-shot	66.0	63.5	-	83.2
KATE - 5-shot	56.9	61.1	44.4	71.1
KATE - 10-shot	59.5	62.0	41.6	73.9
KATE - 32-shot	67.5	62.8	-	80.4
DLN-1	68.5±4.7	55.7±4.5	47.5±2.1	83.2±5.5
DLN-2	83.1±24.7	75.2±14.8	45.7±3.5	85.9±8.7
Table 8: Test accuracy using WizardLM-v1.2 13B as LLM. This open source model seems significantly less able to capture few-shot examples from the context. DLN-1 outperforms ICL on all tasks here.
Method	Nav.	Logic.7	Subj
0-shot	58.0	0.0	65.8
5-shot	56.0	28.0	50.8
DLN-1	61.1	31.0	79.8
Table 9: Test accuracy averaged over three random seeds with  $95\%$  confidence interval. All methods use LLaMA2-70B-Chat as LLM, with DLN + GPT3 employing text-davinci-003 as the backward LLM for prompt and hidden proposals.
Method	Nav.	Date.	Logic.7	Subj
0-shot	42.0±0.0	25.2±0.0	14.4±0.0	62.4±0.0
5-shot	43.2±12.5	21.1±9.7	16.4±2.6	67.7±15.5
DLN-1 + GPT3	43.6±4.0	21.9±5.7	33.1±10.9	80.9±11.5
DLN-1	44.9±6.4	31.6±10.5	38.4±3.6	76.1±4.5
DLN-2 + GPT3	43.7±3.0	51.1±4.0	21.9±4.9	59.1±16.7
DLN-2	68.9±14.4	61.7±17.6	20.0±13.7	63.1±25.4
D Templates
D.1 Forward Templates
"Classification" Template F for 1-Layer LN In the 1-layer LN, we use the following template to elicit the output  $y$  given the input  $x$ . prompt is substituted with the value of the current prompt.
Classification Template F
template:
    {{ prompt}}
    {{ input}}
    Answer:
Residual Classification Template  $\mathbf{F}_r$  for 2-Layer DLN In the 2-layer LN, the last layer just concatenates the input to the output of the first layer,  $\hat{h}$ , before eliciting an answer.
Residual classification template  $\mathbb{F}_r$
template:
    {{ prompt }}
    {{ input }}
    Your thoughts were:
        {{ h }}
    Answer:
Hidden Layer F The variable prompt is substituted with the value of the current prompt  $\pi_0$ . This has the effect of providing additional information about how "Let's think step by step" should behave.
Hidden Layer F
template:
    {{{input}}}
    {{{prompt}}} Let's think step by step.
For 2-Layer DLN on Subj, we use the following hidden template. We couldn't run with the previous template due to lack of time, as we observed that the step by step trigger tended to generate lengthy hidden states.
template:
    {{ prompt}}
    {{ input}}
    Brief Analysis
D.2 Backward Templates (Prompt and Hidden Proposals)
Prompt Proposal Template  $\mathsf{B}_{\pi}$ $\mathsf{B}_{\pi}$  is used to propose new candidate prompts. The template takes as input the current prompt, prompt, and a mini-batch of examples stored in backward_infos. backward_info. input stores the input to the layer ( $x$  if we are proposing prompts for the first layer or  $h$  if it is the second layer); backward_info. target stores the target to the layer ( $h^*$  if it is the first layer or  $y$  otherwise); backward_info. output stores the predictions of the model during the forward pass ( $\hat{h}$  if it is the first layer, and  $\hat{y}$  if it is the second layer). message is substituted with one of the message_alternatives, sampled at random during the DLN training. This induces diversity in the generated prompts and allows emergence of learning to ICL behaviors, where the prompts contain synthetic examples that can help solve the task.
Prompt Proposal Template  $\mathsf{B}_{\pi}$
A student is completing a task that requires producing a text output from a text input. The student receives an instruction that describes how to produce the output given each input.
The student has made some errors. Your task is to improve the instruction such that the student can fix the errors.
This was the instruction.
## Instruction
> {{prompt}}
[END]
Student successes
$\{\%$  for backward_info in backward_infos  $\%\}$ $\{\%$  if backward_info.loss  $= = 0.0\%$  ##Input:
$>\{\{$  backward_info-input  $\}\}$
## Correct Output:
> {{backward_info.target}}
$\{\%$ endif  $\{\%$  endfor  $\{\% \}$
Student errors
$\{\%$  for backward_info in backward_infos  $\%\}$ $\{\%$  if backward_info.loss  $>0.0\%$  ] ## Input:
> {{backward_info-input}}
Student Output:
```python
>>> [backward_info.output]]
## Correct Output:
> {{backward_info.target}}
$\{\%$ endif  $\%\}$ $\{\%$ endfor  $\%\}$
Improve the instruction to fix the student errors. {{message}}
message_alternatives:
- Clarify the instruction by adding few words or a short sentence. Be concise.
- Improve the instruction by providing examples on how to solve the task. Be concise.
- Shorten the instruction by removing superflous words or sentences.
- Rewrite the instruction by providing detailed information to avoid ambiguity. Be concise.
Backward Hidden Templates  $\mathsf{B}_h$  We experiment with multiple backward templates to sample hidden states from the approximate posterior distribution  $q(h)$ . Each vary in the amount of conditioning information. The simpler way of sampling hidden states is to use the same hidden template as in the forward pass. This corresponds to using a posterior distribution  $q(h)$  which is equivalent to the prior distribution  $p(h)$ . The other alternative is to condition the posterior template on the answer  $y$ , as illustrated below:
Backward Hidden Template  $\mathbb{B}_y$  (y conditioning)
{{input}}
Given that the answer is:
{y}
{{prompt}}Let's think step by step.
The next alternative we experiment with is a more verbose template that takes as input the prompt for the final layer  $\pi_1$  in next_prompt, the input  $x$  in input, the hidden state from the forward pass  $\hat{h}$  in h and the ground-truth output  $y$ . We use a similar strategy of sampling different message alternatives to substitute with message to increase diversity of the hidden samples:
Backward Hidden Template  $\mathsf{B}_h$
This is the context needed to solve the problem: {{ next_prompt }}
This is the problem:
These were your thoughts:
{h}
Given that this is the answer:
{{message}}
- Reflect and refine your thoughts for this problem by adding detailed explanations.
- Fix the errors in your reasoning. Add examples to illustrate your thoughts. Be concise.
In practice, we found that sampling from hidden states from a mixture of forward template  $\mathbb{F}$ , i.e.  $p_{\mathrm{LM}}(h|x,\pi_0)$ , and backward template with  $y$  conditioning, i.e.  $q(h|x,y,\pi_0)$  works well. We suspect that this has the effect of capping the KL divergence term between posterior and prior distribution, i.e. the  $\mathrm{KL}(q(h)||p_{\mathrm{LM}}(h|x,\pi_0))$  appearing in the ELBO. In the future, this could be addressed in a more principled way by learning a prompt for the posterior proposal.
E Generalized VI to Multiple Layers
We report the generalized training algorithm for multiple layers in Algorithm 3.
F Learning to In-Context Learn: Additional Examples
In Figure 2, we report examples of prompts found by the 1-layer DLN on the Hyperbaton task, which exhibit either integration or verbalization of task examples. Here, we provide additional examples of prompts found by DLN-1 on the MPQA task.
DLN-1 prompt on MPQA (GPT-3)
Read each sentence, then decide if the sentence is expressing a positive or negative sentiment. For example, if the sentence is "supported", choose "positive", and if the sentence is "derail", choose "negative". Similarly, if the sentence is "victorious", choose "positive", and if the sentence is "would not be a bad idea", choose "positive". Additionally, if the sentence contains multiple words, consider the overall sentiment of the sentence and choose the appropriate option. For example, if the sentence is "counting on", choose "negative", and if the sentence is "peace and stability and prosperity", choose "positive". Note that words like "artificial" tend to have a negative sentiment.
DLN-1 prompt on MPQA (GPT-4)
Determine whether the given input has a positive or negative connotation by analyzing the meaning of the words and phrases in context. If the input expresses a favorable, desirable, or pleasant meaning, choose "positive." If the input expresses an unfavorable, undesirable, or unpleasant meaning, choose "negative." Consider the overall sentiment expressed by the input rather than focusing on individual words or phrases. For example, "calling for" generally has a positive connotation as it implies advocating or supporting something, while "a true Muslim fighter" can be seen as positive, since it refers to someone dedicated to their beliefs. Keep in mind that some phrases may have a positive connotation when they imply improvement or resolution, like "put an end to." Additionally, phrases like "to the contrary" can have a positive connotation when they suggest a differing, yet valid perspective or opinion. When analyzing the input, consider the context in which it is used, as the connotation of a word or phrase can change depending on the situation. For example, "extra vigil" can have a positive connotation when it implies increased awareness and preparedness.
G Examples of 2-Layer Best Weights (GPT-3)
G.1 Navigate (81.6% Dev Acc)
DLN-2 Prompt:  $\pi_0$
Decompose the problem by breaking it down into steps and describing the new position after each step. Note that the direction you are facing stays the same unless specified. Start by specifying the direction you are facing and the coordinates of the starting point. For each step, describe the number of steps taken and the direction of movement (e.g. North, South, East, or West). Specify the new coordinates and direction after each step.
DLN-2 Prompt:  $\pi_1$
Start facing north. Take the specified number of steps in the indicated direction and turn when specified. Make sure to keep track of your direction and the number of steps taken to ensure you return to the starting point.
G.2 Subj (89.9% Dev Acc)
Reflect on the input to produce an output that is either objective (factual) or subjective (involving opinion or value judgment). Objective statements describe events or facts that can be verified, while subjective statements express opinion or personal feelings about a fact, event, or situation that cannot be confirmed as an accurate statement of fact. For example, if the input is "the film was a success at the box office," the output would be "This statement is an objective fact, as it describes events that can be verified." If the input is "the film was an amazing experience," the output would be "This statement is subjective, expressing a personal opinion about the film in question that reflects approval and judgement, and cannot be confirmed as an accurate statement." If the input is "neither Juan Antonio nor Señor Maximiliano know what they are in for when the tables are turned by the sly Carmen," the output would be "This statement is an objective fact, as it describes events that can be verified regarding the actions of Juan Antonio, Señor, and Maximiliano, and the change of circumstances caused by Carmen." If the input is "humorless, self-conscious art drivel, made without a glimmer of intelligence or invention," the output would be "This statement is subjective, expressing a negative opinion about the film in question that reflects disapproval and judgement, and cannot be confirmed as an accurate statement." If the input is a hypothetical situation, such as "how would you feel if when you woke, the nightmare had just begun?", the output would be "This statement is subjective, expressing a personal opinion about a hypothetical situation that cannot be verified."
Read the sentence. Determine if it is expressing a fact or opinion. A fact is an accurate statement that can be confirmed, while an opinion is a personal viewpoint that reflects someone's beliefs. Facts are typically statements about something that happened, such as events, actions, or conditions, or statements that describe a state of being, such as someone's personality or a physical object. Opinions are typically statements that express judgement, approval, or disapproval. Examples of facts include statements about events that occurred, such as "the film was released in 2012," or statements about conditions, such as "the weather is sunny," or statements that describe a state of being, such as "the detective is strong and independent." Examples of opinions include statements about how good or bad something is, such as "the film was terrible" or "the weather is beautiful." Examples of questions that are not facts or opinions include hypothetical questions, such as "how would you feel if when you woke, the nightmare had just begun?" Select "objective" for a fact and "subjective" for an opinion.
H Examples of Hidden States
We report the forward pass produced by a 2-Layer DLN in Section G.1 on Navigate below.
Input  $x$
If you follow these instructions, do you return to the starting point? Take 3 steps. Take 10 steps. Take 4 steps. Take 1 step.
Options:
-Yes
- No
1. Take 3 steps: (3, 0) facing east
2. Take 10 steps: (13, 0) facing east
3. Take 4 steps: (17, 0) facing east
4. Take 1 step: (18, 0) facing east
Answer: No, you do not return to the starting point.
Table 10: Hyperparameter search space.
hyperparam	search space
1-Layer LN
bh_tpl	q_action_prompt:v3.0, q_action_prompt:v3.5
tolerance	-1, 0, 2
use_memory	0, 2
held_out_prompt_ranking	True, False
2-Layer DLN PT + fix 2nd layer
bh_tpl	q_action_prompt:v3.0, q_action_prompt:v3.5
logp_penalty	0., 0.5, 2.
2-Layer DLN PT + fine-tune 2nd layer
bh_tpl	q_action_prompt:v3.0, q_action_prompt:v3.5
logp_penalty	0., 0.5, 2.
2-Layer DLN end-to-end
num_h_samples	5, 10
Output  $\hat{y}$
We report hyperparameter search space in Table 10. A brief description of the hyperparameters is as follows:
- bh_tpl is the type of backward prompt template we use  $\mathsf{B}_{\pi}$ . v3.5 is equal to the  $\mathsf{B}_{\pi}$  we report in Section D. In v3.0, we remove "Be concise." at the end of each message_alternatives. We noticed that in general v3.5 works better as it implements a sort of regularization on the length of the found prompts. Future work could address length regularization in a more principled manner.
- logp_penalty is the coefficient for the exploration reward we mentioned in the paper.
- num_h_samples is the number of  $h$  samples to generate from the approximate posterior distribution.
- use_memory is whether or not we use the backtracking mechanism. Usually 2 works well across tasks.
- held_out_prompt_ranking describes whether we use only half of the mini-batch examples for each prompt proposal, as described in the main paper.
- tolerance describes after how many iterations we reload the best weights found during the last validation if the current validation score is lower than the best score obtained so far.
For the 2-Layer experiments, we have to restrict this search space due to computational costs. We use bh_tpl = "v3.5", tolerance = 2, use_memory = 2, held_out_prompt_ranking = True, logp_penalty = 0.5.
J Pricing
We keep track the number of tokens we interact with GPT-3 via its online API. According to OpenAI's pricing policy, user pays for both the input tokens (prompts) and the output tokens. Using the Hyperbaton task as an example, while training a 1-layer LN, the total number of tokens we use is
Table 11: Test accuracy along with inference cost expressed in tokens (in gray) averaged over three random seeds of a shallow, 1-layer language network (DLN-1) compared to baselines on GPT-4. We also report the  $95\%$  confidence interval on the test accuracy. We emphasize the cost at the testing time because it is more relevant in real-world deployment and the training cost is one-off.
BigBench Hard	NU	Leopard
Method	Hyper.	Nav.	Date.	Logic.7	Mpqa	Trec	Subj	Disaster	Airline
GPT-4
0-shot	64.0±1.0(7.6k)	74.0±1.0(12.9k)	79.2±2.6(23.6k)	68.5±3.5(46.2k)	86.3±0.6(3.7k)	64.8±1.7(7.6k)	72.5±1.5(10.2k)	47.7±0.6(10.1k)	84.5±0.6(9.9k)
5-shot	88.4±2.6(48.0k)	75.7±1.5(79.2k)	79.3±1.1(143.3k)	62.8±1.7(287.5k)	88.0±3.0(24.5k)	82.5±3.8(52.5k)	94.7±3.5(62.6k)	63.6±8.5(63.5k)	88.0±1.0(61.7k)
16-shot	93.3±2.3(136.8k)	75.5±5.1(229.9k)	80.9±5.0(405.1k)	66.4±3.6(817.6k)	91.3±1.5(70.3k)	83.7±0.6(149.0k)	96.5±2.5(177.9k)	67.1±4.0(179.4k)	88.3±2.1(175.2k)
DLN-1	95.2±5.0(77.2k)	77.1±4.7(29.9k)	74.3±1.5(52.3k)	69.1±2.5(68.5k)	91.1±3.2(65.4k)	89.5±2.1(120.7k)	93.1±5.0(46.5k)	82.1±3.8(47.1k)	85.9±1.5(38.2k)
Algorithm 3 Deep Language Network Training Algorithm
1:  $\hat{h} \sim p_{\mathrm{LM}}^t(x)$ : generates a completion of prefix  $x$  with temperature  $t$ .  
2:  $\log p_{\mathrm{LM}}(h|x)$ : return log-prob of  $h$  following  $x$ .  
3:  $N$ : # prompt samples,  $K$ : # posterior samples,  $I$ : # iterations,  $L$ : # layers,  $\mathcal{D}$ : dataset  
4: F: template for the inference (forward pass).  
5:  $\mathtt{B}_{\pi}, \mathtt{B}_h$ : templates for prompt and hidden proposal (backward pass).  
6: Initialize all layers  $\pi_l$  with a generic sentence or task description.  
7: for  $i$  in  $[1, I]$  do  
8:  $x, y \sim \mathcal{D}$   
9:  $\hat{h}_0 \gets x$   
10:  $\hat{h}_l, \dots, \hat{h}_L \gets p_{\mathrm{LM}}^0(\mathsf{F}(\hat{h}_{l-1}, \pi_{l-1}))$   
11:  $h_L^* \gets y$   
12: for  $l$  in  $[L-1, 1]$  do  
13:  $h_l^1, \dots, h_l^K \sim p_{\mathrm{LM}}^{0.7}(\mathtt{B}_h(\hat{h}_l, h_{l+1}^*, \pi_l))$   
14:  $\alpha_l^1, \dots, \alpha_l^K \gets \log p_{\mathrm{LM}}(h_l^k | \mathsf{F}(\hat{h}_{l-1}, \pi_{l-1}))$   
15:  $\beta_l^1, \dots, \beta_l^K \gets \log p_{\mathrm{LM}}(h_{l+1}^* | \mathsf{F}(h_l^k, \pi_l))$   
16:  $q_l^k \gets \exp(\alpha_l^k + \beta_l^k) / (\sum_k \exp(\alpha_l^k + \beta_l^k))$   
17:  $h_l^* \gets \arg \max_{h_l} \{q_l^1, \dots, q_l^K\}$   
18: end for  
19: for  $l$  in  $[L-1, 0]$  do  
20:  $\pi_l^1, \dots, \pi_l^N \sim p_{\mathrm{LM}}^{0.7}(\mathtt{B}_\pi(\{\hat{h}_l, \hat{h}_{l+1}, h_{l+1}^*\}, \pi_l))$   
21:  $s_l^1, \dots, s_l^N \gets \sum_k \sum_{k'} q_l^k q_{l+1}' \log p_{\mathrm{LM}}(h_{l+1}^{k'} | \mathsf{F}(h_l^k, \pi_l^n))$   
22:  $\pi_l^* \gets \arg \max_{\pi_l^n} \{s_l^1, \dots, s_l^N\}$   
23: end for  
24: end for
2,941,360. For a 2-layer DLN, the total number of tokens we use is 13,654,962. According to the current price for GPT-3 (\(0.02/1k tokens), a single run of a 1-layer and 2-layer DLN cost roughly 59 USD and 273 USD, respectively.
In Table 11, we report the cost (lower is better) in terms of total number of tokens for the test set (prompts included). We emphasize the cost at the testing time because it is more relevant in real-world deployment and the training cost is one-off. We can see DLN-1 improves over ICL on 5 out of 9 tasks on GPT-4 at a comparable token cost. Some tasks do not benefit from ICL (i.e. reasoning tasks) while other tasks like Subj, Trec, and Hyper. benefit significantly.