SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We design and conduct a set of experiments to help answer two main research questions:
- Q1: Can we outperform APE and In-Context Learning (ICL) with a DLN-1?
Q2: Does network depth provide further improvement upon DLN-1?
Datasets and Tasks We adopt a set of nine NLP and reasoning tasks commonly used in prior work studying zero- or few-shot learning capabilities of LLMs [23, 10, 39, 42, 1]. We focus on classification tasks. For tasks adopted from BigBench-Hard (BBH) [42] (Hyper., Nav., Date. and Logic.  $7^{2}$ ), we use the 250 data points provided by BBH as test set. We take the remaining data points from BigBench [39] that were not included in BBH, and randomly split them (evenly) into training and validation sets. For tasks adopted from [23] (Mpqa, Trec, and Subj), we randomly sample 400 and 250 data points from their training and test sets, respectively. We use the original validation sets. For tasks adopted from Leopard [1] (Disaster and Airline), we randomly sample 400, 250, and 250 data points as training, valid, and test. We list all tasks and their statistics in Table 3 in the Appendix.
We use accuracy as the evaluation metric. Specifically, given an input, we compare a system's output string against the ground-truth output string provided by the dataset. We score 1 if the two strings are identical and 0 otherwise. Before the comparison, we process the strings from both the model output and the ground-truth to deal with issues like tokenization and capitalization. In all our DLN experiments, we perform a hyperparameter search and run the same hyperparameter setting with three random seeds. We report the test accuracy averaged over three seeds corresponding to the hyperparameter setting that achieves the highest average validation accuracy. We report details of the hyperparameter search in the Appendix I.
Throughout this paper, we use OpenAI's models, specifically GPT-3 (text-davinci-003) and GPT-4, as the backbone to our proposed systems unless otherwise specified. For DLNs, we use a batch size of 20 and train for 20 iterations by early-stopping on validation performance evaluated every 2 iterations. We then report test scores. We sample  $N = 20$  prompt proposals and  $K = 5$  hidden samples.
Baselines We compare the DLN against two classes of baseline systems. First, we test a set of systems equipped with the same backbone (i.e., GPT-3):
- 0-shot: Given an input, the LLM is required to generate the answer in a zero-shot manner.
- 5-shot (ICL): Given an input as well as five data points as in-context examples, the LLM is queried to generate an answer. The five examples are randomly sampled from the training set.
- KATE [20]: Given an input, we retrieve the five most similar data points from the training set using an off-the-shelf sentence encoder, and use them as in-context examples.
- APE [57]: The LLM is queried to generate a pool of candidate prompts for the task given few input-output pair examples. The candidate prompts are evaluated on a validation set to find the best performing instruction prompt. The best instruction is then used for 0-shot evaluation. We optimize the prompt over both 15 and 400 examples (APE-15 and APE-400 respectively).
- CoT [48]: Given an input, the LLM is first queried to generate a reasoning path with the prompt "Let's think step by step". Then, conditioned on the input and its first output, the LLM is queried to generate an answer. This is the zero-shot version of CoT and is a natural baseline for DLN-2: it performs two LLM calls and can be seen as DLN-2 without optimization. We will report performance of this baseline when comparing to DLN-2.
Additionally, we compare against one of the most advanced LLMs to date, GPT-4. We test 0-shot and ICL settings with GPT-4.
5.2 DLN-1
Our first set of experiments evaluates the 1-layer language network (DLN-1) described in Section 2. Table 1 presents results on the full suite of test tasks. We see that it matches the performance of
2We only use the variant with seven objects.
Table 1: Test accuracy averaged over three random seeds of a shallow, 1-layer language network (DLN-1) compared to baselines both on GPT-3 and GPT-4. For trainable systems (i.e., APE and DLN-1) or systems relying on GPT-4, we report the  $95\%$  confidence interval.
BigBench Hard	NU	Leopard
Method	Hyper.	Nav.	Date.	Logic.7	Mpqa	Trec	Subj	Disaster	Airline
GPT-3
0-shot	60.8	64.1	56.4	45.9	88.0	61.9	61.7	81.6	75.6
5-shot	55.6	56.5	62.1	36.7	87.2	80.0	76.4	81.2	82.7
KATE	71.1	56.9	61.1	44.4	88.4	77.6	71.1	76.0	81.6
APE-15	68.5±5.5	67.3±7.7	32.1±28.6	45.5±4.7	85.5±4.6	71.3±5.5	61.3±7.2	54.8±14.6	83.5±3.5
APE-400	65.5±4.7	56.9±32.9	23.5±14.1	45.6±12.4	84.9±9.7	72.0±1.7	63.7±9.2	60.3±37.4	82.3±10.0
DLN-1	91.9±3.0	68.5±4.7	55.7±4.5	47.5±2.1	88.5±2.5	89.7±3.2	83.2±6.5	81.7±6.5	83.2±5.5
GPT-4
0-shot	64.0±1.0	74.0±1.0	79.2±2.6	68.5±3.5	86.3±0.6	64.8±1.7	72.5±1.5	47.7±0.6	84.5±0.6
5-shot	88.4±2.6	75.7±1.5	79.3±1.1	62.8±1.7	88.0±3.0	82.5±3.8	94.7±3.5	63.6±8.5	88.0±1.0
16-shot	93.3±2.3	75.5±5.1	80.9±5.0	66.4±3.6	91.3±1.5	83.7±0.6	96.5±2.5	67.1±4.0	88.3±2.1
DLN-1	95.2±5.0	77.1±4.7	76.7±3.0	69.1±2.5	91.1±3.2	89.5±2.1	93.1±5.0	82.1±3.8	85.9±1.5
DLN-1 prompt on Hyperbaton (GPT-3)
When constructing a sentence with multiple adjectives, the order should be opinion, size, age, shape, color, origin, material, and purpose. Adjectives of the same type should be listed in descending order from largest to smallest. When adjectives of different types are used, the order should be opinion, size, age, shape, color, origin, material, and purpose. For example, in the phrase "massive ancient chair" size (massive) should come before age (ancient). Examples: little old-fashioned Russian silver rectangular ship; silly large old leather hiking chair;brand-new spherical Mexican sweater;enormous old spherical green Nigerian exercise car; medium-size triangular wool eating ship; good square brown Egyptian ship; lovely massive drinking monkey; archaic circular white plastic shoe. In each of the following examples, the adjective order is wrong. Identify the correct adjective order:
DLN-1 prompt on Hyperbaton (GPT-4)
To determine the correct adjective order, follow this sequence: opinion, size, shape, age, color, origin, material, and purpose. For example, choose "large red plastic ball" over "red large plastic ball" since it follows the order: size (large), color (red), and material (plastic). Not all adjectives may be present, but the order should still be maintained. If the options are "ancient prismlike white leather whittling match" and "leather white ancient prismlike whittling match", choose the first option, as it follows the order: age (ancient), shape (prismlike), color (white), material (leather), and purpose (whittling). Remember that opinion always comes before age, so "obnoxious old-fashioned typing shoe" is correct over "old-fashioned obnoxious typing shoe." Ensure opinion adjectives come before other adjectives in the sequence. When comparing options, follow the order of adjectives for each category: size before color, color before origin, and so on. In cases where purpose and material adjectives are switched, like "paper walking monkey" vs "walking paper monkey", choose the option where material comes before the purpose. Additionally, always prioritize the given sequence over the position of adjectives in the sentences. For example, choose "midsize brand-new gray Chinese wood sweater" over "Chinese brand-new gray midsize wood sweater" as it follows the order: size (midsize), age (brand-new), color (gray), origin (Chinese), and material (wood).
Figure 2: The final prompt of the DLN-1 on Hyperbaton includes not only instructions but also examples from the training set. These samples were automatically chosen by the prompt optimization. In a way, this approach combines in-context learning and prompt optimization.
the best GPT-3-based method on Disaster, Mpqa and Airline and narrowly beats the best GPT-3 baseline on Logic.7 and Nav.. On Hyper., Trec, and Subj, DLN-1 significantly outperforms the best GPT-3 baseline (by about 20, 10, and 7 percentage points, respectively). On Hyper., Trec, and Disaster, it even surpasses GPT-4 baselines, unsurprisingly underperforming GPT-4 on all other tasks. DLN-1's excellent performance on Hyper., a BBH task about ordering adjectives according to linguistic convention, is a surprise. To better understand this result, we show the final prompt in Figure 2. We see that the prompt contains both instructions and a list of examples from the training set. These examples were automatically chosen by the optimizer based on their impact on the performance. This can be seen as a combination of KATE, which selects training examples to put in context based on their similarity with the test example, and APE, which selects the prompt based on its performance. On Date., DLN-1 tends to systematically under-perform the 0-shot baseline both for GPT-3 and GPT-4. We observed that DLN-1 overfits due to paucity of examples in the validation set.
Table 2: DLN-2 test accuracy using GPT-3 as LLM.
Method	Nav.	Date.	Logic.7	Disaster	Subj
0-shot	64.1	56.4	45.9	81.6	61.7
CoT	69.3	72.4	41.1	54.4	59.3
APE	67.3±7.7	32.1±28.5	45.5±4.7	54.8±14.6	61.3±7.2
APE-400	56.9±32.9	23.5±14.1	45.6±12.4	60.3±37.4	63.7±9.2
DLN-1	68.5±4.7	55.7±4.5	47.5±2.1	81.7±6.5	83.2±5.5
DLN-2	83.1±24.7	75.2±14.8	45.7±3.5	82.8±2.5	85.9±8.7
5.3 DLN-2
We investigate the effectiveness of depth through experiments with 2-layer language networks (DLN-2) on tasks where we expect depth to be most useful, and on which DLN-1 significantly underperforms the GPT-4 0-shot baseline, i.e., Nav., Date., and Logic.7 [42]. Since the Nav., Date. and Logic.7 tasks from BBH require more complex spatial and temporal reasoning, they are the ones where we most expect a decomposition into subtasks to be helpful. We also include Subj and Disaster as an example where DLN-1 performs well (even outperforming the GPT-4 0-shot baseline), since we are interested to see to what extent DLN-2 can further push performance.
Results for DLN-2 can be found in Table 2. Compared to DLN-1, DLN-2 provides an average boost of  $7.2\%$  absolute score. On Nav. and Date., DLN-2 largely improves the performance of DLN-1, outperforming all single layer networks. On Logic.7, all methods appear to perform similarly. This could point to the fact that the task might be too hard for the base LLM and thus highlights the limits of prompt optimization of a weak base model. On Subj and Disaster, DLN-2 achieves further improvement over DLN-1. Compared to 0-shot GPT-4 results in Table 1, on Subj and Disaster, DLN-2 on average provides more than  $20\%$  in absolute improvement. We encourage readers to find additional experimental results in Appendix C.
Prompt-Based Machine Learning GPT-3 [5] launched a new paradigm in NLP called in-context learning (ICL), now applied beyond traditional NLP tasks [21]. The discovery of chain-of-thought prompts (CoT) marked a major advance in prompting: LLM performance improves markedly when the prompt includes examples of intermediate reasoning steps [48] (few-shot CoT), or simply instructs the model to "think step by step" [17] (zero-shot CoT). Like CoT, DLNs break a problem down into intermediate steps but they operationalize these steps as separate LLM calls, each defined by its own learned prompt. Since the introduction of CoT, prompting techniques have evolved to be more dynamic and iterative. Recent methods often operate recursively. Examples include RECITE [41], Self-ask [33], and related methods for question-answering Creswell et al. [6], Zhou et al. [56]. A similar class of methods relies on "introspection" [14], where an LLM is prompted to ingest, evaluate then possibly act on its own previous output. Self-critique [46], ReAct [54], Reflexion [38], Self-refine [24] fit this mould along with Hao et al. [11], Du et al. [9], Yao et al. [53].
Prompt Optimization Techniques based on notions of self-talk and self-evaluation align naturally with automatic prompt optimization—a core function in DLNs. Early work in this category includes Autoprompt [37] and GRIPS [32]. Deng et al. [7] argue that 'enumeration-then-selection' heuristics for optimizing discrete prompts do not explore the prompt space systematically. They take an RL approach to overcome this problem, training a policy network, via soft Q-learning with a suitably designed and stabilized reward function, to generate effective prompts. Through Gibbs sampling, Reprompting [52] iteratively searches CoT recipes to improve prompt performance automatically. Most relevant to DLNs, Zhou et al. [57] present Automatic prompt engineer (APE). APE optimizes an initial prompt by searching over a pool of candidates to maximize a score function. We use an APE-inspired approach in DLNs and we cast the proposal/scoring functions as elements of variational inference. In a concurrent work, Pryzant et al. [34] proposed using textual gradients in automatic prompt optimization. This algorithm uses LLM's nonparametric feedback to guide prompt generation and selection.
Multi-Layer LLM systems Several recent works compose LLMs as nodes in a computational graph, which is the core idea of DLNs. Some work cited above can be seen as instances of this idea. Similarly, Khot et al. [15] induce an LLM to generate a basic "control flow" that calls distinct LLM modules. Wu et al. [50] propose AI chains, an interactive system of chained LLMs based on a set of "LLM primitive" operations. They conduct a 20-person user study in which participants modify chains, and find this process to improve task performance, transparency, and controllability. Dohan et al. [8] unify LLMs and graphical models as "language model cascades". Specifically, they cast LLM compositions as graphical models with string-valued random variables. They show how scratchpad [30], chain-of-thought [48], tool use [25], and several other prompting strategies fit their formalism. DLNs can likewise be considered an instance of language model cascade, because of that framework's generality. However, going beyond the conceptual work of Dohan et al. [8], we present an effective technique for doing inference in an LLM-based graphical model and we apply learned networks of LLMs to several downstream tasks.
In this paper we introduced an algorithm for joint prompt optimization in deep networks where each layer is an LLM. To do so, we consider outputs of each hidden LLM layer as a latent variable we need to do inference over. From a conceptual perspective, we demonstrated how CoT can be seen as a DLN-2 with a residual connection. Similarly, Generated Knowledge Prompting [19] could be considered as a fixed forward-only DLN-2 where, in the first layer, an LLM generates related knowledge, and in the second layer, another LLM takes the generated knowledge as input and generates the final answer. Other prompting techniques like ReAct [54], Reflexicon [38], and Self-Consistency [46] could all be ensembles of DLN-1s with different prompt initializations.
Although we only tested 1-layer and 2-layer LNs so far, we already show that the performance of smaller LLMs can be boosted when stacked and prompted properly. We believe the modularity of these architectures will make them more adaptable and reusable to new use cases. While accuracy on downstream tasks is an appealing metric, we argue that other considerations are just as important, for example the ease of adapting a model to one's own use case, or the ability to leverage multiple existing models.
We noticed that GPT-3 has a tendency to always produce an answer given an example: this could be due to the particular 0-shot fine-tuning procedure, which biases the model towards generating useful responses. This raises the question of whether we can fine-tune "stackable" LLMs and whether DLNs can be used as a framework to generate training data for that purpose. Second, we engineered our backward and forward templates; in the future, we wish to expand our work to learn parts of such templates: we expect this to make the variational bound tighter and thus easing DLN's optimization. Additionally, while we only proposed 2-layer DLNs, the framework accommodates arbitrary directed acyclic graphs.
Impact statement While we are fully aware of the limitations of addressing societal issues through technical work, we hope that modular approaches like ours will alleviate some of the issues associated with LLMs, like the concentration of power associated with the difficulty to train them. We also hope that, by facilitating the reusability and adaptivity of such models, we shall make them more amenable to a wider variety of use cases. However, while we discuss the performance of these models on artificial benchmarks, we do not address the question of when and how such models should be deployed, nor do we offer additional guarantees against their misuse. We also emphasize that performance on artificial tasks, even if realistic, is neither representative of performance in uncontrolled environments, nor enough to justify the deployment of these models in high stakes situations.
Acknowledgements We would like to acknowledge Silviu Pitis for the useful feedback on the draft, Nikolay Malkin and Tong Wang for their advice during the first steps of this project.
$^{3}$ Earlier work by Miao and Blunsom [27] also treated strings as random variables.
[46] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. (2023a). Self-consistency improves chain of thought reasoning in language models. International Conference on Learning Representations.
[47] Wang, X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. (2023b). Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning.
[48] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS.
[49] Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022). Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214-229.
[50] Wu, T., Terry, M., and Cai, C. J. (2022). Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–22.
[51] Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. (2023a). Wizardlm: Empowering large language models to follow complex instructions.
[52] Xu, W., Banburski-Fahey, A., and Joic, N. (2023b). Reprompting: Automated chain-of-thought prompt inference through gibbs sampling. arXiv preprint arXiv:2305.09993.
[53] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. (2023a). Tree of thoughts: Deliberate problem solving with large language models.
[54] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023b). React: Synergizing reasoning and acting in language models. International Conference on Learning Representations.
[55] Zhang, Z., Zhang, A., Li, M., and Smola, A. (2023). Automatic chain of thought prompting in large language models. International Conference on Learning Representations.
[56] Zhou, D., Scharli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E. (2023a). Least-to-most prompting enables complex reasoning in large language models. International Conference on Learning Representations.
[57] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. (2023b). Large language models are human-level prompt engineers. International Conference on Learning Representations.
Contents in Appendices:
- In Appendix A, we list the contribution of each author to this work.
- In Appendix B, we provide additional experimental details including task statistics and the prompt strings we used to initialize DLN.
- In Appendix C, we provide additional experiments and baselines we compare to.
- In Appendix D, we provide forward and backward templates being used in DLN.
- In Appendix E, we provide an algorithm that generalizes DLN training in a multiple layer setting.
- In Appendix F, we show examples of learned weights that exhibit behavior similar to in-context learning.
- In Appendix G, we show examples of learned weights by 2-Layer DLNs.
- In Appendix H, we show an example of the hidden states produced by a 2-Layer DLN.
- In Appendix I, we provide implementation details, including hyperparameter information.
- In Appendix J, we discuss resource used in DLN development and their pricing.
Table 3: Tasks used in this work.
Task	|train|	|valid|	|test|	|class|	Description
Mpqa	400	256	250	2	Sentiment analysis.
Trec	400	256	250	6	Question type classification.
Subj	400	256	250	2	Determine whether a sentence is subjective or objective.
Disaster	400	250	250	2	Determine whether a sentence is relevant to a disaster.
Airline	400	250	250	3	Airline tweet sentiment analysis.
Hyper.	400	1000	250	2	Order adjectives correctly in English sentences.
Nav.	375	375	250	2	Spatial reasoning given navigation instructions.
Date.	59	60	250	6	Infer a date from context.
Logic.7	225	225	250	7	Deduce the order of seven objects given instruction.
A Contributions
Alessandro Sordoni proposed the general idea of DLN, where multiple prompts are learnt at each layer through backward natural language operations; they proposed to generate synthetic in-context examples and the exploration reward for DLN-2; they wrote the code and ran the experiments; they focused on Sections 2, 3, 4 and contribute writing the rest of the sections.
Xingdi Yuan co-developed the basic idea of DLN and wrote part of the code, they also co-designed and helped conducting experiments. They contributed to the writing of the paper, mainly Sections 5 and 6.
Marc-Alexandre Côté helped with the experiments and the infrastructure to make calls to OpenAI models. They also built a demo to visualize the evolution of DLN's prompts during training and contributed to the writing of the paper, mainly focusing on the algorithms and the appendix.
Matheus Pereira co-coded an earlier, non-variational backwards operator with AT, helped with the APE and DLN-2 layers experiments, implemented the method for estimating the total cost of experiments, build a demo to visualize the evolution of DLN's prompts during training, and contributed to the release of the DLN code.
Adam Trischler helped with template conception and iteration, co-coded an earlier, non-variational backwards operator with MP, and contributed to paper writing, mainly the literature review.
Ziang Xiao helped with the model evaluation and experiment setup and contributed to the paper writing, mainly the literature review and discussion.
Arian Hosseini participated in the development discussions throughout the project and contributed to writing the literature review of the paper.
Friederike Niedtner organized and managed the project, helping the team focus on the right priorities.
Nicolas Le Roux proposed the variational inference formulation and the posterior sharpening. They offered guidance and mentorship for the project. They also contributed to the writing of the paper, mainly sections 1, 2, and 3.
B.1 Additional Task Information
In Table 3, we provide short descriptions for all tasks we use and their statistics.
B.2 Prompt Initialization
We initialize the "classification" layer of the DLNs, i.e. the first layer of the 1-Layer LN and the second layer of the 2-layer DLN, with a question or a task description as reported in Table 4. We use these same initializations to compute the 0-shot performance. Therefore, at initialization, a 1-layer LN is equivalent to the 0-shot baseline. For the hidden layer of the 2-layer DLN, we initialize the prompt to "Decompose the problem to make it simpler:" for Nav. and Subj, and ""