Title: Learning to Reason via Program Generation, Emulation, and Search
Venue: NeurIPS 2024
Accepted: yes
Keywords: language models

instruction tuning

code generation

reasoning

program search

program emulation
Abstract: Program synthesis with language models (LMs) has unlocked a large set of reasoning abilities; code-tuned LMs have proven adept at generating programs that solve a wide variety of algorithmic symbolic manipulation tasks (e.g. word concatenation). However, not all reasoning tasks are easily expressible as code, e.g. tasks involving commonsense reasoning, moral decision-making, and sarcasm understanding. Our goal is to extend a LM’s program synthesis skills to such tasks and evaluate the results via pseudo-programs, namely Python programs where some leaf function calls are left undefined. To that end, we propose, Code Generation and Emulated EXecution (COGEX). COGEX works by (1) training LMs to generate pseudo-programs and (2) teaching them to emulate their generated program’s execution, including those leaf functions, allowing the LM’s knowledge to fill in the execution gaps; and (3) using them to search over many programs to find an optimal one. To adapt the COGEX model to a new task, we introduce a method for performing program search to find a single program whose pseudo-execution yields optimal performance when applied to all the instances of a given dataset. We show that our approach yields large improvements compared to standard in-context learning approaches on a battery of tasks, both algorithmic and soft reasoning. This result thus demonstrates that code synthesis can be applied to a much broader class of problems than previously considered.

Learning to Reason via Program Generation, Emulation, and Search

Abstract

Program synthesis with language models (LMs) has unlocked a large set of reasoning abilities; code-tuned LMs have proven adept at generating programs that solve a wide variety of algorithmic symbolic manipulation tasks (e.g. word concatenation). However, not all reasoning tasks are easily expressible as code, e.g. tasks involving commonsense reasoning, moral decision-making, and sarcasm understanding. Our goal is to extend an LM's program synthesis skills to such tasks and evaluate the results via pseudo-programs , namely Python programs where some leaf function calls are left undefined. To that end, we propose, Co de G eneration and Emulated EX ecution (COGEX). COGEX works by (1) training LMs to generate pseudo-programs, (2) teaching them to emulate their generated program's execution, including those leaf functions, allowing the LM's knowledge to fill in the execution gaps; and (3) using them to search over many programs to find an optimal one. To adapt the COGEX model to a new task, we introduce a method for performing program search to find a single program whose pseudo-execution yields optimal performance when applied to all the instances of a given dataset. We show that our approach yields large improvements compared to standard in-context learning approaches on a battery of tasks, both algorithmic and soft reasoning. This result thus demonstrates that code synthesis can be applied to a much broader class of problems than previously considered. 1

1 Introduction

Recently there have been rapid advances in training language models (LMs) to generate code rather than natural language (NL), following the intuition that code may be more effective than NL for certain tasks, such as those requiring complex calculations, iteration, or data structure manipulation(Chen et al., 2022; Gao et al., 2023). Although successful, these works have mostly studied tasks conducive to a programmatic paradigm, such as symbolic manipulation or algorithmic reasoning, i.e., tasks for which a clear compilable program can be devised. However, it is unclear how to apply this approach to 'softer' reasoning tasks such as commonsense and social reasoning, where algorithmic solutions are less obvious (Zhang et al., 2023a).

### Instruction: Pluralize the given word

### Input: Corpus

CoGEX Instance

### Instruction

Pluralize the given word

### Input

Corpus

### Code

def pluralize_word(word):

# 1. Identify the suffix of the word

ending = identify_ending(word)

# 2. Identify the pluralization rule for the word

pluralization_rule = find_pluralization_rule(word, ending)

# 3. Apply the pluralization rule to the word

plural_form = apply_pluralization_rule(word, ending)

return {

'original_word': word,

'ending': ending,

'rule': pluralization_rule,

'answer': plural_form

}

>>> pluralize_word('Corpus')

### Output

{

'original_word': 'Corpus',

'ending': 'us',

'rule': {'root': 'Latin', 'class': '2nd declension neuter'}

}

2. Plan Generation

3. Code Instantiation

1. Output Reformatting

Instruction from Alpaca

Function Call

Output Dictionary

with Answer

Our goal is to expand an LM's program synthesis skills to such softer reasoning tasks. Our approach builds on the insight that, beyond generating code, LMs can also emulate the execution of code. This includes handling function calls that are defined only by name and documentation, even if they lack a full implementation. We refer to these types of programs-where only the function skeletons are provided without actual code implementations-as pseudo-programs . Such pseudo-programs can encompass both well-defined reasoning steps, such as mathematical or algorithmic operations, as well as function calls representing less precise reasoning, such as commonsense logic. This work investigates whether generating and pseudo-executing such programs can effectively address soft reasoning tasks in addition to traditional algorithmic problems.

To achieve this, we propose a novel approach: training models to follow NL instructions by generating a program and then emulating that program's code execution. Our paradigm, called COGEX, changes the inference process to (1) generate a Python function given an arbitrary instruction and optional input, (2) generate a call to that function, and (3) produce the result of simulating its execution. Unlike other work (Zhang et al., 2023b; Li et al., 2023), we do not use a Python interpreter to execute any code; rather, the model is trained to emulate execution. This allows the generated code to deliberately include calls to underspecified functions (i.e., where only the function name and documentation are included), by allowing the LM emulator to fill in the functions implementations using its latent knowledge. We train the model to not only output the result of pseudo-executing the code but also the results of intermediate function calls in the code. COGEX thus suggests a way to leverage the flexible ad-hoc reasoning abilities of LMs while encouraging programmatic reasoning via the program structure. We train COGEX models by adapting the recent Alpaca instruction tuning dataset (Taori et al., 2023) into a set of analogous Pythonic examples by prompting GPT-4 to perform the conversion, and then use the resulting COGEX dataset to fine-tune smaller (7B and 13B) LMs to answer instructions via code.

The COGEX paradigm allows us to explore a new way to learn new tasks: identifying a general program that applies across a task, such that new task instances can be solved by emulating calls to that one program. Inspired by work on hard prompt tuning (Wen et al., 2024) and example selection for in-context learning (Gupta et al., 2023a), we introduce a search procedure that uses the frozen COGEX model to try out many programs on a set of training examples and identify which single program is optimal for the dataset. The procedure, termed COTACS: CO GEX T ask A daptation via C ode S earch, performs no parameter updates and requires saving nothing more than a program string.

We evaluate over a diverse suite of reasoning tasks, including commonsense QA, text classification, and math datasets. These datasets cover symbolic tasks that could conceptually benefit from programmatic operations and standard natural language tasks whose solutions might not be easily described in code. We find that applying COTACS leads the COGEX models to substantially outperform the comparable NL-based LM using the same original checkpoint and the same set of training examples

available for in-context learning, even in the few-shot regime. COTACS thus gives us one way to fit a model to a new dataset without having to perform any gradient descent or parameter updates, both for algorithmic and softer reasoning datasets. Our contributions are thus:

A novel reasoning paradigm, COGEX, that trains language models to generate and emulate the execution of pseudo-programs. COGEX is a general paradigm that allows LMs to leverage code for different types of reasoning.

A program search method, COTACS, enabling a task-general program suitable for a dataset (rather than a single instance) to be found using a COGEX model.

A dataset, derived from the Alpaca instruction tuning dataset, for training COGEX models.

Overall, this work provides a significant step in showing how code generation can be applied to a much broader class of problems than previously considered.

2 Approach

In this section, we start by formalizing our approach and describing our data construction process (§2.1). We then describe our program search approach to tune a COGEX model on a given task through program search (§2.2).

2.1 Method: COGEX

Formulation. Our goal is for the model to execute a given task by simulating code execution. That means our model will take as input the task description, generate a Python program, and simulate the expected output of executing that program. Formally, given a natural language (NL) task description I , optional input argument A , Python function F , function call C , and output dictionary O designating the output from the program pseudo-execution, the LM will take ⟨ I, A ⟩ as input and generates ⟨ F, C, O ⟩ as output. Since the process is sequential, COGEX models can work as either a reasoner f reasoner ( I, A ) → ( P, C ) → O or as a call-instantiating and execution-emulating model f emulator ( I, A, P ) → C → O that takes a pre-specified program P and applies it to the variable arguments A . This latter formulation enables searching over the space of task-specific programs: searching for one P task to solve a class of problem (e.g., emotion classification) and then applying f emulator ( I, A i , P task ) to emulate its execution on each instance A i of that problem. We expand on program search in §2.2.

Training Data Construction. As we want a general-purpose dataset that spans tasks with diverse reasoning requirements, we choose the Alpaca instruction tuning dataset (Taori et al., 2023). Following Peng et al. (2023), we rely on GPT-4 2 to convert the Alpaca dataset into their COGEX versions. Specifically, every NL instance in the Alpaca dataset is mapped into a corresponding COGEX version. We split the conversion process into three steps, each of which involves prompting GPT-4 with the output from the previous. This stepwise approach proved more effective than directly prompting GPT to convert each instance to code in one shot.

As depicted in Figure 1, the three steps are: (1) converting the outputs and (optional) inputs into Pythonic data structures like strings, lists, and integers whenever relevant as determined by GPT-4; (2) generating an instruction-specific plan, or a series of NL steps that should perform the task for any potential input; (3) instantiating the plan as a Python program whose inline comments are the plan steps and whose output is a dictionary containing all intermediate and final outputs that the LM believes would result from executing each step. Prompts for all steps can be found in Appendix A.

Importantly, we allow for GPT to include undefined functions, e.g., identify_ending() and find_pluralization_rule() in Figure 1. The goal is to leverage the LM knowledge to fill in the semantics of these undefined functions when emulating the execution of a given program. In addition, we include the program's intermediate results in the output dictionary before the final answer to encourage the model to stick to the NL reasoning plan delineated in the program comments. After defining the program, we cue GPT-4 to call the function on an argument, e.g. pluralize_word('corpus') which can reflect the optional Alpaca example input, or can reflect specific details from the instruction itself. Our prompts encourage GPT-4 to write a program that

is as general purpose as possible and not tied to a specific input: e.g. pluralize_word(word) is preferable to pluralize_corpus() .

Fine-tuning any LM on the resulting COGEX dataset creates our desired model, which accepts any task description/input combination and responds by dynamically generating a Python program and then emulating its execution.

2.2 Program Search: COTACS

A COGEX model can generate a new program for any new task instruction and instance; however, some programs might be more or less effective at performing the task. How can we find the optimal program for a specific task, especially when some training data is available? As COGEX relies on argument-accepting pseudo-programs, it naturally enables program optimization. Given multiple examples of a new task, we can search for one program that performs well, and then apply the same program to new test examples by invoking the program with different input arguments.

Our search process, COTACS: CO GEX T ask A daptation via C ode S earch, finds a single program that optimizes a COGEX model for a particular task dataset, enabling adapting a COGEX model to a given task without learning any weight parameters. We learn a new dataset simply by using a finetuned COGEX model to generate and then evaluate many program candidates to find the one that best fits the given dataset. As described in algorithm 1, we split a dataset D of argument and output pairs ( a i , o i ) into a small training set ( n = 300 in experiments) and a larger development set; we then generate a separate code candidate for every training item and retain the programs with decent performance on the training set. 3 We then rank these programs according by their performance on the development set. For certain tasks, we find it beneficial to find multiple programs for a task and then at test time take a majority vote across the COGEX model's responses using each code. To accomplish this, we retain some topk performing codes over the development set.

3 Experiments and Results

Below we describe the training and evaluation of COGEX models. We first show overall performance across a wide array of tasks compared to off-the-shelf baselines (§3.2) then ask a series of follow-up research questions investigating ablated scenarios (§3.3).

3.1 Experimental Setup

Model Training. We fine-tune the 7B and 13B variants of Llama-2 (Touvron et al., 2023). We use parameter-efficient training via Low-rank adaptation (LoRA) (Hu et al., 2021) with a rank of r = 16 , dropout rate of 0 . 05 , and LoRA weights added to the query, key, value, and output matrices in all layers. We train all models for five epochs using a batch size of 32 and a learning rate of 0 . 0003 . As a validation set, we randomly sample 2K examples from the training set and keep the checkpoint with the lowest perplexity on the validation set for testing. Model training was done on a server with 128GB of RAM and 2 Nvidia A6000 48GB GPUs. On our dataset, training a single 7B model and 13B models took around 12 and 20 hours, respectively. To ensure a fair comparison, all the baselines are trained with the exact same hyperparameters.

Datasets. We measure COGEX model performance on a variety of benchmarks ranging from symbolic manipulation to commonsense and social reasoning. We choose these datasets as representatives of tasks that involve complex reasoning and tasks whose solutions cannot be easily described in code. As for symbolic and math reasoning, we use the Word Sorting task from BIG-bench hard (Srivastava et al., 2022; Suzgun et al., 2022), the math word problem dataset SVAMP (Patel et al., 2021), the coin flip tracking dataset from Wei et al. (2022), and the large number arithmetic task (referred to as Sum) from Zelikman et al. (2022). For the last, we use the 5-digit examples for training and 6-digit for testing. We measure string-normalized exact match accuracy for all tasks. Following Zhang et al. (2023b), we evaluate on a series of Text Classification datasets: CoLA (2 labels) (Warstadt et al., 2019), Emotion (6 labels) (Saravia et al., 2018), and SST2 (2 labels) (Socher et al., 2013). We also evaluate on the Commonsense Reasoning datasets CommonsenseQA (Talmor et al., 2019) and Social IQa (Sap et al., 2019), which are 4- and 5-way multiple-choice datasets. We hand-write the instruction I task for these datasets as they do not provide any.

Code Search. For all datasets, we use a maximum of 1000 training examples. We use n = 300 training items to generate candidate codes and evaluate them on the remaining 700 items to identify the most generalizable ones. We experiment with retaining the topk ∈ { 1 , 3 } performing programs for use at test time. For k = 3 , we take a majority vote of answers, breaking ties randomly. We use

sampling temperature t = 0 . 7 when generating candidate codes and t = 0 . 05 to generate answers given a program and argument. We report results on the released dev sets of all considered tasks.

Baselines. We consider two baselines that represent standard practices for adapting an LM to a new task: (1) few-shot prompting using the off-the-shelf Llama-2 and CodeLlama models and (2) zero-shot prompting using the Llama-2 models instruction-tuned from the original Alpaca dataset. 4 For the in-context learning baseline for (1) we use the same 1000 training data-points as COTACS and optimize the examples by retrieving the most similar few-shot examples using BM25. For zero-shot alpaca models, we use the standard Alpaca-7B and -13B models. As COTACS might not require many training examples to achieve strong performance, we compare the 1000-example COTACS run with one that only uses 10 total examples to generate and evaluate candidates.

3.2 Results

Our main results depicting the difference in performance between COGEX models tuned via COTACS versus off-the-shelf few-shot baselines and Alpaca models are shown in Table 1. The COTACS method with 1000 training examples outperforms the baselines for a large majority of tasks and models (8/9 tasks for both Llama-2 7B and 13B). COGEX shows particularly strong gains over the baselines in the Sum and coin flip tracking tasks (+10-20%) as expected due to its code-related nature. We observe that the COTACS method with N train = 1000 training examples performs best on average across the 9 tasks, and still performs better than the baselines with only N train = 10 examples. Retaining the top k = 3 programs instead of 1 improves performance in most cases (+1% average).

Instruction Following. As our models are trained on instruction following in code, can they still perform instruction-related tasks as well as models trained on text-only Alpaca? We verify this by using alpaca-eval to compare Alpaca-7B against our COGEX-7B model trained from the same base Llama model. We find a similar win rate (50% within the 2 SD range) indicating similar instruction following ability. Thus we can see that training on code-based instructions does not hurt standard instruction-following abilities, while opening up many possibilities for program search.

Effect of Code Pre-training. As we are fine-tuning LMs on code data and then evaluating them on tasks that are more or less code-related, a natural question to ask is whether LMs pre-trained on code datasets yield stronger COGEX models. We investigate this by fine-tuning Code Llama (Roziere et al., 2023) on the COGEX dataset instead of Llama. Table 2 shows the resulting change in performance using COTACS ( k =3 ) program search. The Llama-2 models show improved performance on Social IQa (+4%) but much worse on coin flip tracking (-10-12%). These results do not provide conclusive evidence that Llama-2 models are better or worse than Code Llama on particular task categories.

3.3 Ablation Studies

Here we present a series of ablation studies to ask the following questions:

How many training examples are needed for search? In the above experiments, we chose 1000 training examples and 300 program candidates for the COTACS algorithm. This raises the question: how many examples are required to yield the strong performance provided by the search? We investigate this by simulating the algorithm and sampling 1000 trials with varying numbers of training examples and program candidates. Results are shown in Figure 2. In nearly all cases, performance with 50 or 200 training examples is within a couple of points of the full performance with the 300/1000 configuration. The performance when sampling 10 code candidates (green) lands within 2 points of sampling 300 candidates on 5 of the 7 datasets. Benefits do not appear on Word Sorting, as

3

5

10

20

50

100 200

500 1000

0.88

0.90

0.92

Strongest Baseline

SST2

3

5

10

20

50

100 200

500 1000

0.48

0.50

0.52

Word Sorting

3

5

10

20

50

100

200

500 1000

0.375

0.400

0.425

0.450

0.475

SVAMP

3

5

10

20

50

100 200

500 1000

0.550

0.575

0.600

0.625

0.650

Social IQa

3

5

10

20

50

100

200

500 1000

Number of Training Items

0.550

0.575

0.600

0.625

0.650

CoinFlip

3

5

10

20

50

100 200

500 1000

Number of Training Items

0.60

0.62

0.64

Strongest Baseline: 44.5%

Number Summing

3

5

10

20

50

100 200

500 1000

Number of Training Items

0.58

0.60

0.62

0.64

Commonsense QA

# Code Candidates

3

5

10

20

50

100

300

performance lands between 0.51 and 0.515 regardless of configuration. This suggests that the range of quality in generated programs for the Word Sorting dataset is much smaller than the others, so picking between just a few candidates is sufficient. Overall, we see that we can significantly reduce the search space and still see large gains on most tasks.

Is it better to execute an NL plan instead of Python code? We have proposed a mechanism to generate Python programs whose steps are meant to reflect the reasoning process, stated in NL, that answers a given instruction. Is the code necessary, or can a model be trained to generate only the plan and achieve the same performance? We fine-tune a Llama model on a version of the COGEX 52k-item training set where each Python program has been replaced with just the NL steps (removing step 3 of Figure 1). This NL Plan model still returns the same output dictionary with intermediate results. To fit the plan-only model to a dataset, we run the COTACS algorithm but sample and retain the NL plans instead of programs. We see in Figure 4 (orange vs gold) that NL Plan COTACS can match the performance of regular program COTACS on some tasks (CoLA, SST, Word Sorting, Emotion), but performs much worse on others, particularly Coin Flip, SV AMP, and Sum. This follows the intuition that these tasks benefit from a programmatic reasoning paradigm.

Is it better to find one program or generate a new one for each instance? COTACS finds one or multiple programs that can be reapplied to all task instances in a dataset to achieve high performance. Is this better than letting the COGEX model generate a separate program for every instance? It might be the case that the latter allows for catering the program to the specifics of a particular task instance- e.g. in Figure 3, where the left (red) COGEX-generated program has steps specifically crafted to identify actions to be taken by a particular person. Finding a single program disallows this flexibility. We investigate this question by running the model end-to-end on each instance. The COTACS model performs the mapping f ( I task , A i , P COTACS ) → C i → O i for each task instance A i , while the end-to-end model performs f ( I task , A i ) → ( P i , C i ) → O i . We sample from the latter using temperature t = 0 . 05 . 5 Results are shown in Figure 4 (maroon vs gold); end-to-end performance is comparable to COTACS only on Word Sorting and Sum. In all other cases, it is substantially worse.

Llama-2 7B

Llama-2 13B

0

20

40

60

80

QA

CoLA

Llama-2 7B

Llama-2 13B

0

20

40

60

80

SST2

Llama-2 7B

Llama-2 13B

0

10

20

30

40

50

Word Sorting

Llama-2 7B

Llama-2 13B

0

10

20

30

40

50

SVAMP

Llama-2 7B

Llama-2 13B

0

20

40

60

Social IQa

Llama-2 7B

Llama-2 13B

0

20

40

60

QA

Emotion

Llama-2 7B

Llama-2 13B

0

20

40

60

CoinFlip

Llama-2 7B

Llama-2 13B

0

20

40

60

Number Summing

Llama-2 7B

Llama-2 13B

0

20

40

60

Commonsense QA

Ablation

COGEX End-to-End

NL Plan COTACS

COTACS

Is COTACS better than chain-of-thought? A common practice to elicit systematic reasoning from LMs is to prompt it for the reasoning via some version of 'explain your answer step-by-step' (Kojima et al., 2022). How does this compare to COGEX models on a given dataset? We compare COGEX to zero-shot CoT by prompting our Alpaca models with a task-specific instruction, while additionally appending the instruction to 'think step-by-step' before producing the answer. Figure 4 (blue vs gold) shows that CoT prompting performs similarly to the NL plan search method; it can approach COTACS performance on some NLP classification tasks but performs worse over SVAMP, Number Summing, and CoinFlip.

When is COTACS better than fine-tuning? Fine-tuning is a standard practice to adapt an LM to a new task. However, fine-tuning often requires a large amount of training data and storing a new model for each task. Here, we study the impacts of the number of examples on fine-tuning and COTACS. We find that when there are many examples available, fine-tuning achieves stronger performance. However, COTACS is generally better until there are a large number of examples available: it outperforms fine-tuning on 4/9 tasks with 500 examples. This suggests that COTACS can be a lightweight alternative in the low-to-medium shot setup.

100

500

1000

0

.

65

0

.

70

0

.

75

0

.

80

correct

CoLA

100

500

1000

0

.

7

0

.

8

0

.

9

SST2

100

500

1000

0

.

50

0

.

55

0

.

60

Word Sorting

100

500

1000

0

.

2

0

.

4

SVAMP

100

500

1000

0

.

5

0

.

6

0

.

7

Social IQa

100

500

1000

0

.

6

0

.

8

correct

Emotion

100

245

0

.

60

0

.

65

CoinFlip

100

500

1000

0

.

4

0

.

6

Number Summing

100

500

1000

0

.

5

0

.

6

0

.

7

Commonsense QA

Method

COTACS

Fine-tuning

3.4 Qualitative Analysis

Since we rely on the LM as a code emulator, there is no guarantee of correct execution. The generated intermediate outputs allow us to examine if the model can faithfully emulate the program. We observe failure cases where the LM incorrectly simulates the program execution even if the generated program is correct as shown in Figure 6. We also include positive qualitative examples in Appendix B.

4 Related Work

Reasoning via Code. Using code for reasoning is a burgeoning area that has shown improved results on many algorithmic tasks (Chen et al., 2022; Gao et al., 2023). Many approaches ask LLMs to express their reasoning as code and leverage code interpreters to execute them. Recently, and concurrent with our work, some studies investigate training LLMs as code compilers, where the LM is prompted to emulate code execution (Li et al., 2023; Chae et al., 2024; Mishra et al., 2023). These

LM-as-compiler approaches fall into a broader category of work that invokes LLMs as subroutines in programs (Kalyanpur et al., 2022; Weir et al., 2024). Different from ours, these works mainly rely on manually prompting very large models, while we focus on training open-source LMs to both generate and emulate programs. In addition, we aim to achieve task generalization by searching for an optimal program for a given task-different from Chae et al. (2024) who rely on prompting LMs with specific code instructions. Ours is the first work on code-based reasoning that employs search over the program space with the goal of generalizing an optimal program to a task.

Prompt Optimization. Our search procedure, COTACS, has a similar spirit to in-context learning optimization approaches where the goal is to find an optimal set of exemplars (an optimal pseudoprogram, in our case) for a given task. Existing studies (Zhang et al., 2022; Rubin et al., 2022; Ye et al., 2023a; Gupta et al., 2023b; Khalifa et al., 2023a) explored various methods to select optimal in-context examples, leveraging similarity- or diversity-based heuristics-to name a few. Searching for useful task instructions has also been explored (Honovich et al., 2022; Khalifa et al., 2023b; Chen et al., 2023).

Another related area of research is automated prompt engineering (Shin et al., 2020; Deng et al., 2022; Prasad et al., 2023) that bootstraps an effective prompt using some reward function. While LMs have been shown to be effective at producing their own prompts (Zhou et al., 2022; Yang et al., 2024; Pryzant et al., 2023; Ye et al., 2023b), our work shows that LMs can also reason by generating and executing their own generated programs. Our method differs from these studies as it uses the same input instruction and optimizes the intermediate representation, rather than modifying it via prompt optimization. Finding a single program string to solve a class of problems is also related to finding a high-level NL description of a task using one or multiple demonstrations (Weir et al., 2023).

5 Conclusion

We present COGEX, a methodology that trains language models to generate and execute their own Pythonic reasoning programs in response to task instructions. We convert the Alpaca instruction tuning data into COGEX instances that can be used to COGEX-tune any models. We design an optimization algorithm, COTACS, that applies COGEX models to a new dataset by generating and searching through possible programs that can be reapplied to new task items. Applying the COTACS search algorithm yields task performance that exceeds that of few-shot in-context-learning and typical NL instruction following. Our work demonstrates a way to apply LM-based programmatic reasoning to NLP benchmarks that require softer reasoning skills not easily stated in code syntax.

6 Limitations

While our work represents a step towards utilizing code language models for non-algorithmic reasoning tasks, COGEX still suffers from the following limitations:

COGEX is suitable for soft reasoning tasks for which step-by-step programs are difficult to describe. However, when solving algorithmic tasks where a precise step-by-step program is possible, passing the generated code directly to an interpreter may be preferable to emulating code execution via the LM.

We have found that the LM can occasionally produce a result that is inconsistent with the code emulated, as noted in subsection 3.4. In which case, the code does not faithfully reflect the reasoning process followed by the model.

There is an extra computational overhead when emulating code execution via an LM compared to using an interpreter as the LM needs to generate intermediate variables along with the final answer.

References

Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Tzu-iunn Ong, Beong-woo Kwak, Moohyeon Kim, Seonghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, et al. Language models as compilers: Simulating pseudocode execution improves algorithmic reasoning in language models. arXiv preprint arXiv:2404.02575 , 2024.

Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03082 , 2023.

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 , 2022.

Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp. 3369-3391, 2022.

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

Shivanshu Gupta, Matt Gardner, and Sameer Singh. Coverage-based example selection for in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023 , pp. 13924-13950, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.930. URL https://aclanthology.org/2023.findings-emnlp.930 .

Shivanshu Gupta, Matt Gardner, and Sameer Singh. Coverage-based example selection for incontext learning. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pp. 13924-13950, 2023b.

Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10782 , 2022.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Aditya Kalyanpur, Tom Breloff, and David A Ferrucci. Braid: Weaving symbolic and neural knowledge into coherent logical explanations. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pp. 10867-10874, 2022.

Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. Exploring demonstration ensembling for in-context learning. arXiv preprint arXiv:2308.08780 , 2023a.

Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. Few-shot reranking for multi-hop QA via language model prompting. In Anna Rogers, Jordan L. BoydGraber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , pp. 15882-15897. Association for Computational Linguistics, 2023b. doi: 10.18653/V1/ 2023.ACL-LONG.885. URL https://doi.org/10.18653/v1/2023.acl-long.885 .

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35: 22199-22213, 2022.

Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator. arXiv preprint arXiv:2312.04474 , 2023.

Mayank Mishra, Prince Kumar, Riyaz Ahmad Bhat, Rudra Murthy, Danish Contractor, and Srikanth G Tamilselvam. Prompting with pseudo-code instructions. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.168. URL https://aclanthology.org/2021.naacl-main.168 .

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 , 2023.

Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based instruction search for prompting large language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pp. 3845-3864, 2023.

Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 , 2023.

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 2655-2671, 2022.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1454. URL https://aclanthology.org/D19-1454 .

Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pp. 3687-3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1404. URL https: //aclanthology.org/D18-1404 .

Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 , 2020.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://aclanthology.org/D13-1170 .

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https://aclanthology.org/N19-1421 .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html , 3(6):7, 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=1PL1NIMMrw .

Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics , 7:625-641, 2019. doi: 10.1162/ tacl_a_00290. URL https://aclanthology.org/Q19-1040 .

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.

Nathaniel Weir, Xingdi Yuan, Marc-Alexandre Côté, Matthew Hausknecht, Romain Laroche, Ida Momennejad, Harm Van Seijen, and Benjamin Van Durme. One-shot learning from a demonstration with hierarchical latent language. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pp. 2388-2390, 2023.

Nathaniel Weir, Peter Clark, and Benjamin Van Durme. NELLIE: A neuro-symbolic inference engine for grounded, compositional, and explainable reasoning. IJCAI , 2024.

Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems , 36, 2024.

Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=Bb4VGOWELI .

Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. In International Conference on Machine Learning , pp. 39818-39833. PMLR, 2023a.

Qinyuan Ye, Maxamed Axmed, Reid Pryzant, and Fereshte Khani. Prompt engineering a prompt engineer. arXiv preprint arXiv:2311.05661 , 2023b.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems , 35:15476-15488, 2022.

Li Zhang, Liam Dugan, Hainiu Xu, and Chris Callison-burch. Exploring the curious case of code prompts. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE) , 2023a.

Tianhua Zhang, Jiaxin Ge, Hongyin Luo, Yung-Sung Chuang, Mingye Gao, Yuan Gong, Xixin Wu, Yoon Kim, Helen Meng, and James Glass. Natural language embedded programs for hybrid language symbolic reasoning. arXiv preprint arXiv:2309.10814 , 2023b.

Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp. 9134-9148, 2022.

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 , 2022.

A Prompts for Converting Alpaca to COGEX

Figure 7, Figure 8, and Figure 9 display the prompts used to GPT-4 in sequence to convert Alpaca into the COGEX dataset. The prompts (1) convert all inputs and outputs into Pythonic types like strings, lists and dicts, (2) generate plans to answer a given instruction, and (3) instantiate each plan as a Python program with underspecified function calls.

B Further Qualitative Analysis

Figure 10 and Figure 11 show good qualitative examples generated by COTACS, along with outputs from the 2-shot prompting baseline for the text classification and math reasoning tasks, respectively. Wefind that COTACS encourages general-purpose code that is generalizable across multiple examples within the same task. It also enables better interpretability by generating outputs of the intermediate reasoning steps.

Figure 9: Prompt used for instantiating Python programs from NL plans. See repository for full-length prompt.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claim of the paper is a novel reasoning paradigm, dataset for finetuning models on the paradigm, and a task-specific optimization algorithm. We show the empirical benefits of the paradigm and algorithm through our experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No theoretical proofs are given.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Hyperparameters and other details are included in section 2.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Public access to these will be released when not anonymized.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In section 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We only report single runs, following previous work.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In subsection 3.1

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: It follows the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not provide any new risks and discusss risks in section 6.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: They are properly credited via citation and licenses are in their respective Github pages. As they are open-source we do not explicitly discuss the license here and point interested readers to the original authors license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Yes, described in Section 2.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing was done.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing was done.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a fine-tuned LLM that reasons by generating pythonic code and executes it through the model rather than passing it to an external interpreter (CoGEX). They also use the model to generate a set of candidate programs from a training set, among which the top-k performing programs are selected and subsequently used to evaluate the test set (CoTACS). The model shows improved performance on several datasets against benchmark models without the pretraining, including with in-context learning and chain-of-thought reasoning. Several additional analyses are provided for model efficacy, including the number of training items and sampled programs.\n\nSTRENGTHS:\n1. Paper is well-written and easy to follow. All the analyses and results are clearly presented.\n2. The CoGEX model provides a useful datapoint on understanding how well LLMs can benefit from program-like planning without relying on an external program interpreter.\n3. Allowing the model to use ‘primitive’ functions that are difficult to define programmatically offers greater flexibility than relying on interpreters and the ability to query for factual information.\n\nWEAKNESSES:\n1. The ideas do not strike me as particularly novel. As noted in the paper, using LLMs to generate code is not new, and neither is using language models to simulate program executors. CoTACS is simply a top-k search.\n2. It isn’t clear to me what advantage this method has over just passing the code to Python Interpreter. Moreover, what advantage is there to using CoGEX to just using GPT4 given that the dataset was created using GPT4 in the first place?\n3. While I appreciate the completeness of the presented results (for the sake of good science), even for experiments where the improvements are marginal, the results are often marginal at best, and it is unclear in what circumstances this approach is favorable and when it is not. Likewise, the authors use 0-shot CoT as their comparison model (Line 236), which is the weakest CoT baseline possible. Given that using an actual program interpreter is a viable alternative, I would be hesitant to adopt CoGEX without having a clearer understanding of when it outperforms simpler alternatives. In my opinion, the paper would strongly benefit from the following:\n    A. A fair comparison with CoT, either controlling for the number of examples or showing at how many examples the models achieve parity. Since this is the simplest thing to try for most ML practitioners, even parity would have me favor CoT over most alternatives.\n    B. A comparison, if applicable, to actual interpreters (e.g. Python). If this is not possible (e.g. due to calls to undefined functions), how often is that the case? Alternatively, if the model was forced to use predefined functions, how much better is it to use undefined functions?\n    C. I really was hoping for a more extensive Qualitative Analysis section (S 3.4) to get a better understanding of when CoGEX strongly outperforms the baselines. The results in Table 1 demand too much prior knowledge about the individual tasks/datasets from the readers without some guidance on how to think about them.\n\nQUESTIONS:\n1. (Table 1): Could you define what BM25 means? It wasn’t clear to me from the paper exactly how you performed the benchmarks and how you chose the benchmark settings to ensure fair comparison.\n2. (Line 91): What does silver-quality mean?\n3. (Table 3 and Figure 2): It wasn’t clear from the paper whether the model generated meaningfully different programs. Footnote 3 (from Line 231) suggests that the model likely generated very similar outputs (assuming lower temperature means more deterministic outputs).\n4. (Line 103): Few questions about ‘fuzzy’ premitives. \n    A. Does the model come up with its own fuzzy primitive operations, or do they all need to be predefined by the engineer or GPT4? \n    B. How did you determine what level of granularity was appropriate for the dataset? \n    C. Does the model ever produce fuzzy primitives that were not included in the dataset by GPT4 and if so, how does the model deal with it? \n5. (Line 187) Could you provide the actual SD or the 95% interval?\n6. (Figure 2) Could you include baselines for these tasks, e.g. results from the off-the-shelf Llama-2 13B on these problems. (It’s hard to tell if the linear trends are misguiding for me or not because the x-axis is in log-scale while the y-axis is in raw percentages. Logit-scale for y-axis could potentially be more informative.)\n7. (S3.4) Do you observe instances where the generated program is incorrect but the model still gets the problems correct? In other words, does the model actually follow the incorrect program or does the model ignore it?\n\nLIMITATIONS:\nThe authors provide no guarantee of safety (this is acknowledged in the supplements), but not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed approach, Code Generation and Emulated Execution (COGEX), involves training LMs to generate pseudo-programs with some undefined functions and then emulate the execution of these programs. This allows the LM's knowledge to fill in the gaps while maintaining a program like reasoning framework. The COGEX model is adapted to new tasks through program search, optimizing performance across dataset instances. The approach shows improvements over standard in-context learning methods, demonstrating that code synthesis can be applied to a wider range of problems.\n\nSTRENGTHS:\n- The concept of pseudo-program and LLM emulation of program is interesting.\n- The authors evaluated the methodology on many different dataset and tasks.\n\nWEAKNESSES:\nI would consider raising my score if a fair number of the following weaknesses can be fixed by the authors. Many of them should be doable without extra experiments.\n\n- Motivation: While in general I like the idea of pseudo-programs, but I don’t think it is very well motivated or explained in the paper. For this relatively new concept to be introduced, I would like to see a more precise definition that describes each type of reasoning, whether its intuition based with no reasoning, “soft reasoning”, and exact reasoning. State clearly what task falls into the camp of “soft reasoning”.\n- Examples: I think the motivation is insufficient partly because of that the motivating examples (Fig. 1, 3, 10) do not seem to be requiring any “complex reasoning”. By “complex reasoning” I specifically mean things like composition of multiple sources of information, long range dependency of information, the use of logical operations like “and”, “or”, “not”, and “implies”, and etc.\n- Task/Dataset selection: (also related to previous points) I would want some more clarification on why the authors pick the presented tasks and datasets. What makes these dataset special so that COGEX can be applied to them? What additional conceptual benefits do COGEX bring to them? This will also help motivate the methodology you proposed.\n- Formalization: the symbol $f$ for COGEX model is unnecessarily abused. It can be invoked with either 2 or 3 arguments, which does not type check as a well-defined formalization. You should at least create two separate symbols, such as $f_{\\text{reasoner}}$ and $f_{\\text{enumlator}}$.\n- Algorithm: The while loop in line 7-10 of Algorithm 1 does not seem to have termination guarantee. This is a potential flaw in the algorithm. What if the task and dataset is malformed and there exists no proper pseudo-program? Your algorithm should account for this.\n- Comparison to GPT-4: I don’t want to suggest to do new experiments with GPT-4 as baselines, but I do want clarification on why GPT-4 is not used as a zero-shot baseline. I see that authors use GPT-4 to generate datasets used for training. So it’s definitely not the case that authors do not have access or hardware/financial limitations for doing so. Then why isn’t GPT-4 used as a baseline? I would assume that GPT-4 without training can still be used to synthesizing pseudo-programs, right?\n- Claim in the experimental results: authors describes Fig.4 as “Figure 4 (blue vs gold) shows that CoT… can approach COTACS performance on some NLP classification tasks but generally performs substantially worse.” But to me when looking at Fig.4, I don’t see this claim accurate. I can only see significant difference on CoinFlip and Number Summing.\n- Experimental setting: (related to the previous point) the direct comparison with zero-shot CoT might not be fair because your model has undergone fine-tuning, right?\n\nQUESTIONS:\n(I asked majority of questions along with the listed weaknesses)\n- Typo in Figure 3 caption: “SocialIQa” -> “Social IQa”\n\nLIMITATIONS:\n- The author listed the limitations in the Appendix, which I would prefer doing in the main text. This also relates to the first listed weakness that I wrote: giving a proper overview would help people identify where and when your methodology could be applied.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a means of training language models to perform semi-programmatic reasoning, along with an adaptation method based on program search to specialize the resulting models for particular downstream tasks without updating their parameters.\n\nThe authors use GPT-4 to generate programmatic reasoning strategies from Alpaca instruction-tuning examples, allowing GPT-4 to include undefined function calls as the resulting code does not actually need to be executable, only used to guide the downstream language model's inference process.\n\nThe authors evaluate their method by fine-tuning Llama 2 models on their version of the Alpaca dataset, then comparing benchmark performance against few-shot, instruction-tuned zero-shot, and chain-of-thought baselines on several tasks picked to represent both algorithmic and non-algorithmic reasoning.\n\nExperimental results indicate that the proposed method outperforms the considered baselines across most tasks by a good margin.\n\nSTRENGTHS:\n- The proposed method appears to strike a nice balance between the rigidity of code-structured reasoning and freeform CoT, as it performs well across domains where CoT excels and ones where it struggles.\n- Being able to specialize reasoning for a particular task by selecting a pseudo-program is neat. It also appears to work well with a much smaller number of examples than are required to effectively fine-tune a model.\n- The authors do a very good job of proactively answering natural questions with their ablations in section 3.3.\n- As for clarity of exposition, all the writing is easy to follow and the paper is laid out intuitively.\n\nWEAKNESSES:\n1. The authors don't report statistical significance (e.g. through bootstrapping) or variance across runs with different subsamplings of training data.\n2. As far as I can tell, the reported experiments don't really include domains where the correct solution can be reached by actually executing a fully specified program, besides Number Summing (accordingly, a program-of-thought baseline is also missing). In CoGEX the model is responsible for simulating program execution, so it seems likely that it would underperform in these settings. While actually integrating an interpreter into a system that can still handle underspecified functions would be out of scope, it would be good to address this issue at the conceptual level somewhere in the text.\n\nQUESTIONS:\n1. Did you try few-shot with the Alpaca (instruction-tuned) models that you used zero-shot in the paper?\n\nLIMITATIONS:\nThe authors include a short statement to the effect that their models may make reasoning mistakes and the model artifacts are provided without any guarantees of safety. It might be worth including the fact that models may fail to execute generated code consistently, and thus generated code may be seen as an unfaithful explanation of model reasoning.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}