Title: GPT is becoming a Turing machine: Here are some ways to program it
Venue: NeurIPS 2023
Accepted: no
Keywords: Large Language Models

Turing machine

GPT-3

GPT-4

Prompt design

Programming
Abstract: We demonstrate that, through appropriate prompting, GPT-3 family of models can be triggered to perform iterative behaviours necessary to execute (rather than just write or recall) programs that involve loops, including several popular algorithms found in computer science curricula or software developer interviews. We trigger execution and description of Iterations by Regimenting Self-Attention (IRSA) in one (or a combination) of three ways: 1) Using strong repetitive structure in an example of an execution path of a target program for one particular input, 2) Prompting with fragments of execution paths, and 3) Explicitly forbidding (skipping) self-attention to parts of the generated text. On a dynamic program execution, IRSA leads to larger accuracy gains than replacing the model with the much more powerful GPT-4. IRSA has promising applications in education, as the prompts and responses resemble student assignments in data structures and algorithms classes. Our findings hold implications for evaluating LLMs, which typically target the in-context learning: We show that prompts that may not even cover one full task example can trigger algorithmic behaviour, allowing solving problems previously thought of as hard for LLMs, such as logical puzzles. Consequently, prompt design plays an even more critical role in LLM performance than previously recognized.

Abstract

We demonstrate that, through appropriate prompting, GPT-3 can be triggered to perform iterative behaviours necessary to execute (rather than just write or recall) programs that involve loops, including several popular algorithms found in computer science curricula or software developer interviews. We trigger execution and description of iterations by regimenting self-attention (IRSA) in one (or a combination) of three ways: 1) Using strong repetitive structure in an example of an execution path of a target program for one particular input, 2) Prompting with fragments of execution paths, and 3) Explicitly forbidding (skipping) selfattention to parts of the generated text. On a dynamic program execution, IRSA leads to larger accuracy gains than replacing the model with the much more powerful GPT-4. IRSA has promising applications in education, as the prompts and responses resemble student assignments in data structures and algorithms classes. Our findings hold implications for evaluating LLMs, which typically target the in-context learning: We show that prompts that may not even cover one full task example can trigger algorithmic behaviour, allowing solving problems previously thought of as hard for LLMs, such as logical puzzles. Consequently, prompt design plays an even more critical role in LLM performance than previously recognized.

1 Introduction

Large language models (LLMs) [2, 27, 5, 21] are trained on large text datasets, which typically include descriptions of procedures and even computer programs [4]. However, their performance on complex reasoning tasks remains limited despite using advanced prompting methods, such as Chain-of-Thought (CoT) [31, 40, 20, 38, 37, 42, 6, 36, 15, 11, 12]. This implies that despite the massive number of parameters and self-attention to all previous tokens, current LLMs are unlikely to solve problems that require many (or iterated) reasoning steps in a direct, savant-like manner. New benchmarks target these more complex tasks, such as logical deduction and logical grid puzzles in BIG-bench Lite [32], and in-context learning of these problems is typically poor. Practical applications like GitHub Copilot show a mix of promise and limitations: Copilot can auto-generate substantial amounts of code [25, 4], but falls short of expert programmers, lacking execution, state tracking, and debugging abilities (apart from anecdotal evidence, e.g. Fig. 3.7 in [3]; see Section A.3).

LLMs generate tokens in order, each based on many previous tokens in the sequence, whether these tokens were part of the prompt or had just been generated by the LLM itself. Such self-attention could allow an LLM to use all previously generated tokens as the scratchpad for tracking reasoning steps, states, etc. 1 . Such use of generated tokens would resemble a classical Turing Machine with its memory tape [34]. In principle, a non-trivial recurrent transformer model with infinite attention could

be Turing-complete and capable of executing arbitrary routines, as long as the attention mechanism 35 can be controlled stringently enough. But, even in relatively simple settings, LLMs appear to resist 36 strict controls, e.g., slight changes in prompts can yield dramatically different responses [14, 17, 30], 37 because many recurrent patterns in the training data are encoded into a single model, and learned 38 patterns overlap and vary in the context size. Thus it is easy to mislead with a prompt with accidental 39 alphabetical or numerical ordering, or some undetectable semantic bias [41, 16, 19]. 40

41

42

43

44

45

46

47

48

49

50

51

In Section 2, we introduce much stricter attention controls that instruct LLMs to unroll reasoning steps of a procedure with the initially undetermined length, and decide when the solution is found: Iteration by Regimenting Self-Attention (IRSA) . The basic way to achieve such deliberate selfattention control is through highly structured prompting with an example of execution path for one example, as illustrated for Bubble Sort algorithm in Prompt 1, which encourages an LLM to output not just the sorted sequence but also the swap count (response in Prompt A.1 in Appendix), which is a challenging task to solve in a savant manner. We further explore fragmented prompting which combines multiple fragments of execution paths, as well as the strategy of skipping parts of generated text when performing self-attention. We also discuss interpreter/compiler prompts that can translate an algorithm in a high-level programming language into an IRSA prompt that GPT-3 can execute (with details in Section 2.4 in the Appendix).

52

53

54

55

56

57

58

59

60

61

62

63

64

65

We present results on a wide range of algorithms taught in computer science curricula and used to test software engineers in coding interviews, including string manipulations, dynamic programming, and stack operations in Section 3. Our findings point to broader applications for LLMs beyond existing uses like Copilot in areas like software engineering and education [7, 24, 28, 18]. More pressingly, they point out a critical issue in evaluating in-context learning of LLMs, suggesting that current evaluations may underestimate LLMs' abilities if prompts can combine natural language instructions with algorithmic iterative reasoning. The sensitivity of the performance to prompt design may be amplified by the iterative reasoning triggered by the prompt, which will then beg the question: If one LLM beats the other on a task, is it simply because we have not found the right prompt for the second model? For example, IRSA prompting increases the performance of GPT-3 family on logical deduction puzzles from 32% to 76% (Table 1. The discussion in the Appendix also includes an experiment with GPT-4 [21] on a well-known dynamic programming task showing that even the latest member in the family cannot consistently execute code without prompting in IRSA style.

2 Iteration by Regimenting Self Attention (IRSA)

Prompt 1, as well as the prompts 2, A.4, A.5, and A.6 in the Appendix, illustrate the basic IRSA. In 66 each of these examples, a single prompt is provided for a task, which, when combined with a new 67 instance of the task, trigger the execution of an iterative algorithm. The algorithms are single loop 68 (Prompts A.5 and A.6) or double loop (Prompts 1, A.4, and 2); and may have a known or unknown 69 number of iterations until termination. 70

Crucially, the prompts show all state changes and explain each change before it occurs . Although 71 the explanation is colloquial, the structure of it is both rigid and repetitive, strictly regimenting the 72 attention to the rules (corresponding to program instructions) and state changes. In all these examples, 73 this strategy hardens the attention sufficiently to facilitate disciplined procedural reasoning, while 74 leaving non-regimented content open to interpretation. For example, Prompt 1 shows how a sequence 75 of 4 integers can be sorted in some detail, but the same prompt can also be used to sort characters 76 alphabetically or animals by size, and the procedure typically works for both shorter and longer lists. 77

78

79

80

81

82

These prompts could be thought of as an instance of Chain-of-Thought prompting [38, 37, 42, 6, 36, 15, 11]. However, a significant distinction lies in the number of reasoning steps, which is limited and fixed in usual CoT applications. In contrast, the algorithms explored here require an unspecified number of iterations required to complete the execution, and may even incorporate double loops. The prompt contains the condition for declaring the end of execution.

83

2.1 Using IRSA to reason over logical puzzles.

Iterative reasoning is required in solving a number of NLP word prolems, (e.g., [32]), not just in 84 execution of standard algorithms, as these algorithms can be employed after parsing the world 85 problem . The BIG-bench Logical Deduction task asks for the ordering of several objects given 86 their pairwise relationships described in natural language (e.g., a robin is standing on a branch to 87

Prompt 2. Logical deduction puzzle IRSA prompt. Playground link (use with 0 temperature)

the right of a raven, but a sparrow is the left-most). Despite the low number of objects (e.g., five) 88 in these puzzles, LLMs struggle to solve them in zero- or few-shot settings, much like how human 89 solvers typically cannot just see the correct answer instantly without scratch paper. This task is not 90 solved well by LLMs without external search/reasoning/inference algorithms, such as ThinkSum [22]. 91 However, a variant of BubbleSort algorithm adapted to this problem and shown in Prompt 2 can be 92 used to solve 76% of these puzzles. The prompt has a CoT structure that translates the problem into a 93 canonical form, and then, in IRSA style, describes an iterative swapping procedure that rearranges 94 the objects. 95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

2.2 Fragmented prompting.

Another way to trigger iterative behaviour is through fragmented prompting, illustrated in Prompt 3), which relies on:

Complete state specification . In contrast to Prompt 1 where iterative behaviour is induced indirectly through worked-out examples of multiple full loops, Prompt 3 explicitly describes state content in state-to-state transitions, including the iterator i .

Fragmentation. Prompt 3 does not fully cover the entire execution path of any single example. Instead, it follows the first three state changes 2 for the sequence 2 , 3 , 1 , 5 , and then stops in the middle of a sentence. Then it shows 6 additional fragments of execution paths for different problems.

Interestingly, fragmented prompting can also trigger iterative behaviour, where the language model accurately executes the algorithm on a given input and outputs END OF EXECUTION when the termination condition (no new updates on the sequence) is met. Viewing this prompt as an instance of in-context learning, it is challenging to classify it in usual terms. It goes beyond 0-shot learning as it contains explanations specific to the algorithmic sorting task. Yet, as opposed to what the few-shot CoT prompting might do, it does not work out any single example of array sorting. Instead, it provides fragments of patterns that can be stitched together to execute the algorithm (and GPT-3 CODE-DAVINCI-002 does execute it correctly for new inputs).

The potential advantage of such fragmented prompting is that the prompt can be shorter and include a greater variety of situations that may be encountered in new problems. A potential disadvantage is that the language model may get confused by the fragmentation and start hallucinating new independent fragments. In this case, we managed to avoid that by having the first fragment starting from the start of execution, going through several state transitions, and ending mid-sentence. Because of this, when a new problem is given, the language model starts running the execution path from the beginning, and later refers to various cases in the prompt for guidance on how to proceed.

2.3 Skip attention.

Prompt 3 also illustrates the idea of attention skipping. Whether using a single-execution or a fragmented prompt, if the state in the <state>*</state> structure is complete, the attention mechanism can generate the next token without attending to all the generated text. It is sufficient to attend to the prompt and the text generated after and including the last state.

If the skipping is implemented on the server side, akin to stop word functionality, then skipping unnecessary attention saves computation: The state of the model at the end of the prompt is cached and used to continue processing from the latest generated <state> marker, ignoring the text generated in-between. Skip-to-state can also be implemented on the client side, iteratively updating the original prompt by concatenating the latest <state>*</state> structure to the original prompt and calling the generative model with </state> as a stop sequence (We did the latter in our experiments). In both cases, the skip-to-state strategy should increase the number of tokens that can be generated, as self-attention, which grows linearly with the generated text, is the primary cause for the token limitations. Skip-to-state strategy keeps the self-attention cost constant. As IRSA requires the unrolling of potentially long iterative algorithms, these savings are important. For example, running a dynamic program that keeps track of 2D matrices is only practical in this manner. (See also [29] on an external memory approach to dealing with limited attention length. Here we deal with it by skipping parts of generated text, instead). Another advantage of skip-to-state attention is that by only attending to the necessary information, the generative model is less likely to get confused by accidental patterns created in its own generated text. (See more in Section A.3 and Figure A.1.)

Prompt 3. Fragments : An incomplete path for the first few Buble Sort state transitions for one sequence is followed by state transitions involving different sequences at different execution points. Initial part of the response is marked green. Skip attention : The part of the response up to the last state is not needed to continue the generation. Only the prompt, the last <state> * </state> , and the text after it are necessary to generate the next token. Playground link (use with 0 temperature)

2.4 GPT as a machine language: Prompting to interpret/compile a program. 140

Ageneral-purpose computer can execute algorithms that convert the text of a program into its machine 141 code. Analogously, we can design prompts with instructions on how to turn code in some language 142 into execution paths that can then be used in prompting. The details are given in Section A.1 in the 143 Appendix. In fact, we used a 'GPT compiler' in Prompt A.2 to create an execution path for the 144 double loop DP algorithm for finding the longest common subsequence (LCS) which we used in our 145 experiments. 146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

3 Experiments

We evaluated two versions of iteration by regimenting self-attention (IRSA):

Basic IRSA : Prompting with highly structured single execution path examples (Table 1). Although similar to CoT prompting, there are notable differences. CoT prompts typically provide multiple steps of reasoning shown for a few examples and have the LLM perform the same steps on a new example. Conversely, IRSA prompts are designed to trigger iterative reasoning that is repeated until the stop condition is reached and the solution is found. Furthermore, the execution path example for each task is deliberately chosen to be out-of-distribution (e.g., the Bubble Sort prompt features a worked-out example of sorting a four-number sequence in just three passes, while the dataset consists of five-number sequences requiring 2 to 5 iterations and up to 20 state transitions, with varying complexity across problems). Thus in terms of information they provide, these prompts can be seen as somewhere between single-shot and zero-shot prompts.

Skip-to-state IRSA : Prompting as above, but with additional forced attention skipping. In this approach, the LLM is forced to attend only to the prompt and the last generated state as it iterates through the input to find the solution (illustrated at the end of Prompt 3). We also evaluate fragmented prompts (Table 2), where the prompt does not consist of a single complete execution path for an example, but instead shows several state-to-state transitions for different inputs.

Baselines. To make fair comparisons and avoid unnecessary recomputation, we reused existing baselines from [32] wherever possible, denoted by an asterisk (*) (especially considering that these baselines typically perform close to random guessing on certain tasks). We reused these datasets and baselines for the following tasks: Logical deduction, Balanced parenthesis, and Longest common subsequences for long sequences. We created our own datasets and ran baselines for the following tasks: Bubble sort, Longest substring without repeating characters, and Longest common subsequence for short sequences. We include the best result from [32] for the GPT family, as our experiments were mainly conducted using GPT-3. Our baselines included zero or few (5) shot prompting with or without relevant code added to the description of the task in the prompt (e.g. Prompt A.9). Few shot baselines were made with 5 different random choices of examples to be included in the prompt. The 'Guessing' strategy refers to picking the most frequently correct answer for a given task as a guess for each problem in the task, which is different from truly random guessing. Few-shot prompting could prime the answers to pick the most frequently seen answer, even when no understanding of the problem occurs, which makes our 'Guessing' strategy more reflective of the task difficulty.

Models. We have briefly experimented with different members of the GPT-3 family, but ran complete experiments with CODE-DAVINCI-002 for two reasons: TEXT-DAVINICI-002 and 003 often produced qualitatively similar results, and experimentation with the CODE-DAVINCI-002 was easier due to better combination of token quota and availability. Having been tuned on code, this model may have slight advantages over models tuned for more natural language tasks. Nevertheless, as we show in the experiments and discuss in Section A.3, without IRSA, CODE-DAVINCI-002 cannot solve the problems discussed here, even when it can generate the code that could. To induce iterative reasoning in LLMs, it appears that attention needs to be highly regimented through strong structure, and possibly additional attention control, such as the skip-to-state strategy we described in Section 2.3. This also applies to GPT-4 [21]: In Section A.3.3 in Appendix, we show that prompting GPT-4 with straight-forward Prompts A.10, A.11, A.12 does not match the performance of IRSA in GPT-3.

Datasets. To test our proposed methods with various prompting baselines, we focus on challenging programming tasks including computer science algorithms from the school curricula and coding interviews for software engineers as follows.

Bubble sort. We created a dataset of 100 random non-repeating digit sequences of length 5. For each 192 sequence, we ran the bubble sort algorithm to establish the total number of element swaps it requires. 193

The task is to predict the number of swaps for a given sequence. 194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

Longest substring without repeating characters. A classical coding interview question: Given a string of letters, find the longest contiguous substring such that no letter appears more than once. We created a dataset of 100 random strings of length 7, and for each found the length of the longest subsequence without repeating characters. The task is to predict that length for a given sequence.

Logical deduction [32]. We include this task (Section 2.1) in experiments to emphasize the broad importance of triggering iteration in LLMs responses. Enabling LLMs to execute iterative algorithms through effective prompting could help solve numerous reasoning problems. In particualr, this task that involves solving a puzzle about an order of items/objects/persons, such as books on the shelf, birds on a branch, cars, golfers, etc., given several clues, such as 'minivan is more expensive than the car', or 'the robin is to the left of the finch.' We focus on a subtask involving 5 items, with varying sets of items and the types of ordering across the puzzles. While in-context learning with LLMs consistently solves less than 35% of puzzles, a recent combination of GPT-3 and probabilistic reasoning [22] was able to solve 77% of the puzzles. We reach a similar performance through IRSA, without an additional external reasoning mechanism.

Valid parentheses [32]. The task is the first of the two in the cs-algorithms challenge in BIGbench. The goal is to evaluate LLMs ability to perform reasoning equivalent to the classical stack manipulations needed to check if a sequence of parentheses of different types is balanced. LLMs (including GPT) tend to do about the same as chance ( 50% ), except for PaLM with 3 shots, which gets around 75% accuracy.

Longest common subsequence (long) [32]. The second task in BIG-bench cs-algorithms involves solving the classical dynamic programming problem. Defining a subsequence of a sequence to be a sequence of symbols one could get by skipping arbitrary stretches in the original sequence, the task is to find the length of the longest subsequence common to two given sequences. LLMs do not do much better than chance on this task ( ∼ 10%).

Longest common subsequence (short). We created this dataset in the same manner as the above one from the BIG-bench, but with the constraint on the sequence lengths, limiting them to a maximum of 6 characters. This allows us to evaluate IRSA on more cases, ensuring it does not run out-of-memory (tokens) in generation 3 .

Basic IRSA results. The basic IRSA results are summarized in Table 1. For Bubble Sort evaluations, we show the results using both Prompt 1, and Prompt A.4. The latter is a single execution path for the same problem ( 2 , 3 , 1 , 5 ), but in the style of Fragmented Prompt 3 by continuing the execution path initiated by Prompt 3, without incorporating fragments from other paths. The former had an accuracy of 74% for inferring the numbers of swaps necessary to sort different sequences, while the latter achieved 100% . Note that while the execution path for the example 2 , 3 , 1 , 5 requires three iterations of the outer loop and three iterations in each inner loop, the dataset contains sequences of length 5 and thus requires four iterations in the inner loop and a variable number of iterations of the outside loop - anywhere from 2 to 5 - and yet the model can execute the correct number of iterations based on the stoppage criterion (that in the inner loop, no changes were made to the sequence).

For the logical deduction puzzles, we used the Prompt 2 Appendix. Note that the logic of the iterative reasoning there is faulty as it may enter an infinite loop. When that happens, the generation runs out of tokens and we simply used the answer after the 4th iteration in evaluation. Further

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

discussion in Section A.3 suggests the potential for creating more effective prompts. Nevertheless, with this prompt to induce iterative reasoning, we still reach the state-of-the-art results, comparable only with [22], which uses an external reasoning mechanism in conjunction with prompting. To solve the longest substring without repeating characters problems, we developed Prompt A.5 based on asingle-pass algorithm (Section A.2), which, interestingly, trades computation for memory. To address the parentheses problem, we used the single execution path that demonstrates stack operations for determining whether the sequence is balanced or not. The beginning and the end are shown in Prompt A.6 and discussed in Section A.2.1 in the Appendix.

Skip-to-state attention results. The dynamic programming solution to the longest common subsequence (LCS) problem requires its state including a ( M +1) × ( N +1) matrix storing the solution for all prefixes of the two sequences of lengths M and N . Without skip-to-state attention (Section 2.3), the API calls run out of tokens before reaching the end for all but the shortest problems. Using the approach described in Section 2.4, A.1, we compiled an execution path in Prompt A.3, and then used it to induce IRSA on LCS short (LCS-S) and LCS long (LCS-L) problems. Even with skip attention, the state was too large to fit the token limit for most of the problems in LCS-L from BIG-bench. Yet, IRSA with skip attention still beats the state-of-the-art significantly (Table 2). On shorter problems in LCS-S, where IRSA with skip-attention does not run out of tokens, the performance was a respectable 93% . Note that even GPT-4, without IRSA, can only reach 69% accuracy on LCS-S (Section A.3.3).

We tested fragmented prompting of Bubble Sort execution (Table 2). For each selected number of fragments - 7, 13, 19, 25 - at least one of five randomly generated prompts achieved 100% accuracy. These prompts followed the format in Prompt 3, starting with the few state transitions from the beginning for the sequence [2 , 3 , 1 , 5] and then listing additional 6, 12, 18, or 24 fragments. Bubble Sort has 6 different transitions, and fully balanced instruction listing one, two, three, or four of each type, with a random sequence in the state, leads to a slightly better performance than having a completely randomly chosen execution path fragments. These six basic transitions, illustrated in Prompt 3, involve two ways of ending an iteration depending on the swap flag and four ways of changing the state: two possibilities for inequality being true or not, combined with two possible previous values of the swap flag. We found that the prompt sensitivity causes different prompts to fail for different test cases: Each of the fragmented prompt collections yields 100% as an ensemble.

4 Conclusion

We demonstrated that GPT-3 can be triggered to execute iterative algorithms, including double loops, 266 with variable termination conditions. This has several consequences discussed in Appendix (Section 267 A.3). For example, IRSA may find applications in sofware engineering and education. If LLMs are 268 Turing Machines (in addition to being natural language translators and analyzers), their evaluation 269 probably needs to be rethought, esp. in cases where models are expected to make inferences for 270 which we have algorithms, because in-context learning would cover prompts designed to execute 271 them (Section A.3). Regimenting self-attention for a given task may require a different level of effort 272 (Section A.3.2, but even GPT-4 cannot execute programs consistently without IRSA (Section A.3.3). 273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

References

Yoshua Bengio. The consciousness prior. arXiv preprint arXiv:1709.08568 , 2017.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Neural Information Processing Systems (NeurIPS) , 2020.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712 , 2022.

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435 , 2022.

Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of human cognition. arXiv preprint arXiv:2011.15091 , 2020.

Daniel Kahneman. Thinking, fast and slow . Macmillan, 2011.

Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406 , 2022.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 , 2022.

Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336 , 2022.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science , 378(6624):1092-1097, 2022.

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt3 ? arXiv preprint arXiv:2101.06804 , 2021.

Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad 322 Shoeybi, and Bryan Catanzaro. Multi-stage prompting for knowledgeable dialogue generation. 323 In Findings of the Association for Computational Linguistics: ACL 2022 , pages 1317-1337, 324 Dublin, Ireland, May 2022. Association for Computational Linguistics. 325

326

327

328

329

330

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics.

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Coherence boosting: When your pretrained language model is not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8214-8236, 2022.

Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 , 2023.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022.

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021.

OpenAI. Gpt-4 technical report, 2023.

Batu Ozturkler, Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Thinksum: Probabilistic reasoning over sets using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , 2023.

Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014 , 2023.

Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255 , 2022.

Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity: Evidence from github copilot, 2023.

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350 , 2022.

Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446 , 2021.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 , 2023.

Dale Schuurmans. Memory augmented large language models are computationally universal. arXiv preprint arXiv:2301.04589 , 2023.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.00093 , 2023.

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 4615-4629, Online, November 2020. Association for Computational Linguistics.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging bigbench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022.

Alan M. Turing. On computable numbers, with an application to the Entscheidungsproblem. Proceedings of the London Mathematical Society , 2(42):230-265, 1936.

Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. Science , 185(4157):11241131, 1974.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Rationale388 augmented ensembles in language models. arXiv preprint arXiv:2207.00747 , 2022. 389

390

391

392

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Selfconsistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022.

393

394

395

396

397

398

399

400

401

402

403

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 , 2022.

Eric Zelikman, Yuhuai Wu, and Noah D Goodman. STaR: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203.14465 , 2022.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning , pages 12697-12706. PMLR, 2021.

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale 404 Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables 405 complex reasoning in large language models. arXiv preprint arXiv:2205.10625 , 2022. 406

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a way to prompt GPT-3 to exhibit behavior simulating execution of iterative programs. Authors propose the following prompt constructs: providing structured examples of program execution; using fragments of execution; not using self-attention on some parts of the generated text. Authors compare the results to baselines and show significant improvements\n\nSTRENGTHS:\n- Authors introduce a method that enables GPT-3 to mimic the execution of iterative programs. They achieve this by supplying the model with intermediate steps and outcomes.\nThis is somewhat novel and could be useful for using LLMs to solve problems that require iterative processing. \n- The use of path fragments may prove beneficial in situations where the context size is insufficient for comprehensive examples.\n- The strategy of confining self-attention to particular segments of the output might be advantageous when the context size needs to be considered\n- The examples provided in the paper are well written\n\nWEAKNESSES:\n- Authors' approach requires the manual construction of prompts for each problem at hand. It is not automated and not scalable. This limits the usefulness of the approach in practice.\n- Authors compare their approach to simple baselines. There should be a comparison to at least chain-of-thought reasoning. \n- Paper is hard to read and accept as a standalone without appendices. Authors refer to the content in the appendices too much.\n- The significance of the work is low. It is known that LLMs can produce iterative output. Although the authors have enhanced the quality of such outputs via structured prompting, structured prompting is not entirely novel. Same can be said about using fragments in prompt/context.\n\nEDIT: I have raised my evaluation of the paper from 3 to 4. Authors have promised to address the issues I and other reviewers have raised. However, in my opinion, such changes would require a major rewrite of the paper. I am not confident if these changes can be done well for the publication.\n\nI have read the author’s rebuttal and further discussion with authors based on my questions and feedback. \nAuthors' rebuttal addressed some of my concerns by more in depth discussion of IRSA relationship to chain-of-thought reasoning and possible automated generation of IRSA prompts.\n\nQUESTIONS:\nIs there a generic prompt structure that could be used for any algorithm or some class of algorithms?\n\nIn Prompt 3 is the “Final List: 6, 7, 3, 5” intentional? It is not sorted.\n\nLIMITATIONS:\nIt would be good if authors explored the limitations of what can be achieved by their approach. At some point the LLM \"execution\" (simulation really) of the program should fail. The points at which the simulation would fail likely depend on the input/output/context size. It may also depend on the algorithm semantic and/or algorithmic complexity. (Those are different complexities and may affect the breaking point differently). These questions could be explored and would be useful to know.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMaking LLMs follow procedural rules precisely, as done when executing a program, is been a challenging task. In this paper, the authors introduce iterations by regimenting self-attention (IRSA), a prompting technique to make large-language models (LLMs) *execute* a hand coded programs (on novel inputs) precisely. The authors propose three techniques for IRSA: 1) by prompting the LLM with a step-by-step example showing many state-transitions in details, 2) By prompting with fragments of the state-to-state transitions only (along with the latest state for which it predicts the transition), and finally 3)skipping attention on intermediate state-to-state transitions. LLMs are shown to be significantly more successful at tasks such as sorting arrays, finding longest sub-sequence in a string, or simpler logical puzzles etc. when prompted with IRSA.\n\nSTRENGTHS:\n### Originality\n\nRecently, many new proposals for prompting LLMs, including scratchpad, and Chain-of-Though prompting have been proposed. However, most approaches focus on making LLMs reason and solve problems/puzzles. Instead, in this paper, the focus is on making LLMs *follow instructions accurately* (which then can be used for solving certain puzzle). This is a novel and original direction.\n\n### Quality\nThe paper is well presented, including the figures, and the tables. Additionally, the experiments have been conducted on many tasks to sho\n\n### Clarity\nThe paper is very clearly written and well presented. Specifically, I found the prompt examples very useful in understanding the paper's ideas.\n\n### significance\nGetting LLMs to precisely execute procedural rules is of intereset to the research community at large. LLMs inability to successfully tackle procedural problems (such as multiplication) of complexity beyond the training set complexity has raised questions regarding its ability tackle compositional problems. This paper provide an important perspective and countering result that will further enrich this discussion.\n\nWEAKNESSES:\nThe two main drawbacks of the paper are motivation and experiments. \n\n### Motivation\nThe paper does not sufficiently motivate the problem statement. Why should we care about making LLMs execute programs - perform iterative behavior? When the process is deterministic and easy to programmatically describe, why would we prefer LLMs over a deterministic typical program (one can even use programs which explicitly *show* transition rules applied as well)? The authors mention education or software engineering vaguely, but there is no concrete motivation in these use-cases (when would a LLM be more suitable for this task over a REPL-like loop with python/cpp?).\n\n\n### Experiments\n\n1) The authors do not evaluate on significantly larger sequences sizes. Since fragmented prompting seems to allow arbitrarily large sequence of state-transitions, I believe authors can indeed use IRSA on larger sequence problems. The trend between success rate and sequence length would be insightful.\n2) I strongly appreciate the authors for showing the negative result in Figure A.1 (Appendix section A.3.2). This shows that despite using IRSA, the model may end up performing wrong state-transitions based on its correlation to patterns in recent history. If multiple previous state transitions contain sequences where the statement \"2 < x = True\" appears, then when asked \"2 < 1 =\" the LLM has higher likelihood of filling True than False. This seems to directly negate the claim of this paper that we can make LLMs execute programs precisely. It clearly seems to be affected by the prompt history which can make them act in unreliable ways.\n\nQUESTIONS:\nI would appreciate if the authors can respond to the weaknesses raised above.\n\nLIMITATIONS:\nYes the authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed several novel prompting methods that could trigger GPT-3 to perform iterative behavior for executing algorithms with loops. The main technique the paper presented, IRSA, is to use highly structured prompts that contains information about unrolled execution trace, program states, and a detailed explanation of the motivation of a specific action, thus bringing strict attention controls to LLMs and help them reason about the procedures to get the solution. \n\nBased on the proposed IRSA method, the authors also introduced two alternative prompting methods: Fragmented prompting and Skip attention. The fragmented prompting technique strips out some of the iterations in the full execution trace, thus enabling the prompt to contain more diverse scenarios under a limited prompt length. The skip attention technique explicitly marks program state information with a special token and puts emphasis on the original prompt that serves as a demonstration to the LLM and the last execution state where the LLM could continue its execution from, thus also bringing LLM server side and client side optimization opportunities. Meanwhile, the authors also briefly discussed the automatic generation of the proposed prompts using  LLMs. \n\nThe proposed prompting methods were evaluated on various tasks whose solutions involved loops. It is shown by the evaluations that the proposed methods could achieve state-of-the-art results on multiple tasks.\n\nSTRENGTHS:\n1. The paper showed great originality and insights in that the authors identified the failing reason of the LLM on tasks involving iterations, considered related Turing machine concepts, and designed several novel prompting methods incorporating highly structured information about unrolled execution trace, program states, and a detailed explanation of the motivation of a specific action, bringing strict attention controls to LLMs and help them reasoning about the procedures to get the solution. \n\n2. The proposed prompting methods showed state-of-the-art results on multiple loop-involving tasks.\n\n3. The proposed Fragmented prompting method can be a promising technique that could encode diverse scenarios while keeping the prompt relatively short. This can be helpful for working with LLM APIs.\n\n4. The proposed Skip attention prompting method can also be promising in that it emphasizes the concept of program state in the context of using LLMs as general Turing machines. This technique could also help reduce prompt length, and with adequate supporting modifications and implementations on LLMs, this can be a solid base for future works.\n\nWEAKNESSES:\nThe overall presentation of the work can be relatively hard to comprehend for the readers. Especially for the presentation of the proposed Skip attention, it could be better if the authors provided a figure that briefly demonstrates the idea of the server and client-side implementations of the method.\n\nQUESTIONS:\n1. From the presentations of the work, it seems like the potential of the proposed Skip attention method was not fully demonstrated due to the prompt length limitations of the LLM APIs. Can the authors further evaluate the method on other LLMs (with online APIs or a local deployment) that accept longer prompts? Is it possible to adapt some encoding scheme that further reduces the prompt length, thus enabling the method to be applied to more tasks?\n\n2. Could you provide some quantitative results of the experiments on prompting to compile a program? For example, the success rate of triggering GPT3 to execute iterative algorithms on a certain task such as LCS.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the use of regimented self-attention (IRSA) to prompt GPT-3 to perform iterative behaviors necessary for executing programs involving loops. The authors investigate three approaches to trigger the execution and description of iterations. The results suggest that IRSA leads to larger accuracy gains than using the more powerful GPT-4 for dynamic program execution. The authors highlight the potential applications of IRSA in education and discuss the implications for evaluating large language models (LLMs). While LLMs have limitations in complex reasoning tasks, prompt design plays a crucial role in their performance.\n\nSTRENGTHS:\n+ Interesting problem and approach\n+ Providing examples of prompts\n\nWEAKNESSES:\n- Presentation\n- Concern about reliability\n- Concern about \"Turning machine\" claims\n\nQUESTIONS:\nThere are a couple of points that could benefit from further clarification and discussion:\n\n1. The paper mentions (Line 71-73): \"it is easy to mislead with a prompt with accidental alphabetical or numerical ordering, or some undetectable semantic bias,\" and \"slight changes in prompts can yield dramatically different responses.\" However, the authors do not provide any specific syntax/structure for the IRSA query prompting scheme using the CoT paradigm. It is unclear whether changing a certain text ordering in these prompts will produce the same performance. In other words, this raises concerns about the reliability and consistency of LLMs in generating accurate outputs. Can you elaborate on the robustness of IRSA to different prompt variations and its sensitivity to prompt design choices?\n\n2. Table 1: What does guessing mean for the longest substring, logical deduction?\n\n3. The experimental results show significant performance improvements when using IRSA on logical deduction puzzles. However, it would be valuable to understand the generalizability of these findings. How does IRSA perform on other complex reasoning tasks beyond logical puzzles?\n\n4. The paper claims that IRSA outperforms GPT-4 on dynamic program execution but lacks a thorough comparison or analysis of GPT -4's performance in the main paper. Can you provide more insights into the limitations of GPT-4 and why it fails to consistently execute code without IRSA prompting?\n\n5. Line 268-269: Is there proof of this? It seems a far-fetched & ambitious statement that LLMs are similar or equivalent to Turing machines. Moreover, the authors mention \"... becoming a Turing machine\" in the title (that's a huge claim), but there are only two other places in the paper that mention \"Turing machine\" in the sentence. The authors should add more details to justify this more concretely.\n\n6. Github URL seems to be not anonymous\n\n7. The names of the methods \"Iteration by Regimenting Self Attention (IRSA),\" and \"Skip attention\" are misleading. It makes the reader feel that it is related to the internal attention mechanism of a GPT, but actually it deals with query/prompt engineering.\n\n8. The paper seems to be written in a hurry, so there are a couple of typos and some discontinuity while reading.\n\nLIMITATIONS:\n* The authors make huge claims, but do not discuss the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces Iterations by Regimenting Self-Attention (IRSA) which is a set of LLM prompting techniques for producing repetitive, algorithm-like behavior that can be useful for a range of tasks, such as carrying out a sorting algorithm or solving a logic puzzle. There are 3 techniques discussed: 1) \"Basic IRSA\" which is a chain of thought prompt that looks a bit like an execution trace of some natural-language-like pseudocode - theres a lot of repetitive structure, the current state is verbosely repeated after each step, and changes to the state are explicitly describe before the happen. 2) \"Fragments\" which is the idea that instead of prompting with a full trace of an algorithm you can just prompt with random unordered individual steps of the algorithm to prepare the model for executing a random step. 3) \"Skip Attention\" where only the most recently produced state is attended to (plus the original, fragment-based prompt), since changes to the state should be independent of the history of states – this cuts down on computation and helps the LLM not get confused by patterns in its recent output.\n\nSTRENGTHS:\n- Skip attention and fragments (which pair well together) are great ideas, and are original as far as I know – other reviewers can correct me if I'm wrong. In many algorithms (and in fact, in the execution of interpreted code in general) only the current state matters as opposed to the history of how the state has changed. Only showing the most recent state to the LLM makes a lot of sense. It saves on computation cost and makes long running algorithms feasible, since there's no need to attend over the whole history of generations (which would become a huge problem as an algorithm runs for dozens or hundreds of steps). As the authors point out, this Markovian setup of not looking at the history of states also means that the LLM won't get confused by patterns in its recent history.\n  - The \"fragments\" approach is a clever way to get the LLM used to this idea of seeing somewhat random states and needing to do a single algorithmic step for each one.\n  - Skip attention makes so much sense, I'm surprised past work like \"Show Your Work\" (Nye et al 2021) didn't take an approach like this, since I imagine it would work fine with executing interpreted Python programs (where the state is the set of local variables/values along with the current line number in the program, and the LLM just has to output a next set of local variables and next line number).\n- More generally, getting LLMs to do things that look more like rigid computation can be difficult and I think that this is a paper with a pretty good evaluation of a particular approach to this problem, and would be useful for the NeurIPS community to see.\n- The evaluation is reasonable and shows unsurprisingly that the skip attention method can work great and generalize to very long sequences (eg bubble sort with 25 steps).\n\nWEAKNESSES:\n- The descriptions of what IRSA is were quite difficult for me to understand. The first line of section 2, the section describing IRSA, is \"Prompt 1, as well as the prompts 2, A.4, A.5, and A.6 in the Appendix, illustrate the basic IRSA.\" (line 66). Written as is this feels a bit overwhelming as it suggests that I need to look at 5 different prompts (including 3 in the appendix) and try to look for the common features among them to figure out the method. Also, in reality many of these references (2, A.4, A.5, A.6) are actually going to show up later on in places where they're discussed so it's okay if I don't look in detail at them now, but since I haven't been told that I feel some need to dig them all up before continuing.\n    - A flow for section 2 that would be much more understandable to me (and I believe others) would be the following. This is just one suggested way of doing it and I think there are many valid ways that would be widely understandable (you dont need to do the below), but the current flow is difficult to understand:\n        - Give a brief but concise description of the key feature of IRSA (similar to lines 71-73 right now) so we're primed with looking for that *before* we're told about any prompts to look at. I might even suggest that instead of putting the CoT comparison at the end (lines 78-82), it might flow nicer to actually frame it *in terms of CoT / as an extension building on CoT* since that is a closely related framework many readers know about. In general after reading the paper I actually still find I have trouble precisely articulating what makes something count as \"basic IRSA\", so presenting it from the start in terms of its relation to CoT might be helpful.\n        - Tell us to look at Prompt 1, and briefly walk us through what we're looking at / why this is IRSA. \n        - Mention that the precise keywords/format of Prompt 1 (\"EXECUTION\", \"Prep\", \"EndPrep\" \"Iteration\", the indentations, \"State:\") are not important (IRSA is not a set of specific keywords to use) and point to Prompt 2 as an example of something that looks different on the surface level but is still IRSA.\n        - At this point, you might parenthetically refer to the 3 appendix prompts as additional examples used in the evaluation that the reader can look to if they want more.\n    - Again, to be totally clear, I'm not prescribing this format, I just find the current flow difficult to understand so I took a stab at restructuring it, but there are many other ways of doing so that would also flow well.\n- I'd like to see some discussion of how this relates to the paper \"Show Your Work: Scratchpads for Intermediate Computation with Language Models\" (Nye et al 2021) which is currently just referenced by the paper in the list of CoT related works without specific discussion. In that work, the authors showed that while LLMs are bad at directly predicting the output of a Python function called on certain inputs, they could instead have the LLM repeatedly output the current state (what the variables are set to) plus the next line of the program to run. That was essentially a form of CoT with extra structure. IRSA seems somewhere in between the strict state/instruction format of Show Your Work and the free flowing reasoning of more general CoT. I think an explicit comparison to that paper (and/or any other paper that does some form of rigid CoT) is important, so it's clear how this work should be viewed in relation to others that have structured CoT.\n\n- I'm generally coming out of this paper still somewhat unsure what precisely \"basic IRSA\" is (i.e., without fragmenting or skip attention), and I felt I could only gesture towards some of its important features when writing the Summary section above.\n    - It feels related to CoT and I'd like to understand it in terms of that. One section comparing CoT to IRSA says \"a significant distinction lies in the number of reasoning steps, which is limited and fixed in *usual* CoT applications\" (emphasis is mine) (lines 78-82) but this is not always true (e.g. in the Show Your Work paper above – so does that make Show Your Work and instance of IRSA and this present paper is proposing a general framework encompassing that?).\n\n- (minor weakness) Section 2.4 sounds interesting but is largely confined to the appendix. It doesn't really flow with the rest of the story and as far as I can tell isn't used in the evaluation alter. But I'm a bit torn because it is actually quite cool and maybe it doesn't hurt to have as just an aside (though maybe with a slightly more clear verbose explanation). I don't terribly hold this one against the paper, it just feels a little out of place. \n\n**Overall** I think that, though it is difficult to follow the flow of this paper in places so it took me quite a while to understand, and I'm still not totally clear on what makes something \"basic IRSA\" or how it relates to prior work like Show Your Work, I think that in particular given the contributions of skip attention and fragments this would still be valuable work for the NeurIPS community to see. The positives outweigh the negatives in my view, but with revisions around the points mentioned above I would be more supportive.\n\nQUESTIONS:\n- (as discussed in Weaknesses) How does IRSA relate to the \"Show Your Work\" paper?\n- (as discussed in Weaknesses) How, precisely, does IRSA relate to CoT?\n- While full LLM prompts can be useful, using 3 full pages for three full-page LLM prompts is a lot, and you might consider abbreviating some of these prompts to keep the key bits (while leaving the full version in the appendix).\n- \"world\" -> \"word\" typo in the bolded text of line 85\n- \"the Prompt 2 Appendix\" (line 233) seems like some sort of typo, is this prompt 2 or prompt A.2 or something else?\n- For Table 1 Instead of saying Prompt 1 and Prompt A.4 I would say things like \"Base IRSA (Prompt 1)\" or something, so that at-a-glance you can understand these results without having to remember / look up what exactly prompt 1 and prompt A.4 are. It'd also be helpful for the entries like \"Longest substring\" to actually say which variant of IRSA was used for this result (or to put that in the caption, or to put it as separate columns, or do anything else that just makes it easy at a glance to see which IRSA method you're talking about).\n\nLIMITATIONS:\nThe authors address limitations adequately", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}