Large Language Models as General Pattern Machines
Authors: Suvir Mirchandani
Fei Xia
Pete Florence
brian ichter
Danny Driess
Montserrat Gonzalez Arenas
Kanishka Rao
Dorsa Sadigh
Andy Zeng
Venue: CoRL 2023 Poster (2023)
Keywords: large language models
in-context learning
language for robotics

Abstract: We observe that pre-trained large language models (LLMs) are capable of autoregressively completing complex token sequences—from arbitrary ones procedurally generated by probabilistic context-free grammars (PCFG), to more rich spatial patterns found in the Abstraction and Reasoning Corpus (ARC), a general AI benchmark, prompted in the style of ASCII art. Surprisingly, pattern completion proficiency can be partially retained even when the sequences are expressed using tokens randomly sampled from the vocabulary. These results suggest that without any additional training, LLMs can serve as general sequence modelers, driven by in-context learning. In this work, we investigate how these zero-shot capabilities may be applied to problems in robotics—from extrapolating sequences of numbers that represent states over time to complete simple motions, to least-to-most prompting of reward-conditioned trajectories that can discover and represent closed-loop policies (e.g., a stabilizing controller for CartPole). While difficult to deploy today for real systems due to latency, context size limitations, and compute costs, the approach of using LLMs to drive low-level control may provide an exciting glimpse into how the patterns among words could be transferred to actions.

Large Language Models as General Pattern Machines
Suvir Mirchandani', Fei Xia², Pete Florence', Brian Ichter?, Danny Driess Montserrat Gonzalez Arenas?, Kanishka Rao², Dorsa Sadigh' Andy Zeng² SStanford University, GGoogle DeepMind, 'TU Berlin
Abstract: We observe that pre-trained large language models (LLMs) are capable au- toregressively completing complex token sequences from arbitrary ones procedurally generated by prohabilistic context-free grammars (PCFG), to rich spatial par- tems found in the Abstraction and Reasoning Corpus (ARC), general AI benchmark, prompted in the style of ASCII art. Surprisingly, pattern completion proficiency can be partially retained even when the sequences are expressed using tokens randomly sampled from the vocabulary. These results suggest that without any additional training. LLMs can serve as general sequence modelers, driven by in-context learning. this work, we investigate how these zero-shot capabilities may applied to problems in robotics from extrapolating sequences of numbers that represent states over time complete simple motions, least-to- tostoottttttttooooo prompting of discover and represent closed-loop policies (e.g.. a stabilizing controller for CartPole). While difficult to deploy today for real systems due to latency, context size limitations, and compute costs, the approach of using LLMs to drive low-level control may provide an exciting glimpse into how the pattems among words could be transferred to actions.
trajectories that
Keywords: large language models, in-context learning, language for robotics
1
Introduction
Large language models (LLMs) are trained to absorb the myriad of patterns that are woven into the structure of language. They not only exhibit various out-of-the oofthhe-box capabilities such as generating chains of reasoning [1, 2]. solving logic problems [3, 4], and completing math puzzles [5), but also have been applied in robotics where they can serve as high-level planners for instruction following tasks [6, 7, 8, 9, 10, 11, 12), synthesize programs representing robot policies [13, 14], design reward functions [15, 16), and generalize user pref- erences [17]. These settings rely on few-shot in-context examples in text prompts that specify the domain and input-output format for their tasks [18, 19]. and remain highly semantic their inputs and outputs.
key observation of our work-and perhaps contrary to the predominant intuition-is that LLM's ability to represent, manipulate, and extrapolate more abstract, nonlinguistic patterns may allow them to serve basic versions of general pattern machines. To illustrate this idea, consider the Abstraction and Reasoning Corpus [20], a general Al benchmark that contains collections of 2D grids with patterns that evoke abstract concepts (c.g., infilling, counting, and rotating shapes). Each problem provides small number of input-output examples, followed by test input(s) for which the objective is to predict the corresponding output. Most methods (based on program synthesis) are manually engincered with Nomain-specitf languages [21, 22, 23, 24] evaluated on simplified extensions or subsets of the benchmark [25, 26, 27]. and-tt--end machine learning methods only solve handful of test problems [28]; however, our experiments indicate that L.MMs in-context prompted in the style of ASCII art (see Fig. 1) can correctly predict solutions for up to 85 (out of 800) problems exceeding some existing recent systems [21, 22, 24]. without additional model training or fine-tuning. Surprisingly, we find this extends beyond ASCII numbers, and that when they
Anpuc: 0, 8,
Snout:
impats 9, 8, 8. -h,
e. 8, 0,
a.
8,
a,
a.
6. a.
s. B. 6, a.
a.
6. 6. o.
Fig. I: LLMS out-of-the-hox can complete (highlighted) complex ARC patterns [20] expressed arbitrary tokens.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Sequence Transformation
Sequence Completion Simple function classes (numeric) S Snusold Extropolation
Sequence Improvement
Pattern transforinations (symbolic)
Online policies (numeric symbolic)
Asstract Reasoning Corpus
CartPole
200
Max reweard
Total Rowara
improvement
rolloutt
78, 76, 72, 85 60, 53,
Episodes
Object Reerrergeement
Wisteboard Drawing
Pushing vie Cliker Training
(Scripted)
Completion
Clickrr
Taple Sweeppnng
(Kinettweiic)
Completion
# Steps
Fig. Pre-trained LLMs out-of-the-box may serve as basic versions of general pattern machines can recogmize and complete sequences of numeric or arbitrary (symbolic) tokens expressing abstract problems roboties and sequential decision-making. Experiments show that extent, LLMs can in-context learn (i) sequence transformations (e.g.. reason spatial rearrangements of symbols, for dynamics modeling and images), (ii) completion of simple functions (e.g.. to extrapolate kinesthetic demonstrations), or (iii) meta- meeta-patterns to improve return-couu conditonn policies (e.g.. to discover oscillatory behaviors to stabilize a CartPole).
prediction on downsampled
replaced with mapping to randomly sampled tokens in the vocabulary, LLMs can still generate some valid solutions. These results suggest intriguing insight: that LLMs may exhibit more general capabilities of representing and extrapolating symbolic patterns, invariant to the specific tokens involved. This is in-line with-and complementary to-recent observations that using random or abstract label mappings for in-context classification retains some performance compared to ground-truth labels [29, 30]. We hypothesize that the capabilities that drive pattern reasoning on the ARC may allow general pattern manipulation various levels of abstraction useful for robotics and sequential decision making [31, 32]. wherein a diverse array of problems involve patterns that may be difficult to reason about precisely in words. For example, procedure for spatially rearranging tabletop objects could be represented using arbitrary tokens (see Fig. 2). As another example, optimizing trajectory with respect to a reward function can be framed as extrapolating sequence consisting of state and action tokens with increasing returns.
Orthogonal and complementary to efforts that develop multi-task policies by pre-training on large amounts of robot data [33], robotics foundation models [34] that can be fine-tuned for downstream tasks [35, 36, 37), our goal is instead (i) assess the zero-shot capabilities that LLMs may already contain perform some degree of general pattern manipulation, and (ii) investigate how these abilities can be used in robotics. These capabilities are certainly not sufficient to replace specialized algorithms; nonetheless, they are useful to characterize, and doing so may help inform priorities for training generalist models in robotics.
We assess LLMs as pattern machines categorized into three areas: sequence transformation, sequence completion, and sequence improvement (Fig. 2). First, we show that LLMs are capable of generalizing certain sequence transformations of increasing complexity with a degree of token invariance, and posit that this can carry over to spatial reasoning capabilities in robotic tasks. Next, we assess LLMS ability to complete patterns from simple functions (c.g., sinusoids) and show this can be applied to robotic tasks like extending a wiping motion from kinesthetic demonstrations, or drawing patterns on a whiteboard. The combination of in-context sequence transformation and extrapolation further enables LLMs to do basic forms of sequence improvement. We show that providing reward-labeled trajectories as context, coupled with online interaction, can enable an LLM-based agent learn navigate through small grid, discover simple CartPole controller, and optimize simple trajectories via human-in loop "clicker" reward training. Code, benchmarks, and videos made available at https://ennrral-paie machines, machines.gituu..o.

---

2 Related Work
In-Context Learning. Pattern reasoning by prompting pre-trained LLMs with few-shot input-output examples is driven by in-context learning [38, 39]. The examples serve form of task specification, where the model is expected to complete further instances of the task by predicting what comes next. In- context learning extends the concept of "task prefixes" (predefined token sequences, e.g., [40]), but swapped in with examples instead. Brown et al. [39] observe that it improves (in particular, out-of-disstiibution generalization) from scaling model size. This is in contrast to scaling models for pre-training + fine-tuning, which has been shown to not necessarily improve OOD generalization on language tasks [41]. Nonetheless, despite compelling OOD generalization abilities, in-context learning still comes at cost, as it continues to lag behind in terms of absolute performance benchmarks compared to task-specific fine-tuning [38, 42].
Explanations of In-Context Learning. In-context learning is explicitly trained for by packing examples from the same task and dataset into a context buffer that is fed as input to an LLM with an unsupervised autoregressive objective [39], sometimes referred to as meta-training. However, it can also emerge implicitly from training on datasets where tokens exhibit a Zipfian distribution [43] on Transformer architectures, but not necessarily with recurrent architectures (e.g.. vanilla RNNs) [43]. Other works have shown that in- context learning with Transformers and can generalize to seemingly unbounded number of tasks (when trained on tasks from the same task family) better than multitask MLPs [47], with Bayesian interpretations of this phenomenon [48] [49].
learn simple function classes on par with least squares [44, 45, 46),
In-Context ys. In- Weights Learning, In-context learning occurs during inference without gradient updates to the model weights, and can be differentiated from in-weights learning, which relies on information stored in the model weights during LLM training [50] (and can be useful for completion tasks such as "Abraham Lincoln was born in "). Chan et al. [50] observes that generalization of in in-context learning can be characterized as more "exemplar-basee"" (on the basis of similarity to in-contex, examples [51]), as opposed to generalization of in weighss learning which tends to be more "rule-based" (on the basis of minimal features that support category boundaries in the training data [52]). The vast capabilities of LLMs [39, 53, 54, 55, 56] have been driven by a combination of both forms of learning. In this work, we are particularly interested in in-context learning, and (depending on the task) using the semantic priors of numeric tokens to drive capabilities such as sequence completion (Section 5) and improvement (Section 6).
LLMs and Robotics. LLMs have been applied across several areas in robotics- suutc as decomposing high-level task descriptions to mid-level plans [6. 7, 57, 58, 59, 60), robot code [13, 17, 14, 61], and plan- ning domain definition languages [10]. These methods leverage semantic priors stored in LLMs to compose plans or parameterize primitive APIs, but whether L.MS can directly influence control (e-g., at the level of trajectories) in a zero-shot manner remains an open problem. We explore how pattern reasoning capabilities of L.LMs may drive various control tasks, to extend or optimize low-level sequences. While it is possible to explicitly train models for these capabilities [62, 63, 64, 65], this work focuses on the inherent abilities of LLMs out-of out-of-the-box. thx--eoo, which may have implications for the role of language pre- me-ttrining for building em- bodied Al systems. Related to our work are [42] which studies how LLMs perform on non-language classi- fication and regression tasks; [66] which examines analogical reasoning in various text tasks; and [67] which studies how LLMs can represent a rollout policy and world model in-context and then uses Q-learning to drive policy improvement across a collection of toy environments with linguistic representations. Our use of I.LMs for sequence improvement can be as a simplification of in-context policy iteration that supports learning from demonstrations and in-context RL, driven by the generality of LLMs pattern machines.
3 Language Models as General Pattern Machines
The capacity of LLMSs to act as general pattern machines is driven by their ability to perform in in-context
learning on sequences of numeric or arbitrary tokens. An LLM typically represents sequence modeling
autoregressively, with a decoder-only Transformer [68], by factorizing the probability of a sequence I, which is sequence of symbols (31, ..., sn), into the product of conditional probabilities p(x) [T=:((ss//s |s|51, ..., s,-1). To perform in-context learning, the model can be conditioned with a prompt that provides the initial tokens in the sequence $ = (81, ..., sk) and uses the model to complete $k+im.
3

---

The adaptability of in-context learning lies in the amount of flexibility that can be packed into $1.A this prompt sequence can itself contain many sequences, each an input-output pair, and perhaps additional task conditioning [38, 29}. Specifically, a model can in-context learn to complete a prompt which is a set of N examples $1:8 = (x*; x², x²)) where each x² is a variable-length sequence (s1 82, ..., s
Rather than investigating in-context learning with natural language tasks [39], in this work we interested in investigating more abstract notions of non- -on-liiggiitic pattern.. The following sections evaluate these capabilities across LLMs, and show how they can be used in robotics. By varying the notion of what each x² should be, we can characterize in-context pattern learning capabilities into the following 3 categories.
Sequence Transformation (Section 4) each x² ... N is a sequence-fo-sequencc input-output pair; i... x = (mipu each subsequence of variable length, and xVV is the query input (=
Sequence Completion (Section 5) rather than containing input-output pairs, and rather than containing many examples of different sequences, the prompt x= 2==(s1 ....5)) corresponds to discrete samples from a single function, e.g., of the form s; -a -isin(bii), which can be extrapolated.
Sequence Improvement (Section 6): each =², ..., It N -1 a collection of trajectories (potentially labeled with corresponding total rewards), and x prompts the model "improve" the sequences by inferring a better one, e.g., with least-to-most prompting [69]-this process can be iterative and applied to a variety of formulations, e.g., offline trajectory optimization online in in-context reinforcement learning.
4 Sequence Transformation
LLMs are capable of in-context learning the distribution of functions that represent sequence transformations by completing abstract patterns observed among examples of input- sequences x² = (a ipi caupu) of arbitrary tokens, each drawn from a fixed alphabet A. For example, suppose that we are given a string of input-output examples such as " 5 3 0, 5; 7 1, 6 7; 9 2 3, 2 9; 4 8 5," Here A consists of tokens that represent space-prefixed digits 0-9, a comma token to separate inputs from outputs, and semi-colon token delineate examples from each other. A general pattern machine should infer the completion 8 4" by recognizing that the pattern is swap the first 2 tokens, then remove the 3rd.
We use the ARC [20] to evaluate LLMs on such sequence transfornations that are substantially more complex, covering
Method
Total (of 800)
(g4) gpt-4-0613
77
a range of abstract spatial tasks: infilling, counting, rotating
(d3) text-davinci-003
85
shapes, etc. Each task has input-output examples (3.3 on aver-
(d3) random A
44-16
age), and 1-3 test inputs which can be represented 2D grids. Input and output sizes may differ. LLMs can be used for the ARC by flattening grids and predicting output grid items in row-major order, which naturally supports vuriable-length out- puts. While I.LMs are not specifically trained for rasterizing spatial outputs, we hypothesize that a general pattern machine would be capable of implicitly recognizing long-range depen- dencies between rows (using positional encoding as bias [71)) pick up patterns that extend across the 2nd dimension.
(d2) text-davinci-002 [53]
64
(p) PaLM [55, 56]
42
(dl) text-davinci-001 [39]
11
(dl) finetuned
9
Ainooson al., 2023 [23]
1300
Kaggle 1st Place, 2022 [70]
# #164
Xu et al., 2022 [22]
1157
Alford et al., 2021 [24]
**2
Ferré et al., 2021 [21]
32
Numbers averaged across 5 randomly sampled alphabets.
Result: ARC benchmark. Table 1 shows that LLMs (PaLM, InstructGPT series in acronyms dI - d3) prompted with input grids represented as tokens drawn from an alphahet of digits, can correctly infer solutions for up to 85 problems. Surpris- ingly, this outperforms some recent systems [21, 22] based on program synthesis that use manually engineered domain-specific to surpass brute-force search [23] to compose functions from handcrafted API of grid operators, they do exhibit non-trivial performance. (We address the important caveat that parts of the ARC may be present in the training data of L.MSs later helow.) Note that while we are concerned with LLM performance over raw patterns, concurrent work finds improvements via object representations [72] and hypothesis search [73].
"assdd brute force search hand-desigeed DSL.
# Reeported out of 400 train tasks, among candidates. Reporied out of subset of 160 object- orientel problems. "Bused on program synthesis, out 36 symmetry tasks,
Tab. I: LLMs out-of-the-box can solve non- trivial number of ARC problems.
languages (DSLs). While LLMs have yet
Observation: consistent tokenization matters. The ARC can be found among the suite of tasks in BIG- Bench [74], but has often been overlooked since many language models appear to perform poorly (near

---

at zero performance). We observe this occurs due to the formatting of the benchmark, where grid elements are represented as neighhoring characters i.e., "8686" (instead of 8 8 6"). While subtle, this difference is enough for certain Byte-Pair Encoding (or Sentencelieece) tokenizers [75, 76] (that do not tokenize per digit) to group multiple grid elements ("88" and "6"" into a single token (*86") which maps to a different token embedding. This causes inconsistencies with how patterss are expressed at the token level. For example, given task expressed "8686, 6868; 7979," if the tokenizer groups pairs of digits 86, 68, 79, respectively, the sequential inductive patterns of the task (to swap and repeat individual digits) are lost. A simple work-around is to directly pass token indices or embeddings to the language model, or use token alphabets unlikely to be grouped together (which involves some knowledge about the tokenizer). Even beyond the ARC, we observe it is beneficial to tokenize consistently with the pattern being represented.
Observation: token mapping invariance. The hypothesis that L.LMS can serve as general patterm machines stems from the observation that they can still solve a non-trivial number of ARC problems using alphabets A sampled randomly from the LLM's token vocabulary. For instance, given a particular alphabet: 8 + falls, 6 + +H, 7 + U1, 9 + Chev, + E 2 + 2010}, a pattern machine at sufficient proficiency can be expected to complete the prompt "falls +# falls +#, +# falls +# falls; UI Chev UI Chev, Chev UI Chev UI; e 2010 d 2010,"' " by predicting " 2010 & 2010 " For example, text-davinci-0033 [53, 39] with the following mapping ( 0 + offence, 1 + Subject, + Lub, + Fail, + Chev, + symb. + swung. 7 + u1, + escalate, + Chromebook) solves 52 ARC problems, and across 5 random alphabets solves an average of 43.6 problems. Interestingly. we find that token mapping invariance holds to extent patterns over randomly sampled embeddings as well (not associated with any token in the vocabulary, see Appendix A.3).
The implications of token mapping invariance are two-fold. First, note that it is possible that parts of the ARC are present in the LLM's training data (i.e., due to contamination). Thus, measuring the performance L..MS under random alphahets may provide closer estimate of their underlying sequence transformation capabilities. (As further evidence that these abilities are not simply due to memorization, we provide a new moocedurally-geneaaae pattern transformation benchmark described below.) Second, we hypothesize that the pattern manipulation capabilities implied by token invariance could help drive positive transfer from patterns learned across Internetsscale language data to new modalities or symbolic representations for robot reasoning, As an example, (i) Fig. 10 (top) in the Appendix shows a grasp (Skittles) detector which outputs target coordinates within a downsampled image (with 6 in-context examples), and (ii) Fig. (bottom) shows spatial rearrangement via predicting simple forward dynamics where the red bowl moves the green plate (with 9 in-context examples of downsampled images as inputs and outputs). The generality of what the arbitrary tokens could represent may allow pattern transformation capabilitie- especially as LLMss improve-to to be leveraged at various levels of abstraction in robotics (e.g., pixels or joint positions).
Result: PCFG benehmark. The ARC is a difficult bench- mark, and the perfomnance falloff can be steep (and relatively uninformative) across L.MSs with decreasing model size and data scale, making it difficult measure incremental progress
Method
Accuracy (%)
(d3) text-davinci-003
75
(d3) w/ random .A
158 1
(p) PaLM [55, 56]
74
towards pattern machines that could be used for sequence trans- formation in robotics. Therefore, we introduce an adjustable- difficulty benchmark, where the transformations are procedu- rally generated using the probabilistic context-free grammar (PCFG) in Hupkes et al. [77]. These transformations include collection of lexical rules that may be composed (e.g., re- verse, shift, swap, repeat, etc.) over the tokens in the input sequence mimpu to generate s'uupuu Example transforma- tions are given in Table 4 in the Appendix. The complexity of these transformations can be controlled by varying the number of tokens & used to express sequences x' ='= and the number of lexical rules w used to define the transformation. This is simply the iden- tity function when w 0, and progressively appears more complex as w oo. Table aggregates PCFG pattern completion accuracy across different LLMs over sequence length = [1,2,44,8,16 and com- plexity = (0,1,3,7,15,311 each with 100 runs (see Appendix A.4 for ablations of k,w)). This benchmark
(d2) text-davinci--022 [53]
69
(dI) text-davinci-001 [39]
60
(cl) text-curie-001
54
(b1) text-habbage-001
50
(al) text-ada-001
39
Numbers averaged across randomly sampled alphabets. Tab. 2: LLMs of varying sizes are capable of completing pattems procedurally generated with PCFG, averaged over range of k and w.
5

---

provides a more unbiased evaluation of pattern reasoning capabilities in LLMs; PCFG completion accuracy improves with model scale, and correlates with ARC performance. We use PCFG for evaluation only (rather than for training (77, 78)) so that one can measure how pre-training regimes or modalities may improve general pattern capabilities across sequence transformations. We have released the PCFG benchmark.
5 Sequence Completion
Completion of sinusoids. We start with simple example where LLMs extrapolate function of the form f(x)= (00)=ansin()) As in Section 4, tokenization matters; we found it effective to discretize outputs among integers 0-100, as these integers are represented by single tokens in the tokenizers of the LLMs we tested.
Fig. 3 shows completions of the sine wave by text- davinci-003 over 11 trials given 3 and periods context, as well as average distance (computed by Dy- namic Time Warping) of the generated predictions to the ground truth function values across several LLMs. Multiple LLMs produce near-perfect continuations of the sine wave, especially with more context (i.e.,
10
100
300 (oor
100
100
200
more periods of the sine wave). We additionally test
2x
2m
the function family ar sin sbz( in which the am- plitude of the oscillations increases with z- values. Here, the LLM must extrapolate to new values un- seen in the context, which highlights the utility of using a metric space for the outputs (0-100) where the LLM has priors over the scale of the different to- kens. These functions also contain a "meta-pattem": the y-values increase, decrease, and then increase in single period-and the amplitude of the function also increases over time. This is a form of least-to to most-to-moot prompting [69], an ability we find useful later for a sequence improvement in Section 6. We also test the function ga -sin(bx). Across these three functions, we observe that greater context and larger scale LLMs yield higher quality predictions.
100
Fig. 3: L.MSs (d3 shown) can extrapolate various func- tions y = a s-ninbbx) (top row). y az six()x) (middle row), and y= 22 sin(brr (bottom row) given amounts of context. Overall, larger models make better predictions with lower error rates (right column). More context also helps prediction accuracy (light vs. dark).
Completion of periodic motions. We emphasize that the Sequence Completion capability above is domain- agnostic- i.c., we do not use any specialized prompts explaining what function should be completed, nor do we provide any linguistic grounding for the metric tokens. We can therefore operationalize this zero-shot capability of LLMs to simple open-loop motion extrapolation problems in robotics, e.g., by encoding a series of positions sampled from a demonstration, and predicting future positions. We test two simple tasks on a mobile robot manipulator: Table Sweeping and Whiteboard Drawing (both shown in Fig. 2).
In Table Sweeping, the goal is to continue human- provided kinesthetic demonstration of sweeping portion of table (see middle Fig. 2). We encode the demonstration as series of end-efficttrr poses at approximately 3 Hz. Each demonstration lasts roughly 20-30 seconds. We represent the 7-dim end-effector pose as a concatenation of Cartesian position and the quaternion, where each value is binned to an integer between 0 and 100, and the dimensions are delimited by spaces. We collect 30 demonstrations that demonstrate the sweeping motion. Note that demonstrations in this task are noisier and higher dimensional than the stylized sinusoid functions above. For each demonstration, we construct context consist of the first two-thirds of the provided demonstration, and treat the last one-third as the ground truth for the LLM to predict. Larger models quantitatively perform better with generally lower variance (see Appendix).
In Whiteboard Drawing, the goal is to continue a scripted demonstration of drawing loops on whiteboard (see Fig. 2). Loops are defined by parametric equations of the form x = a₂ cos + d, and y = We execute the motions using position control and record the end-effector positions at 5 Hz, then discretize states in between and 300, as finer motion needed for this task. We provide part of the loop pattern in-context, and assess the ability extrapolate from loops do a third loop. LLMs, e.g.. text-davinci-003 perform well-we we show more completions with different loop styles in the Appendix.

---

Sequence Improvement
In this section, we explore the synergies between sequence transformation and completion- and investigate improving sequence, such as trajectories in a sequential decision process, along some metric, such as a reward function. Here, we use an LLM to generate new sequences x conditioned on previous sequences (a', ... N which can represent previous iterations of the same sequence (or policy it represents). The improvement also be retum-condiitoned, given a reward function r(.). By inserting the first token(s) of each sequence its corresponding total reward = (r(2),,11 ..... $k), we prompt the model to conditionally "improve" by "just asking" [79] for a higher reward than those seen in-context (i.e., prompting LLMs to act as Decision Transformers [80]. New "rollouts" can yield new reward labels that then replace the original desired rewards with actual rewards. Iteratively performing this inference and accumulating trajectories may jointly use the model's general notion of pattern transformation and extrapolation to perform improvement of sequences, which can be represented by numeric or symbolic tokens. Note that there are practical considcrations, c.g. depending on the task model, not all sequences can fit in context, options could be keep the most recent, the ones with the highest rewards if available (see Appendix for more discussion). In this section, we perform series of targeted experiments on simple tasks, aiming to explore the possibility of using LLMs for sequence improvement in trajectory and policy optimization.
Extrapolating simple patte-p among trajectories. Sequence improvement with LLMs enables simple form of trajectory optimization for Marker in Cup task on a Franka Panda robot, where define the prefixed reward of trajec- tory to be the negative distance between the final end-effector position and the cup (normalized between 0-100), and initial- ize the context with a collection of trajectories (stopping at 20%, 40%, 60%, and 80% of the way to the cup), delimited by newlines and prefixed by rewards (ranging roughly from 70- 90; see Appendix). For this task, we represent trajectories as sequences of Cartesian positions, each dimension normalized between 0-100. We find that text-davinci-003. to an extent, is able to generalize the pattern and generate a trajectory that achieves a reward >> 90. For this extrapolation to occur, we observe that meta-patterms in the context are crucial: in Fig. 4 (left), we compare the average reward achieved by text-davinci-O00 over trials (each with a different goal position) given contexts with different trajectory orderings (sorted by reward, randomly permuted, or with/without reward annotations).
100
Retmm
Ri2tt
p-000
Return
Seriit No Recam
Pu100
Return
Fig. 4: LLM agents can generate new trajecto- ries with increasing returns a Marker in Cup task (right). Performance varies with different. ways of building the context (left).
Sampling higher-reward trajectories online. While LLMs can extrapolate from trajectories that exhibit clear meta-patterns among them, we find that this ability is more limited for less trivial setups. Consider a simple 9 x 9 Grid navigation environment with a random goal position and a fixed starting position at the grid center. Episodes terminate after 20 timesteps, and the return based on the distance from the agent the goal the final time step. This environment is inspired by the Dark Room environment from [62] but with a continuous reward function, reducing the exploration challenge. The agent may take actions (1 (1-5) corresponding to moving right, up, left, down, and no-op. We initialize the context buffer with 20 trajectories of agent grid positions generated by a random policy, sorted by total cumulative rewards. These trajectories exhibit a more complicated meta meta-pattem than in the Marker in Cup task; we do not find that L.LMs can generate trajectories of higher reward immediately. consider an iterative, online setting, in which the LLM acts an agent that interacts with the environment in closed-loop. The context consists of the highest reward trajectories in sorted order, appended with higher reward than was seen in the context, plus states and actions from the current partial trajectory (see Appendix). Once an episode terminates, its trajectory is relabeled with the reward achieved, and inserted into the context at the appropriate position. In Fig. 5, we plot the maximum return attained by al-d3 over 50 episodes, compared to random exploration, averaged over trials. We find that al-dl tend to sometimes "exploit" the suboptimal behaviors represented in the context (which initially contains trajectories with rewards ranging from 6-78), whereas d3 can consistently find a solution to Grid within 50 episodes.
95
Max Retum
90
15
80
75
70
Fig. 5: Average max return for LLM agents al -d3 on Grid compared to random exploration (r).
With that said, we can
7

---

200
100
Discovering a simple CartPole controller. We show that using LLMs as agents in an online, closed-loop setting can discover a simple controller for CartPole (where observations consist of pole angle and velocity, normalized to 0-100, actions are (left) and (right), maximum horizon is 200). Fig. 6 (left) shows that return (number of steps the CartPole is kept upright) improves average across various LLMs over 100 episodes (where the first 100 are generated by random exploration). Fig. (right) shows the evolution of trajectories over episodes of d3, demonstrating that it discovers oscillatory behaviors to keep the CartPole upright.
80
1.50
60
100
100
120
140
160
150
200
-50
100
150
200
Eptindd
Timeerp
Fig. 6: Different LLM agents (d3 - cl) on average can improve trajectories (total rewards) with more CartPole episodes (left), and discovers "oscillatory behaviors" (right) keep the CartPole upright (later episodes are brighter).
Online human-guided trajectory optimization. LLMs can also react to sparse binary reward sig- nals (e.g. subjectively provided by human) to adjust trajectories online. This is analogous to an implementation of "clicker training" [81, 82] used for training dogs, but instead applied robots. In this setup, every time step (2s), the robot exe- cutes an action corresponding to movement of its end-effector in particular direction. The human observes the action and chooses whether to give reward (i.e., by using the clicker) to encourage or discourage similar behaviors. Episodes reset after 30 seconds, and the first two episodes are generated by random exploration. The (reward, state, action) tuples are added as in-context examples (with negative examples followed by positives, and an equal number of each) to generate the next action based on the current state. An example context format is given in the Appendix. As shown in Fig. 7, applying LLMs' sequence improvement capabilities in this way enables human to guide the robot to push an object.
"Culinn isscomnent E presining
Rowand sigrall (oaline)
State space
Glei reached
Stnte
M (x)
Epiaadde joacts
Stops
Fig. LLMs can in-context react to sparse reward signals online to encourage an end effector to reach a desired goal.
7 Discussion
We are excited about the opportunities of LLMs pattern machines for robotics from reasoning and extrapolating complex patterns as a prior for control, to online optimization of closed-loop policies via sequence improvement. These capabilities present several implications. including (i) perspectives on the role of language pre-training for end-to-end robot learning models [31, 32], and (ii) in-context learning of arbitrary pattemss as a driving mechanism for policy improvement. LLMs also show promise for mixed autonomy settings-es e.g., real-time pattern extrapolation for assistive teleoperation. We expect many these abilities continue improving as large models expand from learning patterns within language-only datasets to multimodal domains (e.g., images, videos). While this work investigates in-context generalization fairly simple settings without additional data collection model training, these capabiliiiess presumably may be significantly improved via domain-specine objectives and finetuning [83, 84, 64, 65, 42).
Limitations & Future Work. Today, the inference costs (and monetary costs) using LLMs in the control loop are quite high. Predicting the next token for every sequence, e.g., every dimension of every time step in trajectory, involves querying an LLM. State-action spaces which are higher dimensional and/or greater precision also result in longer representations, and thereby the extent which they can be extrapolated sequence optimized is bounded by the context length of models. These limitations may prevent deploying these models on more complex tasks in practice; however, they may be partially mitigated by incorporating mechanisms like external memory, and by current efforts to drive improvements in LLM quantization [85] and inference efficiency [86]. An additional limitation lies in the fact that, for best performance, some care must be taken to represent patterss with consistent tokenization (which requires knowledge of the model's tokenization scheme). Finally, as with any other language-only model, LLM-bbsed control may (i) be unpredictable, and (ii) lack visual/physical grounding; thus, it is not currently suitable for application outside of constrained lab settings. We leave the exploration of these important topics for future work.
8

---

Acknowledgments
The authors would like to acknowledge Jie Tan, Peng Xu, Carolina Parada, Alexander Herzog, Jensen Gao,
Joey Hejna, and Megha Srivastava for valuable feedback and discussions.
References
[1] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NewrIPS), 2022.
[2] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. In Advances Neural Information Processing Systems (NeurlPS), 2022.
[3] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Lc, E. H. Chi,
D. Zhou, et al. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. arXiv:2210.0926), 2022.
[4] A. Creswell, M. Shanahan, and I. Higgins. Selection-Infent Inference: Exploiting large language models for interpretable logical reasoning. arXii:2205.09772, 2022.
[5] A. Lewkowyez, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag,
T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[6] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, al. Do as I can, not I say: Grounding language in robotic affordances. In Conference Robot Leurning (CoRL), 2022.
[7] W. Huang. P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv:201.007200 2022.
[8] Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh. Translating natural language to planning goals with language models. arXiv:2302.05/2 2023.
[9] Y. Ding, X. Zhang, C. Paxton, and S. Zhang. Task and motion planning with large language models for object rearranement. arXiv:2303.0622. 2023.
[10] Liu, Jiang. X. Zhang. Q Liu, Zhang, J. Biswas, and P. Stone. LLM+P: Empowering large language models with optimal planning proficiency. arXiv:2304.11477 2023.
[11] F. Zelikman, Huang, G. Poesia, N. D. Goodman, and N. Haber. Parsel: A (de-) compositional framework for algorithmic reasoning with language models, 2rXiv:2212.10566 2023.
[12] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Texi2Motion: From natural language instructions feasible plans. arXiv:2303.1D.
2023.
[13] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. lehter, P. Florence, and A. Zeng. Code as Policies: Language model programs for embodied control. In International Conference Robotics and Automation (ICRA), 2023.
[14] 1. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu. J. Tremblay, D. Fox. J. Thomason. and A. Garg. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. In International Conference Robotics and Automation (ICRA), 2023.
[15) M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh. Reward Design with Language Models. In International Conference on Learning Representations (ICLR), 2023.
[16] H. Hu and D. Sadigh. Language Instructed Reinforcement Leamingg for Human Al Coordination. In International Conference on Machine Learning (ICML), 2023.
[17] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng. S. Song, J. Bohg. S. Rusinkiewicz, and T. Funkhouser. TidyBot: Personalized Robot Assistance with Large Language Models. In International Conference on Intelligent Robots and Systems (IROS), 2023,
[18] J. Liu, D. Shen, Y. Zhang, B. Dolan, I. Carin, and W. Chen, What Makes Good In-Context Examples for GPT-3. In Proceedings of Deep Learning Inside Out (Deel1O 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures 2021.
[19] Z. Zhao, E. Wallace, S. Feng. D. Klein, and S. Singh. Calibrate before use: Improving few-shot performance of language models. In Interaatonaa Conference Machine Learning (ICML), 2021.

---

[20] E. Chollet. On the measure of intelligence. urXiv:191101147, 2019.
[21] S. Ferré. First Steps of an Approach to the ARC Challenge based on Descriptive Grid Models and the Minimum Description Length Principle. 1rXiv:2112.0086 2021.
[22] Y. Xu, E. B. Khalil, and S. Sanner. Graphs, Constraints, and Search for the Abstraction and Reasoning Corpus. In AAAI Conference on Artificial Intelligence, 2022.
[23] J. Ainooson, D. Sanyal, J. P. Michelson, Y. Yang, and M. Kunda. An approach for solving tasks on the Abstract Reasoning Corpus, 094445 2023.
[24] S. Alford. A Newrosmbollic Approuch to Abstraction and Reasoning. PhD thesis, Massachusetts Institute of Technology, 2021.
[25] R. Assouel, P. Rodriguez, P. Taslakian, D. Vazquez, and Y. Bengin. Object-centric Compositional Imagination for Visual Abstract Reasoning. In ICLR Workshop on the Elements Reasoning: Objects, Structure and Causality, 2022.
[26] A. Moskvichev, V. V. Odouard, and M. Mitchell. The ConceptARC Benchmark: Evaluating Understanding and Generalization the ARC Domain, arXiv:2305.07 2023.
[27] V. Kolev, B. Georgiev, and S. Penkov, Neural abstract reasoner. In 4th Knowledge Representation and Reasoning Meets Machine Learning Workshop (KR2ML) at NeurlPS, 2020.
[28] Paparaju. ARC Competition : EDA + PyTorch CNN. https//www..aagl...ooooo//mmmmmp 2022. Accessed: 2023-05-30.
[29] S. Min, X. Lyu, A. Holtzman, Artetxe, M. Lewis, H. Hajishirzi, and Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In- Context Learning Work? In Conference on Empirical Methods in Natural Language Processing, 2022.
[30] J. Pan, T. Gao, H. Chen, and D. Chen. What In-Context Learning "Learss" In-Context: Disentangling Task Recognition and Task Learning. In Findings of the Assuciation for Computational Linguistics, 2023.
[31] K. Lu, A. Grover, P. Abheel, and I. Mordatch. Pretrained transformers as universal computation engines. In AAAI Conference on Artificial Intelligence, 2022.
[32] M. Reid, Y. Yamada, and S. S. Gu. Can wikipedia help offline reinforcement learnin? In International Conference on Learning Representations (ICLR), 2023.
[33] A. Brohan, N. Brown, J. Carbajal, Y. Chehotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, al. RT-1: Robotics transformer for woald control at scale. In Proceedings of Rohotics: Science and Systems (RSS), 2022.
[34] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bersstein J. Bohg, A. Bossclut, Brunskill, et al. On the opportunities and risks of foundation models. arXiv:2108.07258 2021.
[35] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3M: A universal visual representation for robot manipulation. In Conference on Robot Learning (CoRL), 2022.
[36] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. Conference on Robot Learning (CoRL), 2023.
[37] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language-Driver Representation Leaming for Robotics, Proceedings of Robotics: Science and Systems (RSS), 2023.
[38] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[39] Brown, B. Mann, N. Ryder, M. Subbiah, Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry,
A. Askell, et al. Language models are few-shot leamers. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[40] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer leaming with a unified text-to-text transformer. The Journal of Machine Learning Research (JMLR), 21(1):5485-555 2020.
[41] D. Hendrycks, X. Liu, E. Wallace, A. Dziedzic, R. Krishnan, and D. Song. Pretrained transformers improve out-of-distribution robustness. In Annual Meeting of the Association for Computational Linguistics, 2020.
10

---

[42] T. Dinh. Y. Zeng. R. Zhang, Z. Lin. M. Cira, S. Rajput, J-yy Sohn. D. Papailiippoulos and K. Lee. LIFT. Language-interfaeed fine-tuning for non-language machine learning tasks. In Advances in Neural Information Processing Systems (NewrIPS). 2022.
[43] Chan, A. Santoro. A. Lampinen, Wang, A. Singh. P. Richemond. J. McClelland, and F. Hill. Data distributional properties drive emergent in-context learning in transformers, In Advances in Neural Information Processing Systems (NewrIPS), 2022.
[44] S. Garg, D. Tsipras, S.
Liang, and G. Valiant. What can transformers leam in-context? a case study of simple function classes. In Advances in Neurul Information Processing Systems (NeurIPS), 2022.
[45] E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in cn-context learning? investigations with linear models. In International Conference on Learning Representations (ICLR), 2022.
[46] J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers leam in-context by gradient descent. In International Conference on Machine Learning (ICML), 2023.
[47] I.. Kirsch, J. Harrison, J. Sahl-Dickstein, and I.. Metz. General-purpose in-context leamning by meta-leaming transformers. In Workshop on Meta-l earming at NeurlPS, 2022.
[48] S. M. Xie, A. Raghunathan, P. Liang, and T. Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations (ICLR), 2022.
[49] X. Wang. W. Zhu, and W. Y. Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning, arXiv:2301. 1916, 2023.
[50] S. C. Chan, I. Dasgupta, J. Kim, D. Kumaran, A. K. Lampinen, and F. Hill. Transformers generalize differently from information stored in context in weights. arXiv:2210.0567,, 2022.
[51] R. N. Shepard and J.-J. Chang. Stimulus generalization in the learning of classifications. Journal of Experimental Psychology, 6514))94 1963.
[52] Ashby and J. Townsend. Varieties perceptual independence. Psychological Review, 93(2):154, 1986.
[53] L. Ouyang, Wu, X. Jiang, Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray. et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NewrIPS), 2022.
[54] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Bunda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. 21X1v:2107.03374, 2021.
[55] A. Chowdbery, S. Narang. J. Devlin. M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, al. PaLM: Scaling language modeling with pathways. 2rX1v:2244.02311, 2022.
[56] R. Anil, A. M. Dai, O. Firat, M. Johnson. D. Lepikhin, A. Passos, S. Shakeri, E, Taropa, P. Bailey, Z. Chen, al. PaLM 2 Technical Report. arXiv:2305. 10403, 2023.
[57] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, Mordatch, Y. Chehotar, Sermanct, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner Monologue: Embodied Reasoning through Planning with Language Models. In Conference on Robot Learning (CoRL), 2022.
[58] A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic Models: Composing shot multimodal reasoning with language. In International Conference on Learning Representations (ICLR), 2023.
[59] C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su. LLM-Planner: Few-shot grounded planning for embodied agents with large language models. arXiv:2212.04088 2022.
[60] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. Florence, I. Mordatch, S. Levine, K. Hausman, al. Grounded Decoding: Guiding text generation with grounded models for rohot control. arXiv:2303. 00855, 2023.
[61] G. Wang, Xie, Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305. 16291, 2023.
[62] M. Laskin, L. Wang. J. Oh, E. Parisotto, S. Spencer, R. Steigerwald, D. Strouse, S. Hansen, A. Filos, E. Brooks, et al. In-context reinforcement learning with algorithm distillation. In International Conference on Learning Representations (ICLR), 2023,
11

---

[63] M. Xu, Y. Shen, S. Zhang, Y. Lu, D. Zhao, J. Tenenbaum. and C. Gan. Prompting decision transformer for few-shot policy generalization. In International Conference on Machine Learning (ICML), 2022.
[64] Y. Zhang, D. Huang. B. Liu, S. Tang, Y. Lu, L. Chen, L. Bai, Q Chu. N. Yu, and W. Ouyang. MotionGPT: Finetuned LLMs are
Motion Generators, arXiv:2306.1000 2023.
[65] J. N. Lec, A. Xie, A. Pacchiano, Y. Chandak, C. Finn, O. Nachum, and E. Brunskill. Supervised Pretraining Can Leam In-Context Reinforcemenet Learning, arXiv:2306. 14892, 2023.
[66] T. Webb, K. J. Holyoak, and H. Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, pages 1-16, 2023.
[67] E. Brooks, L. Walls, R. L. Lewis, and S. Singh. In-Context Policy Iteration. arXiv:2210.08821, 2022.
[68] A. Vaswani, N. Shazeer, N. Pannar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (Neur[PS), 2017.
[69] D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang. D. Schuurmans, O. Bousquet, Q. Lc, and E. Chi. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In International Conference on Learning Representations (ICLR), 2023.
[70] Abstraction and Rasoning Challenge 1st place solution.
http://www..kaggl.ccoo//mmmmttie
abstraction and---ndddddddd wead--rasnnnn-ccieeee shallenne//discussioo//5555 2020.
[71] Su, Lu, S. Pan, Murtadha, Wen, and Y. Liu. RoFormer: Enhanced Transformer with Rotary Position Embodding. aXii:2104409864 2021.
[72] Y. Xu, W. Li, Vaczipoor, S. Sanner, and E. B. Khalil. Lims and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. arXiv:2305. 18354, 2023.
[73] R. Wang, Zelikman, G. Poesia, Y. Pu, N. Haber, and N. D. Goodman. Hypothesis search: Inductive reasoning with language models. arXiv:2309.0566) 2023.
[74] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research (TMLR), 2022.
[75] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Annual Meeting of the Association for Computational Linguistics, 2015.
[76] T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Conference on Empirical Methods in Natural Language Processing. 2018.
[77] D. Hupkes, V. Dankers, M. Mul, and E. Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Arifiiial Intelligence Research (JAIR), 2020.
[78] Z. Allen-Zhu and Y. Li. Physics of Language Models: Part Context- Grammar, axXi:2305.13667, 2023.
[79] E. Jang. Just Ask for Generalization. In http:://waaag..oo02222////////ee
2022.
[80] L. Chen, K. Lu, A. Rajeswaran, K. Loc, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
[81] F. Kaplan. P.-Y. Oudeyer, E. Kubinyi. and A. Miklósi. Robotic clicker training. Robotics and Autonomous Systems, 38(34))19772066, 2002.
[82] C. Chiandetti, S. Avella, E. Fongaro, and F. Cerri. Can clicker training facilitate conditioning in dogs? Applied Animal Behaviour Science, 184:109-116, 2016.
[83] S. Li, X. Puig, C. Paxton, Y. Du, C. Wang, L. Fan, T. Chen, D.-A. Huang, E. Akyürek, A. Anandkumar, et al. Pre-trained language models for interactive decision-making. In Advances in Neural Information Processing Systems (NewrIPS), 2022.
[84] D. Driess, F. Xia. M. S. Sajjadi, C. Lynch. A. Chowdhery, B. Ichter. A. Wahid, J. Tompson, Q. Vuong. T. Yu. al. PaLM-E: An embodied multimodal language model. v.Xiv:2303.03378 2023.
[85] O. Zaffir, G. Boudoukh, P. Izsak, and M. Wasserblat. Q8bert: Quantized Sbit bert. In Fifth Workshop on Energy Efficcinnt Machine Learning and Cognitive Computing NNurrIP Edition (EMC2-NIPS), pages 36 39, IEEE, 2019.
12

---

[86] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and effficeent exact attention with io-awareness, Advances in Neural Information Processing Systems (NewrIPS), 2022.
[87] F. Paischer, T. Adler, V. Patil, A. Bitto-Nemling, M. Holzleitner, S. Lehner, H. Eghhal-7adeh,, and S. Hochreiter. History compression via language models in reinforcement leamnin. In International Conference Machine Learning (ICML), 2022.
[88] K. Ellis, C. Wong, M. Nye, M. Sablé-Meyer, L. Morales, Hewitt, L. Cary, A. Solar-Lezama, and Tenenbaum. DreamCoder: Bootstrapping Inductive Program Synthesis with Wake Wake-Sleep Library Learning.. In ACM SIGPLAN International Conference on Progranming Language Design and Implementation (PLDI), 2021.
[89] Ellis, Morales, M. Sablé-Meyer, Solar-Lezama, and Tenenbaum. Learning libraries subroutines for neurally-guided bayesian program induction. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[90] M. F. Cusumano- Towner, A. Saad, A. K. Lew, and V. K. Mansinghka. Gen: A General- General-Purpose Proba- bilistic Programming System with Programmable Inference. In ACM SIGPLAN Interational Conference on Programming Language Design and Implementation (PIDI), 2019.
13

---

A Sequence Transformation
A.1 Abstraction and Reasoning Corpus: Additional Details and Examples
In Section 4 of the main paper, we describe how ARC problems require reasoning about range of different types of pattern operations- inntilinn,,,,liinng.. counting, translating and rotating shapes, and more. In Fig. show sample problems among the 800 ARC problems for which text-davinci-000 correctly generalizes the pattern shown in few train examples to a test example. Fig. 9, we show sample problems that are not correctly solved by text-davinci-003. In Listing I, we show example context for ARC problem encoded as integers.
Input
Output
Inpat
Output
Input
Output
Input
Output
Inpur
Output
Train Examples
Test Example
Fig. 8: Sample ARC problems that correctly solved by text-davinci-000
Input
Output
Input
Output
Imput
Outppt
Input
Output
Input
Output
+
+
Train
Examples
Test Example
x
Fig. Sample ARC problems that correctly solved by text-davinci-003.
input: 0,
3, 4,
7, 6,
0
0 8,
0
output:
0 0 4
8
0
0 8
0
6
input:
e 8
0
5. 6
14

---

0, 8 3
0
output:
Listing I: Example context format for an ARC problem (only one input-output example shown, along with a query input.
A.2
Patterns over Low-Resolution Images
In Fig. 10, we show an example in-context grasp detector which outputs target coordinates in downsampled image, given 6 in-context examples, as well as an example of simple forward dynamics model predicting spatial rearrangement of a red bowl into a green plate, given in-context examples. As L.LMs progress on the benchmarks discussed in Section 4, they may become more robust and precise at performing such tasks.
Inpot
Input (Low-Res)
Input & Ourpur (Tokens)
Output (Rendered)
inpet:
124
S7
62
86
91
BE
87
123
Iuttuut:
E3
63
61
58
E3
83 53
62
02
62
92
62
SLSPUE:
62
Fig. 10: Example LLM prodiction as an in-context grasp detector (top) and a simple forward dynamics model (bottom).
A.3 Token Invariance for New Token Embeddings
In Section 4, we have argued that LLMs are, to a certain extent, invariant to the choice of alphabet pattern encoded with, in line with prior work on mappings from semantically meaningful tokens to random tokens in pre-trained language model [29, 30, 87]. Here, we present an experiment that investigates token invariance even further by introducing new token embedding vectors the model has not seen during training.
We sample many new embedding vectors Gaussian samples using the mean and or 2 standard deviations of the original LLM embedding matrix statistics in each of embedding vector dimension. This way, we create a new token embedding matrix. that mainly consists of the newly sampled token embeddings the model has not seen during training. Additionally, we add embeddings from the original matrix that correspond separating tokens (comma, period) to build the input prompts. Although the model has never seen the new embedding vectors during training, we can feed them into the transformer as input and compute cosine similarities the output analogously to how the original embedding matrix treated.
Fig. 11 shows the success rate of correctly choosing the target token in single-token prediction task when using the newly sampled embeddings in comparison with the native embedding matrix. The tasks
considering are of the form (1, 1, 2) + or (1, 2, 2) + 1. We provide in-context examples build prompt of the form "1, 2, 2, in 3, 4, 4, \n 5, 6, 6,
\n 7, 8, 8, where the correct prediction should be '7" in this example. Note that the numbers "1", "2" etc. are randomly mapped to the newly sampled token embeddings for indexing purposes and in particular do not enter the LLM. As one can see in Fig. 11, for lo noise sampling, the model is able to solve the task with the new embeddings with similar performance as with the native embeddings. In case of 20, the performance degrades. Although these are relatively simple single-token prediction tasks, this experiment shows that LLMs show pattern recognition abilities even when prompted with out-of-distribution continuous input embeddings. The results are obtained with K 100, averaged over 3 random seeds when sampling the token embeddings, 30 instances each, and context length of 5, 10, or 20 examples. The L.M the parameter variant of [55].
15

---

100
80
Success Rate (%)
40
20
10 # Examples Gaussian Emb. le
20
Native Emb.
Gaussian Emb., 20
Fig. 11: Foken-invarianee experiment with newly sampled token embeddings the model has not seen during training. Shown success rates when using randomly sampled token embecklings from the native embedding matrix, newly sampled embeddings.
A.4 PCFG Benchmark: Additional Details and Ablations
Our PCFG benchmark is a procedurally generated, adjustable-difficll benchmark for measuring abstract sequence transformation capabilities LLMs, based on the PCFG from [77]. In Table 3, we show illustrations of the primitive operations in the PCFG that be applied tokens. In Table 4, we show examples benchmark, which are composed of the primitive operations. In Table 5, show independent ablations of sequence length (number tokens) l and complexity (number of rules) w in the sequence transformations, illustrating the way in which the solve rate decreases either factor increases. In Listing 2, show example context for a PCFG problem on integer sequences.
two sequences of two transformations (of different complexities) from our
Unary Functions
Binary Functions
Name
Input
Output
Name
Input
Output
copy
append
reverse
prepend
shift
remove_first
swap
remove_second
repeat
echo
Tab. 3: Illustrations depicting the unary and binary operators from Hupkes et al. 2020, which we use for our PCFG benchmark.
6
7
9
9
1
9
9
7
6;
4
0
5 2
8: 5 0
8
4;
3
3
0
Listing 2: Example context format for PCFG problem (two input-output examples are shown, along with query input).
16

---

Function
Example Inputs
Example Outputs
remove_second(swap(1) 52), 13)
ccho prepend [ swap(echo(s,s))), 835) $38059999110
Tab. 4: Illustrations of transformations in PCFG benchmark. Row shows transformation composed of w== operations over ==3 3 tokens, row shows a transfomaatin composed operations over k= tokens, respectively. For each transformation function, we show two example inputs and the corresponding outputs.
text-davinci-000
text-davinci-003 w/ random
A
PaLM
k
0
3
7
15
31
k
15
31
k:
0
1
3
7
15
31
1
100
92
100
2
100
100
2
91
92
100
100
4
100
100
100
100
100
100
8
100
74
82
16
100
86
59
47
16
84
64
32
17
22
16
100
78
57
51
58
32
100
74
32
14
12
22
32
83
40
13
8
9
12
32
100
68
23
18
22
34
Tab. 5: Solve rate (%) for PCFG across number of tokens & and number of rules for different models.
A.5
PCFG Benchmark: Program Synthesis
We have run DreamCoder [88] on our PCFG benchmark to contextualize the hardness the task, and present the results in Table 6. We ran DreamCoder with two different sets of initial primitives:
PCFG Ops. In this version, we provide DreamCoder with an initial set of primitives that corresponds to the exact set of unary and binary functions (from [77)) that the PCFG benchmark based on: copy, reverse, shift, swap, repeat, echo, append, prepend, remove first, remove second. We also include slicing operator slice, length, and integers 1-10.
List Ops. this version, we provide DreamCoder with a set lf primitives: length, empty, singleton, range, append, map, reduce, true, not, and, sort, add, negate, equal, reverse, index, filter, slice. These primitives are not specially designed for PCFG and are based on those used [88, 89).
In both cases, the provided primitives are sufficient to define the transformations in the PCFG benchmark. For each (k,wr) pair in Table train 100 task instances and report the number of tasks which get solved (i.e. a correct program that satisfies the training examples is discovered). We ran DreamCoder for 4 iterations and use the default hyperparameters and timeout. As we would expect, the version of DreamCoder with "oracle" access to the PCFG operations performs well, in several cases matching or exceeding the performance LLMs. This is especially true when the search problem easier (i.e. when number of functions w is smaller). The version of DreamCoder with access to list primitives is also able to solve many of the tasks with small values of w. but there a sizeable dropoff the complexity of the tasks increases. These results help to contextualize the difficulty of the PCFG benchmark when given access to different amounts of comain-speciffi information. We also note that we would expect brute-force force search over the PCFG operators to eventually solve these tasks. Doubling the computation time budget for the version with oracle access to the PCFG operators leads to increased success rates (k: :==8, w =0=3) increases from 80 +8; (k 16, 7) increases from 33 +41.

---

DreamCoder w/ PCPG Ops.
DreamCoder w/ List Ops.
w
1
15
31
k
0
7
15
31
100
1
100
100
100
2
100
100
100
100
100
51
100
100
80
8
100
81
23
25
16
100
98
64
38
38
16
100
63
26
4
17
32
100
96
41
24
26
37
32
100
66
6
3
5
11
Tab. 6: Solve rate (%) for PCFG across number of tokens and number of rules w DreamCoder initialized with two different sets of primitives.
B Sequence Completion
B.1 Sinusoids: Structure Learning Comparison
While the sinusoid extrapolation could be easily performed with standard regression techniques, we contextualize the task with another method that has specific prior knowledge of the function being extrapolated. We include the structure learning baseline from [90], implemented with Gen [90]. This method uses Gaussian Process with inferred covariance kernel model time series data. The covariance kemel is inferred using MCMC sampling over a PCFG of covariance functions (e.g. squared exponential, periodic). We run the algorithm for 100 iterations and sample from the resulting GP, shown below in Fig. 12. The training data is generally fit with low error. However, the quality of the completion differs for the various functions; the sine wave is generally extrapolated well whereas the sinusoids yield high variance samples. Similar to the LLMs, greater context generally yields lower error completions. Note however that the outputs of the structure learning algorithm are high- variane by design, and there are multiple ways to utilize the outputs of the algorithm. We also note that [90] was tested on larger set of functions than those we look at here. Though not the goal of our work, would be interesting future work to evaluate how LLMs extrapolate patterns generated by wider array of function classes. We also refer to [42] for an extensive comparison language models to baselines on regression tasks when formulated with natural language interface well study the effects of fine- fine-tuning.
m mmm
wwwm
Ze
Ze
100
300 1ine
100
100 300
The structure learning approach (g) extrapolates various functions y sin (bz)) (top row), = az sin (bx) (middle row), and 2 sin (bx) (bottom row) with different degrees of erro. More context also generally helps prediction accuracy (light vs. dark).
Fig. 12:
B.2 Table Sweeping: Additional Details
In Section 5 of the main paper, we demonstrate how sequence completion capabilities can be applied to continuation of partial motions, such as sweeping a table. In Fig. 13, we show the average DTW distance between predicted and ground truth trajectory completions in the Table Sweeping task, given 66% of the trajectory as context, over 30 trials. Each full trajectory consists table. We compare completions made by various language models. We find that larger models generally
sweeping motions across
18

---

160
Prediction laroor
120
c1
d3
Fig. 13: LLM trajectory predictions Table Sweeping improve with larger models.
perform better; text-davinci-003 performs the best, and also has the lowest variance. On our website, we show qualitative examples of text-davinci-003 completing table sweeping motion given by human demonstration.
B.3
Whiteboard Drawing: Qualitative Results
Fig. 14, we show example completions for three different loop styles by text-davinci-003 over three trials. The completions generally match the overall shape shown in the two loops given context. However, the results also qualitatively illustrate that fine motion patterns can be challenging to predict precisely.
Trial #1
Trial #2
Trial #3
700)
Narrow
(OOD
000
Mcdium
Wide
COC
CQO
coCo
Fig. 14: Sampled drawings produced performing in-context completion (of loop. highlighted in green) given a scripted demonstration two loops. Each is different loop style (narrow, medium, wide), and each column different trial. Results are shown for text eext-davinci-00:.
C Sequence Improvement
C.1 Marker in Cup: Additional Details
In this task, we use LLMs to generate improved trajectories (according to reward metric) given context of trajectories that have increasing returns. For this task, states are Cartesian (æ., y, 2) positions, with each dimension normalized between 0 and 200, trajectories are series of states that can be executed via position control, and the return of trajectory is proportional to the negative distance to the goal (cup) plus an offset. We form the trajectories in the context follows: we take full trajectory which attains reward of 100 and construct trajectories that stop moving 20%, 40%, 60%, and 80% of the way to the goal (such that all trajectories 50 timesteps). We condition the LLM to generate 100-reward trajectory by prompting with "100: start state". An excerpt an example context shown Listing 3. The results in Figure from the main paper are over 11 trials, each with different goal position.
71:
104 83 123, 104 83 123,
72:
104
83
123,
104
83 123,
80:
104
83
123,
104
83 123.
19

---

90:
104
83
123,
104
83
123.
184
83
123,
104
83
123,
104
83
123,
104
83
123
104
83
123
104
83
123
104
83
123
104
83
123
104
83
123
104
123
104
123
104
123,
104
83
123,
105
83
123,
105
83
123
186
83
123
106
83
123.
107
83
123,
108
83
122.
109
83
122.
110
83
122
111
83
121
112
82
120
113
82
119
113
82
118
114
81
118
115
81 117,
115
81
116,
115
80
115
116
80
114
116
80
113
117
79
112,
117
111
110
118
78
118
78
169
118
78
189
118
78
109
118
78
109,
118
78
109
118
78
109
118
78
109
118
109,
118
78
109,
118
78
169,
118
78
109,
118
78
109,
118
78
109
108:
104
83
123
Listing 3: Example context (excerpt) for Marker in Cup, illustrating the (reward: state, state, state..) format.
C.2 Grid: Additional Details
the Grid environment, observations are z,y positions represented by integers 0-8 for each coordinate. There are five possible actions (1, 2. 3, 4, corresponding to (right, up, left, down) movement by space and no-op. A goal is randomly placed in the grid. The agent (which is initialized at the center position) receives a reward of 100 10 distance from the goal to the agent's final position. Episodes terminate after 20 time steps. For our experiments, we limit the context length to 1024 tokens. At each iteration, the I.MMs is prompted to generate trajectory with the maximum retum from the buffer plus randomly selected offset of up to 20.
C.3 CartPole: Additional Details
We use simplified version of the CartPole enviorment in OpenAI Gym. Observations are two woo-imensiooa. (conresponding to pole angle and velocity, normalized 0-100) and the maximum time horizon 200. There are two possible actions (1, 2) corresponding to (left, right), and the agent gets +1 reward for every time step that the CartPole kept upright. I Listing 4, we show an example context excerpt for CartPole, where trajectory history is appended with an encoding of the current trajectory.
52:
40
50
1, 40 54,
2,
41 49
1, 41 54, 1,
60:
45
50
2,
45 45
1, 44
50
2, 44 45, 1,
75:
52
50
1,
52
55
2, 53
50,
2, 53 46 2,
98:
44
50,
1, 55, 2, 45
50,
Listing 4: Example context format for CartPole run. A trajectory history (with each trajectory in the format reward: observation, action, observation, action ...) is followed by encoding of the current trajectory, up the current observation.
Below, we discuss some additional considerations for forming the context from the trajectory history.
Context Length. When context length longer, trajectories fit in the context (which yields more in-context "training data" that could potentially be used to generalize to higher rewards, but also requires the LLM to attend over more tokens). Context length is limiting factor of using current LLMs in our trajectory improvement setting: the number of tokens required to represent a trajectory history scales with the observation dimensionality, action dimensionality, time horizon, and number of trajectories. For our CartPole experiments, we limit the context to 1024 tokens (which is the maximum context length for text-ada-001, text- cext-babbage-000 and text-curie-O00 models).
Action Representation. In initial experiments, we found that the tokens used to represent the action space (e.g. "0" for left, "1" for right) can seemingly affect the ability of an LLM to improve trajectories in the online setting. For example, we observed that if "0" is included in the action space, L.MMs may "default" sampling "0" (likely due token-specific priors). Therefore, for our experiments, integer action representations, which appears to alleviate the bias towards choosing a particular action. The fact that action representation can sometimes affect performance complements our observations in the Sequence Transformation section, in which we find that token mapping invariance holds to some extent, but not entirely.
1-indexed
C.4 Clicker Training: Additional Details
In our clicker training example, the observation consists of the end-effector position and the approximate object position as determined by visual input, with the (Izz12) values normalized between 0 and 300.
20

---

Actions correspond to movements of the end-effector (normalized between 0 and 100, such that 50, 50,50 represents no movement). A sample context is given in Listing 5.
0: 80, 49, 19,,,,3,,110 , 133;
45 45,44, 44,4,,5 ,55
0: 125,2,115.10 10,,
48,59, 48
0: 82,32, 155. 109 554,133, 133,
48 08,59,00 48
38.9,,154.109.54
15,54,45
1: 85,36,149 109, 54,,
,7,54,45 46
1: 23,40,142, 109, 54, 1333
44, ,2,
1:
Listing 5: Example context format for clicker training. (Reward, observation, action) tuples are ordered by reward (with click corresponding to reward of I) with an equal number of reward O and reward I transitions represented in the context.
21


Reviews:
[{"review_id": "xXzXmbaFzxa", "reviewer": "Reviewer_seqs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper evaluates the LLM’s abilities to model general pattern sequences including transferring a sequence from a source format to a target format via in-context learning (sequence transformation), extrapolating a given sequence (sequence completion), and improving the sequence (in decision-making tasks) via in-context learning via up-side-down RL conditioned on rewards (Sequence Improvement). In addition, the paper shows that such abilities have the potential to transfer to robotic tasks. ", "summary_of_recommendation": "In general, I think the evaluation study of LLM on modeling general patterns is insightful and valuable. But I feel the experiments in robotic tasks are too simple and limited, and don’t well support the storyline. I would like to give a borderline, but there is no such option. Given the soundness and the value of the evaluation study in the general patterns, I tend to accept this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1.\tWhy there is no evaluation for GPT-4? Given the much better reasoning ability of GPT-4 compared to text-davinci-003, more practical and realistic robotic tasks might be doable. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength\n\n1.\tThe evaluation study on LLM’s ability to model general patterns is rich and convincing. \n2.\tThis paper provides valuable engineer experience on LLM (e.g., the influence of the tokenizer) which I think is beneficial to many readers in various domains. \n3.\tThe paper writing is well.\n\nWeakness\n\n1.\tExperiments on robot tasks, like reaching targets or continuing the previous action patterns, are too simple and limited. Therefore, I feel that one of the paper’s main ideas, the opportunities to use LLMs as pattern machines for robotics, is not so persuasive.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "jto_R--LoyM", "reviewer": "Reviewer_fBNs", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper investigates the potential use of LLM in robotics, without fine-tuning or retraining, for pattern generation in a sequential context. Tokenization and sequence construction methods were developed for tasks, under three categories which the authors call \"sequence transformation, sequence completion, and sequence improvement\", including some pattern completion tasks from the Abstract Reasoning Corpus, the classical closed-loop control task CartPole, open-loop operations of robotic arms, and a few others. Systematic and rigorous studies of the effectiveness of the methods were carried out to conclusively demonstrate that there is real potential in using LLM for pattern generation in robotics.", "summary_of_recommendation": "Clearly an excellent paper, possibly an outstanding one for CoRL. It has its limitations in terms of the depth of the study, but one can only do so much in a first work in a novel direction. The work here is very likely going to be impactful and stimulate substantive follow-up studies.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "### Typo\n\n- Line 355: \"many these\" => \"many of these\"\n\n### Substantive Issues\n\n##### 1. \"Related Work\" section\n\n1.1 It might help if a table could be included or at least some heads or conceptual points highlighted so that the *themes* that matter could be grokked by the reader. \n\n1.1 Also, it will be helpful, e.g. for supporting a more adequate analytical framework (see above and below), if relevant studies beyond LLM but rather from neuroscience about pattern generation and from robotics about sequence handling could be discussed.\n\n##### 2. \"Language Models as General Pattern Machines\" section\n\n2.1 What are the rationales behind the three-way distinction of \"sequence transformation, sequence completion, and sequence improvement\"? Does it come from robotics practice? Is it a consequence of first generating a sequence and then execute a sequence? Or is it merely consequent of seq2seq? Note that without enough framing here, it is a bit hard for one to tell which experimental studies and how many of them are enough for making the case the paper wants to make. \n\n2.2 Concretely, for example, how should one understand the importance of sorting (Figure 5 and line 303-305) the sequences? Is it something one should expect? Is it surprising? Is it acceptable to sort in practice? (See 3. below for more.)\n\n##### 3. Grounding and Challenging the Method\n\n3.1 Tokenization\n\n3.1.1 While the paper specifically addresses the importance of tokenization, a general point could be made: much \"intelligence\" comes from choosing the right tokanization scheme suitable for the specific tasks and environments. (Claim of a \"generalist\" rests on the intelligence that went into choosing tokenization. This is a point not adequately addressed by Gato and the need for training there made it much less interesting than the current study.)\n\n3.1.2 Case in point: the ASCII art encoding for ARC does a lot of work and it requires privileged knowledge of tokens the LLM depend on. The study here was done decently adequately by including random choice of tokens, but analysis of tokenization as a system dependency and acknowledgement of the associated limitation seem to be absent. (E.g. discussions in subsection starting on Line 169 is both a contribution and comes with a limitation.)\n\n3.1.3 Another case in point: discussion in Lines 189-195 has very important implication for robotics architecture, concerning how to understand the interface between a \"perception module\" and a \"planning module\". What is being illustrated here seems to be that there is no need for a vocabulary with fixed semantics to serve at such an “interface”. Instead, so long as there is adequately token differentiation with in-context semantics/correspondence, that may be enough. While this could mean less human-oriented interpretability, it could provide better generalization.\n\n3.1.4 Time and tokenization: the temporal dimension of tokenization (and discretization) also matters as is clear from discussions around \nLine 260 (3Hz) and Line 270 (5Hz). This is a specific place where prior study in neuroscience on Central Pattern Generator (cf.  https://en.wikipedia.org/wiki/Central_pattern_generator) becomes relevant. If we are to develop towards an end-to-end, fully adaptive method, with something like an LLM in the middle serving as pattern generator, then issues with real time must be handled in a principled approach.\n\n3.1.5 Sequence forming: sorting discussed in Lines 303-305 (also mentioned above in 2.2) should be considered an issue belonging in the same family as tokenization. Again, a stronger analytical framework may be helpful for raising the right questions here or at least pointing out the limitations here for future work.\n\n3.2 Extra mechanisms: control flow and external memory required for running the sequence improvement either online or offline could ideally also be contextualized in a more adequate framework.\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is overall an an excellent paper. It investigates the potential of using LLM for robotics from a relatively refreshing perspective. It nicely blends robotics-specific concerns and general concerns about sequence handling and learning generalization, and thus provides a principled study in robot learning.\n\nThe perspective and methods are novel and impactful. A fairly clearly favorable comparison here is that with Gato -- a much larger engineering effort -- in terms of arriving at less intuitive results, raising interesting questions, generating follow-up studies, and possibly leading to eventual real-world use.\n\nThe paper is generally clearly written and the video clips on the website were quite helpful.\n\nThe main concern is with the inadequate grounding of the analytical framework, specifically in terms of (1) situating the study in prior literature about pattern generation (e.g. from neuroscience), (2) motivating the three-way distinction of \"sequence transformation\", \"sequence completion\", and \"sequence improvement\" through situating them in robotics, and (3) raising and at least acknowledging, if not also systematically investigating, the real conceptual and architectural issues with tokenization and sequence preparation in the context of robotics and embodiment, beyond just affirming their importance. This concern leads the reviewer to believe that the limitation section as is currently written is inadequate (see more details below under \"Questions For Rebuttal\"). Another related concern is that because of the inadequate articulation of a strong analytical framework, the reader gets the sense that the number of studies reported is more than necessary but penetration into the substantive issues for future studies is still limited.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Myfy2tFQ3H", "reviewer": "Reviewer_A4xe", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Summary:\nThis paper examines the role of large language models as a pattern reasoner and tries to operationalize pattern reasoning capabilities of LLMs to different types of robotics tasks involving sequence reasoning, or working with serialized data (such as rasterized images).\n\nThe authors present three use cases of pattern reasoning: sequence transformation, sequence extrapolation, and sequence evolution where the LLM output is expected to improve a labeled score of the sequence.  \n\nThe authors present each of these sequence reasoning problems as a case study with observed performance and some discussion of the relevance of the observed results.", "summary_of_recommendation": "My primary concerns with the paper is that this is an attempt to apply LLM methods to a well studied area of ML. In doing so, the paper must adequately establish baseline performance on the problems it is considering, and as described in prior section, the authors have not done an adequate job of establishing the state of the art. \n\nTo my knowledge, establishing the baseline performance does undermine the claim that LLMs are general pattern machines to some extent. However, the ideas explored in the paper are interesting by themselves, and a more in depth discussion of the non-LLM baselines will only strengthen the paper", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1) Could you please clarify the non-LLM state of the art performance on the first two sequence reasoning tasks? Again this does not invalidate the approach, but is important to establish the best performance on these benchmarks. \n\n2) What are the input output encodings for the ARC benchmarks for non-LLM approaches, do they consider image inputs or is there a recommended input, output representation format. \n\n3) Non-LLM approach on PCFG benchmark? Dreamcoder [1] would be a valid baseline\n\n4) What was the motivation for the construction of the PCFG benchmark? I notice that the all the operations involve a reuse of the input symbols. There are other sequence reasoning benchmarks in prior work that also handle substitution operations [4]\n\n\n[4] - Ellis, K., Solar-Lezama, A. and Tenenbaum, J., 2015. Unsupervised learning by program synthesis. Advances in neural information processing systems, 28.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Major comments:**\n\n*Representing the state-of-the-art non-LLM approaches:* I am not quite sure what the source of the authors number for Alford 2022 [ref-23]. If I read correctly, the author tested on smaller subset of symmetry problems for ARC and achieved a high success rate with an algorithm based on the dreamcoder framework [1] that compresses primitive operations into macros. Also, the state-of-the-art Kaggle success rate at present is 175/400 and not 130/800 [2]. Therefore, I urge the authors to remove the claim that LLMs are the best generalist solutions to ARC challenge. Similarly, the authors do not compare to function extrapolation methods for the sequence extension task. The automated time series structure discovery [3 sec 7.2 and refs therein]  performs extrapolation for a various function classes detected automatically, and it is in some sense general purpose, though with a different notion of generalization than an LLM. The purpose once again is not a direct comparison of the approaches, but a more accurate representation of what the state-of-the-art performance is in each of the domains considered by the authors and testing the LLM on an appropriate diversity of input values before making generalization claims\n\n*Impact of tokenization:* The tokenization and compression of each feature vector into an integer between 0-100 for the sequence extrapolation and improvement tasks, and consistent tokenization for the sequence transformation task appears to be key to obtaining reasonable performance. it would be interesting if the authors experimented with another set of non-numerical but ordinal tokens (a-z or A-Z) representing the trajectories, or by associating a non-linear grounding to the numbers. The grounding problem of transforming the LLM output DSL (simple as it may be) into robot states is still completely solved manually by the user. \n\n*Latency in continuous control tasks:* While the use of LLMs might be interesting as an intellectual excercise, in all but quasistatic robotics tasks, the latency of token generation might not be enough to operationalize a robot controller. The cost of each token generated by an LLM is a lot larger than computational cost of even using a policy network. This along with the inability of hosting the models locally, makes them particularly unsuitable for robotics task as identified by the authors in the limitations section. I would like the authors to reflect more on why an LLM-based approach might have benefits in the first place?\n\n*Structure of the paper:* Currently, the paper very much reads as a series of case-studies, and I wonder if a restructuring of the text in terms of methodologies, tests/hypothesis, results and discussion in each subproblem would serve the draft a lot better. \n\n[1] - Ellis, K., Wong, L., Nye, M., Sable-Meyer, M., Cary, L., Anaya Pozo, L., Hewitt, L., Solar-Lezama, A. and Tenenbaum, J.B., 2023. DreamCoder: growing generalizable, interpretable knowledge with wake–sleep Bayesian program learning. Philosophical Transactions of the Royal Society A, 381(2251), p.20220050.\n[2] - https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/discussion/154597\n[3] - Cusumano-Towner, Marco F., Feras A. Saad, Alexander K. Lew, and Vikash K. Mansinghka. \"Gen: a general-purpose probabilistic programming system with programmable inference.\" In Proceedings of the 40th acm sigplan conference on programming language design and implementation, pp. 221-236. 2019.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "HXWh23ApE_Z", "reviewer": "Reviewer_JNxR", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors apply LLMs to non-language tasks via ICL (in-context learning; examples are kept in context buffer rather than used to update weights). They test this ability for general pattern manipulation in abstract tasks on grids of symbols (as \"sequence transformation\") and in hand-tailored discretizations of position sequence extrapolation in 1d, 2d, and 3d (as \"sequence completion\" and \"sequence improvement\").  Their highlighted implications include use of LLMs for control (as priors or for online improvement) and the broad possibility that pretraining on one data modality can provide pattern processing capabilities useful for other data modalities (with a nod to the prospect of pretraining in additional modalities apart from language).  They note primary limitations of the high resource demand of using LLMs relative to standard control approaches, potential scaling issues with context buffer size, and unpredictability and lack of visual grounding of LLM-based control.", "summary_of_recommendation": "As someone with a general interest in deep learning, I found the paper stimulating and do not regret having spent time with it.  However, as a robotics practitioner and robot learning researcher, I would hesitate to suggest to my collaborators within these topic areas that they spend time on this paper.  Its clear presentation and decent attention to caveats/limitations somewhat limit the risk of adding more noise to a research area that has an overwhelming pace of activity, and I don't necessarily think a new idea needs to have immediately competitive results in order to be worthy of attention.  However, the level of care that went into this paper seems out of proportion with its questionable meaningfulness and appropriateness for what it is aiming to solve.  I wouldn't want it to crowd out other submissions that more clearly relate to the core challenges of robotics and the frontiers of motion generation and control.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "It is hard to know what to request from the authors as I'm not sure one could get this approach to a stage relevant to roboicists without first passing through a good portion of the territory presented here (which I find less obviously relevant to roboticists).  The authors have also been largely forthcoming and transparent about the assumptions and limitations of their work, and though I could suggest some more caveats to incorporate, it would not address my core reservations.\n\nI am nonetheless interested to hear the authors' response to my assessment and would pose the following as some key points to respond to:\n1. The necessity of carefully designed tokenization/discretization, ranking schemes etc make it harder to distinguish this approach as \"generalist\" in a way that domain-specific languages are not.  \n2. The generalization shown on ~continuous systems looks more like steering (directly modulating open-loop behavior) than adapting (responding to novel stimulus in a way that is not trivial to pre-specify); I think the latter is much more key to advancing robotics.\n3. I posit that language has specifically evolved to solve the more abstract levels of decision-making and pattern processing -- more like the \"code as policies\" level and above, that are _complementary_ to the sorts of high-dimensional spatiotemporal reactivity / fine motor skills that are still insufficient in so many robotics applications and which we as humans acquire primarily by nature and practice rather than symbolic reasoning and communication.  Therefore it is little surprise to me that casting such problems as general pattern processing appears to be a strained exercise.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of positives / generous interpretation of paper:\n\nThis paper stimulates an interesting conversation about the relationship between domain-specific pattern processing and generalist pattern processing in large ML models.  The focus on in-context learning keeps it fresh and distinct from the related and more developed topics of meta-learning and transfer learning.  The authors establish good breadth by defining three separate robotics-adjacent paradigms, and multiple environments per paradigm, in which to test the performance of several LLMs.  Writing and overall structure are clear and relatively easy to digest, and the Related Work section is appropriately scoped.  Decent attention to detail e.g. PCFG sequence complexity and probing which aspects of domain specification are crucial.  Limitations and assumptions present in the example domains are adequately highlighted and the authors remain sober in discussing the future prospects for leveraging LLMs in this way.  While it is not clear that the general pattern capability of LLMs maps onto frontier challenges in robot control and motion generation, this paper may help us think more precisely about the different kinds of pattern-based reasoning required for those challenges and where we should focus our efforts for developing appropriate generalist resources for them.\n\n\nSummary of negatives / critical interpretation of paper:\n\nAs interesting as this paper is from a deep learning perspective, its relationship to robot control is rather superficial and the approach still requires significant hand-tailoring (discretization/tokenization, ranking, etc) that debatably isn't so distinct from efforts needed with domain-specific languages.  There is a sense in which what is shown is that it is possible to engineer strained analogies that enable solving basic problems in one domain with powerful, resource-hungry tools from another domain.  I think the interesting thing about this paper is that LLMs have general pattern manipulation capability (as the title conveys) that can be leveraged through ICL, and not that general pattern manipulation capability can solve carefully-defined low-dimensional position sequencing problems.\nA roboticist needs to know what a general approach, e.g. search-based planning, is suitable and scalable for before they decide to formulate their application as such.  Likewise I think roboticists need to know whether it is useful or natural to formulate a control or motion generation problem as general pattern manipulation problem rather than to merely know that it is possible at a basic level. \n\nFurther comments on details of example problems:\n\n* Sequence Transformation \n    * The ~spatial operations in ARC and similarly the lexical rules of PCFG benchmarks are completely discrete (e.g., rotation only by exactly 90 degree intervals), aren't they?  If there is any sense in which these setups (such as recognizing that a raster of a 75-degree rotated image of a color wheel was \"close\" to the discretely 90-degree rotated raster of that color wheel), it would be interesting to know something about that, but I assume it is not the case and that \"spatial\" is a rather qualified term in this situation.\n    * Flattening 2d grids to 1d sequences -- because the output is autoregressive, this would imply that model performance might substantially vary between one input and its transpose?  Part of me feels like these problems aren't \"truly\" 2d but I suppose tasks like infilling are something of a counterpoint.\n    * I appreciate the sharing of details of how the tokenization matters, but I also would consider this fact to support my criticism that the hand-designed interface is a big part of the solution. \n    * I also like the examination of token mapping invariance; it's a nice way to try to distill out the non-language component of what an LLM is doing.\n\n* Sequence Completion / Sinusoids\n    * Given the use of the 0-100 tokens which I am not surprised for LLMs to grasp as representing a continuous range, and the basic concept of neural nets from the simplest MLPs as \"function approximators\", it's hard to feel much from this section's results.\n    * The function \"reminiscent of a stabilizing controller\" doesn't seem to add anything other than possibly inviting false assumptions from someone taking a quick look at the paper; there is no fighting of perturbations or opposing forces or dissipating energy here, it's just a pattern that becomes increasingly trivial with time.\n\n* Sequence Completion on robot arm\n    * Nit picks: the EE pose is a 7-dim value representing 6dof and 1 constraint, and it's not clear that there's much significant happening in any of the angular DoF.\n    * The noisiness of the human demos is the main aspect of this problem that resembles the difference between robotics and abstract pattern games on symbol spaces, but as evidenced by the \"partially successful example\" video in the supplement it doesn't seem like anything more significant than averaging/smoothing is going on.\n\n* Sequence improvement \n    * I found this section to be the most interesting and the idea of in-context RL is compelling for getting a generalist system to solve a novel problem on the fly, including with human guidance.\n    * The analysis of Fig 5 feels a little double-edged, though, by again revealing that nontrivial hand specifications are necessary to get the generalist agent to solve simpe problems in the novel domain.  The result feels like makeshift gradient ascent.\n    * Cartpole -- For brevity I'll just say that the boolean action space makes this only the most rudimentary proof-of-concept.\n\nLastly, I very much appreciate the direct acknowledgment by the authors of the high costs of using LLMs for even these very pared-down applications.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the ability of LLMs to recognize and manipulate general patterns within in-context learning with abstract sequences, divides the problem into three subcategories, and shows first transfers to robotic domains.
While in general the reviewers tend to agree on the novelty and interestingness of the idea – beside the clear presentation and appreciated thoroughly executed studies – there are mixed opinions on the applicability to robotics and the investigated problems with respect to the (robotic) claims. Some main concerns raised tackle the positioning of the study in prior literature, the task complexity in the evaluations, and the ‘general utility’ of the approach (speaking simplified and overexaggerated).
Nevertheless, the paper shows a refreshing approach and thinking, that may spark many interesting follow up research questions, and quite often such novel ideas produce split opinions at first. The authors provided an extensive update during the rebuttal and there was a nice and intensive discussion.
While I share some of the concerns of the reviewers with respect to the robotics/CoRL relevance, I also see the value of discussing refreshing and novel ideas in a different context and feel the novel connection of LLMs and robotics – in combination with the high quality of the paper – is of great value to the CoRL audience.

**Importantly though, the updated paper is a full page too long. The authors must shorten the paper significantly without removing important and significant content!**
Decision: Accept (Poster)