Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The paper investigates the potential use of LLM in robotics, without fine-tuning or retraining, for pattern generation in a sequential context. Tokenization and sequence construction methods were developed for tasks, under three categories which the authors call "sequence transformation, sequence completion, and sequence improvement", including some pattern completion tasks from the Abstract Reasoning Corpus, the classical closed-loop control task CartPole, open-loop operations of robotic arms, and a few others. Systematic and rigorous studies of the effectiveness of the methods were carried out to conclusively demonstrate that there is real potential in using LLM for pattern generation in robotics.
review: This is overall an an excellent paper. It investigates the potential of using LLM for robotics from a relatively refreshing perspective. It nicely blends robotics-specific concerns and general concerns about sequence handling and learning generalization, and thus provides a principled study in robot learning.

The perspective and methods are novel and impactful. A fairly clearly favorable comparison here is that with Gato -- a much larger engineering effort -- in terms of arriving at less intuitive results, raising interesting questions, generating follow-up studies, and possibly leading to eventual real-world use.

The paper is generally clearly written and the video clips on the website were quite helpful.

The main concern is with the inadequate grounding of the analytical framework, specifically in terms of (1) situating the study in prior literature about pattern generation (e.g. from neuroscience), (2) motivating the three-way distinction of "sequence transformation", "sequence completion", and "sequence improvement" through situating them in robotics, and (3) raising and at least acknowledging, if not also systematically investigating, the real conceptual and architectural issues with tokenization and sequence preparation in the context of robotics and embodiment, beyond just affirming their importance. This concern leads the reviewer to believe that the limitation section as is currently written is inadequate (see more details below under "Questions For Rebuttal"). Another related concern is that because of the inadequate articulation of a strong analytical framework, the reader gets the sense that the number of studies reported is more than necessary but penetration into the substantive issues for future studies is still limited.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Clearly an excellent paper, possibly an outstanding one for CoRL. It has its limitations in terms of the depth of the study, but one can only do so much in a first work in a novel direction. The work here is very likely going to be impactful and stimulate substantive follow-up studies.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: Summary:
This paper examines the role of large language models as a pattern reasoner and tries to operationalize pattern reasoning capabilities of LLMs to different types of robotics tasks involving sequence reasoning, or working with serialized data (such as rasterized images).

The authors present three use cases of pattern reasoning: sequence transformation, sequence extrapolation, and sequence evolution where the LLM output is expected to improve a labeled score of the sequence.  

The authors present each of these sequence reasoning problems as a case study with observed performance and some discussion of the relevance of the observed results.
review: **Major comments:**

*Representing the state-of-the-art non-LLM approaches:* I am not quite sure what the source of the authors number for Alford 2022 [ref-23]. If I read correctly, the author tested on smaller subset of symmetry problems for ARC and achieved a high success rate with an algorithm based on the dreamcoder framework [1] that compresses primitive operations into macros. Also, the state-of-the-art Kaggle success rate at present is 175/400 and not 130/800 [2]. Therefore, I urge the authors to remove the claim that LLMs are the best generalist solutions to ARC challenge. Similarly, the authors do not compare to function extrapolation methods for the sequence extension task. The automated time series structure discovery [3 sec 7.2 and refs therein]  performs extrapolation for a various function classes detected automatically, and it is in some sense general purpose, though with a different notion of generalization than an LLM. The purpose once again is not a direct comparison of the approaches, but a more accurate representation of what the state-of-the-art performance is in each of the domains considered by the authors and testing the LLM on an appropriate diversity of input values before making generalization claims

*Impact of tokenization:* The tokenization and compression of each feature vector into an integer between 0-100 for the sequence extrapolation and improvement tasks, and consistent tokenization for the sequence transformation task appears to be key to obtaining reasonable performance. it would be interesting if the authors experimented with another set of non-numerical but ordinal tokens (a-z or A-Z) representing the trajectories, or by associating a non-linear grounding to the numbers. The grounding problem of transforming the LLM output DSL (simple as it may be) into robot states is still completely solved manually by the user. 

*Latency in continuous control tasks:* While the use of LLMs might be interesting as an intellectual excercise, in all but quasistatic robotics tasks, the latency of token generation might not be enough to operationalize a robot controller. The cost of each token generated by an LLM is a lot larger than computational cost of even using a policy network. This along with the inability of hosting the models locally, makes them particularly unsuitable for robotics task as identified by the authors in the limitations section. I would like the authors to reflect more on why an LLM-based approach might have benefits in the first place?

*Structure of the paper:* Currently, the paper very much reads as a series of case-studies, and I wonder if a restructuring of the text in terms of methodologies, tests/hypothesis, results and discussion in each subproblem would serve the draft a lot better. 

[1] - Ellis, K., Wong, L., Nye, M., Sable-Meyer, M., Cary, L., Anaya Pozo, L., Hewitt, L., Solar-Lezama, A. and Tenenbaum, J.B., 2023. DreamCoder: growing generalizable, interpretable knowledge with wake–sleep Bayesian program learning. Philosophical Transactions of the Royal Society A, 381(2251), p.20220050.
[2] - https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/discussion/154597
[3] - Cusumano-Towner, Marco F., Feras A. Saad, Alexander K. Lew, and Vikash K. Mansinghka. "Gen: a general-purpose probabilistic programming system with programmable inference." In Proceedings of the 40th acm sigplan conference on programming language design and implementation, pp. 221-236. 2019.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: My primary concerns with the paper is that this is an attempt to apply LLM methods to a well studied area of ML. In doing so, the paper must adequately establish baseline performance on the problems it is considering, and as described in prior section, the authors have not done an adequate job of establishing the state of the art. 

To my knowledge, establishing the baseline performance does undermine the claim that LLMs are general pattern machines to some extent. However, the ideas explored in the paper are interesting by themselves, and a more in depth discussion of the non-LLM baselines will only strengthen the paper
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors apply LLMs to non-language tasks via ICL (in-context learning; examples are kept in context buffer rather than used to update weights). They test this ability for general pattern manipulation in abstract tasks on grids of symbols (as "sequence transformation") and in hand-tailored discretizations of position sequence extrapolation in 1d, 2d, and 3d (as "sequence completion" and "sequence improvement").  Their highlighted implications include use of LLMs for control (as priors or for online improvement) and the broad possibility that pretraining on one data modality can provide pattern processing capabilities useful for other data modalities (with a nod to the prospect of pretraining in additional modalities apart from language).  They note primary limitations of the high resource demand of using LLMs relative to standard control approaches, potential scaling issues with context buffer size, and unpredictability and lack of visual grounding of LLM-based control.
review: Summary of positives / generous interpretation of paper:

This paper stimulates an interesting conversation about the relationship between domain-specific pattern processing and generalist pattern processing in large ML models.  The focus on in-context learning keeps it fresh and distinct from the related and more developed topics of meta-learning and transfer learning.  The authors establish good breadth by defining three separate robotics-adjacent paradigms, and multiple environments per paradigm, in which to test the performance of several LLMs.  Writing and overall structure are clear and relatively easy to digest, and the Related Work section is appropriately scoped.  Decent attention to detail e.g. PCFG sequence complexity and probing which aspects of domain specification are crucial.  Limitations and assumptions present in the example domains are adequately highlighted and the authors remain sober in discussing the future prospects for leveraging LLMs in this way.  While it is not clear that the general pattern capability of LLMs maps onto frontier challenges in robot control and motion generation, this paper may help us think more precisely about the different kinds of pattern-based reasoning required for those challenges and where we should focus our efforts for developing appropriate generalist resources for them.


Summary of negatives / critical interpretation of paper:

As interesting as this paper is from a deep learning perspective, its relationship to robot control is rather superficial and the approach still requires significant hand-tailoring (discretization/tokenization, ranking, etc) that debatably isn't so distinct from efforts needed with domain-specific languages.  There is a sense in which what is shown is that it is possible to engineer strained analogies that enable solving basic problems in one domain with powerful, resource-hungry tools from another domain.  I think the interesting thing about this paper is that LLMs have general pattern manipulation capability (as the title conveys) that can be leveraged through ICL, and not that general pattern manipulation capability can solve carefully-defined low-dimensional position sequencing problems.
A roboticist needs to know what a general approach, e.g. search-based planning, is suitable and scalable for before they decide to formulate their application as such.  Likewise I think roboticists need to know whether it is useful or natural to formulate a control or motion generation problem as general pattern manipulation problem rather than to merely know that it is possible at a basic level. 

Further comments on details of example problems:

* Sequence Transformation 
    * The ~spatial operations in ARC and similarly the lexical rules of PCFG benchmarks are completely discrete (e.g., rotation only by exactly 90 degree intervals), aren't they?  If there is any sense in which these setups (such as recognizing that a raster of a 75-degree rotated image of a color wheel was "close" to the discretely 90-degree rotated raster of that color wheel), it would be interesting to know something about that, but I assume it is not the case and that "spatial" is a rather qualified term in this situation.
    * Flattening 2d grids to 1d sequences -- because the output is autoregressive, this would imply that model performance might substantially vary between one input and its transpose?  Part of me feels like these problems aren't "truly" 2d but I suppose tasks like infilling are something of a counterpoint.
    * I appreciate the sharing of details of how the tokenization matters, but I also would consider this fact to support my criticism that the hand-designed interface is a big part of the solution. 
    * I also like the examination of token mapping invariance; it's a nice way to try to distill out the non-language component of what an LLM is doing.

* Sequence Completion / Sinusoids
    * Given the use of the 0-100 tokens which I am not surprised for LLMs to grasp as representing a continuous range, and the basic concept of neural nets from the simplest MLPs as "function approximators", it's hard to feel much from this section's results.
    * The function "reminiscent of a stabilizing controller" doesn't seem to add anything other than possibly inviting false assumptions from someone taking a quick look at the paper; there is no fighting of perturbations or opposing forces or dissipating energy here, it's just a pattern that becomes increasingly trivial with time.

* Sequence Completion on robot arm
    * Nit picks: the EE pose is a 7-dim value representing 6dof and 1 constraint, and it's not clear that there's much significant happening in any of the angular DoF.
    * The noisiness of the human demos is the main aspect of this problem that resembles the difference between robotics and abstract pattern games on symbol spaces, but as evidenced by the "partially successful example" video in the supplement it doesn't seem like anything more significant than averaging/smoothing is going on.

* Sequence improvement 
    * I found this section to be the most interesting and the idea of in-context RL is compelling for getting a generalist system to solve a novel problem on the fly, including with human guidance.
    * The analysis of Fig 5 feels a little double-edged, though, by again revealing that nontrivial hand specifications are necessary to get the generalist agent to solve simpe problems in the novel domain.  The result feels like makeshift gradient ascent.
    * Cartpole -- For brevity I'll just say that the boolean action space makes this only the most rudimentary proof-of-concept.

Lastly, I very much appreciate the direct acknowledgment by the authors of the high costs of using LLMs for even these very pared-down applications.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: As someone with a general interest in deep learning, I found the paper stimulating and do not regret having spent time with it.  However, as a robotics practitioner and robot learning researcher, I would hesitate to suggest to my collaborators within these topic areas that they spend time on this paper.  Its clear presentation and decent attention to caveats/limitations somewhat limit the risk of adding more noise to a research area that has an overwhelming pace of activity, and I don't necessarily think a new idea needs to have immediately competitive results in order to be worthy of attention.  However, the level of care that went into this paper seems out of proportion with its questionable meaningfulness and appropriateness for what it is aiming to solve.  I wouldn't want it to crowd out other submissions that more clearly relate to the core challenges of robotics and the frontiers of motion generation and control.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper evaluates the LLM’s abilities to model general pattern sequences including transferring a sequence from a source format to a target format via in-context learning (sequence transformation), extrapolating a given sequence (sequence completion), and improving the sequence (in decision-making tasks) via in-context learning via up-side-down RL conditioned on rewards (Sequence Improvement). In addition, the paper shows that such abilities have the potential to transfer to robotic tasks. 
review: Strength

1.	The evaluation study on LLM’s ability to model general patterns is rich and convincing. 
2.	This paper provides valuable engineer experience on LLM (e.g., the influence of the tokenizer) which I think is beneficial to many readers in various domains. 
3.	The paper writing is well.

Weakness

1.	Experiments on robot tasks, like reaching targets or continuing the previous action patterns, are too simple and limited. Therefore, I feel that one of the paper’s main ideas, the opportunities to use LLMs as pattern machines for robotics, is not so persuasive.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: In general, I think the evaluation study of LLM on modeling general patterns is insightful and valuable. But I feel the experiments in robotic tasks are too simple and limited, and don’t well support the storyline. I would like to give a borderline, but there is no such option. Given the soundness and the value of the evaluation study in the general patterns, I tend to accept this paper.
review_title: null

Meta-review: The paper investigates the ability of LLMs to recognize and manipulate general patterns within in-context learning with abstract sequences, divides the problem into three subcategories, and shows first transfers to robotic domains.
While in general the reviewers tend to agree on the novelty and interestingness of the idea – beside the clear presentation and appreciated thoroughly executed studies – there are mixed opinions on the applicability to robotics and the investigated problems with respect to the (robotic) claims. Some main concerns raised tackle the positioning of the study in prior literature, the task complexity in the evaluations, and the ‘general utility’ of the approach (speaking simplified and overexaggerated).
Nevertheless, the paper shows a refreshing approach and thinking, that may spark many interesting follow up research questions, and quite often such novel ideas produce split opinions at first. The authors provided an extensive update during the rebuttal and there was a nice and intensive discussion.
While I share some of the concerns of the reviewers with respect to the robotics/CoRL relevance, I also see the value of discussing refreshing and novel ideas in a different context and feel the novel connection of LLMs and robotics – in combination with the high quality of the paper – is of great value to the CoRL audience.

**Importantly though, the updated paper is a full page too long. The authors must shorten the paper significantly without removing important and significant content!**
Decision: Accept (Poster)