# A Survey of Robot Intelligence with Large Language Models
## 4 Language Models for Robotic Intelligence
### Low-Level Control  
Research is also being conducted on generating commands that directly control a robot's actuators (i.e., enabling low-level control) through various applications of LLM models. Among these projects, the Google research team developed RT-1 [8], which consists of film-conditioned EfficientNet-B3, TokenLearner, and Transformer. RT-1 is a model that receives images and natural language instructions at a rate of 3 Hz and outputs discretized robot actions. RT-1 was trained on a vast demo dataset with over 130k episodes from more than 700 tests collected over 17 months using 13 robots. A notable feature of RT-1 is its ability to enhance performance by learning from data gathered from heterogeneous robots or simulations. In the study [8], the authors evaluated the performance of a model trained exclusively on data from the EveryDay Robot (EDR) against a model trained using data from both EDR and Kuka IIWA robots. They recorded a 12% improvement in the bin-picking test. Another experiment compared model performance using data from real environments and simulations for items not encountered in actual settings. The findings indicated that incorporating simulation data in RT-1 training enhances performance over using purely real environment data, suggesting that RT-1 can substantially improve model performance by integrating diverse data from robots of varied morphologies or simulations while sustaining existing task capabilities. RT-2 [9] is defined as a vision-language-action (VLA) model that facilitates fine-grained control of robots through vision and language commands. RT-2 is fine-tuned with robotic trajectory data based on VLM models such as PaLM-E [140], which has 12 billion parameters and is trained on VQA data, alongside PaLI-X [141], which has parameter sizes ranging from 5 billion to 55 billion. The RT-2 system operates as an integrated closed-loop robotic system that combines low-level and high-level control policies. Despite not explicitly learning certain capabilities during pre-training, RT-2 exhibits improved task performance via real-world generalization involving diverse objects, visual scenes, and instructional contexts. The paper [9] quantitatively assesses RT-2's emergent capabilities in areas such as reasoning, symbol understanding, and human recognition. Furthermore, applying chain-of-thought prompting techniques to RT-2 has proven effective in solving more complex semantic inference tasks, such as using a rock as an improvised hammer or offering an energy drink instead of a carbonated beverage to a thirsty person. In comparison with the earlier study on RT-1, RT-2 demonstrates enhanced performance in both familiar and novel tasks. AutoRT [10] is a follow-up study based on the research results of RT-1 and RT-2, establishing an orchestration of large-scale robotic agents for data collection in real-world scenarios. AutoRT employed 53 robots to gather 77,000 real robot episodes over seven months through both teleoperation and autonomous robot policies. At the heart of AutoRT is a robust foundation model that generates 'task proposals' based on given visual observations. Notably, AutoRT introduces a 'Robot Constitution' using constitutional prompting to ensure actions during the task proposal process do not compromise the safety of the robot or nearby individuals. This Robot Constitution, inspired by [PERSON]'s three laws [142], comprises basic rules, safety rules that identify unsafe or unwanted tasks, and embodiment rules that clarify the robot's operational boundaries. AutoRT enhances data collection by initially scanning the surroundings to identify interesting scenes or tasks (exploration). It interprets the given context through a VLM and proposes potential tasks via an LLM (task generation). Subsequently, tasks suggested by the LLM are screened (afordance) to assess their feasibility and the need for human intervention, employing the Robot Constitution. During this procedure, viable tasks are chosen and performed, while pertinent data are gathered (data collection).