# A Survey of Robot Intelligence with Large Language Models
## 4 Language Models for Robotic Intelligence  
### Reward Design in Reinforcement Learning  
Research in reinforcement learning, closely associated with the field of robotics, has actively incorporated studies using LLM models. Specifically, Nvidia has developed a GPU-based multi-environment reinforcement learning platform. Utilizing its Omniverse 3D virtual environment platform, Nvidia created Isaac Sim, which is dedicated to robot simulation. [PERSON] published research findings on Isaac Gym (Preview), which achieved significant reductions in reinforcement learning training times through GPU-based multi-environment approaches. Subsequently, Isaac Gym (Preview)'s features were integrated into Isaac Sim and released as Omni Isaac Gym. Later, Nvidia introduced Orbit [133], facilitating the simulation of PhysX 5.1-based cloth, soft-body, fluid, and rigid-body dynamics, along with RGBD, LiDAR, and contact sensor simulation. Orbit also incorporates various robot platforms into the simulation environment. Recently, Orbit was updated to Isaac Lab and integrated into Isaac Sim 4.0. Nvidia has continuously advanced dynamic simulation environment technologies for reinforcement learning using GPU parallel computation. Leveraging this GPU reinforcement learning, they launched Eureka [11], which automates the design of reward functions for reinforcement learning using LLMs. Following this, Nvidia introduced DrEureka [134], an automated platform addressing the Sim2 Real problem [135] in reinforcement learning based on Eureka. Eureka (Evolution-driven Universal REward Kit for Agent) [11], shown in Figure 4, automatically generates reward functions for various tasks using different robots, eliminating the need for specific templates or tailored reward functions for the robot's form or explanations for reinforcement learning tasks. Eureka consists of three main components: _environment-as-context_, _evolutionary search_, and _reward reflection_. Environment-as-context generates executable reward functions in a zero-shot manner by utilizing virtual environment source (Python) code as context. Evolutionary search iteratively generates reward function candidates and proposes enhanced functions based on previously generated and best-performing ones, while also creating new functions through mutation. Reward reflection offers a text summary of reward function quality based on training statistics recorded during reinforcement learning, which assists in generating subsequent reward functions as feedback for the performance of previous functions. The reward functions generated outperformed expert-generated functions in 83% of benchmark tests. Moreover, Eureka solved the pen spinning problem where a robot hand must spin a pen as much as possible according to predefined rotations, a task previously considered unsolvable through manual reward engineering. Eureka introduces a universal reward function design algorithm based on a code LLM and in-context evolutionary search, facilitating human-level reward generation for various robots and tasks without the need for prompt engineering or human intervention. Following Eureka, DrEureka [134], shown in Figure 5, was developed to address the sim-to-real problem by automatically configuring appropriate reward functions and domain randomization for physical environments. DrEureka's reward-aware physics priors mechanism defines the lower and upper bounds of physical environment parameters based on policies trained through initial reinforcement learning, facilitating reinforcement learning across various physical environment domains. This randomization enables the trained model to excel in actual environments. Consequently, DrEureka achieved benchmark success in real-world quadruped locomotion with walking globe and cube-rotation manipulation using real robots, all without human supervision. [PERSON] [136] introduced _Text2 Reward_, a framework that automatically generated dense reward functions for reinforcement learning using LLMs.