CogLoop: Bridging Planning and Control for Robot Manipulation by Embodied Feedback
Authors: Chuhao Jin, Jiange Yang, Wenhui Tan, Bei Liu, Ruihua Song, Limin Wang, Jianlong Fu


Reviewer paper summaries: This paper aims at solving the coordination problem of LLMs' generated plans and downstream controlling. For this, this paper introduces a dataset LayoutBlock including 35 high-level tasks of multi-step text plans and paired observation-action sequences. Besides, the paper also proposes a closed-loop multi-modal embodied planning model. Experiments on the LayoutBlock dataset show the proposed model can outperform both existing open-loop and closed-loop methods.
Summary of strengths: - The paper is well-written and easy to understand and follow.
- The collected dataset can benefit the community a lot.
- The proposed method to fine-tune the multi-modal model and experiments provide insights on the advantages and disadvantages of fine-tuning and GPT based closed-loop and open-loop methods.
Summary of weaknesses: - In the LayoutBlock dataset, the evaluation is based on the accurate position of blocks. However, the key to the task is the shape construction no matter where the whole shape is placed. This poses strong constraints on the flexibility of shape construction tasks.
- Code generation is good at tasks requiring accurate spatial reasoning. So related baselines such as Code as Policies should be discussed.
Comments, suggestions and typos: - The paper should also provide details about the simulated environment.
Overall assessment scores: 4