Title: Multi-module GRPO: Composing Policy Gradients and Prompt Optimization for Language Model Programs
Primary area: reinforcement learning
Keywords: chain of thought, online reinforcement learning, in-context learning, multi-hop reasoning, prompting techniques, bettertogether

Abstract: Group Relative Policy Optimization (GRPO) has proven to be an effective tool for post-training language models (LMs). However, AI systems are increasingly expressed as modular programs that mix together multiple LM calls with distinct prompt templates and other tools, and it is not clear how best to leverage GRPO to improve these systems. We begin to address this challenge by defining mmGRPO, a simple multi-module generalization of GRPO that groups LM calls by module across rollouts and handles variable-length and interrupted trajectories.We find that mmGRPO, composed with automatic prompt optimization, improves accuracy by 11% on average across classification, many-hop search, and privacy-preserving delegation tasks against the post-trained LM—and by 5% against prompt optimization on its own. mmGRPO is released as an open-source optimizer for compound-AI systems.

TL;DR: Extends GRPO to modular LM programs; shows that composing GRPO with prompt optimization works the best