Multi-module GRPO: Composing Policy Gradients and Prompt Optimization for Language Model Programs

## Abstract

Abstract Group Relative Policy Optimization (GRPO) has proven to be an effective tool for post-training language models (LMs). However, AI systems are increasingly expressed as modular programs that mix together multiple LM calls with distinct prompt templates and other tools, and it is not clear how best to leverage GRPO to improve these systems. We begin to address this challenge by defining mmGRPO , a simple multi-module generalization of GRPO that groups LM calls by module across rollouts and handles variable-length and interrupted trajectories. We find that mmGRPO , composed with automatic prompt optimization, improves accuracy by 11% on average across classification, many-hop search, and privacy-preserving delegation tasks against the post-trained LM—and by 5% against prompt optimization on its own.
We open-source mmGRPO in DSPy as the dspy.GRPO optimizer. https://github.com/stanfordnlp/dspy

## 1 Introduction

Modern natural language processing (NLP) systems are increasingly implemented as modular systems, in which each module is responsible for a well-specified subtask that contributes to solving a broader objective.
A canonical example is “multi-hop” research, where the system responds to a question by iteratively using a query generation LM module to produce a search query, passing that query to a retriever, and finally feeding all iteratively retrieved passages into a response generation LM module to produce the final output.
The explicit modularization of such systems makes their behavior controllable, akin to conventional software, and allows for structured optimization of individual components, leveraging the priors of the LM differently for each module.

Group Relative Policy Optimization (GRPO; Shao et al. 2024) has recently emerged as a powerful method for fine-tuning language models (LMs) in the final stages of training.
By leveraging relative rewards within groups of “reasoning” rollouts that share the same prompt, GRPO offers a simple alternative to Proximal Policy Optimization (PPO; Schulman et al. 2017).
However, GRPO was originally designed for single-stage settings where each rollout consists of a single autoregressive LM call, and it is not obvious how to best extend it to systems composed of multiple such calls with distinct prompt templates.

In this paper, we ask whether post-training RL algorithms such as GRPO could be applied effectively to such multi-module LM programs, in which each rollout may invoke several distinct LM modules, each with its own prompt template and context.
This could prove challenging in practice, as such the rollouts generated from the same input to the program can differ in both number of steps and structure, due to variations in control flow or early termination from, e.g., parsing failures, and often produce disjoint intermediate inputs and outputs.

In response to these challenges, we implement mmGRPO, a simple and extensible framework for applying GRPO to multi-module setups.
The core idea is to relax GRPO’s requirement for shared inputs by grouping rollouts at the *module-level*, aligning structurally comparable module calls across different trajectories.
This approach enables GRPO-style policy gradient updates without requiring shared histories or module-level inputs across rollouts, and it offers a first strong baseline for online policy-gradient RL methods applied to LM programs.
We open-source mmGRPO as an off-the-shelf optimizer for arbitrary compound AI systems as part of the DSPy library at [dspy.ai](https://dspy.ai).

Ours is the first implementation of GRPO that applies to sophisticated pipelines of LMs.
This enables us to conduct a controlled comparison of three approaches to optimizing modular AI systems: prompt optimization (PO), online reinforcement learning via mmGRPO, and their combination using the BetterTogether framework (Soylu et al., 2024).
Our evaluation spans three diverse LM program tasks: classification (Banking77; Casanueva et al. 2020), multi-hop claim verification (HoVer; Jiang et al. 2020, and privacy-conscious delegation (PAPILLON; Siyan et al. 2024).
Each involves different reasoning styles and control flow structures.
Experiments are run using two open-source LMs, llama3.1-8b-instruct (Grattafiori et al., 2024) and qwen3-8b (Yang et al., 2025).

Our results are summarized in [Table 1](#S1.T1).
Across these settings, mmGRPO improves performance by 7% on average against the model’s unadapted reasoning performance.
While mmGRPO does not always surpass the prompt optimized programs via MIPROv2 (Opsahl-Ong et al., 2024), it complements them effectively: staging MIPROv2 and mmGRPO—à la BetterTogether—consistently yields higher performance than either method alone, improving by 5% and 3% compared to MIPROv2 and mmGRPO, respectively; and by 11% compared to the model’s unadapted reasoning performance.
These findings suggest that policy gradient RL and PO offer complementary benefits for LM program training, and we advocate for future work exploring their integration in both offline and online settings.

**Table 1: Performance of different learning algorithms across three LM programs: a single-stage program, Banking77, and multi-stage programs, PAPILLON and HoVer4-HOP. MIPROv2 represents a prompt optimization baseline, while Vanilla CoT refers to vanilla chain-of-thought prompting. Both mmGRPO and MIPROv2 improve over the untuned baseline, though neither consistently dominates the other. The best overall performance is achieved by the BetterTogether variant of mmGRPO, which first applies prompt optimization using MIPROv2 and then fine-tunes using mmGRPO. We report dev set accuracy for each cell, averaged over 3 seeds. The dev set is used strictly for evaluation and not for optimization.**
| Strategy | Banking77 | PAPILLON | HoVer4-HOP | Avg Scores |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| llama3.1 | qwen3 | llama3.1 | qwen3 | llama3.1 | qwen3 | llama3.1 | qwen3 | All |  |
| Baseline Strategies: |  |  |  |  |  |  |  |  |  |
| Vanilla CoT | 58.4 | 64.6 | 76.2 | 78.3 | 59.5 | 60.6 | 64.7 | 67.8 | 66.3 |
| MIPROv2 (PO) | 59.4 | 65.9 | 83.9 | 78.1 | 63.4 | 69.3 | 68.9 | 71.1 | 70.0 |
| mmGRPO Strategies: |  |  |  |  |  |  |  |  |  |
| mmGRPO | 63.7 | 64.9 | 83.9 | 83.3 | 60.2 | 71.0 | 69.3 | 73.1 | 71.2 |
| BetterTogether(PO, mmGRPO) | 63.7 | 69.1 | 86.5 | 81.1 | 68.3 | 71.5 | 72.8 | 73.9 | 73.4 |

## 2 Preliminaries

GRPO is an online policy gradient method for LM fine-tuning that operates over *groups* of trajectories sharing the same input prompt in single-stage tasks.
The GRPO objective encourages the current policy $p_{\theta_{\text{old}}}$, parametrized by LM weights $\theta_{\text{old}}$, to upweight relatively high-reward completions within a group, while applying PPO-style clipping and KL divergence regularization to ensure stable updates.
This results in an updated policy $p_{\theta}$.

GRPO also makes use of a reference policy $p_{\theta_{\text{ref}}}$ in the KL-divergence penalty, seeking to prevent the updated policy from drifting too far from its original distribution.
Here, we express the original GRPO objective in [Equation 1](#S2.E1) in terms of the prompt–output–reward triples $(q,o_{i},r_{i})$ to facilitate the extension to the multi-module setting.

$$ $\displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\,$ $\displaystyle\mathbb{E}_{\{(q,o_{i},r_{i})\}_{i=1}^{G}}\text{, where }\theta\text{ indicates the parameters for an LM shared by all groups}$ $\displaystyle{\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\left\{\min\left(\omega_{t}\hat{A}_{i},\,\operatorname{clip}\left(\omega_{t},\,1-\epsilon,\,1+\epsilon\right)\hat{A}_{i}\right)-\beta\mathbb{D}_{\mathrm{KL}}[p_{\theta}\,\|\,p_{\theta_{\text{ref}}}]\right\}}$ (1) $\displaystyle\text{where}\quad\omega_{t}$ $\displaystyle=\frac{p_{\theta}(o_{i,t}\mid q,o_{i,<t})}{p_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}\text{, and }\hat{A}_{i}\text{ is derived from the observed reward $r_{i}$ (below)}$ $$

Each GRPO group is defined as a set of triples $\mathcal{G}=\{(q,o_{i},r_{i})\}_{i=1}^{G}$, constructed by first sampling a fixed prompt from a distribution of questions $q\sim P(Q)$, and then generating a batch of $G$ completions $\{o_{i}\}_{i=1}^{G}\sim p_{\theta_{\text{old}}}(O\mid q)$ from the current policy. Finally, a scalar reward $r_{i}$ for each $o_{i}$ is computed with a reward function.
The term $\omega_{t}$ denotes the importance sampling ratio between the new and old policies for the $t$th token in a given output.
The scalar reward $r_{i}$ is then normalized within the group to compute an advantage $\hat{A}_{i}$ in the outcome supervision formulation of GRPO, which is applied uniformly across all tokens $t$ in the corresponding completion, as shown in [Equation 2](#S2.E2).

$$ $\displaystyle\hat{A}_{i}$ $\displaystyle=\frac{r_{i}-\text{mean}(\mathcal{R})}{\text{std}(\mathcal{R})},\quad\mathcal{R}=\{r_{i}\text{, reward for }o_{i}\}_{i=1}^{G}$ (2) $$

#### LM program formulation

An LM program $\Phi$ is composed of LM modules and other tools orchestrated by the control flow of $\Phi$.
Let $\mathcal{M}=\{M_{1},\ldots,M_{|\mathcal{M}|}\}$ denote the set of LM modules used therein, each of which communicates via natural language.

Given a structured input $x$ (e.g., a record with fields such as question), executing $\Phi(x)$ orchestrates module invocations, transforming inputs and routing outputs between modules.
In other words, $\Phi(x)$ defines a distribution from which we can sample $y,\rho$ pairs, where $y$ is the final output and $\rho$ is the trajectory of module calls:

$$ $\displaystyle(y,\rho)\sim\Phi(x),\quad\rho=[\zeta_{1},\zeta_{2},\dots,\zeta_{|\rho|}],$ (3) $$

Here, the trajectory $\rho$ records the sequence of module calls, and each trace $\zeta_{t}=\langle M_{t},q_{t},o_{t}\rangle$ captures the module identity as well as the module-level inputs and outputs at module invocation $t$ within the program trajectory.
The trajectory $\rho$ logs only the LM-level calls in their execution order and omits any other control logic.

Each module $M\in\mathcal{M}$, which may appear zero or more times in a given $\rho$, is parameterized by a prompt template $\pi_{M}$ and LM weights $\theta_{M}$.
During execution at module invocation $t$, the prompt template $\pi_{M_{t}}$ transforms the input $q_{t}$ into a materialized prompt: $q_{t}\leftarrow\pi_{M_{t}}(q_{t})$.
This prompt is then passed to an LM parameterized by $\theta_{M_{t}}$, which samples an output $o_{t}\sim p_{\theta_{M_{t}}}(\cdot\mid q_{t})$, returned to the control flow of $\Phi$ for subsequent steps.(^1^11It is useful to consider how this setup differs from standard multi-turn LM generation settings, where the LM prompt is expanded serially in each turn (Jin et al. 2025; Zeng et al. 2025; Wang et al. 2025). In arbitrary LM programs, the control flow dictates what context is visible to each module by selecting its inputs, enabling more modular and interpretable execution, but presenting new challenges for learning.)

This modularity offers several benefits.
It allows for privacy-preserving delegation, e.g., a module may call a proprietary LM that should not access previous interactions, as in our PAPILLON task, and better context length management, which is particularly important in RAG-style pipelines like HoVer, where large numbers of retrieved passages may need to be processed independently.
This is a core reason why multi-step GRPO formulations wouldn’t be suitable for LM programs out-of-the-box and motivates us to explore alternative multi-module formulations.
Throughout this paper, we treat LM policy inputs as being defined strictly at the module-level.

#### LM program optimization

Let $\mathcal{D}=\{(x,m)\}$ be a dataset of inputs $x$ and optional metadata $m$ (e.g., final answer, documents to retrieve, or PII to redact). The goal is to learn the parameters of the given LM program $\Phi$, namely, the prompt templates $\pi_{M}$ and LM weights $\theta_{M}$ for each module $M\in\mathcal{M}$, such that we maximize the expected reward $\mathbb{E}_{(x,m)\sim\mathcal{D};\,(y,\rho)\sim\Phi_{\Pi,\Theta}(x)}[\mu(y,\rho,m)]$.

Here, the reward function $\mu(y,\rho,m)$ scores the execution, typically based on the final output $y$’s correctness.
Any metadata $m$ (e.g., gold answers) is not visible to the program during execution but may be used by $\mu$ for evaluation.

## 3 Applying GRPO to multi-module LM programs

Given a dataset $\mathcal{D}$ and a reward function $\mu$, our goal is to optimize an LM program $\Phi$ consisting of modules $\mathcal{M}$ by updating the weights $\theta_{M_{i}}$ of each module.
In standard GRPO, each group contains trajectories from a single auto-regressive LM call—i.e., one prompt and its full output. LM programs typically comprise multiple modules, each invoking its own LM with a custom prompt, raising the question of how to best extend GRPO grouping to this multi-module setting.
To set a strong baseline in this space, we explore the simplest possible design with mmGRPO, particularly one that allows our implementation to remain largely modular with respect to existing GRPO implementations.

mmGRPO starts by sampling full program trajectories, forming a meta-group of trajectories, each with many module invocations.
It then aligns module calls across these trajectories and creates GRPO groups at the module level, each containing input–output–reward triples for a specific module. We default to uniform credit assignment, setting each reward to correspond to the final program reward.
A modified GRPO loss is then applied independently to each group, updating only the LM weights of the module that produced the group’s data.
In practice, the same LM is often shared across all modules.
Section [5](#S5) validates that this approach is able to improve realistic LM programs and to compose effectively with prompt optimization.
We focus on the high-level design in this section, deferring implementation details to [Appendix A](#A1).

Additionally, mmGRPO allows sampling trajectories not only from the student program but also from a list of fixed *teacher* programs.
This enables flexible training setups, including warm-starting from prompt-optimized programs or learning from more capable LMs.
When used on single-module programs without teachers, mmGRPO reduces exactly to standard GRPO.

The meta-group of trajectories used in mmGRPO consists of multiple executions of the same program on a shared program-level input $x$, i.e., $(y,\rho)\sim\Phi(x)$, where $y$ is the final program output and $\rho=[\zeta_{1},\zeta_{2},\dots,\zeta_{|\rho|}]$ is the trajectory of module calls.
Each $\zeta_{t}$ is a triple containing the invoked module $M_{t}$, the prompt $q_{t}$ sent to the corresponding module LM $\theta_{M_{t}}$, and the resulting output $o_{t}$.
The program-level output reward for the entire trajectory is computed as $r=\mu(y,\rho,m)$, where $m$ is any additional metadata associated with the example.

To construct GRPO groups, mmGRPO aligns module calls across trajectories based on both the module identifier and the relative order in which it appears within the trajectory.
This alignment process yields module-level GRPO groups, each of the form $\{(q_{i},o_{i},r_{i})\}^{G}_{i=1}$, where $q_{i}$ and $o_{i}$ are extracted from a group of aligned traces all generated by a specific module $M$, and $r_{i}$ is set to the corresponding program-level output reward for the trajectory that generated each trace.

$$ $\displaystyle\mathcal{J}_{\text{mmGRPO}}(\hbox{\pagecolor{HighlightColorTheta}$\theta_{M}$})=\,$ $\displaystyle\mathbb{E}_{\{(\hbox{\pagecolor{HighlightColorQ}$\scriptstyle q_{i}$},o_{i},r_{i})\}_{i=1}^{G}}\text{, where }\hbox{\pagecolor{HighlightColorTheta}$\scriptstyle\theta_{M}$}\text{ indicates the LM weights for module $M$}$ $\displaystyle\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}$ $\displaystyle\left\{\min\left(\omega_{t}\hat{A}_{i},\,\operatorname{clip}\left(\omega_{t},\,1-\epsilon,\,1+\epsilon\right)\hat{A}_{i}\right)-\beta\mathbb{D}_{\mathrm{KL}}[\hbox{\pagecolor{HighlightColorTheta}$\scriptstyle p_{\theta_{M}}$}\,\|\,\hbox{\pagecolor{HighlightColorTheta}$\scriptstyle p_{\theta_{M_{\text{ref}}}}$}]\right\}$ (4) $\displaystyle\text{where}\quad\omega_{t}$ $\displaystyle=\frac{\hbox{\pagecolor{HighlightColorTheta}$p_{\theta_{M}}$}(o_{i,t}\mid\hbox{\pagecolor{HighlightColorQ}$q_{i}$},o_{i,<t})}{\hbox{\pagecolor{HighlightColorTheta}$p_{\theta_{M_{\text{old}}}}$}(o_{i,t}\mid\hbox{\pagecolor{HighlightColorQ}$q_{i}$},o_{i,<t})}\text{, and }\hat{A}_{i}\text{ is computed from $r_{i}$ via \autoref{eq:grpo_outcome_supervision}}$ $$

In practice, not all trajectories generated by $\Phi$ given the same program-level input $x$ follow the same structure; the program logic may diverge (e.g., by invoking different modules or terminating early), or errors such as module-level parsing failures may halt execution.
To accommodate this, mmGRPO optionally pads smaller groups to a fixed size before applying the loss, described in more detail in [Appendix A](#A1).
Once the groups are formed, mmGRPO loss in [Equation 4](#S3.E4) is applied independently to each module-level group, with two key differences from the original GRPO objective ([Equation 1](#S2.E1)).
First, rather than updating a shared LM, each group updates only the weights of the module it corresponds to.
Second, unlike GRPO where completions share a single prompt, datapoints in a module-level group may have different prompts $q_{i}$, reflecting variation in upstream context.

## 4 Composing Online RL with Prompt Optimization via BetterTogether

BetterTogether (Soylu et al., 2024) demonstrated that combining prompt optimization (PO) with weight optimization yields stronger results than using either technique alone, specifically in the context of offline RL via rejection fine-tuning on outcome-filtered trajectories.
Rather than applying weight optimization directly to an unmodified program, the authors first optimize the program’s prompt templates and then apply weight optimization on the resulting prompt-optimized program.

We extend this approach to the online RL setting using mmGRPO, and combine it with a state-of-the-art prompt optimizer, MIPROv2 (Opsahl-Ong et al., 2024).
Soylu et al. (2024) also experiment with alternative compositions, such as running prompt optimization after weight tuning, but in our work, we focus on the former: applying mmGRPO to a prompt-optimized program.

## 5 Experiments

### 5.1 LMs and datasets

The LM programs for each of the tasks we use for evaluation, along with example inputs and program trajectories, are shared in [Appendix B](#A2).
We use the LM program implementations open-sourced by Tan et al. (2025) as our starting point for all tasks, but make modifications for HoVer.
For more information on the LMs and datasets used along with their license information, refer to [Appendix C](#A3).

#### LMs

We run our experiments on two open LMs: llama3.1-8b-instruct (Grattafiori et al., 2024) and qwen3-8b (Yang et al., 2025).
Although mmGRPO allows for different LM copies to learn separate weight updates for the different modules of a program, we use the same underlying LM weights for each module for lightweight training and deployment in a multi-task manner.

#### Classification

Banking77 is an intent classification benchmark involving $13,083$ labeled customer service queries from the banking domain Casanueva et al. (2020).
The task is to assign each user query to one of 77 intent classes.
We implement a simple program for this task using a single Chain-of-Thought (CoT) module (Wei et al., 2022), which first produces a reasoning trace before predicting the intent label.
For evaluation we compute the exact match between the ground-truth label and the generated label.
Since the program we have for Banking77 has only a single module, running the mmGRPO algorithm on it is the same as the standard GRPO setup.
For training and evaluation, we randomly sample $250$ training examples and $500$ for development.

#### Privacy-conscious delegation

The Private User Prompt Annotations (PUPA) benchmark constructed by Siyan et al. (2024) focuses on privacy-preserving question answering, where the goal is to respond to user queries without exposing private information to external APIs.
We use PAPILLON, also from Siyan et al. (2024), a two-module pipeline that generates a redacted version of a private user query, sends the redacted query to an untrusted but more powerful external model, and then uses the response of that powerful model to generate the final response.
We utilize openai/gpt-4.1-mini-2025-04-14 (OpenAI, 2025) as the external LM.
As described in Siyan et al. (2024), the evaluation metric is a composite score which takes into account the content of the response and the amount of private information that was leaked, both of which are judged by the same large LM.
We evaluate this setup using $111$ training examples and $221$ for development.

#### Multi-hop claim verification

HoVer (Hoppy Verification, Jiang et al., 2020) is a claim verification benchmark where the task is to extract facts from multiple relevant Wikipedia articles and deciding whether a given claim is supported.
The claims in HoVer are multi-hop in that they require multi-hop reasoning by connecting information found in different articles.
The original dataset has $18,171$ train and $4000$ development and test examples derived from the examples in the HotPotQA dataset (Yang et al., 2018).
Our program for HoVer consists of 2 modules, a query generation module and a fact summarization module, called iteratively over 4 hops, along with a ColBERTv2 (Santhanam et al., 2021) retriever indexed on the short snippets from the Wikipedia (2017) dump provided with the HotPotQA dataset, shared with HoVer.
We refer to the particular 4-hop variant Hover program we use with HoVer4-HOP, in order to differentiate it from the one provided in Tan et al. (2024).
The program returns up to 100 passages at the end, and the final metric evaluates whether the gold passages are found within the returned passages using Recall@100.
We build our splits from the original train split, randomly sampling $500$ examples each for our train and development splits; while ensuring that we don’t sample any two examples derived from the same HotPotQA question.

### 5.2 Baseline and method details

We evaluate each of our LM and task pairs with vanilla Chain-of-Thought (CoT) and a prompt optimizer, to serve as baselines.
We demonstrate our mmGRPO optimizer in two flavors: mmGRPO, and BetterTogether mmGRPO.
While each method assumes access to a program-level evaluation metric, none relies on an external oracle dataset.
Instead, we generate training data dynamically by running the program itself and bootstrapping from model outputs and associated program-level metrics.
We use the DSPy framework (Khattab et al., 2024) to run our baseline experiments and develop our new mmGRPO optimizers.
We use DSPy’s RL training library, Arbor (Ziems et al., 2025), which draws inspiration from the Verifiers library (Brown, 2025).

#### Inference

We use the vLLM (Kwon et al., 2023) engine for sampling with max context length of $32,768$ tokens for inference.
We set max tokens to $1032$ and re-try each query up to $3$ times in case of module parsing errors.
For qwen3-8b, we use $\mathrm{sampling_temperature}=0.6$, $\mathrm{top_p}=0.95$ and $\mathrm{top_k}=20$ following the parameters used for its instruction training as noted in Yang et al. (2025).
For llama3.1-8b-instruct, we use $\mathrm{sampling_temperature}=0.6$ and $\mathrm{top_p}=0.9$ following the official model card’s generation configuration in HuggingFace (MetaAI, 2024).

#### Vanilla CoT

We adopt the Chain-of-Thought (CoT) prompting method introduced by Wei et al. (2022), where each module’s prompt instructs the language model to first generate a reasoning field before producing its final answer.
Unless stated otherwise, both the prompt-optimization and mmGRPO methods described below begin training from this base CoT prompt.
We refer to this initial prompt configuration as the “Vanilla CoT” program.

#### MIPROv2

We use the state-of-the-art prompt optimizer Multiprompt Instruction PRoposal Optimizer Version 2 (MIPROv2; Opsahl-Ong et al. 2024) as our prompt-optimized baseline.
For our experiments, we use the auto=medium setting, which uses 12 trials; 12 few-shot and 6 instruction candidates, and automatically uses a 80% of the train set for validation.
We refer to the program we optimize using MIPROv2 with these settings as the prompt-optimized program and re-use it for the BetterTogether strategy below.

#### mmGRPO

We train our models using the HuggingFace GRPOTrainer, each with a maximum context length of $8192$ tokens.
Training is performed with a temperature of $0.6$, a learning rate of $1\times 10^{-5}$, gradient accumulation steps of $20$, with per device train batch size of $1$.
We use $\beta=0.01$ and gradient norm clipping of $0.1$ for qwen3-8b; and $\beta=0.04$ and gradient norm clipping of $0.5$ for llama3.1-8b-instruct.

We run mmGRPO for 750 steps, using 4 training examples per step.
At each step, we randomly draw 4 examples from the training dataset.
For each example, we generate 12 rollouts, which are then grouped into module-level GRPO groups using the procedure in [Algorithm 2](#alg2).
We use a train context length of $\mathrm{8,192}$ tokens, which is used to filter any trajectory with a module level prompt and completion longer than this.
We apply Low-Rank Adaptation (LoRA, Hu et al. 2021) with rank $\mathrm{r}=16$, $\mathrm{lora_alpha}=64$, $\mathrm{lora_dropout}=0.05$, targeting the projection modules $[\mathrm{q},\mathrm{k},\mathrm{v},\mathrm{o},\mathrm{up},\mathrm{down},\mathrm{gate}]$.
We run all of our mmGRPO experiments below using these same settings.
Pseudocode of the mmGRPO algorithm can be found in [Algorithm 1](#alg1).

#### mmGRPO with BetterTogether

We further experiment with a setting where we combine prompt optimization with the weight optimization of mmGRPO following the BetterTogether algorithm introduced by Soylu et al. (2024).
Specifically, instead of directly optimizing the weights used in an LM program, we first use prompt optimization to find high quality prompts to be used by the LM program.
The prompts are then kept fixed in the LM program and the program weights are then optimized with mmGRPO.
We refer to this setup as BetterTogether(PO, mmGRPO) for short.

### 5.3 Main results

Our main experimental results are shared in [Table 1](#S1.T1), evaluated on the dev set and averaged over $3$ seeds.
The dev set is used exclusively for evaluation and plays no role in optimization.

#### mmGRPO and BetterTogether(PO, mmGRPO ) consistently improve over their respective baselines.

We can see that the mmGRPO row is consistently higher than the “Vanilla CoT” row, 7% on average.
Similarly, BetterTogether(PO, mmGRPO) shows consistent gains over the “MIPROv2 (PO)” row, 5% on average.
These show that mmGRPO is effective at finding better policies for the provided program across all LM–task pairs.

#### PO is competitive with lower computational budgets.

When averaged across all tasks and models, MIPROv2 alone improved upon the Vanilla CoT strategy by 5% compared to mmGRPO’s 7% improvement.
However, MIPROv2 achieved these results significantly faster while using fewer GPU-hours.
On average, our vanilla mmGRPO experiments took 18.7 hours using 2 H100 GPUs whereas MIPROv2 took only 1.4 hours on average and only required 1 H100 GPU.
These results indicate that PO approaches like MIPROv2 are likely much more feasible for settings which have lower computation budgets.

#### BetterTogether(PO, mmGRPO ) performs the best in most task pairs.

BetterTogether(PO, mmGRPO) approach improves over the Vanilla CoT by 11%, MIPROv2 by 5%, and vanilla mmGRPO by 3%.
This shows the value of high-quality rollouts at the start of mmGRPO training, as performing PO generates stronger rollouts, leading to a more robust training signal early in the training runs.

## 6 Related work

#### Prompt optimization

Much recent work has explored methods that adapt prompt strings to fit data.
This includes methods focused on prompting LMs to generate instructions (Yang et al., 2024; Zhou et al., 2023; Pryzant et al., 2023; Fernando et al., 2024), using gradients to optimize the prompt (Shin et al., 2020; Wen et al., 2023), and RL-based prompt optimizers (Deng et al., 2022; Zhang et al., 2023; Hao et al., 2023), among many others.

#### Weight optimization

Proximal Policy Optimization (PPO) has been widely used for post-training language models with reinforcement learning, particularly when aligning language models with human preferences or feedback (Schulman et al., 2017; Ouyang et al., 2022).
Recently, Direct Preference Optimization (DPO) algorithms emerged as a simpler alternative that avoids explicit reward modeling and instead learns from contrastive preference pairs (Rafailov et al., 2023).
Similarly, Group Relative Policy Optimization (GRPO) offers an efficient alternative to PPO by avoiding the need for a value model and instead relying on estimated advantages through relative rewards within a group of rollouts (Shao et al., 2024).

#### Optimization of LM Programs’ Prompts & Weights

Existing work has explored optimizing LM programs with prompt optimizers, including those that focus primarily on rejection sampling (Khattab et al., 2024) and others that extend this to use Bayesian optimization for selecting the instruction-demonstration candidates that are most promising (Opsahl-Ong et al., 2024).
Additional work (Soylu et al., 2024) has explored combining weight optimizers with prompt optimizers for additional benefit, but in the context of offline RL. However, adapting some techniques to LM Programs requires making a number of decisions ([Section 2](#S2)) and presents substantial implementation challenges. The present work describes how we generalize GRPO to LM programs composed of multiple modules.

## 7 Conclusion

We introduce mmGRPO, a novel extension of GRPO that enables online weight optimization for multi-module LM programs by propagating final rewards backward across disjoint modules.
Our experiments demonstrate that mmGRPO consistently outperforms standard baselines across tasks and models, validating its effectiveness in navigating the challenging credit assignment problem without requiring intermediate supervision.
We further show that combining mmGRPO with state-of-the-art prompt optimization methods via BetterTogether yields the strongest overall performance in the majority of settings, revealing that complementary relationship between weight and prompt optimization holds for online RL methods.

## 8 Limitations

While our experiments demonstrate the promise of multi-module RL formulations, this work has several limitations.
First, we use 8-billion parameter language models, which may not reflect how mmGRPO performs with larger models.
Second, we rely on LoRA for fine-tuning; while efficient, this may limit training performance compared to full-parameter updates.
Third, we evaluate only one mmGRPO implementation despite many possible alternative formulations.
Finally, while Banking77 is a well-understood classification task, we study it in a limited-feedback setting where models only receive rewards derived from bootstrapped rollouts, not supervised intent labels.
While supervised training enables encoder models to perform well on this task, we investigate whether GRPO or MIPRO can achieve similar performance from reward signals alone. Our results suggest that this is not yet the case.

## Acknowledgments

D.S. thanks the members of the StanfordNLP group for their support and feedback.
The authors also thank Zoom, Inc., for providing access to their compute clusters for running the later batch of experiments.

## References

- Brown (2025)
William Brown.
Verifiers: Reinforcement learning with llms in verifiable environments.
[https://github.com/willccbb/verifiers](https://github.com/willccbb/verifiers), 2025.
- Casanueva et al. (2020)
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić.
Efficient intent detection with dual sentence encoders.
In *Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI*, pp. 38–45, Online, 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.nlp4convai-1.5.
URL [https://aclanthology.org/2020.nlp4convai-1.5](https://aclanthology.org/2020.nlp4convai-1.5).
- Deng et al. (2022)
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu.
RLPrompt: Optimizing discrete text prompts with reinforcement learning.
In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pp. 3369–3391, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.emnlp-main.222.
URL [https://aclanthology.org/2022.emnlp-main.222/](https://aclanthology.org/2022.emnlp-main.222/).
- Fernando et al. (2024)
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel.
Promptbreeder: self-referential self-improvement via prompt evolution.
In *Proceedings of the 41st International Conference on Machine Learning*, pp. 13481–13544, 2024.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang,
Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur,
Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher,
Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh
Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna
Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim
Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan,
Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi
Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma.
The llama 3 herd of models.
*ArXiv preprint*, abs/2407.21783, 2024.
URL [https://arxiv.org/abs/2407.21783](https://arxiv.org/abs/2407.21783).
- Hao et al. (2023)
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei.
Optimizing prompts for text-to-image generation.
*Advances in Neural Information Processing Systems*, 36:66923–66939, 2023.
- Hu et al. (2021)
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
*ArXiv preprint*, abs/2106.09685, 2021.
URL [https://arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685).
- Jiang et al. (2020)
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal.
HoVer: A dataset for many-hop fact extraction and claim verification.
In *Findings of the Association for Computational Linguistics: EMNLP 2020*, pp. 3441–3460, Online, 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.findings-emnlp.309.
URL [https://aclanthology.org/2020.findings-emnlp.309](https://aclanthology.org/2020.findings-emnlp.309).
- Jin et al. (2025)
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
*ArXiv preprint*, abs/2503.09516, 2025.
URL [https://arxiv.org/abs/2503.09516](https://arxiv.org/abs/2503.09516).
- Khattab et al. (2024)
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts.
Dspy: Compiling declarative language model calls into self-improving pipelines.
In *ICLR*, 2024.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.
- MetaAI (2024)
MetaAI.
Meta llama 3.1-8b-instruct: Generation configuration.
[https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/generation_config.json](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/generation_config.json), 2024.
Accessed: 2025-07-29.
- OpenAI (2025)
OpenAI.
Gpt-4.1-mini.
[https://platform.openai.com/docs/models/gpt-4.1-mini](https://platform.openai.com/docs/models/gpt-4.1-mini), 2025.
Accessed: 2025-07-17.
- Opsahl-Ong et al. (2024)
Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab.
Optimizing instructions and demonstrations for multi-stage language model programs.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 9340–9366, Miami, Florida, USA, 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.525.
URL [https://aclanthology.org/2024.emnlp-main.525/](https://aclanthology.org/2024.emnlp-main.525/).
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744, 2022.
- Pryzant et al. (2023)
Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng.
Automatic prompt optimization with “gradient descent” and beam search.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 7957–7968, Singapore, 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.494.
URL [https://aclanthology.org/2023.emnlp-main.494/](https://aclanthology.org/2023.emnlp-main.494/).
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in neural information processing systems*, 36:53728–53741, 2023.
- Santhanam et al. (2021)
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia.
Colbertv2: Effective and efficient retrieval via lightweight late interaction.
*ArXiv preprint*, abs/2112.01488, 2021.
URL [https://arxiv.org/abs/2112.01488](https://arxiv.org/abs/2112.01488).
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*ArXiv preprint*, abs/1707.06347, 2017.
URL [https://arxiv.org/abs/1707.06347](https://arxiv.org/abs/1707.06347).
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*ArXiv preprint*, abs/2402.03300, 2024.
URL [https://arxiv.org/abs/2402.03300](https://arxiv.org/abs/2402.03300).
- Shin et al. (2020)
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.
In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp. 4222–4235, Online, 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.emnlp-main.346.
URL [https://aclanthology.org/2020.emnlp-main.346](https://aclanthology.org/2020.emnlp-main.346).
- Siyan et al. (2024)
Li Siyan, Vethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, and Zhou Yu.
Papillon: Privacy preservation from internet-based and local language model ensembles.
*ArXiv preprint*, abs/2410.17127, 2024.
URL [https://arxiv.org/abs/2410.17127](https://arxiv.org/abs/2410.17127).
- Soylu et al. (2024)
Dilara Soylu, Christopher Potts, and Omar Khattab.
Fine-tuning and prompt optimization: Two great steps that work better together.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 10696–10710, Miami, Florida, USA, 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.597.
URL [https://aclanthology.org/2024.emnlp-main.597/](https://aclanthology.org/2024.emnlp-main.597/).
- Tan et al. (2025)
Shangyin Tan, Lakshya A Agrawal, Arnav Singhvi, Liheng Lai, Michael J Ryan, Dan Klein, Omar Khattab, Koushik Sen, and Matei Zaharia.
Langprobe: a language programs benchmark.
*ArXiv preprint*, abs/2502.20315, 2025.
URL [https://arxiv.org/abs/2502.20315](https://arxiv.org/abs/2502.20315).
- Tan et al. (2024)
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica.
Judgebench: A benchmark for evaluating llm-based judges.
*ArXiv preprint*, abs/2410.12784, 2024.
URL [https://arxiv.org/abs/2410.12784](https://arxiv.org/abs/2410.12784).
- Wang et al. (2025)
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li.
Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.
*ArXiv preprint*, abs/2504.20073, 2025.
URL [https://arxiv.org/abs/2504.20073](https://arxiv.org/abs/2504.20073).
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language models.
*ArXiv preprint*, abs/2201.11903, 2022.
URL [https://arxiv.org/abs/2201.11903](https://arxiv.org/abs/2201.11903).
- Wen et al. (2023)
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein.
Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery.
*Advances in Neural Information Processing Systems*, 36:51008–51025, 2023.
- Xu et al. (2025)
Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter.
Not all rollouts are useful: Down-sampling rollouts in llm reinforcement learning.
*ArXiv preprint*, abs/2504.13818, 2025.
URL [https://arxiv.org/abs/2504.13818](https://arxiv.org/abs/2504.13818).
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report.
*ArXiv preprint*, abs/2505.09388, 2025.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Yang et al. (2024)
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen.
Large language models as optimizers.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=Bb4VGOWELI](https://openreview.net/forum?id=Bb4VGOWELI).
- Yang et al. (2018)
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning.
HotpotQA: A dataset for diverse, explainable multi-hop question answering.
In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pp. 2369–2380, Brussels, Belgium, 2018. Association for Computational Linguistics.
doi: 10.18653/v1/D18-1259.
URL [https://aclanthology.org/D18-1259](https://aclanthology.org/D18-1259).
- Zeng et al. (2025)
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong.
Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment.
*ArXiv preprint*, abs/2505.11821, 2025.
URL [https://arxiv.org/abs/2505.11821](https://arxiv.org/abs/2505.11821).
- Zhang et al. (2023)
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez.
TEMPERA: Test-time prompt editing via reinforcement learning.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=gSHyqBijPFO](https://openreview.net/forum?id=gSHyqBijPFO).
- Zhou et al. (2023)
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.
Large language models are human-level prompt engineers.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=92gvk82DE-](https://openreview.net/forum?id=92gvk82DE-).
- Ziems et al. (2025)
Noah Ziems, Lakshya A Agrawal, Dilara Soylu, Liheng Lai, Isaac Miller, Chen Qian, Meng Jiang, and Omar Khattab.
Arbor: Open source language model post training.
[https://github.com/Ziems/arbor](https://github.com/Ziems/arbor), 2025.

## Appendix

## Appendix A mmGRPO algorithm

### A.1 Overview

The mmGRPO algorithm extends GRPO to the multi-module setting by improving the LM weights of each module in a program through module-level policy gradients.
Two core abstractions distinguish mmGRPO in [Algorithm 1](#alg1): (1) the ability to sample trajectories from multiple teacher programs, and (2) the construction of module-level GRPO groups based on relative invocation order.
These components are highlighted in the algorithm and explained in more detail in [Section A.2](#A1.SS2) and [Section A.3](#A1.SS3), respectively, while the remaining steps follow standard GRPO procedure and are included for completeness.

Figure: Algorithm 1 mmGRPO: GRPO for multi-module LM programs

mmGRPO takes as input a student program $\Phi$, a training dataset $\mathcal{D}$, a reward metric $\mu$, an optional set of teacher programs $\mathcal{T}$, and optional hyper-parameters (Line [1](#alg1.l1a)).
If unspecified, the set of teacher programs $\mathcal{T}$ defaults to a singleton set containing only the student program.
At each training step (Line [2](#alg1.l2a)), the algorithm samples a batch $\mathcal{B}$ of examples from the training dataset $\mathcal{D}$ using the configured batch size $B$ (Line [3](#alg1.l3a)). For each example $(x,m)\in\mathcal{B}$ (Line [4](#alg1.l4a)), the algorithm collects rollouts from the teacher programs via the SampleTeacherRollouts function (Line [5](#alg1.l5a)), which returns a set of output-trajectory tuples.
These rollouts are passed to FormModuleLevelGroups from [Algorithm 2](#alg2) (Line [6](#alg1.l6a)), which constructs module-level GRPO groups and returns them along with the corresponding references to the module-level LM weights $\theta_{M}$ to be updated.
The algorithm then iterates over each group and its associated LM weights (Line [7](#alg1.l7a)), and applies the GRPO loss (as defined in [Equation 4](#S3.E4)) independently to each group (Line [8](#alg1.l8a)), using the specified training hyper-parameters.
After $N_{\text{steps}}$ iterations, the algorithm returns the updated student program $\Phi$, preserving its original prompt templates while incorporating improved LM weights (Line [9](#alg1.l9a)).

### A.2 Sampling with teacher programs

In addition to the student program, mmGRPO accepts a list of optional teacher programs, which are used to generate the set of trajectories that populate the $\mathtt{runs}$ list.
At each GRPO step, rather than sampling all rollouts from the student program alone, mmGRPO samples trajectories from a specified mixture of teacher programs.
This list must include the student itself.
All teacher programs share the same structural interface, meaning they operate over the same LM program and module-level input/output fields, but may differ in their module-level prompt-templates (e.g., alternative instructions or few-shot examples) or LM weights (e.g., larger LMs).
These variations enable the mmGRPO framework to support training that is online but partially off-policy, providing greater flexibility in guiding learning using curated or higher-performing policies.

The SampleTeacherRollouts function samples trajectories from each teacher program in $\mathcal{T}$, using a rollout configuration $K$ that specifies the number of rollouts to generate per teacher.
This per-teacher control enables flexible data mixtures across programs.
For each rollout, the function extracts the final output $y$ and trajectory $\rho$, and collects the resulting $(y,\rho)$ pairs into the rollout set $\mathcal{R}$ returned for training.(^2^22When using teacher programs to sample trajectories, the modules $M$ recorded in the traces reflect those of the teacher rather than the student program. In practice, however, mmGRPO ensures that the module keys used to form module-level GRPO groups correspond to the student program’s modules for each respective teacher module, since it is required that student and teachers programs share the “same structure”.)

### A.3 Forming module-level groups

Figure: Algorithm 2 FormModuleLevelGroups: Create module-level GRPO groups for mmGRPO

We now describe how mmGRPO constructs GRPO-style groups at the module level for LM programs.
Once the rollouts are sampled , mmGRPO construct module-level GRPO groups via the FormModuleLevelGroups function described in [Algorithm 2](#alg2).
Each GRPO group is defined as a list of $G\leq R$ triples $\{(q_{i},o_{i},r_{i})\}_{i=1}^{G}$, where each element consists of a module-level input prompt $q$, the corresponding output $o$, and the final trajectory-level reward $r$.
In practice, one can use $G<R$, the number of rollouts, to leave room for post-hoc adjustments to group size (discussed later in this section).

Given the program $\Phi$, the list of output–trajectory tuples $\mathcal{R}$, and the desired GRPO group size $G$, FormModuleLevelGroups iterates over each output–trajectory pair in $\mathcal{R}$ (Line [3](#alg2.l3a)), computing a corresponding score $r=\mu(y,\rho,m)$ (Line [4](#alg2.l4a)).
If the corresponding trajectory is incomplete, a fallback reward is assigned (e.g., a formatting error penalty).
Following this, it iterates over the traces in each trajectory (Line [6](#alg2.l6a)).
Each trace contributes a triple $(q,o,r)$ consisting of the module-level input, output, and final trajectory reward.
This triple is added to the group corresponding to $(M,k)$, where $k$ is the relative invocation index of $M$ in the trajectory (Line [7](#alg2.l7a)), where the relative index is incremented after each occurrence (Line [8](#alg2.l8a)).
To ensure uniform group sizes despite variability in module invocation counts across trajectories, Lines [9](#alg2.l9) and [10](#alg2.l10) apply post-processing steps that adjust each group to have exactly $G$ elements, as detailed later in this section.
Finally, Line [11](#alg2.l11) constructs a list of LM weight references, one corresponding to each group, and both this list and the final GRPO groups are returned (Line [12](#alg2.l12)).

As a result, FormModuleLevelGroups creates GRPO groups by both the module identity and their relative position within the trajectory with respect to the other calls to the same module.
Let $K_{M_{i},\rho_{j}}$ denote the number of times module $M_{i}$ is invoked in trajectory $\rho_{j}$ for $(y_{j},\rho_{j})\in\mathcal{R}$; then the total number of GRPO groups formed across all trajectories is $\sum_{i}\max_{j}K_{M_{i},\rho_{j}}$, where $M_{i}\in\mathcal{M}$ for the given $\mathtt{runs}$.
Each resulting group is a list of module-level $(q,o,r)$ triples, corresponding to structurally aligned invocations of a given module at a specific position in the trajectory.
In contrast to standard GRPO, which produces a single group per set of rollouts in single-stage settings, mmGRPO yields a list of groups, one for each module and relative invocation position.
To ensure uniform group sizes and handle variation across trajectories, mmGRPO apply two post-processing steps: PadGroups and SelectKDiverseElements, described next.

#### Handling variably invoked trajectories with PadGroups

If every module $M_{i}$ in the student program is invoked the same number of times $K_{M_{i},*}$ across all trajectories $\rho_{j}$ where $(y_{j},\rho_{j})\in\mathcal{R}$, then each constructed GRPO group will contain exactly $R$ triples prior to the call to Line [9](#alg2.l9) in [Algorithm 2](#alg2).
For example, suppose the LM program consists of two modules, $M_{1}$ and $M_{2}$, and $R=3$ trajectories are sampled.
If, in every trajectory, the program calls $M_{1}$ exactly twice and $M_{2}$ exactly once, then mmGRPO will form three GRPO groups: two for $M_{1}$ (corresponding to its first and second calls) and one for $M_{2}$.
Each of these groups will contain exactly three triples, one from each trajectory, without requiring any padding or truncation.
This scenario arises when all executions yield structurally identical trajectories and none encounter parsing or runtime errors.

However, in practice, these conditions may not hold: some modules may be invoked fewer times due to variation in control flow, while others may terminate early due to parsing failures or other runtime errors.
In such cases, certain module, module invocation level GRPO groups may contain fewer than $N$ elements.
To address this, mmGRPO applies post-processing strategies to ensure that each group has a uniform size, with a call to the PadGroups function, described here.

The behavior of PadGroups is controlled by a padding_mode hyper-parameter (not explicitly noted in the function call to it in [Algorithm 1](#alg1)), which supports two values: truncate and fill.
Under the truncate strategy, it discards all GRPO groups for module $M_{i}$ whose invocation index exceeds $\min_{j}K_{M_{i},\rho_{j}}$, ensuring that only groups with complete representation across all trajectories are retained.
Under the truncate strategy, it discards all GRPO groups for a module $M_{i}$ whose invocation index exceeds $\min_{j}K_{M_{i},\rho_{j}}$, ensuring that only those invocation positions represented in every trajectory are retained.
We use the fill setting for the experiments reported in this paper.

#### Ensuring diversity in groups with SelectKDiverseElements

After standardizing group sizes across trajectories, mmGRPO further adjust seach group to ensure it contains exactly $G$ elements, the target GRPO group size.
Rather than sampling elements uniformly at random,it invokes the SelectKDiverseElements function, which selects (or duplicates) elements to form a group of size $G$ while maximizing diversity within the group.
This function handles both down-sampling (when the group has more than $G$ elements) and up-sampling (when it has fewer), favoring selections that increase reward variance in the sampled prompt-output pairs.
Contemporaneously, Xu et al. (2025) propose a similar variance-based selection strategy, demonstrating that promoting diversity in GRPO groups improves held-out generalization.

## Appendix B Task details

The DSPy implementations for the LM programs for Banking77, PAPILLON, and HoVer are presented alongside example datapoints in their respective sections.
Code snippets assume respective DSPy imports are available.

### B.1 Banking77

The following snippet presents an example datapoint from Banking77.
The task is to predict the appropriate label given a user query.

A DSPy program for Banking77 is shared in [Python Snippet 1](#LST1).

Figure: Python Snippet 1: DSPy program for Banking77.

### B.2 PAPILLON

The following snippet shows an example datapoint from PAPILLON.
The task is to generate the target response using a proprietary language model, while ensuring that no personally identifiable information (PII) is leaked to it.

A DSPy program for PAPILLON is shared in [Python Snippet 2](#LST2).

Figure: Python Snippet 2: DSPy program for Papillon.

### B.3 HoVer

The following snippet shows an example datapoint from HoVer.
The task is to retrieve all gold Wikipedia titles that support the given claim.

A DSPy program for HoVer is shared in [Python Snippet 3](#LST3).

Figure: Python Snippet 3: DSPy program for HoVer.

## Appendix C Asset information

The license information for the models and datasets we used are shared below.
All models and datasets are access via HuggingFace.

qwen3-8b is shared with the Apache License 2.0, accessed via the HuggingFace model identifier Qwen/Qwen3-8B

llama3.1-8b-instruct is shared with the Meta Llama 3 Community License, accessed via the HuggingFace model identifier meta-llama/Meta-Llama-3.1-8B-Instruct

Banking77 is shared with CC BY 4.0 license

HoVer is shared with CC BY 4.0 license

PAPILLON is shared with the MIT License license

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 2
presentation: 1
questions: 1- [suggestion] The paper can benefit from a teaser/motivating figure that readily explains the idea/motivation behind the method.

2- [suggestion] A concrete example on LM program would have been helpful for the reader to understand what you mean by Control Flow, trajectory of module calls, etc.

3- [Line 191] certain terms get used for the first time without properly being introduced before. What does the author refer to student/teacher program In Section 3? 

4- Do authors consider a separate module for the last step in each tasks? like label generation for classification, final response generation for PUPA etc.

5- why not including the trajectory-level GRPO as another baseline to better understand the contribution of grouping the trajectories (module-level)?
soundness: 3
strengths: - The proposed method is addressing an increasingly relevant challenge in modern AI systems—optimizing multi-component or modular architectures rather than isolated models
summary: This paper proposed Multi-module GRPO, an extension to GRPO algorithm/implementation which allows grouping LM calls by module and weight optimization for each module separately. They motivate this through the need for optimizing compound AI systems that consist of multiple modules (e.g., deep research) and allow privacy-preserving applications. They also showed combining mmGRPO with prompt optimization techniques consistently improve performance across 3 tasks of single-stage intent-classification, two-stage privacy-conscious delegation program, and four-stage claim verification.
weaknesses: - First, the paper is really poorly written, and suffers from significant clarity and presentation issues. It lacks a clear scientific structure and concrete running examples that could help readers understand the core problem and motivation. Introducing a concrete example early in the introduction would greatly improve readability. Certain terms get used for the first time without prior definition or explanation (see details in question section). Overall, the paper, in its current state, is not ready for publication and would benefit from substantial rewriting to improve presentation and clarity.

- The results section is also underdeveloped. The empirical evaluation is limited to a single table (Table 1) with no additional analysis, and ablation studies to justify the design choices or isolate the contributions of different components. The comparison baselines are also quite narrow.

- Finally, the advantages of mmGRPO over simpler methods such as prompt optimization, which require substantially less computational overhead, are not convincingly demonstrated (and are even acknowledged by the authors)

---

confidence: 2
contribution: 2
initial_rating: 4
presentation: 2
questions: Please address the weaknesses.
soundness: 3
strengths: - The paper appears solid overall, providing a convincing answer to the research question it poses.

- Although it does not make a particularly strong claim, the motivation and reasoning are coherent.

- The proposed approach is conceptually simple, as it mainly applies GRPO to a multistep system rather than introducing a new or overly complex algorithm.
summary: This work introduces MMGRPO, a multi-module extension of  GRPO designed for modular AI systems that combine multiple language model calls and tools. MMGRPO groups calls by module and manages variable or interrupted trajectories. When combined with automatic prompt optimization, it improves accuracy over post-trained LMs and over prompt optimization alone. The optimizer is released as an open-source tool.
weaknesses: - The ablation study and experimental analysis, from a quantitative point of view, are limited. Reducing the empirical depth of the analysis.

- The evaluation relies exclusively on accuracy, which is inadequate for large language model assessment.
The authors should adopt standard LLM evaluation metrics, such as pass@k (and ideally maj@k, as used in DeepSeek), to provide a more comprehensive and reliable performance evaluation.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. merge/split: what's the design criteria for choosing the boundary? also include ablation of merge/split of modules.
2. specify the message q scheme. what information about all prefix goes into the message that by right is sufficient to encode everything?
3. following (2), curious to see a state-Incorporated module (learned latent / prompt embedding). Try a learned state appended to q: a small encoder s = f(upstream traces) whose vector is serialized (text or special tokens) or fused via a prompt embedding adapter.
4. would grouping benefit from context awareness, not simply same call, same module name? Add a variant grouping by (module, k, cluster(q \oplus s)). If this outperforms (module, k) at the same keep-rate/compute, it indicates heterogeneity at fixed k was hurting.
soundness: 2
strengths: 1. It's relative easy, practical implementation. Sampling full runs does not require architectural changes; Integrates with LoRA, optional prompt optimization, and teacher traces.
2. Though many assumptions, they are stated properly. The main result is scoped correctly: directional equivalence of trajectory- vs module-level gradients under explicit conditions.
summary: The paper studies reinforcement learning for multi-module LM programs by moving GRPO updates from full trajectories to module-level groups, and articulates when this decomposition is preferable. Trajectory-level GRPO is feasible for fixed, single-shape pipelines, but becomes awkward when control flow varies (skips, repeats, early stops) or when module boundaries are administered separately. The proposed MMGRPO samples full runs, then groups tokens by (module id, k-th invocation) and applies a GRPO loss to that module’s tokens using the final program reward. The method rests on a Markov-given-message assumption: once routed to module M, next-token probabilities depend only on M’s input message q (which is expected to summarize upstream state) and its own prefix; routing and tools are treated as environment. Under three conditions, (one shared LM being optimized, equal or linearizable structure, and determinism given the same generations), the paper shows the sum of per-module gradients has the same direction as the trajectory-level gradient up to scale, which justifies the decomposition in fixed-shape settings. In practice, full-run sampling lets each run contribute to every module that fired and allows reuse of teacher traces by aligning their module calls; with LoRA updates and an optional prompt-optimization stage, results are comparable to trajectory-level training on fixed structures and remain usable when control flow varies, subject to how well q captures the necessary context.
weaknesses: 1. Narrow utility window. Benefits seem tied to variable control flow or administrative boundaries; outside that, a single trajectory loss is simpler and likely stronger.
2. Arbitrary partitioning risk. Module boundaries are designer choices; outcomes can depend on where you cut. The paper offers no merge/split robustness or guidance.
3. Strong Markov-given-message assumption. It is assumed, not validated, that q fully summarizes upstream state; if serialization is lossy, gradients are biased.
4. Heterogeneous groups. Grouping by the k-th call mixes unlike contexts/prompts; apples-to-oranges grouping can inflate variance and misdirect updates.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: * Sec 5.1: Do any of these setups use more than 1 teacher? i.e. does the list of teacher programs in L666 consist of any teacher other than the student program?  
* Sec 5.1 : It would be good to present examples of rollouts for each task, showing the trajectory of module calls, and some examples where these might lead to differing lengths.
* Appendix A3: It would be good to present some examples (inputs/outputs) showing how Algorithm 2 (FormingModuleLevelGroups) works since that is a critical component of the proposed approach.
soundness: 2
strengths: * Proposes a novel framework to extend GRPO to a setting where multiple module LMs are called. 
* Demonstrates that the method performs better than a chain of thought baseline.
* Shows that the method in combination with prompt optimization performs better than using prompt optimization alone.
summary: This paper presents an extension of Group Relative Policy Optimization (GRPO) to a setting where multiple module LM programs are invoked, each with their own prompts/contexts. This is challenging for regular GRPO because of varying number of steps, differing control flows etc. The key idea is to generate full program trajectories, align module calls across trajectories and apply GRPO to each module separately. Results are shown on three tasks - intent classification (Banking 77), privacy conscious delegation (PAPILLON), and multi-hop claim verification (HoVer). Two baseline approaches are compared: a chain of thought (CoT) baseline and prompt optimization (MIPROv2). The proposed method  improves upon CoT and sometimes performs better than MIPROv2. When used in combination with prompt optimization using the better together framework, it consistently outperforms both baselines.
weaknesses: * The method does not outperform the prompt optimization baseline on its own.
* It would be good to report a baseline where GRPO can be applied to the multi-module LM setting in some adhoc manner for a specific setting, e.g. by requiring that rollouts have the same number of steps. Even if such a setting is artificial, it would allow us to measure the benefit of the proposed approach which does not impose such requirements.
* Though the approach is general and allows for multiple teachers, it seems like all evaluations are reported with a single teacher.
* L227: The Markovian assumption is a limitation but is not mentioned in the Limitations section. 
* L285: Though the approach allows for different LMs for each module, results are presented only for the scenario where the same LM is used for each module.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - Instead of grouping module calls by their relative invocation order, have the authors considered alternative alignment strategies, such as grouping based on the *semantic similarity* of module inputs, which might create more coherent groups for policy learning?
- Given the challenge of sparse rewards in long-horizon tasks, was any form of *reward shaping* or intermediate reward mechanism explored? 
- How does the framework differentiate between a low-reward trajectory resulting from a single catastrophic failure versus one from an accumulation of minor imperfections?
soundness: 2
strengths: - The paper addresses the timely and relevant problem of optimizing compound AI systems, which is a clear departure from traditional single-call LM fine-tuning.
- The proposed MMGRPO method is an intuitive and elegant extension of the existing GRPO framework, making it straightforward to understand and implement.
- The empirical evaluation is solid, demonstrating the method's effectiveness across three distinct tasks and showing the complementary benefits of combining weight optimization with prompt optimization through the BetterTogether framework.
summary: This paper introduces MMGRPO, a practical extension of Group Relative Policy Optimization (GRPO) designed for post-training multi-module language model programs. The core contribution is a module-level grouping strategy that enables policy gradient updates across structurally diverse trajectories, and the authors demonstrate that combining this method with prompt optimization yields strong performance improvements on a variety of tasks.
weaknesses: - My main concerns lie in the uniform credit assignment strategy. Assigning the final trajectory's reward to every intermediate module call is a considerable *simplification*. In multi-step reasoning or tool-use scenarios, the importance of each module's contribution can vary greatly; an early-stage error might derail the entire process, while a late-stage refinement might only offer marginal improvement. This approach masks the complexity of temporal credit assignment and may not be robust for tasks with longer horizons or more intricate module dependencies, where the signal-to-noise ratio for early modules would be low.
- The method's reliance on a deterministic program structure limits its applicability to more dynamic agentic systems where the control flow itself might be learned or stochastic.
- The experiments are conducted on 8-billion parameter models, and it is unclear how these findings on policy optimization will translate to much larger foundational models.


Decision: Reject