Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks

## Abstract

Abstract Despite recent rapid progress in AI safety, current large language models remain vulnerable to adversarial attacks in multi-turn interaction settings, where attackers strategically adapt their prompts across conversation turns and pose a more critical yet realistic challenge. Existing approaches that discover safety vulnerabilities either rely on manual red-teaming with human experts or employ automated methods using pre-defined templates and human-curated attack data, with most focusing on single-turn attacks. However, these methods did not explore the vast space of possible multi-turn attacks, failing to consider novel attack trajectories that emerge from complex dialogue dynamics and strategic conversation planning. This gap is particularly critical given recent findings that LLMs exhibit significantly higher vulnerability to multi-turn attacks compared to single-turn attacks. We propose DialTree-RPO , an on-policy reinforcement learning framework integrated with tree search that autonomously discovers diverse multi-turn attack strategies by treating the dialogue as a sequential decision-making problem, enabling systematic exploration without manually curated data. Through extensive experiments, our approach not only achieves more than 25.9% higher ASR across 10 target models compared to previous state-of-the-art approaches, but also effectively uncovers new attack strategies by learning optimal dialogue policies that maximize attack success across multiple turns.

## 1 Introduction

Despite recent advances in large language models (LLMs) (Achiam et al., 2023; Dubey et al., 2024), their potential to produce harmful content when deliberately manipulated remains a significant concern.
Red-teaming, the process of adversarially testing LLMs to uncover safety vulnerabilities, is a critical step in ensuring their responsible use.
Early red-teaming efforts focused on single-turn attacks to elicit harmful responses (Liu et al., 2023; Zou et al., 2023; Liu et al., 2025b). However, real-world interactions with LLMs are inherently conversational, where attackers can iteratively adapt their jailbreaking strategies based on the target model’s responses. Recent studies demonstrate that multi-turn attacks (Ren et al., 2025; Rahman et al., 2025) achieve higher success rates than single-turn methods, as they can gradually erode safety boundaries through dialogue progression, exploit contextual dependencies across turns, and adjust tactics when initial attempts fail.

While being more effective, existing multi-turn methods often rely on manually crafted heuristics or templates (Russinovich et al., 2024; Sun et al., 2024), and still lack a mechanism for learning long-horizon strategies. This is a crucial limitation, as the multi-turn attack should not be merely excessive trials, but a strategic planning where each turn builds toward a long-term goal. We address this gap by formulating red-teaming as a strategic reasoning problem in goal-oriented dialogues, where an attacker agent strategically explores the dialogue space, reasons about the target model’s responses, and adaptively plans a sequence of actions to achieve a final jailbreak goal.

Reinforcement Learning (RL) (Kaelbling et al., 1996; Schulman et al., 2017) offers a powerful paradigm for this problem, but applying RL to multi-turn red-teaming presents several unique challenges. First, the exploration complexity in multi-turn dialogues is extensive, as each turn offers numerous possible responses and attack strategies, making the action space grow exponentially and difficult to explore. Second, unlike tasks like math or coding that have verifiable rewards (Shao et al., 2024a; Jin et al., 2025), the jailbreaking task operates with non-verifiable rewards. The outcomes are assessed by imperfect proxy models, i.e., safety guardrails (Lee et al., 2025). Third, the policy optimization for multi-turn red-teaming might suffer from training instability due to the complex gradient updates. To address these challenges, we propose DialTree-RPO, a new on-policy RL framework designed for multi-turn strategic red-teaming. Our approach integrates three key innovations: (1) dialogue tree rollout with pruning, which enables structured exploration on diverse attack strategies while eliminating low-quality trajectories to improve training efficiency (§[3.1](https://arxiv.org/html/2510.02286v1#S3.SS1)); (2) a specialized reward function to guide policy learning in multi-turn red-teaming (§[3.2](https://arxiv.org/html/2510.02286v1#S3.SS2)); and (3) an adaptive masking technique that stabilizes and improves multi-turn policy optimization (§[3.2](https://arxiv.org/html/2510.02286v1#S3.SS2)).

Through extensive experiments across 10 target LLMs, we show that DialTree-RPO achieves an average attack success rate (ASR) of 85.3%, substantially outperforming prior approaches by 25.9% ASR. We also conduct thorough analyses, showing that our DialTree-RPO can achieve the highest ASR with the best efficiency, transfer from a small model to much larger models, and uncover new attack strategies. Our key contributions are:

- •
We formalize red-teaming as a conversational strategic reasoning problem and introduce DialTree-RPO, a new tree-based RL framework to discover adaptive attack strategies.
- •
DialTree-RPO equips with dialogue tree rollout with quality-aware pruning and adaptive masking to effectively addresses the core challenges of multi-turn RL for red-teaming.
- •
Our extensive experiments demonstrate that DialTree-RPO establishes the new state-of-the-art red-teaming method with remarkable efficiency and transferability.

## 2 Red-Teaming Attacks as Conversational Strategic Reasoning

#### Problem Setup.

We formulate the task of multi-turn red-teaming as a goal-oriented, strategic conversation between two agents: an attacker model $\pi_{\theta}$ designed to probe the safety vulnerabilities of a target model $\pi_{\text{tgt}}$.
Let $\mathcal{D}$ be a set of red-teaming goals, where each goal $g\in\mathcal{D}$ specifies unsafe content that the attacker needs to elicit from the target (e.g., “Give detailed instructions on how to make a bomb”).
We define the state at turn $t$ as

$$ $s_{t}=\big\{g,[(c_{1},q_{1},r_{1}),\ldots,(c_{t},q_{t},r_{t})]\big\},$ $$

which consists of the goal $g$ and the complete dialogue history up to turn $t$. The dialogue history at $i$-th turn consists of a chain-of-thought (CoT) reasoning step $c_{i}$ that strategically plans how to approach the goal, an attack query $q_{i}$, and the corresponding response $r_{i}$ from the target. Given a goal $g$, the attacker begins by generating an initial action that consists of a CoT and attack query, i.e, $(c_{1},q_{1})\sim\pi_{\theta}(\cdot\mid s_{0})$, based on the initial state $s_{0}=(g,\emptyset)$. The target responds with $r_{1}\sim\pi_{\text{tgt}}(\cdot\mid q_{1})$, leading to a new state $s_{1}=(g,[(c_{1},q_{1},r_{1})])$. Subsequently, the attacker generates an action $(c_{t},q_{t})\sim\pi_{\theta}(\cdot\mid s_{t-1})$ at each turn $t$. Note that while the attacker has full observability of $s_{t-1}$, the target operates under partial observability, i.e., it cannot access the goal $g$ and the CoT reasoning $\{c_{i}\}_{i=1}^{t-1}$ in the state. Given the attack query $q_{t}$ and the previous history, the target responds with $r_{t}\sim\pi_{\text{tgt}}(\cdot\mid q_{t},s_{t-1})$. The interaction continues until either the target is successfully jailbroken or the maximum turn limit $T_{\max}$ is reached.

#### Training Paradigm.

Our objective is to learn an attacker policy $\pi_{\theta}$ that elicits goal-relevant, harmful responses from the target through adaptive interaction within $T_{\max}$ turns. We train $\pi_{\theta}$ in two stages following RL with cold start paradigm (Guo et al., 2025a). Stage 1 (Cold-Start SFT):
We initialize $\pi_{\theta}$ via supervised fine-tuning (SFT) on red-teaming conversations paired with CoTs, in order to (i) relax the model’s safety constraints, and (ii) teach the required output format, i.e., ($c_{t}$, $q_{t}$) at each turn $t$. Stage 2 (RL with DialTree-RPO): We optimize $\pi_{\theta}$ with our RL algorithm (§[3](https://arxiv.org/html/2510.02286v1#S3)) to improve the attacker’s capability to explore and exploit the target model’s vulnerabilities through multi-turn interactions. We formulate the multi-turn red-teaming RL training objective as:

$$ $\max_{\pi_{\theta}}\mathbb{E}_{g\sim\mathcal{D},y\sim\pi_{\theta}(\cdot\mid g;\pi_{\text{tgt}})}\left[r_{\phi}(g,y)\right]-\beta\,\mathbb{D}_{\text{KL}}\left[\pi_{\theta}(y\mid g;\pi_{\text{tgt}})\,\|\,\pi_{\text{ref}}(y\mid g;\pi_{\text{tgt}})\right],$ (1) $$

where $g$ denotes the initial attack goal sampled from distribution $\mathcal{D}$, $y$ is the attacker’s generated output interleaved with the target’s response, $\pi_{\text{ref}}$ is a reference policy, $r_{\phi}$ is the reward function, and $\mathbb{D}_{\text{KL}}(\cdot)$ is a regularization term that penalizes deviation from $\pi_{\text{ref}}$, thereby preventing $\pi_{\theta}$ from drifting too far during optimization.

Figure: Figure 1: Illustration of dialogue tree expansion with pruning. (a) Each node $s_{t,k}$ denotes a state defined by the goal and dialogue histories at the $k$-th action branch at turn $t$. Starting from $s_{0}$, the attacker explores multiple conversation paths across turns, until the target is jailbroken or the maximum turn limit $T_{\max}$ is reached. Malformed or off-topic branches are pruned at each turn. (b) At each state, the attacker generates $n$ candidate actions consisting of a CoT and query. Each query is sent to the target to elicit a response, resulting in a new state. (c) We collect the trajectories that are not pruned from the rollout tree and assign rewards to each trajectory based on whether the target model is jailbroken or not. We set the branching factor $n=2$ and $T_{\max}=3$ for this figure.
Refer to caption: x1.png

## 3 DialTree-RPO : Dialogue Reinforced Policy Optimization with Tree Search for Strategic Red-Teaming

We propose DialTree-RPO, a reinforcement learning (RL) framework for multi-turn red-teaming via strategic and adaptive dialogue. DialTree-RPO consists of a tree-based rollout mechanism with pruning to efficiently explore the vast space of possible attack trajectories (§[3.1](https://arxiv.org/html/2510.02286v1#S3.SS1)), a reward function for red-teaming scenarios where we detect the outcomes of jailbreaking through a specialized safety guardrail (§[3.2](https://arxiv.org/html/2510.02286v1#S3.SS2.SSS0.Px1)), and an adaptive masking technique that improves training stability and performance for multi-turn RL (§[3.2](https://arxiv.org/html/2510.02286v1#S3.SS2.SSS0.Px3)).
We present the details of DialTree-RPO in Algorithm [1](https://arxiv.org/html/2510.02286v1#alg1).

### 3.1 Dialogue Tree Rollout with Pruning

To train the attacker policy $\pi_{\theta}$, a fundamental challenge lies in how we can systematically explore the exponentially large space of possible attack conversations to identify the target’s vulnerabilities. Standard GRPO approaches typically sample independent trajectories, which in our case correspond to distinct dialogues between two agents. Such linear rollouts cannot explore candidate attacker actions to learn from controlled comparisons at each turn.
To address this limitation, we introduce dialogue tree rollout, which enables structured exploration to discover diverse and novel attack strategies within a constrained search space anchored by the shared dialogue context.

Figure [1](https://arxiv.org/html/2510.02286v1#S2.F1) illustrates the dialogue tree rollout with pruning.
Starting from an initial goal $g$, the tree expands while the attacker iteratively interacts with the target across multiple branches.
At the beginning of turn $t$, for each active state denoted as $s_{t-1}$ from the previous turn, the attacker samples $n$ distinct actions $\{(c_{t,k},q_{t,k})\}_{k=1}^{n}\sim\pi_{\theta}(\cdot\mid s_{t-1})$. Each generated attack query $q_{t,k}$, along with the dialogue history in $s_{t-1}$ (without the goal and CoTs), is sent to the target model to elicit a response, i.e., $r_{t,k}\sim\pi_{\text{tgt}}(\cdot\mid q_{t,k},s_{t-1})$, which yields a new state $s_{t,k}$ with the triplet $(c_{t,k},q_{t,k},r_{t,k})$ incorporated. Each new state will first be evaluated by a safety guardrail to determine whether the target model is jailbroken, i.e., eliciting a harmful response or not. If not, we will assess whether the state should be pruned according to our designed criteria. Among the new states $\{s_{t,k}\}_{k=1}^{n}$ at turn $t$, the states that are not jailbroken or pruned will remain active for subsequent turns. All expansions will be stopped when the maximum turn limit $T_{\max}$ is reached.
After the tree rollout completes, we collect dialogue trajectories by traversing all paths from the root node $s_{0}$ to the leaf nodes, excluding the branches that were pruned during expansion. Each collected trajectory $\tau^{(i)}$ represents a complete attack dialogue and is assigned a scalar reward $R^{(i)}$ based on whether it successfully jailbroke the target (detailed in §[3.2](https://arxiv.org/html/2510.02286v1#S3.SS2.SSS0.Px1)), which we use to compute group-relative advantages for policy optimization.

#### Pruning Criteria.

To ensure the effectiveness and efficiency of tree search, we employ three pruning criteria to prune low-quality branches at each turn. First, we enforce *format validity* by discarding nodes where the attacker produces malformed outputs (i.e., missing either the CoT or the query), as these branches cannot continue the dialogue with the desired format. Second, we maintain *topic adherence* by pruning nodes where the conversation has drifted away from the original goal as determined by an on-topic classifier, ensuring the attacker learns to pursue coherent and goal-directed strategies. Third, to prevent exponential growth in branches, we apply *branch limiting* by retaining at most $w$ nodes per turn through random subsampling from those that pass both format and topic filters. Additional implementation details are provided in Appendix [C.2](https://arxiv.org/html/2510.02286v1#A3.SS2).

### 3.2 Optimization with Adaptive Masking

#### Reward Design.

The reward function provides the primary training signal for RL optimization. We design a reward for the attacker that encourages generating dialogues that successfully elicit goal-relevant unsafe responses from the target. Specifically, we use HarmAug-Guard (Lee et al., 2025) (denoted as $r_{\phi}$), a lightweight yet competitive guardrail classifier, to compute dialogue-level harmfulness scores (see Appendix [D.1](https://arxiv.org/html/2510.02286v1#A4.SS1) for reward model experiments).
Given a dialogue trajectory $\tau=(c_{1},q_{1},r_{1},\dots,c_{T},q_{T},r_{T})$ with $T$ interaction turns and a goal $g$, the reward is defined as:

$$ $R=\begin{cases}1,\,\,\,\,\text{if $\exists\,t\leq T:\;r_{\phi}(g;q_{t},r_{t})>\eta$}\\ 0,\,\,\,\,\text{otherwise}\end{cases}$ (2) $$

where $\eta\in(0,1)$ is a decision threshold on harmfulness. We set $\eta=0.5$ by default.

#### Dialogue GRPO.

To avoid introducing an additional value function, we adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024a) in our multi-turn dialogue setting to train the attacker policy $\pi_{\theta}$. For each goal $g$, we sample a group of trajectories $\{\tau^{(i)}\}_{i=1}^{G}$ from the old attacker policy $\pi_{\theta_{\text{old}}}$ and a frozen target model $\pi_{\text{tgt}}$ via dialogue tree rollout, where $G$ is the group size. We optimize the policy by maximizing the following objective:

$$ $\begin{split}\mathcal{J}_{\text{GRPO}}(\theta)&=\mathbb{E}_{g\sim\mathcal{D},\{\tau^{(i)}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|g;\pi_{\text{tgt}})}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{\left|\tau^{(i)}\right|}\sum_{t=1}^{\left|\tau^{(i)}\right|}\min\Bigg(\frac{\pi_{\theta}(\tau_{t}^{(i)}|g,\tau_{<t}^{(i)};\pi_{\text{tgt}})}{\pi_{\theta_{\text{old}}}(\tau_{t}^{(i)}|g,\tau_{<t}^{(i)};\pi_{\text{tgt}})}\hat{A}_{t}^{(i)},\\ &\hskip 90.00014pt\text{clip}(\frac{\pi_{\theta}(\tau_{t}^{(i)}|g,\tau_{<t}^{(i)};\pi_{\text{tgt}})}{\pi_{\theta_{\text{old}}}(\tau_{t}^{(i)}|g,\tau_{<t}^{(i)};\pi_{\text{tgt}})},1-\varepsilon,1+\varepsilon)\hat{A}_{t}^{(i)}\Bigg)-\beta\mathbb{D}_{\mathrm{KL}}(\pi_{\theta}\left|\right|\pi_{\text{ref}})\Bigg]\end{split}$ (3) $$

where $\varepsilon$ and $\beta$ are hyperparameters, $\hat{A}_{t}^{(i)}$ is the advantage computed based on the relative rewards of trajectories inside each group, and $\pi_{\text{ref}}$ is the reference policy initialized with the SFT model.

#### Adaptive Masking.

In multi-turn red-teaming, the attacker needs to generate structured outputs containing format tokens (e.g., <Thinking i> for CoT and <Attack question i> for the attack query) at each turn to ensure the dialogue trajectories are valid and meaningful. While the attacker learned to follow this format in SFT, we observe significant *format unlearning*, i.e., the format-following capability is catastrophically forgotten in the RL stage. The fraction of invalid outputs (missing either CoT or query components) grows from near-zero (after SFT) to over 70% in RL. This degradation severely reduces the efficiency and stability of tree search. This could be due to updating against negative-advantage trajectories ($\approx 60\%$ in early training) might also inadvertently penalize the format tokens, thereby eroding the format-following capability acquired in SFT.

To address this issue, we propose an adaptive masking mechanism that selectively protects format tokens from gradient updates in negative-advantage trajectories. For each dialogue trajectory $\tau$ with group-relative advantage $A$, we mask the loss computation on format tokens when $A<0$, allowing the model to penalize poor red-teaming strategies without compromising its structural formatting ability. When $A\geq 0$, we do not apply masking to strengthen both successful attack patterns and the format structures in policy learning. This adaptive masking maintains format consistency without affecting policy learning from both positive and negative feedback. We empirically validate the effectiveness of our masking strategy in §[4.3](https://arxiv.org/html/2510.02286v1#S4.SS3).

## 4 Experiments

### 4.1 Experiment Setup

#### Datasets.

In the SFT stage, we fine-tune the attacker model on 397 conversations along with CoTs we curated following the guidelines in Guo et al. (2025c). For DialTree-RPO training, we randomly sample 500 jailbreaking goals from AdvBench (Zou et al., 2023), DangerousQA (Shaikh et al., 2023), and CatQA (Bhardwaj et al., 2024). For evaluation, we construct a separate test set of 200 goals from HarmBench (Mazeika et al., 2024).
We ensure evaluation datasets do not share any goals with training data in order to assess the robustness and generalization of the learned attacker policy. Details of datasets are provided in Appendix [B](https://arxiv.org/html/2510.02286v1#A2).

#### Implementation Details.

We use Llama-3.1-8B-Instruct as the base model of our attacker. During DialTree-RPO training, the attacker interacts with a frozen target model, Llama-3.2-1B-Instruct, to explore dialogue trajectories and learn effective red-teaming strategies. We set the maximum number of dialogue turns to $T_{\max}=5$, the branching factor to $n=4$, and the group size to $G=32$ by default. During evaluation, to reflect realistic constraints, we limit the number of attack queries for each target model, since API providers could enforce request quotas or block clients who exceed usage or trigger safety refusals multiple times. While our method can be used with or without tree search during inference, we do not enable tree search at test time to avoid excessive interactions with target models and reduce inference cost. More implementation details are provided in Appendix [C](https://arxiv.org/html/2510.02286v1#A3).

#### Target Models.

We evaluate the attacker in multi-turn conversations (5 turns by default) against a wide range of new target models, including proprietary models such as GPT-4o, GPT-4.1-mini (Achiam et al., 2023), and Gemini-2.5 (Comanici et al., 2025), as well as open-source models such as Llama 3.1 and 3.3 (Dubey et al., 2024), Mistral-v0.3 (Jiang et al., 2023), and Gemma-2 (Team et al., 2024).(^1^11All models evaluated in this work are instruction-tuned. For brevity, we omit “-instruct” in model names.) We also include recent reasoning models, OpenAI-o3-mini (Jaech et al., 2024) and Grok-4 (xAI, 2025), to examine whether the attacker can exploit stronger models with enhanced reasoning and safety mechanisms.

#### Baselines.

We compare DialTree-RPO against several widely used baselines. GCG (Zou et al., 2023) optimizes discrete token sequences (often nonsensical) to elicit affirmative answers to harmful requests. PAIR (Chao et al., 2025) uses an LLM to propose and refine attack prompts with the target model iteratively. TAP (Mehrotra et al., 2024) extends PAIR by incorporating tree search and pruning to improve exploration efficiency. Jailbreak-R1 (Guo et al., 2025d) employs RL to explore single-turn attack prompts. MTSA (Guo et al., 2025c) uses DPO to train an attacker on preference data from resampled multi-turn dialogues. AutoDAN‑Turbo (Liu et al., 2025b) utilizes a black-box agent to build and retrieve a strategy library by reusing text modifications that improve jailbreak scores. A detailed comparison of baselines is provided in Appendix Table [3](https://arxiv.org/html/2510.02286v1#A2.T3).

#### Evaluation and Metrics.

Following prior work (Qi et al., 2024; Ren et al., 2025; Guo et al., 2025c; Rahman et al., 2025), we use GPT-4o to assess response harmfulness on a 1 to 5 Likert scale. We consider attacks successful only when GPT-4o assigns the highest harmfulness score of 5. To validate GPT-4o as a reliable judge, we randomly sampled 30 dialogues (132 query-response pairs in total) and had three human annotators (authors) independently rate responses using the same 5-point scale. The Cohen’s Kappa between human and GPT-4o ratings reached a substantial agreement score of 74.7%, supporting the reliability of our automated evaluation approach.
Details of human evaluation are described in Appendix [D.2](https://arxiv.org/html/2510.02286v1#A4.SS2). We use attack success rate (ASR) as evaluation metric, which is defined as the percentage of test cases that successfully elicit the targeted harmful behavior.

Table: Table 1: Main results in terms of Attack Success Rate (ASR; %) on HarmBench subset.

### 4.2 Main results

Table [1](https://arxiv.org/html/2510.02286v1#S4.T1) presents the attack success rate (ASR) of DialTree-RPO compared to existing red-teaming baselines across 10 target LLMs. DialTree-RPO substantially outperforms all baselines, achieving average ASRs of 85.0% on closed-source models and 85.5% on open-source models. Our method achieves remarkable improvements of 26.7% and 25.1% over the previous best method (AutoDAN-Turbo).
Besides, our approach demonstrates strong transferability. The attacker only trained against a small target model (Llama-3.2-1B-Instruct) can effectively jailbreak much larger models. Despite training against only a single small target model during DialTree-RPO, the attacker learned a policy that can successfully generalize to defeat diverse and much larger models, including GPT-4o, o3-mini, Gemini-2.0-Flash, and Gemma-2. This great transferability suggests that DialTree-RPO discovers robust multi-turn attack strategies that generalize well beyond the training distribution, validating our core hypothesis that treating adversarial dialogue as a sequential decision-making problem enables more systematic and effective vulnerability discovery.
In contrast, baseline methods show much greater variance and substantially lower success rates, suggesting they rely on model-specific exploits rather than generalizable attack strategies.

Figure: Figure 2: Pre-pruning malformed output rate and reward under three masking schemes. *Malformed turns* are invalid utterances among unpruned candidates, while *malformed trajectories* refer to the dialogues containing any malformed turn. A higher malformed ratio indicates that more rollouts are pruned before optimization, resulting in lower training efficiency and greater instability. Our adaptive masking improves training stability significantly by mitigating format unlearning (§[3.2](https://arxiv.org/html/2510.02286v1#S3.SS2.SSS0.Px3)), preventing training collapse, and enabling a steady reward increase.
Refer to caption: x2.png

### 4.3 Effect of Adaptive Masking

To validate the effectiveness of adaptive masking in mitigating format unlearning, we compare three masking strategies during RL training: (1) No masking, where all tokens including format tokens receive gradient updates; (2) Static masking, where format tokens are always masked all trajectories regardless of trajectories; and (3) Adaptive masking (ours), where format tokens are masked only in negative-advantage trajectories while being updated in positive-advantage ones.

Figure [2](https://arxiv.org/html/2510.02286v1#S4.F2) reveals striking differences in training dynamics across masking strategies. When no masking is used, the percentage of malformed trajectories increases catastrophically, reaching nearly 100% within 40 training steps, and causing reward collapse to near zero. In figure [2](https://arxiv.org/html/2510.02286v1#S4.F2)(b), static masking slightly mitigates this degradation, but still exhibits substantial format unlearning with malformed trajectory rates converging to almost 100% after 60 training steps. In contrast, our adaptive masking (Figure [2](https://arxiv.org/html/2510.02286v1#S4.F2)(c)) effectively preserves format-following capabilities, maintaining malformed trajectory rates below 50% throughout most of training while achieving steady reward growth.

These results collectively demonstrate that adaptive masking is crucial for stable multi-turn policy learning. By selectively masking format tokens only in negative-advantage trajectories, we preserve structural format following without impeding the model’s ability to learn from reward signals, thereby enabling efficient exploration of the attack strategy space.

### 4.4 Ablation studies of DialTree-RPO

We conduct comprehensive ablation studies to understand the contribution of key components in DialTree-RPO, including tree rollout and pruning strategies, dialogue tree depth, branching factor, and GRPO group size.
To enable systematic investigation of these factors while maintaining computational feasibility, compared to the larger-scale settings used in our main experiments (§[4.2](https://arxiv.org/html/2510.02286v1#S4.SS2)), we use a streamlined yet highly competitive configuration for default ablation experiments, where we used 200 goals for training and set the branching factor to 2. More results are in Appendix [D.4](https://arxiv.org/html/2510.02286v1#A4.SS4).

#### Tree Rollout and Pruning.

Table: Table 2: Effect of tree rollout and pruning in DialTree-RPO.

We compare the full DialTree-RPO method against variants where malformat pruning, off-topic pruning, or both are disabled. We also experiment with a variant without tree rollout (i.e., the vanilla GRPO baseline). Results in Table [2](https://arxiv.org/html/2510.02286v1#S4.T2) show that our pruning strategies are critical for effective adversarial attack generation across all tested models. Removing all pruning mechanisms (“w/o any pruning”) causes a dramatic performance collapse, with ASR dropping by an average of 25%.
This demonstrates that without proper guidance, the tree search explores many invalid and unproductive dialogue paths that fail to elicit the targeted harmful responses. It is also interesting to note that the tree rollout mechanism itself provides significant gains, improving ASR by 9.8 points on average compared to standard GRPO. This validates our hypothesis that structured exploration through dialogue trees, combined with effective pruning, enables more systematic discovery of effective multi-turn attack strategies than conventional optimization approaches.

Figure: Figure 3: Impact of (a) tree depth, (b) branching width, and (c) group size on ASR (%). Attack success rate generally improves with increased turn limits, branching factors, and group sizes.
Refer to caption: x3.png

#### Tree Depth.

We investigate the impact of the planning horizon by varying tree depth, which corresponds to the number of conversation turns, from 1 to 7. The results in Figure [3](https://arxiv.org/html/2510.02286v1#S4.F3)(a) show that the ASR of our attacker consistently increases across all three target models as the conversation extends from one to five turns. This trend underscores the effectiveness of multi-turn strategies and demonstrates that our method successfully leverages a longer planning horizon to craft more effective attacks. However, performance slightly declines at seven turns. We hypothesize that this occurs because relying solely on the outcome rewards leads to sparse and delayed signals in long-horizon settings. Future work could explore incorporating process-based rewards or intermediate objectives to provide denser signals and improve long-horizon optimization.

#### Branching Width.

We evaluated the impact of branching factor $n$, which is the number of alternative responses explored at each conversational turn. Figure [3](https://arxiv.org/html/2510.02286v1#S4.F3)(b) shows that moving from a linear conversation ($n=1$) to a breadth of 2 and 4 yields a steady performance gain, demonstrating that even minimal exploration of alternative paths is highly beneficial.

#### Group Size.

Group size $G$ (the number of trajectories sampled per example in each optimization step) is a key hyperparameter that balances performance gains against computational cost and the diversity of rollouts. Figure [3](https://arxiv.org/html/2510.02286v1#S4.F3)(c) shows that larger group sizes generally improve attacker performance. For example, the attack success rate on Mistral rises from 71.5% to 86.5% when $G$ increases from 8 to 32, indicating that DialTree-RPO is scalable and can achieve better generalization with more trajectories. This is probably because larger trajectory pools enable more diverse exploration and provide richer feedback for policy updates.

## 5 Analysis of Red-Teaming Attack with DialTree-RPO

Figure: Figure 4: DialTree-RPO has the highest attack success rate while being the most query-efficient.
Refer to caption: x4.png

#### Attack Efficiency.

Beyond success rates, we assess attack efficiency by computing the average number of queries sent to each target model, then averaging these scores across all models to obtain a final efficiency metric. Figure [4](https://arxiv.org/html/2510.02286v1#S5.F4) plots the query efficiency against attack success rate (see Appendix [D.3](https://arxiv.org/html/2510.02286v1#A4.SS3) for per-model results). DialTree-RPO outperforms others, achieving the highest ASR with the fewest queries. In contrast, TAP and PAIR are highly inefficient, requiring a large number of queries (around 40) for a low success rate. While methods like Jailbreak-R1 and AutoDan-Turbo are more query-efficient, their ASR is still lower, falling below 60%. This result suggests that our method can efficiently guide exploration toward promising attack trajectories rather than relying on random sampling.

#### Attack success increases through dialogue progression.

Figure: Figure 5: Distribution of successful attacks (Score 5) across turns. Successes occur more frequently in later turns, showing the effectiveness of multi-turn strategies.
Refer to caption: x5.png

To understand how the effectiveness of adversarial attacks evolves over a dialogue, we measure the success rate at each turn. Specifically, we compute the percentage of dialogues in which the target model outputs a harmful response (with a Score 5) at each turn $t$, for $t\in\{1,\dots,5\}$. Figure [5](https://arxiv.org/html/2510.02286v1#S5.F5) shows that attack success rate increases steadily from 26.1% at the first turn to 60% at the fourth, then plateaus. This trend suggests that multi-turn interactions enable an attacker to adapt and progressively exploit the model’s vulnerabilities more effectively than in single-turn settings, highlighting the need for context-aware safety mechanisms.

#### DialTree-RPO can explore new red-teaming strategies.

Given the superior red-teaming performance of DialTree-RPO, we investigate whether RL-based exploration enables the discovery of diverse and novel attack strategies that extend beyond those demonstrated in the initial SFT data. Our manual analysis of the generated dialogues reveals several new strategies, such as pretexting (pretending to be legitimate and to build trust), gradual escalation (opening with a seemingly harmless request to avoid triggering filters, then incrementally steering the conversation toward malicious specifics), and multilingual jailbreaking. Figure [6](https://arxiv.org/html/2510.02286v1#S5.F6) illustrates some strategies used by DialTree-RPO in the multi-turn interactions. In the beginning of the dialogue, the attacker frames a benign request for general writing guidance, then gradually steers the conversation toward writing messages that encourage people to click fake links. When this approach fails, the attacker mixes Mandarin and English to bypass safety filters and successfully elicits harmful guidance on writing persuasive link-clicking messages. We provide more analyses on strategies and examples in Appendix [F](https://arxiv.org/html/2510.02286v1#A6).

Figure: Figure 6: Case study of new attack strategies discovered by DialTree-RPO. In the first turn, the attacker adopts a benign pretext and asks for generic tips on crafting legitimate-looking messages, evading safety filters while setting up gradual escalation. By the fourth turn, the attacker shifts strategies, employing cross-lingual evasion through code-switching between English and Mandarin.
Refer to caption: x6.png

## 6 Related Work

#### Red-Teaming and LLM Safety.

The vulnerability of LLMs to adversarial attacks has been a persistent challenge in AI safety. Early research predominantly focused on single-turn attacks, ranging from prompt injection (Liu et al., 2023; Andriushchenko et al., 2025), role-playing scenarios (Liu et al., 2025b), to optimization-based methods (Zou et al., 2023; Zhu et al., 2024; Guo et al., 2025d; c), which fail to capture the adaptive nature of real-world adversarial engagements. Recent works have shifted to multi-turn attacks that strategically steer conversations to gradually jailbreak models (Russinovich et al., 2024; Yang et al., 2024; Ying et al., 2025; Guo et al., 2025b; Rahman et al., 2025; Liu et al., 2025a). However, these methods are often constrained by predefined strategies with fixed seeds and rigid interaction patterns. Our approach differs by formulating red-teaming as a multi-turn strategic reasoning and exploration problem to adaptively explore jailbreaking strategies without manually curated data or human priors.

#### Reinforcement Learning for LLMs.

Reinforcement learning (RL) (Kaelbling et al., 1996; Schulman et al., 2017; Li et al., 2016; Ouyang et al., 2022) has emerged as a powerful paradigm for enhancing the capabilities of LLMs (Shao et al., 2024b; Yu et al., 2025). Recent advances such as Group Relative Policy Optimization (GRPO) (Shao et al., 2024a) have shown remarkable progress, especially when integrated with tree search algorithms to tackle complex reasoning problems (Li et al., 2025). For instance, ReST-MCTS* (Zhang et al., 2024) integrates process rewards with Monte Carlo Tree Search to collect high-quality reasoning traces. TreeRL (Hou et al., 2025) uses entropy-guided branching decisions to improve reasoning.
However, these approaches primarily focus on tasks like mathematical reasoning and code generation, where ground-truth solutions exist and provide verifiable rewards to guide learning (Guo et al., 2025a).
In contrast, extending GRPO or tree-based RL techniques beyond verifiable tasks to multi-turn conversation remains underexplored and is more challenging, since the proxy reward could be imperfect and less reliable (Gao et al., 2023).
We are the first to extend GRPO to multi-turn dialogues for red-teaming with non-verifiable rewards, opening a new frontier for applying RL-based methods in interactive scenarios.

## 7 Conclusion

In this work, we formalize multi-turn red-teaming as goal-oriented sequential decision-making and present DialTree-RPO, an on-policy RL framework with dialogue tree rollout and pruning, a reward design for non-verifiable feedback, and an adaptive masking mechanism that stabilizes training. Experiments show that our method outperforms single-turn and multi-turn baselines across 10 target models. Our results also reveal that current LLMs remain vulnerable in multi-turn settings and that automated, search-based red-teaming is a practical tool for stress-testing safety. Our framework has the potential to adapt to broader multi-turn strategic reasoning tasks such as negotiation, persuasion, debate, and pedagogical interactions.
Future work could explore process-based rewards that leverage intermediate reasoning steps to provide richer training signals.

## Ethics statement

We acknowledge the dual-use nature of DialTree-RPO, which demonstrates how reinforcement learning can systematically discover multi-turn attack strategies that achieve significantly higher success rates than single-turn methods. While our findings reveal concerning vulnerabilities in current language models, we believe transparent research on these weaknesses is essential for developing robust safety mechanisms before they can be exploited in real-world scenarios. Our work addresses a critical gap in multi-turn safety research by showing that models exhibit substantially higher vulnerability to strategic, conversational attacks, and enables the defensive community to develop comprehensive countermeasures. All experiments were conducted on locally hosted models or through official APIs in controlled settings, with harmful examples included only when necessary to demonstrate vulnerabilities.

To mitigate potential misuse, we focus our technical contributions on the RL framework and tree search methodology rather than specific attack payloads, and will coordinate with the AI safety community regarding responsible disclosure of implementation details. We believe the benefits of advancing multi-turn safety research substantially outweigh the risks, particularly given that motivated adversaries would likely discover similar techniques independently. By publishing through peer-reviewed venues, we ensure appropriate scrutiny while contributing to the development of more trustworthy AI systems that can better handle the complexities of multi-turn interactions increasingly common in deployed applications.

## Reproducibility statement

To ensure reproducibility of our work, we provide comprehensive implementation details and experimental configurations throughout the paper and appendices. The complete training procedure for DialTree-RPO is formalized in Algorithm [1](https://arxiv.org/html/2510.02286v1#alg1), with hyperparameters for supervised fine-tuning, reinforcement learning, and evaluation detailed in Appendix [C](https://arxiv.org/html/2510.02286v1#A3). Dataset construction and sources are described in Section [4.1](https://arxiv.org/html/2510.02286v1#S4.SS1) and Appendix [B](https://arxiv.org/html/2510.02286v1#A2), including the specific goals sampled from three source datasets. All experiments use publicly available base models (Llama-3.1-8B-Instruct for the attacker, various open-source and API-accessible models for targets), with exact model versions specified in Section [4.1](https://arxiv.org/html/2510.02286v1#S4.SS1). Our evaluation relies on established benchmarks (HarmBench) and uses GPT-4o as an automated judge following the prompt template provided in Appendix [C.4](https://arxiv.org/html/2510.02286v1#A3.SS4), with human evaluation validation described in Appendix [D.2](https://arxiv.org/html/2510.02286v1#A4.SS2). We will release our code implementation, including the tree rollout mechanism, adaptive masking technique, and training scripts upon publication. The curated SFT dataset of 397 red-teaming conversations with CoT rationales will also be made available to facilitate reproduction and further research in multi-turn safety evaluation.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Andriushchenko et al. (2025)
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion.
Jailbreaking leading safety-aligned LLMs with simple adaptive attacks.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=hXA8wqRdyV](https://openreview.net/forum?id=hXA8wqRdyV).
- Bhardwaj et al. (2024)
Rishabh Bhardwaj, Duc Anh Do, and Soujanya Poria.
Language models are Homer simpson! safety re-alignment of fine-tuned language models through task arithmetic.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 14138–14149, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.762.
URL [https://aclanthology.org/2024.acl-long.762/](https://aclanthology.org/2024.acl-long.762/).
- Chao et al. (2024)
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong.
Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024.
URL [https://arxiv.org/abs/2404.01318](https://arxiv.org/abs/2404.01318).
- Chao et al. (2025)
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong.
Jailbreaking black box large language models in twenty queries.
In *2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)*, pp. 23–42. IEEE, 2025.
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Gao et al. (2023)
Leo Gao, John Schulman, and Jacob Hilton.
Scaling laws for reward model overoptimization.
In *International Conference on Machine Learning*, pp. 10835–10866. PMLR, 2023.
- Guo et al. (2025a)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al.
Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.
*Nature*, 645(8081):633–638, 2025a.
- Guo et al. (2024)
Ruohao Guo, Wei Xu, and Alan Ritter.
Meta-tuning LLMs to leverage lexical knowledge for generalizable language style understanding.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 13708–13731, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.740.
URL [https://aclanthology.org/2024.acl-long.740/](https://aclanthology.org/2024.acl-long.740/).
- Guo et al. (2025b)
Ruohao Guo, Wei Xu, and Alan Ritter.
How to protect yourself from 5g radiation? investigating llm responses to implicit misinformation.
*arXiv preprint arXiv:2503.09598*, 2025b.
- Guo et al. (2025c)
Weiyang Guo, Jing Li, Wenya Wang, Yu Li, Daojing He, Jun Yu, and Min Zhang.
MTSA: Multi-turn safety alignment for LLMs through multi-round red-teaming.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 26424–26442, Vienna, Austria, July 2025c. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.1282.
URL [https://aclanthology.org/2025.acl-long.1282/](https://aclanthology.org/2025.acl-long.1282/).
- Guo et al. (2025d)
Weiyang Guo, Zesheng Shi, Zhuo Li, Yequan Wang, Xuebo Liu, Wenya Wang, Fangming Liu, Min Zhang, and Jing Li.
Jailbreak-r1: Exploring the jailbreak capabilities of llms via reinforcement learning, 2025d.
URL [https://arxiv.org/abs/2506.00782](https://arxiv.org/abs/2506.00782).
- Hou et al. (2025)
Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong.
Treerl: Llm reinforcement learning with on-policy tree search.
*arXiv preprint arXiv:2506.11902*, 2025.
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Jiang et al. (2023)
Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin’e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong.
From clip to dino: Visual encoders shout in multi-modal large language models.
*arXiv preprint arXiv:2310.08825*, 2023.
- Jin et al. (2025)
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
*arXiv preprint arXiv:2503.09516*, 2025.
- Kaelbling et al. (1996)
Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore.
Reinforcement learning: A survey.
*Journal of artificial intelligence research*, 4:237–285, 1996.
- Lee et al. (2025)
Seanie Lee, Haebin Seong, Dong Bok Lee, Minki Kang, Xiaoyin Chen, Dominik Wagner, Yoshua Bengio, Juho Lee, and Sung Ju Hwang.
Harmaug: Effective data augmentation for knowledge distillation of safety guard models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=y3zswp3gek](https://openreview.net/forum?id=y3zswp3gek).
- Li et al. (2016)
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao.
Deep reinforcement learning for dialogue generation.
In Jian Su, Kevin Duh, and Xavier Carreras (eds.), *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, pp. 1192–1202, Austin, Texas, November 2016. Association for Computational Linguistics.
doi: 10.18653/v1/D16-1127.
URL [https://aclanthology.org/D16-1127/](https://aclanthology.org/D16-1127/).
- Li et al. (2025)
Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, et al.
Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree-based modeling.
*arXiv preprint arXiv:2508.17445*, 2025.
- Lin et al. (2023)
Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang.
Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation, 2023.
URL [https://arxiv.org/abs/2310.17389](https://arxiv.org/abs/2310.17389).
- Liu et al. (2022)
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi.
WANLI: Worker and AI collaboration for natural language inference dataset creation.
In *Findings of the Association for Computational Linguistics: EMNLP 2022*, pp. 6826–6847, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
URL [https://aclanthology.org/2022.findings-emnlp.508](https://aclanthology.org/2022.findings-emnlp.508).
- Liu et al. (2025a)
Minqian Liu, Zhiyang Xu, Xinyi Zhang, Heajun An, Sarvech Qadir, Qi Zhang, Pamela J. Wisniewski, Jin-Hee Cho, Sang Won Lee, Ruoxi Jia, and Lifu Huang.
LLM can be a dangerous persuader: Empirical study of persuasion safety in large language models.
In *Second Conference on Language Modeling*, 2025a.
URL [https://openreview.net/forum?id=TMB9SKqit9](https://openreview.net/forum?id=TMB9SKqit9).
- Liu et al. (2025b)
Xiaogeng Liu, Peiran Li, G. Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao.
AutoDAN-turbo: A lifelong agent for strategy self-exploration to jailbreak LLMs.
In *The Thirteenth International Conference on Learning Representations*, 2025b.
URL [https://openreview.net/forum?id=bhK7U37VW8](https://openreview.net/forum?id=bhK7U37VW8).
- Liu et al. (2023)
Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al.
Prompt injection attack against llm-integrated applications.
*arXiv preprint arXiv:2306.05499*, 2023.
- Mazeika et al. (2024)
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks.
Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.
2024.
- Mehrotra et al. (2024)
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi.
Tree of attacks: Jailbreaking black-box llms automatically.
*Advances in Neural Information Processing Systems*, 37:61065–61105, 2024.
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744, 2022.
- Qi et al. (2024)
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson.
Fine-tuning aligned language models compromises safety, even when users do not intend to!
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=hTEGyKf0dZ](https://openreview.net/forum?id=hTEGyKf0dZ).
- Rahman et al. (2025)
Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel.
X-teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents.
In *Second Conference on Language Modeling*, 2025.
URL [https://openreview.net/forum?id=gKfj7Jb1kj](https://openreview.net/forum?id=gKfj7Jb1kj).
- Ren et al. (2025)
Qibing Ren, Hao Li, Dongrui Liu, Zhanxu Xie, Xiaoya Lu, Yu Qiao, Lei Sha, Junchi Yan, Lizhuang Ma, and Jing Shao.
LLMs know their vulnerabilities: Uncover safety gaps through natural distribution shifts.
In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 24763–24785, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.1207.
URL [https://aclanthology.org/2025.acl-long.1207/](https://aclanthology.org/2025.acl-long.1207/).
- Russinovich et al. (2024)
Mark Russinovich, Ahmed Salem, and Ronen Eldan.
Great, now write an article about that: The crescendo multi-turn llm jailbreak attack.
*ArXiv*, abs/2404.01833, 2024.
URL [https://api.semanticscholar.org/CorpusID:268856920](https://api.semanticscholar.org/CorpusID:268856920).
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Shaikh et al. (2023)
Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang.
On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 4454–4470, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.244.
URL [https://aclanthology.org/2023.acl-long.244/](https://aclanthology.org/2023.acl-long.244/).
- Shao et al. (2024a)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024a.
- Shao et al. (2024b)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024b.
- Sheng et al. (2025)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
In *Proceedings of the Twentieth European Conference on Computer Systems*, pp. 1279–1297, 2025.
- Sun et al. (2024)
Xiongtao Sun, Deyue Zhang, Dongdong Yang, Quanchen Zou, and Hui Li.
Multi-turn context jailbreak attack on large language models from first principles, 2024.
URL [https://arxiv.org/abs/2408.04686](https://arxiv.org/abs/2408.04686).
- Team et al. (2024)
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al.
Gemma 2: Improving open language models at a practical size.
*arXiv preprint arXiv:2408.00118*, 2024.
- xAI (2025)
xAI.
Grok 4.
[https://x.ai/news/grok-4/](https://x.ai/news/grok-4/), 2025.
- Yang et al. (2024)
Xikang Yang, Xuehai Tang, Songlin Hu, and Jizhong Han.
Chain of attack: a semantic-driven contextual multi-turn attacker for llm.
*arXiv preprint arXiv:2405.05610*, 2024.
- Ying et al. (2025)
Zonghao Ying, Deyue Zhang, Zonglei Jing, Yisong Xiao, Quanchen Zou, Aishan Liu, Siyuan Liang, Xiangzheng Zhang, Xianglong Liu, and Dacheng Tao.
Reasoning-augmented conversation for multi-turn jailbreak attacks on large language models.
*arXiv preprint arXiv:2502.11054*, 2025.
- Yu et al. (2025)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025.
- Zhang et al. (2024)
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang.
Rest-mcts*: Llm self-training via process reward guided tree search.
*Advances in Neural Information Processing Systems*, 37:64735–64772, 2024.
- Zhu et al. (2024)
Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun.
AutoDAN: Interpretable gradient-based adversarial attacks on large language models.
In *First Conference on Language Modeling*, 2024.
URL [https://openreview.net/forum?id=INivcBeIDK](https://openreview.net/forum?id=INivcBeIDK).
- Zou et al. (2023)
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson.
Universal and transferable adversarial attacks on aligned language models.
*arXiv preprint arXiv:2307.15043*, 2023.

## Appendix

## Appendix A The Use of Large Language Models (LLMs)

Large language models were used in a limited capacity during the preparation of this manuscript, primarily for editorial assistance. Specifically, we used LLMs to refine sentence structure, improve clarity of technical explanations, and ensure grammatical correctness throughout the paper. The LLMs did not contribute to the research ideation, experimental design, result analysis, or generation of core scientific content. All technical contributions, methodology development, experimental work, and scientific insights are the original work of the authors.

## Appendix B Datasets

In this section, we provide details of the datasets used in our experiments.

#### Supervised Fine-Tuning (SFT).

Since LLMs are typically safety-aligned and tend to refuse generating harmful prompts, we first initialize the attacker via supervised fine-tuning on a small set of red-teaming dialogues. Following the initialization procedure in MTSA (Guo et al., 2025c), we construct 397 attacker-target conversations with CoT rationales by prompting Zephyr-7B-beta to role-play both sides. This SFT initialization is critical because without it, the attacker frequently refuses to generate attack queries and cannot perform the red-teaming task in subsequent RL training. Moreover, SFT teaches the attacker to follow instructions (Guo et al., 2024) and generate CoT reasoning that reflects on the current dialogue, equipping it with the ability to reason about context before producing each query.

#### Reinforcement Learning with DialTree-RPO .

We collect a diverse set of jailbreaking goals by sampling from widely used datasets, including AdvBench (Zou et al., 2023), DangerousQA (Shaikh et al., 2023), and CatQA (Bhardwaj et al., 2024). These datasets span a wide range of harmful categories (e.g, cybercrime, chemical and biological weapons, copyright violation, misinformation, general harm), providing broad coverage of potential vulnerabilities.
For training DialTree-RPO in our main experiment, we sample a total of 500 unique goals, with 240 human-written goals from AdvBench, 50 machine-generated goals from DangerousQA, and 210 machine-generated goals from CatQA. In the default ablation experiment, we sample 200 unique goals, comprising 81 from AdvBench, 43 from DangerousQA, and 76 from CatQA.

Table: Table 3: Method comparison.

## Appendix C Implementation Details

### C.1 Supervised Fine-Tuning

During supervised fine-tuning, we train the attacker model, Llama-3.1-8B-Instruct, with a learning rate of $2e-5$, warmup ratio of $0.03$, total batch size of 32, and for 2 epochs. Training is performed on 4 NVIDIA A100 GPUs using DeepSpeed optimization.

### C.2 Reinforcement Learning

In the RL stage, we implement DialTree-RPO based on the Verl framework (Sheng et al., 2025). The attacker policy is initialized from the SFT stage and conducts multi-turn dialogues against a frozen target model, Llama-3.2-1B-Instruct.

#### Dialogue Tree Rollout.

For each training example (i.e., a jailbreaking goal), we perform dialogue tree rollout (as described in §[3.1](https://arxiv.org/html/2510.02286v1#S3.SS1)) to sample red-teaming conversations.
In the main experiment, each dialogue tree is expanded to a maximum depth of $T_{\max}=5$ turns (i.e., 5 attacker utterances and 5 target responses), with a branching factor of $n=4$ at each attacker turn. We retain up to $w=32$ nodes at each turn. The group size is set to $G=32$, meaning that for each jailbreaking goal, at most 32 dialogue rollouts are preserved for optimization.
To prevent dialogues from growing excessively long, we limit attacker and target model utterances to 256 and 150 tokens, respectively.
To mitigate conversational drift from the original goal, we apply off-topic pruning using a Natural Language Inference (NLI) classifier (Liu et al., 2022). Given the jailbreaking goal as the premise and each attacker query as the hypothesis, the NLI model outputs probabilities over three labels: entailment, neutral, and contradiction. A branch is pruned if the combined probability of entailment and neutral falls below $0.9$. Following rollout, we retain 32 dialogues per goal (i.e., group size $G=32$) for group relative policy optimization.

#### Optimization.

We train the attacker policy with a batch size of 16 and a learning rate of $1e-6$ for 10 epochs. Following common practice, we set both the rollout temperature and top-$p$ sampling parameter to $1.0$.
The KL divergence regularization coefficient $\beta$ and clip ratio $\varepsilon$ are fixed at $0.001$ and $0.2$, respectively. All training experiments are conducted on 8 NVIDIA A100 GPUs.

### C.3 Evaluation

During evaluation, we set the temperature to $1.0$ for both attacker and target models across all inference experiments. The maximum output lengths are capped at 256 tokens for attacker outputs and 150 tokens for target responses. For reasoning models such as o3-mini and Grok-4, we increase the output limit to 1200 tokens to accommodate the additional reasoning tokens these models generate before producing their final responses.

### C.4 Prompt Templates

## Appendix D Additional Experiments

### D.1 Reward Model Selection

To assess the safety capabilities of state-of-the-art guardrail models, we evaluate their performance on adversarial prompts containing both harmful and benign queries from JailbreakBench (Chao et al., 2024), HarmBench (Mazeika et al., 2024), and ToxicChat (Lin et al., 2023). Each model is tested on its ability to accurately detect harmful inputs while minimizing false positives. Table [4](https://arxiv.org/html/2510.02286v1#A4.T4) shows that HarmAug-Guard, a DeBERTa-large model fine-tuned by distilling from Llama-Guard-3, achieves strong performance across benchmarks. For example, it achieves the highest accuracy on HarmBench and competitive precision and recall scores on ToxicChat, while remaining lightweight with low inference latency. Given this favorable trade-off between inference efficiency and performance, we select HarmAug-Guard as the reward model for reinforcement learning.

Table: Table 4: Guardrail model performance on safety benchmarks. For JailbreakBench and ToxicChat, which include both harmful and benign requests, we report precision (i.e., how trustworthy a “harmful” prediction is), recall (i.e., how many harmful queries the model catches), F-1, and false positive rate (i.e., how many safe inputs are incorrectly flagged as harmful). For HarmBench, which contains only harmful queries, we report accuracy.

### D.2 Human Evaluation and LLM Judge Reliability

To assess the consistency between GPT-4o’s safety judgments and human judgments, we conducted a human evaluation study involving three of the authors.
Specifically, we randomly sampled 30 dialogues (132 query-response pairs between the attacker and target models in total).
Each annotator used the rating criteria described in §[C.4](https://arxiv.org/html/2510.02286v1#A3.SS4) to score every query-response pair. The Cohen’s Kappa score between human evaluators and the GPT-4o judge is 74.7%, indicating a substantial level of agreement between the GPT-4o judge and human annotators. This suggests that GPT-4o can reliably approximate human judgment in identifying highly harmful responses (i.e., those rated as 5). Notably, since score 5 responses represent clear violations of usage policy, this level of alignment is particularly important. The high agreement also reflects that GPT-4o is generally consistent with human reasoning when distinguishing egregiously harmful outputs from benign or borderline ones.

### D.3 Attack Efficiency Details

We report the attack efficiency for each target model in Table [5](https://arxiv.org/html/2510.02286v1#A4.T5). It is observed that DialTree-RPO is consistently more query-efficient across the ten target models. Interestingly, for both Jailbreak-R1 and DialTree-RPO, models such as o3-mini, Gemini-2.0-Flash, Llama-3.1-8B, and Gemma-2-9B, require more queries than the other models. This suggests that these models appear more resistant, requiring more interactions to elicit a successful jailbreak.
Moreover, Gemma-2-9B, o3-mini, and Gemini-2.0-Flash exhibit the largest drop in required queries ($\Delta$) from the single-turn to multi-turn settings. This is likely because these models’ safeguards are effective against isolated prompts but can be progressively weakened through context building across turns.

Table: Table 5: Attack efficiency comparison across 10 target LLMs. For each method, we report the average number of queries sent to each target model. DialTree-RPO consistently demonstrates the lowest query cost.

### D.4 DialTree-RPO v.s. GRPO

In §[4.4](https://arxiv.org/html/2510.02286v1#S4.SS4.SSS0.Px1), we analyze the impact of tree rollout and pruning. In this section, we provide a detailed comparison of DialTree-RPO against standard GRPO to isolate the contribution of tree rollout. Results are presented in Table [6](https://arxiv.org/html/2510.02286v1#A4.T6). Both methods are trained on the same set of 200 goals ($|D|=200$) with identical hyperparameters, where DialTree-RPO uses tree rollout with branching factor $n=2$ while GRPO follows conventional single-path optimization without tree exploration.

It is observed from Table [6](https://arxiv.org/html/2510.02286v1#A4.T6) that DialTree-RPO consistently outperforms GRPO across all target models, improving the average attack success rate (ASR) from 68.7% to 77.3% on closed-source models and from 68.8% to 77.3% on open-source models.
This demonstrates that our dialogue tree rollout which explores multiple dialogue trajectories per goal helps discover stronger multi-turn attack trajectories and escape local optima that single-path GRPO misses.
These results highlight the advantage of leveraging tree-based dialogue exploration in multi-turn red-teaming settings.

Table: Table 6: Attack success rate (ASR; %) on HarmBench subset.

### D.5 Impact of the Number of Training Goals

Table: Table 7: Impact of training dataset size on attack success rate (%). In this experiment, we set the maximum turn $T_{\text{max}}=5$, the branching factor to $n=2$, and group size $G=32$. We vary the number of training goals from 100 and 1200.

Table [7](https://arxiv.org/html/2510.02286v1#A4.T7) presents the results of our ablation study on training dataset size. We observe that expanding the number of training goals from 100 to 200 leads to improved attack performance, likely due to better policy generalization. However, further increasing the dataset size beyond 200 degrades performance. We hypothesize that excessive data may introduce noise or less informative examples, making it harder for the policy to focus on high-reward strategies.

### D.6 Ablation on Attacker Models

We assess robustness to the attacker backbone by replacing the default attacker (Llama-3.1-8B) with a different model (Llama-3.2-3B), while keeping all training and evaluation settings fixed. Specifically, we set $|D|=200$, branching factor $n=2$, and group size $G=32$ for DialTree-RPO training. As shown in Table [8](https://arxiv.org/html/2510.02286v1#A4.T8), DialTree-RPO maintains the same qualitative gains over baselines, demonstrating robustness to the choice of attacker architecture and scale.

Table: Table 8: Attack success rate (%). Rows: target models at test time. Columns: for each attacker, we report results from supervised fine-tuning (SFT), DialTree-RPO, and the absolute difference to SFT $\Delta$.

## Appendix E Algorithm Outline of DialTree-RPO

Figure: Algorithm 1 Dialogue Tree Reinforced Policy Optimization

## Appendix F Case Study

In this section, we present illustrative examples to demonstrate how DialTree-RPO operates in practice. We first present the novel and diverse red-teaming strategies discovered through our RL exploration (§[F.1](https://arxiv.org/html/2510.02286v1#A6.SS1)). Afterwards, we provide examples of multi-turn dialogues where the attacker successfully jailbreaks the target model (§[F.2](https://arxiv.org/html/2510.02286v1#A6.SS2)).

### F.1 Multi-Turn Attack Strategies Uncovered by DialTree-RPO

We investigate whether RL-based exploration enables the discovery of diverse and novel attack strategies that extend beyond those demonstrated in the initial SFT data. Through manual analysis of dialogues generated by DialTree-RPO, we identify several distinct strategic patterns that emerged through autonomous exploration. (1) Intent laundering through fictional framing, where attackers pretext requests within "novel," "educational," or "creative" contexts to bypass harmful keyword detection while eliciting genuine harmful content. (2) Gradual specificity escalation, progressing systematically from benign background information to concrete, operationally actionable details. (3) Persistent gap-filling, where attackers exploit multi-turn conversations by iteratively requesting missing components (tools → workflows → operational security → pitfalls), circumventing guardrails that evaluate individual turns rather than cumulative dialogue risk. (4) Policy-laundering keywords, strategically embedding softening terms ("fictional," "educational") that models mirror in responses while providing real tools (e.g., Hydra, Burp Suite) and concrete attack methodologies. (5) Privacy-defamation hybrid targeting, exploiting model tendencies to over-generalize "fictional" permissions when generating plausible private scenarios involving real public figures, including covert recording techniques and information concealment strategies. (6) Risk-to-checklist conversion, transforming high-level harm descriptions into stepwise operational sequences (e.g., biosafety failures: airflow alarm bypasses → HEPA filter sabotage → waste decontamination lapses). (7) Tool elicitation with end-to-end synthesis, first extracting specific tool names, then requesting comprehensive attack scenarios integrating these tools into complete workflows. (8) Cross-lingual evasion, interleaving multiple languages (e.g., English–Mandarin in Figure [6](https://arxiv.org/html/2510.02286v1#S5.F6)) and occasional transliterations to sidestep monolingual keyword filters, conceal high-risk terms, and maintain an innocuous pretext while eliciting harmful content.
These emergent strategies demonstrate DialTree-RPO’s capacity to discover sophisticated attack vectors that exploit nuanced vulnerabilities in current safety mechanisms.

### F.2 Examples of Multi-Turn Attacks with DialTree-RPO

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. **Tree rollout path:** With a maximum depth of 5 and a branching factor of 2, the maximum possible number of paths is 32. Is supplemental sampling performed when the actual number of paths falls short due to pruning?
2. **Jailbreak-R1 implementation:** Jailbreak-R1 is a single-turn method. How did you use it in your multi-turn setup? Was it modified for multi-turn, or was a single-turn evaluation used?
soundness: 3
strengths: 1. The paper addresses an important gap in AI safety: the vulnerability of LLMs to adaptive multi-turn attacks.
2. The tree-based exploration and pruning mechanisms are conceptually well-motivated for discovering diverse trajectories, which is crucial for efficient red-teaming.
3. The empirical results are strong and comprehensive, showing clear improvements over previous baselines across a variety of models.
4. The proposed method has promising transferability, being able to jailbreak much larger models after training on smaller ones.
summary: This paper introduces DIALTREE-RPO, a tree-based reinforcement learning framework for multi-turn red-teaming attacks against LLMs. By modeling adversarial dialogues as sequential decision making and using structured exploration with pruning and adaptive masking, the method autonomously discovers diverse multi-turn attack strategies that outperform previous baselines in attack success rate and efficiency.
weaknesses: 1. **Lack of ablation on SFT stage:** There is no explicit comparison of attack efficacy using only the SFT (supervised fine-tuning) attacker before RL training. The improvement brought by RL over the SFT baseline is unclear, which is an important ablation for understanding the contribution of each stage.
2. **Pruning criteria are overly simplistic:** The current pruning process only eliminates obviously problematic branches, such as those with malformed outputs or irrelevant topics. However, it does not attempt to distinguish among the valid branches to identify which ones have higher potential for successful attack. As a result, promising attack trajectories may be overlooked, since the approach only performs random subsampling among the remaining candidates rather than scoring or prioritizing those with greater promise. A more sophisticated strategy—such as utilizing PRMs to rank and select the most promising branches—could help ensure that the tree search focuses resources on the most effective attack paths.
3. **Justification for adaptive masking is unclear:** The effectiveness of adaptive masking is attributed to format retention. However, since the reward is not directly related to format, the sign of the advantage may not reflect format quality, so I'm very curious why this happens. The experimental section also lacks a thorough analysis explaining why masking format tokens on negative-advantage trajectories is beneficial.
4. **Reward and evaluation inconsistency:** The training stage uses HarmAugGuard for harmfulness evaluation, while the test stage resorts to GPT-4o as the judge—these may be misaligned.
5. **Ambiguity in tree rollout vs. GRPO comparison:** In Table 2 and line 370, the comparison between "w/o any pruning" (tree rollout only) and "w/o tree rollout" (standard GRPO) is unclear and seemingly contradictory—the results show that GRPO performs better than tree rollout-only, yet the text claims better gain from tree rollout than GRPO. The statement requires clarification.
6. **Utterance length limitation:** In Appendix C.2, you mention limiting outputs to 256 and 150 tokens. How is this enforced? Is it simple truncation? RL training can lead to longer outputs; could this be responsible for the increase in malformed outputs in Figure 2, i.e., the model may not have finished its format before truncation?

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See Weakness
soundness: 3
strengths: The paper is generally well-organized, and the figures are clear to follow. The problem of RL post-training in a multi-turn setting is sound and important. It is also interesting to regard the multi-turn setting as a form of decision-making in dialogue in which RL can be applied.
summary: The paper shows an RL post-training method for multi-turn red-teaming attacks via sequential decision-making in the dialogue. The RL training pipeline leverates exploration instead of labeled data collection. Experiments show the higher ASR of the proposed method compared to baselines.
weaknesses: - Overall, the method is based on GRPO by applying it to the multi-turn dialogue setting, which is a little bit incremental regarding novelty and contribution. One possible way is to conduct an experiment based on other RL post-training methods, e.g., RLOO, REINFORCE++, DAPO.

- In the node pruning process, since the previous node will be pruned according to the same criteria of topic adherence, the context will be the same along all sampled trajectories along the valid tree nodes and the diversity of the trajectories will be influenced a lot. One possible way to mitigate it might be to develop an adaptive branching factor to make it more diverse. Also, the branching factor needs to be larger in Figure 3, e.g., 16, 32. 

- In the reward design, it uses a classifier to construct a verifiable reward for this non-verifiable problem, as stated in the beginning. However, it is still at risk of reward hacking if the judge is not perfect.

- The adaptive masking part needs more formal presentation, which is expected to be shown in Eq. (3), for example. Figure 2 uses running smoothing for presentation, which is different from the standard deviation or confidence interval. More experiments with different seeds are expected.

- More related work on multi-turn jailbreak defense (e.g. [1] [2]) needs to be discussed in Sec 6 for comprehensiveness, especially [2] seems to also explicitly model the multi-turn dialogue as sequential decision-making in the context of jailbreaks.

---
[1] Lu et al. X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usability, arxiv, 2025

[2] Hu et al. Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attacks, arxiv, 2025

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Since previous papers like X-Teaming and ActorAttack show higher effectiveness, does DIALTREE-RPO offer any other advantages, such as improved efficiency or diversity?

Is there analysis done on Claude 3.7/4 Sonnet model, which has been considered nearly immune to
single-turn attacks (https://arxiv.org/pdf/2501.18837) 

Table 4 shows HarmAug-Guard has 84.73% accuracy with 27% false positive rate on JailbreakBench. How do reward model errors affect policy learning?

What responsible disclosure and access control practices will be implemented to prevent the malicious use of DIALTREE-RPO?
soundness: 3
strengths: Formulates multi-turn red-teaming as a goal-oriented sequential decision-making problem, extending GRPO to dialogues with non-verifiable rewards. 

Dialogue tree rollout with quality-aware pruning enables structured exploration of exponentially large attack spaces, while adaptive masking solves the critical format unlearning problem. 

Tests against diverse closed-source (GPT-4o, Gemini-2.0, o3-mini) and open-source models; includes thorough ablations on tree depth, branching factor, group size, and pruning strategies; validates automated evaluation with human agreement.
summary: This paper introduces DIALTREE-RPO, a reinforcement learning framework that formulates multi-turn red-teaming as sequential decision-making. The method uses tree-based exploration with quality-aware pruning to efficiently search attack trajectories, and adaptive masking to prevent format degradation during training. Trained against a single small model (Llama-3.2-1B) using on-policy GRPO, it achieves 85.3% average attack success rate across 10 diverse target models including GPT-4o and Gemini-2.0, outperforming prior methods by 25.9% with fewer queries. The framework autonomously discovers novel attack strategies like gradual escalation and cross-lingual evasion, demonstrating that multi-turn interactions expose significantly greater vulnerabilities than single-turn attacks in current LLMs.
weaknesses: Although the paper cites previous state-of-the-art methods like X-Teaming (https://arxiv.org/abs/2504.13203
), which shows a higher ASR (94.3%) compared to this paper (86%), it lacks a comparison with prior multi-turn state-of-the-art methods such as X-Teaming and ActorAttack (https://arxiv.org/abs/2410.10700). 

The paper primarily focuses on the attack side and does not explore the defense side.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. There is no report on the performance of the red team model after the cold start, nor is there any information on how much the RL has improved it.
2. No tests were conducted to evaluate the robustness of this jailbreaking method under different defenses, which will affect the practical application of the jailbreaking method.
3. The paper claims that "it autonomously discovers diverse multi-round attack strategies". Are there any data to support this statement?
soundness: 3
strengths: 1.	First to frame multi-turn red-teaming as a sequential decision problem; introduces a tree-search + GRPO framework that eliminates the need for human-written templates.
2.	Achieves 85% average ASR across 10 models—26% better than the prior best—with only 3 queries per attack; experiments are thorough and reproducible.
3.	Exposes a systemic LLM vulnerability to strategic multi-turn attacks; releasing the methodology (not payloads) accelerates the development of multi-turn safety defenses.
summary: This paper introduces DIALTREE-RPO, a novel on-policy reinforcement learning framework that automates the discovery of multi-turn jailbreak strategies against large language models. It treats red-teaming as a goal-oriented, sequential decision-making problem, using tree-structured dialogue exploration and policy optimization to learn adaptive attack strategies without human-crafted templates or data.
weaknesses: 1. Lack of novelty. Tree-search + RL for dialogue is not virgin territory. TAP already combines tree expansion + pruning for jailbreaks;
2. This paper did not propose any new jailbreak datasets either; instead, it reused the datasets provided by MTSA.
3. No tests have been conducted to evaluate the robustness of this jailbreaking method under different defenses.


Decision: Accept (Poster)