Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
Authors: Yusong Wu
Stephen Brade
Teng Ma
Tia-Jane Fowler
Enning Yang
Berker Banar
Aaron Courville
Natasha Jaques
Cheng-Zhi Anna Huang
Venue: ICLR 2026 Poster (2026)
Keywords: Real-time Music Accompaniment
Music Generation
Reinforcement Learning
Adversarial Machine Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Most applications of generative AI involve a sequential interaction in which a person inputs a prompt and waits for a response, and where reaction time and adaptivity are not important factors. In contrast, live jamming is a collaborative interaction that requires real-time coordination and adaptation without access to the other player’s future moves, while preserving diversity to sustain a creative flow. Reinforcement learning post-training enables effective adaptation through on-policy interaction, yet it often reduces output diversity by exploiting coherence-based rewards. This collapse, known as ``reward hacking'', affects many RL post-training pipelines, but is especially harmful in live jamming, where musical creativity relies on dynamic variation and mutual responsiveness. In this paper, we propose a novel adversarial training method on policy-generated trajectories to mitigate reward hacking in RL post-training for melody-to-chord accompaniment. A co-evolving discriminator separates policy trajectories from the data distribution, while the policy maximizes the discriminator output in addition to coherence rewards to prevent collapse to trivial outputs. We evaluate accompaniment quality and output diversity in simulation with both fixed test melodies and learned melody agents, and we conduct a user study with the model deployed in a real-time interactive system with expert musicians. Quantitative evaluation and user feedback demonstrate improved output diversity, harmonic coherence, adaptation speed and user agency. Our results demonstrate a simple yet effective method to mitigate reward hacking in RL post-training of generative sequence models.

Published as a conference paper at ICLR 2026
GENERATIVE ADVERSARIAL POST-TRAINING MITI- GATES REWARD HACKING IN LIVE HUMAN-AI MUSIC INTERACTION
Yusong Wu', Stephen Brade', Aleksandra Teng Ma', Tia-Jane Fowlers, Enning Yang*, Berker Banar", Aaron Courville Natasha Jaques;; CaquessCCheng-ZZi Cheng-Zhi Anna Huang" Milla, Quebee Artificial Intelligence Institute, Université de Montréal "Caadda CIFAR AI Chair
3 "Massachusetts Institute of Technology
*Georgia Institute of Technology
UUniversity of Washington
'MeGill University
7 "Independent Researcher
wu nu.yysongemi la. quebec, njecsswwssinggeeeee....e... www.ington.edd......
huangcza@mitt.o
ABSTRACT
Most applications of generative AI involve a sequential interaction in which a per- inputs prompt and waits for response, and where reaction time and adaptiv- ity are not important factors. In contrast, live jamming is a collaborative interac- tion that requires real-time coordination and adaptation without access to the other player's future moves, while preserving diversity to sustain a creative flow. Re- inforcement learning post-training enables effective adaptation through on-policy interaction, yet it often reduces output diversity by exploiting coherence-based re- wards. This collapse, known "reward hacking", affects many RL post-training pipelines, but is especially harmful in live jamming, where musical creativity relies on dynamic variation and mutual responsiveness. In this paper, we pro- pose novel adversarial training method on policy-generated trajectories to mit- igate reward hacking in RL post-training for melody-to-chord accompaniment, A co-evolving discriminator separates policy trajectories from the data distribu- tion, while the policy maximizes the discriminator output in addition to coherence rewards to prevent collapse to trivial outputs. We evaluate accompaniment qual- ity and output diversity in simulation with both fixed test melodies and learned melody agents, and conduct user study with the model deployed in real- time interactive system with expert musicians. Quantitative evaluation and user feedback demonstrate improved output diversity, harmonic coherence, adaptation speed and user agency. Our results demonstrate simple yet effective method to mitigate reward hacking in RL post-training of generative sequence models.'
INTRODUCTION
The combination of large-scale transformer-based models and reinforcement learning (RL) post- training has revolutionized AI, with over billion people now using large language models (LLMs) trained with these techniques (OpenAI, 2025; Perez, 2025). However, most applications of gener- ative Al still involve slow back-and-forth interaction, where the user inputs request, and then Somts-ss several minutes-for a response. Further, RI. post-training techniques are lim- ited by their vulnerability to "reward hacking", where the policy exploits the reward produce either unintelligible outputs or trivial, low-diversity content that scores well yet fails to engage users (Skalse et al., 2022; Lewis al., 2017).
"Equal contribution as senior authors.
CCode: https: / ttt.tht...tut////////////......t ccm / www.wws/ww/ae...oooo.aa real chords chords-pytordh Audio examples: https:/ real chorda- thub. i

---

Published as a conference paper at ICLR 2026
RL Post-training
Generative Adversrial Post-Training (GAPT)
A
Fm
R(z, v)
Live Interaction
Discriminator De(p)
R(2,y) Buds
Ryte log (1 (y()))
CCFF
Figure I: Left: RL post-training enables real-time adaptation for melody-to-chord accompaniment but vulnerable to reward hacking: the policy exploits the coherence reward R(x, y) by repeating simple, high scoring chords, which reduces diversity and breaks creative flow. Right: We propose an adversarial reward signal to prevent reward hacking. A discriminator (vy trained to distinguish policy rollouts from data, with its realism estimation added to the reward. This regularizes the policy toward natural accompaniment while preserving input coherence, preventing diversity collapse.
Live music jamming is collaborative interaction requiring real-time coordination and adaptation without seeing the partner's 's future moves, and demanding creative expression supported by diversity. Imagine stepping onto stage jam with musician you have never met. Within the first few beats, you establish alignment, plan your next move without knowing the future moves of your partner, anticipating the partner to maintain coherent harmony, and recover quickly from errors (Keller, 2008), all while keeping progressions varied to support creative flow (Wrigley & Emmerson, 2013) and emotional connection (Trost et al., 2024).
Training generative music models that can jam live with humans is rare in existing works and de- manding from both engineering and algorithmic perspectives, as such systems must run at low la- lency, recover from errors, and adapt on the fly without access to future input. Supervised maximum likelihood training is straightforward, yet such models often fail at deployment because curated cor- pora are well composed and rarely contain mistakes, corrective maneuvers, or co-adaptive behavior (Wu et al., 2024; Jiang et al., 2020). Reinforcement Learning (RL) post-training offers a promising alternative by simulating interactive sessions and optimizing rewards on accompaniment coherence (Wu et al., 2024; Jiang et al., 2020). However, optimizing a learned reward can induce behavior similar to RL post-training of dialogue models, where the policy maximizes reward parameterized by another learned model (Skalse et al., 2022). This leaves it vulnerable to reward hacking, where discovers outputs that adversarially trick the reward model into assigning spuriously high scores for bad inputs. In dialogue, this can sometimes look like manipulating the reward model into thinking it has satisfied user's preferences when it has not. In music, this appears as non-varying repeti- tive accompaniment that is highly harmonic, but simple (Fig. 2); it diminishes user experience by reducing perceived control and agency in creative jamming (Fig 3). This type of "reward hacking" behavior is an inherent limitation of RL post-training, and can be frequently observed in dialogue applications as well (Wan al., 2025).
In this paper we propose Generative Adversarial Post- dest-Trainig (GAPT, Fig. 1), an adversarial augmentation of RL post-training for real-time live music jamming inspired by Generative Adver- sarial Networks (Goodfellow et al., 2014; Ho & Ermon, 2016). Specifically, we target melody-to chord accompaniment, where a policy generates chords online response to a live melody stream without access to the partner's future moves. Alongside the coherence-based task reward, we train a discriminator to distinguish policy trajectories from data. As in GANs, the discriminator is up- dated online the policy improves to get progressively better at making fine distinctions between the generated samples and the real data (Goodfellow et al., 2014). Unlike GANs, gradients cannot be backpropagated through sequence sampling, we use RL optimize the output signal of the discriminator, in the spirit of Generative Adversarial Imitation Learning (Ho & Ermon, 2016).
However, adversarial training is effective only if the policy can effectively increase the discrimina- tor's "realism" estimation. This is difficult because discriminator updates make the induced reward nonstationary, and an overpowered discriminator impedes the policy from optimizing the reward sig- nal (Arjovsky et al., 2017). We therefore introduce two-phase training schedule that yields stable, casy-to-optimize reward. Phase one warms up the discriminator with fixed-int intervall updates, while
2

---

Published as a conference paper at ICLR 2026
Melody Input
C5
GAPT w/o Adv.
GAPT (Ours)
Gbm
24
Beats
Figure 2: Under the same melody input stream (first row) in live accompaniment setting, the model trained without adversarial reward (second row) produces harmonically coherent yet unnatural pro- gressions with repetitive, trivial, and low-coverage chord choices that hinder human-Al interaction. In contrast, GAPT (third row) produces coherent, natural, and diverse live chord accompaniment by jointly training the policy with discriminator that supplies an adversarial reward.
phase two applies adaptive updates: we update the discriminator only when the moving average of discriminator reward over recent policy steps exceeds threshold, and keep it frozen otherwise.
Together, the discriminator and the coherence reward provide complementary constraints that miti- gate reward hacking: repetitive, low-variation progressions that exploit the coherence reward can be easily identified, resulting in low realism reward, whereas outputs that chase realism but ignore the live melody receive poor coherence reward. The two-phase adaptive discriminator update balances learning speeds, reduces oscillation, and supports steady growth in reward. In this way, GAPT T plays similar role to the KI. constraint commonly used in RI post-training (Jaques et al., 2017). However, we find that the KI. constraint is insufficient to mitigate reward hacking in our setting. Adversarial training necessary to preserve realism while learning to adapt and optimize for accompaniment coherence across diverse inputs, unlocking the benefits of RL post-training for real-time applications demanding robust, adaptive responses to many users.
To evaluate the effectiveness of GAPT, we simulate live interaction with two sources of input: fixed test set melodies and a learned counterpart agent. We then deploy the trained policy in a real-time interactive system and conduct user study with 12 expert musicians. We evaluate accompaniment quality and diversity on the generated music across all three settings, and in the user study we additionally collect quantitative participant ratings. Across the extensive evaluations, method improves adaptability, preserves diversity, and increases perceived user agency compared with pure RL post-training baselines.
The contributions and findings of our research are summarized as follows:
We propose Generative Adversarial Post-Training (GAPT), method that mitigates reward hacking when fine-tuning Transformer-base generative sequence models. It uses a two phase joint optimization of the policy and discriminator, where the discriminator is first warmed up, then updated adaptively only when the policy meaningfully increases the realism reward.
We apply GAPT to the challenging setting of real-time musical accompaniment, which demands both harmonic coherence and progression diversity without access to the partner's future moves.
We evaluate the model in simulation and deploy it in a real-time system that interacts live with expert musicians. The adversarially augmented model improves diversity, harmonic coherence, adaptation speed, and perceived agency over baselines.
We release training datasets, model checkpoints, and code for an RL. training infrastructure for Transformer- music agents and for the real-time interactive system.
RELATED WORKS
Reward Hacking in RL Post-training RL post-training is widely used to align pretrained gener- ative sequence models with human preferences (Ouyang et al., 2022). However, research has found that maximizing a learned reward can induce models to exploit weaknesses in the reward signal (Gao et al., 2023), resulting in trivial, repetitive outputs that obtain high scores but drift from natural syntax and semantics, or do not correspond to desired behavior (Skalse et al., 2022; Lewis et al., 2017). This phenomenon often referred "reward hacking" (Skalse al., 2022), also called
3

---

Published as a conference paper at ICLR 2026
"alignment tax" (Askell et al., 2021) "language drift" (Lee et al., 2019). A common mitigation constrains deviation from the pretrained model by adding a Bullback-Leibble (KL) penalty to the training objective (Jaques et al., 2017; Ouyang al., 2022; Lee et al., 2023). Recent work finds that the KL constraint can be insufficient and proposes additional approaches such as elastic reset (Noukhovitch et al., 2023), reward shaping (Fu et al., 2025; Yan et al., 2024), personalized rewards (Wan et al., 2025). Orthogonal to these policy-side mitigation, Bukharin et al. (2025) introduce ad- versarial training in the reward model mitigate reward hacking. In this work, we propose simple yet effective mitigation by training discriminator that provides adversarial rewards together with policy regularizing. The discriminator acts as regularizer, effectively constraining the policy from collapsing to trivial outputs.
Real-time Music Accompaniment Systems Early real-time accompaniment systems relied on non-neural methods, divided into score following and rule-based or corpus-based generation. Score- following approaches align a performance to a fixed score and play pre-defined accompaniment accordingly (Dannenberg, 1984; Raphael, 2010; Cont, 2008). Generative and co-creative systems synthesize accompaniment using rules or by recombining material learned from a corpus predefined or constructed on the fly (Lewis, 2003; Assayag et al., 2006; Nika & Chemillier, 2012; Nika et al., 2017). Recent deep learning systems model musical context and interaction directly. Early systems primarily used supervised maximum likelihood, for example LSTM--aaed BachlDut (Benetatos et al., 2020) and the Transformer-CRS pipeline in SongDriver (Wang et al., 2022). Recently, sys- tems adopt RL post-training to improve adaptability by optimizing on interactive trajectories. RL- Duet (Jiang et al., 2020) pioneered this approach, employing RL fine-tuning to adaptively generate music. ReaLchords (Wu al., 2024) introduced transformer-basee -ased models fine-tuned via RL with knowledge distillation and self-superrvsed coherence reward, and ReaLJam (Scarlatos et al., 2025) constructs a real-time human-A AI jamming interface based on ReaLchords model. Building upon Re- aLchords, we diagnose diversity collapse under coherence-only RL as reward hacking and propose a novel Generative Adversarial Post- -rainirin that mitigates reward hacking while preserving coor- dination. Moreover, we extend evaluation beyond model-data simulation to include model-model interaction and a human study with musicians using a real-time interactive system.
Generative Adversarial Learning Generative adversarial learning trains generator and dis- criminator in two-player game, where the generator aims to produce samples that the discriminator cannot distinguish from data (Goodfellow al., 2014). For more than half a decade from 201 4-220,,,, this paradigm was successfully applied to improve images (Karras et al., 2020), audio (Kumar et al., 2019), and text (Yu al., 2017). In RL, similar generative adversarial objectives were widely used for off-policy imitation learning, where discrininator-induued reward aligns the policy's 's occu- pancy measure with expert demonstrations, as in Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) and Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al., 2017). However, for the most part these approaches have fallen out of favor in the modern post-LLM era, with recent works use adversarial training merely for sequence generation et al., 2024; Wang et al., 2024; Yu et al., 2023). Peng et al. (2021) trains a discriminator along with robotic control policy to increase the naturalness of the motion, but their application is limited to robotics with simple policy model. Our work shows that they still hold value for mitigating reward hacking, and demonstrates their effectiveness for challenging real world live interaction task.
specific task (Zhang
3 METHODS
3.1 BACKGROUND
We study collaborative music co-creation where two agents (either a trained model or a human player) act concurrently to produce joint sequence (Z1,91)..... (r). At each discrete step t, both agents observe the shared history &eet Net and simultaneously emit the next melody token I, and chord token yı- We define the simultaneous generation process conditionally independent given the shared history:
Pr(r,, Ve
Tet. Wer) = Pr(xx I<e, Mex) Pr(ys (et))e)))
(1)
In the general setting, the melody x and chords y co-involve via their shared history (I Vex), corresponding to musicians adapting to each other they play. As a first step, we focus the
4

---

Published as a conference paper at ICLR 2026
accompaniment setting where the model is trained with melody p(2;, z<()) fixed. The accompani- ment setting assumes the melody is taking the lead and the chord is following the melody. We also assume "cold-start" coordination where the two agents do not have prior knowledge of each other and do not have shared context at the beginning of the accompaniment. We train a chord-generation policy wo for real-time accompaniment with the following online dependency:
To x) II To(Ue Ve)) t=1
(2)
The factorization in Eq. 2 enforces the online constraint, since TT(ye | Tict, [=et,,ee)) does not depend on 2yz or any future tokens, which enables the model to be deployed live, to jam in real-time with a person. We first pretrain To by maximun-likelihoe estimation (MLE) on paired melody and chord sequences from a dataset D of (z, y) pairs:
Σ log Adve mee, mett
max B
Purely supervised online models trained on DD often fail at deployment due to exposure bias (Wu al., 2024). Curated datasets of composed scores rarely include mistakes, corrective maneuvers, or co-adaptive behavior, so the model does not practice recovery or adaptation. During inference, the policy conditions on its own past outputs, which leads to error accumulation and out-of-distributioo states if it never practices recovery after a misprediction or adapts to changes in the input distribution.
3.2 REINFOR LEARNING POST-TRAINING
RL ppos-ttraiiig equips the melody-to toochord accompaniment policy with two online skills that MLE alone lacks: anticipating upcoming inputs and adapting distributional changes during interaction. In particular, by sampling from its own policy to produce rollouts and updating on their successes and failures, RL reduces the mismatch between training and deployment and is less vulnerable to encountering out-of-distribution states when generating from its own outputs. To implement RL post-training for music generation, we sample batch of melodies x from the dataset and roll out the policy online according to Eq. 2. producing chord trajectory Unlike pairs from the curated dataset, on on-policy trajectories (x. y) naturally include adaptation, recovery, and anitcinniii events that arise during learning. Afterwards, we compute scalar episode reward R(z, y) using an ensemble that combines (i) self-supervised coherence rewards ($3.4), (ii) rule-based penalties ($3.4), and (iii) an adversarial reward signal that scores how data like the generated chord trajectory is ($3.3). The policy is then updated by optimizing a KL-regularized objective:
[(xx,)= 3 DKL(TT()
|x) (-(1)=)) +7 Σ H( ( [ (=cD)))))) (4)
max 0
DND. goota (-Ix)
where Σt= H(TTl x<te Uct)) is an entropy regularization loss to promote output diversity; B and 7 are coefficients balancing the KI. objective and the entropy loss. We use Proximal Policy Optimization (PPO) (Schulman et al., 2017) for the reward maximization objective. We initialize both the policy and the value model in PPO from the MLE pretrained checkpoint. Following prior work (Wu et al., 2024), the KL anchor &w is a trained offline model that conditions on the full input:
T
( (2) = om x. x.
(5)
1=1
One common practice in RL post-training of LLMS is to use the initialization of policy anchor to prevent reward hacking. However, previous work (Wu et al., 2024) shows that RI. finetuning with MLE KL anchor fails to train good model in the online accompaniment setting.
3.3 GENERATIVE ADVERSARIAL POST-TRAINING
To counter reward hacking, we introduce a discriminator (-).) implemented as a Transformer en- coder that maps a policy-generated trajectory y realism estimation (y). During on-policy RI.
5

---

Published as a conference paper at ICLR 2026
post-training, D& co-evolves with the policy via binary classification: sequences from the dataset are labeled positive, and sequences 9 generated by the current policy while interacting with the input are labeled negative. In the interactive setting, train Dg. only on the outputs of the model not the full interaction trajectory so that it captures an input-agnostic prior that transfers to unseen inputs.
We incorporate the discriminator signal into the policy objective by rewarding sequences that deems data-like. Following (Ho & Ermon, 2016), we define an adversarial reward Radv = - -log(1 (v)), where (y( [o, 1] is the discriminator's estimate that the policy-generated chord trajectory comes from the data distribution. The task reward and the adversarial reward create complementary pressures: sequences that achieve high task reward by exploiting unrealis- tic shortcuts result in low realism estimation and are penalized through Radv, while sequences that appear data-like yet fail to optimize the task objective receive low task reward. This combination steers the policy toward diverse, well-formed and realistic outputs that follows data distribution by adaptively penalizing reward-hace -aarrd- behavior during training.
A practical challenge is that updating Dy jointly with the policy introduces constraints that hin- der learning stability. We know from the rich history of work on GANs that if Dy advances too quickly, the policy receives vanishing or uninformative gradients, which impedes reward maximiza- tion (Arjovsky et al., 2017). Because Dy is updated throughout training, the reward signal is also nonstationary, which destabilizes optimization. We therefore adopt two-phase update schedule with adaptive discriminator update. Phase 1 performs a short warmup with a fixed update ratio to roughly match learning speeds: one D& update after every five PPO policy updates for the first 200 steps. Phase 2 switches to adaptive, confidence-gated updates that address both constraints. Let Rady be the moving average of the adversarial reward over the most recent three PPO updates. We enable a discriminator step when Radv > T (we use T = 1.0), otherwise we keep Dy: frozen. Gating holds Dw static when its signal would be unstable or overpowering, then advances De only once the policy has caught up enough for the reward signal to be informative. To reduce overfitting in the diseriminator, we apply label smoothing with a = 0.1 to the binary cross-entropy targets (Szegedy et al., 2016). See Alg. for pseudocode of the training process.
3.4 REWARD MODELS AND REGULARIZATION
Self-supervised coherence rewards Following and extending Real.chords (Wu et al., 2024), we construct R(x,, y) from an ensemble of self-supervised rewards and compute a single per-episode score for each rollout. We do not use multi-scale rewards as in ReaLchords. All reward models are trained and evaluated at the full sequence length, equal to the maximum context of the accom- paniment model. A contrastive model encodes melody and chord into embeddings and is trained with an InfoNCE objective (Oord al., 2018; Radford et al., 2021) to align true pairs within a batch. At evaluation time, the cosine similarity between the melody and chord embeddings provides a global harmonic-alion harmonic-alignment signal. A discriminative model takes the full pair (æ, y) and outputs the probability that the pair is real rather than a randomly re-paired negative, which provides com- plementary temporal-coherenc signal. To mitigate bias introduced by pitch-shift augmentation, we include rhythm-only variants of both models that strip pitch and retain onset, hold, and silence to- kens. For each reward type and input variant, we train two seeds and ensemble their normalized scores, then average across all components obtain the per-episode reward R R(x,)) v).
Rule-based penalties We include four penalties during RL to regularize training: (i) an invalid output penalty for format violations, (ii) silence penalty applied when more than 4% of frames are silent while the melody is active, with grace period covering the first 8 frames of each sequence, (iii) an early stop penalty when an end of sequence token is emitted before the melody ends, and (iv) a repetition penalty when the same chord repeats for more than four consecutive times.
4 EXPERIMENTS
Overview We evaluate whether the GAPT improves adaptation to live input while preserving di- versity. We consider three settings that progressively increase interactivity: (i) fixed melody sim- ulation, where the accompaniment policy responds online to held-out melodies; (ii) model-model interaction, where a learned melody jamming agent co-adapts with the chord policy, which better approximates playing with human partner who adapts online to the accompaniment and probes

---

Published as a conference paper at ICLR 2026
Figure 3: Participant ratings for real-time jamming with each model. Error bars show standard error. GAPT has the high- on all three evaluation ques- tions and significantly improves adap- tation speed and perceived control and agency over ReaLchords (p < 0.05). The improved user experience benefits from higher diversity under generative adversarial post-training.
Online Real chords GAPT (Ours)
Preference
Adactation Qualiyy
Adaptation Spood
Control and Agenoy
mutual adaptation; and (iii) a real-time user study with expert musicians jamming with the model deployed an interactive system. We report adaptation quality, output diversity, and user ratings. We report model architecture and training details in $C.
4.1 DATASET AND DATA REPRESENTATION
We train our models on three datasets: Hooktheory (Donahue et al., 2022), Nottingham (Allwright al., 2003), and POP909 (Wang* et al., 2020). All datasets comprise monophonic melodies and chords from pop folk songs. We apply data augmentation by randomly transposing each piece by semitones with & E {-6...., Wikifonia dataset (Simonetta al., 2018), which is excluded from training. The details of dataset are included in $B.2
,,,...,,;,, 6). For evaluation, we additionally evaluate the policy on the
We follow the frame-based representation used in ReaLchords Wu et al. (2024). For the melody (z1....,xy) } and the chord sequence .91.....rr'" each time step t corresponds to a time quan- tized to sixteenth note frame and carries pair (æt, 3t) of discrete tokens. Melody tokens use the vocabulary of ON {a} and HOLD (p), where p is the MIDI pitch active at frame f. We emit ON (p} on the onset frame of note with pitch p, and HOLD (p) on subsequent frames until that note ends. Chord tokens use the similar vocabulary of ON_(c} and HOLD_ [(c), where e is a chord symbol. We maximum sequence length 7' 256 for both and y. To respect the online dependency in Eq. 2, the policy is trained on an interleaved stream {y1, I1, 1/, 22 BT, ITJ.
4.2 SYSTEMS COMPARED
Online MIE. The accompaniment policy trained only with MLE via Eq. (supervised learning). ReaLchords. A reproduction of Wu et al. (2024) trained with the ensemble of coherence rewards and penalties (33.4, no entropy term (7 = 0), and trained only on the Hooktheory dataset. GAPT, Our method: on-policy PPO with the ensemble of coherence rewards and penalties ($3.4) combined with adversarial reward ($3.3) and the entropy term. GAPT w/o Adv. Training. An ablation that removes the adversarial reward and keeps all other components identical. In the study compare Online MLE, ReaLchords, and GAPT.
4.3 EVALUATION SETTINGS
Fixed melody simulation We stream each held-out melody x and roll out the accompaniment policy online according to Eq. 2 to obtain y. This setting isolates online adaptation to real melodies without a co-adapting partner. We evaluate both on combined test set of three datasets used to train the model, as well as out-of-diitributtin dataset Wikifonia that is never seen during training.
Model-model interaction To study co-adaptation, we train a melody jamming agent that gener- ates melody given shared history: < y) = IT. <<; Iee, Vee). We train this melody jamming agent on all three datasets used for training the chord policy, as well as the unseen Wikifor- nia dataset. This is because we would like to test the generalization of the chord agent to partner that has novel musical styles may not have encountered. Our goal is to create a simulated adaptive agent with different experience than the chord agent to better estimate how the chord agent would perform with real human musicians. We train the melody jamming agent using the same RL objec- tive and reward formulation the chord policy, well as the adversarial reward. We also train

---

Published as a conference paper at ICLR 2026
CAPT
Rexuchorcs GAPT
chonte
Homong
0.44
0.60
642
0.12
5.0
0.39
Onlino M.E.
Ontine MLE
0.00
0.30
22
25
30
Diseriily
Divenily
(a) Test Set
(b) Held-out Dataset
(c) I-SNE of Test Set Generation
Figure 4: GAPT advances the Pareto frontier for diversity versus harmony. In simulated interaction on the test set (a) and on an out-of- of-distribution dataset (b), GAPT attains higher diversity while pre- serving strong harmony. By contrast, Online MLE without RL produces diverse outputs but fails harmonic coherence during interactive generation. ReaLchords and GAPT without adversarial train- ing achieves strong harmony at cost of diversity. The t-SNE visualization of test set generations (c) likewise shows that GAPT covers broader region of the accompaniment space.
offline melody model (x y) that conditions on full chord context. During evaluation, the two online policies act simultaneously according to Eq. 1, conditioning on the shared history.
Real-time Interactive System and User Study We deploy client-server system adapted from .Jam (Scarlatos et al., 2025) that generates music in chunks with fixed lookahead which main- tains buffer of planned outputs to handle network latency (see details in $B.4). We recruit 12 experienced musicians, most with more than 10 years of instrument practice and some with im- provisation experience. Each session is in person. After a short demo and a one minute famil- iarization, participants interact with three anonymized systems in counterbalanced order. For each system, they complete three tasks in increasing adaptability: (1) play a fixed melody, (2) improvise in one key and modulate second key midway, and (3) co-improvise while attending upcoming chords. Each task lasts 1-2 minutes. After each system, participants answer three 5-point Likert- scale items: Adaptation quality (the harmony matched my melody), Adaptation speed (the model adapted quickly to changes), and Control and Agency (I felt that I had control and agency during the session). We report per- per-question average score across different systems. We also conduct a brief free-form interview at the end.
4.4 EVALUATION METRICS
Adaptation quality: note-in-chord ratio We measure adaptation with the note-in-chord ratio, computed as the proportion of frames where the melody pitch class belongs to the concurrent chord. For example, if the melody token at time é has pitch C and the chord token at that time is C major, that frame contributes to the ratio. We evaluate only frames where melody note active, then average this indicator across frames and examples to report the overall ratio.
Diversity: Vendi Score We assess chord diversity with the Vendi Score (Friedman & Dieng, 2023;
Pasarkar Dieng, 2024). First, we embed each chord sequence ((()) into a vector 2; using the chord encoder from the contrastive reward model ($3.4), Next, we compute pairwise similarities between embeddings via cosine similarity to form an N by N Gram matrix, where N is the total number of chord sequences. We normalize this matrix and compute the Vendi Score the Shannon entropy of the eigenvalues the normalized matrix. The Vendi Score reflects the effective number of distinct patterns in the set, with higher values indicating greater diversity.
We note that neither metric suffices in isolation. Maximizing harmony at the expense of diversity yields repetitive accompaniment, whereas maximizing diversity without harmony yields disorder. Therefore an ideal music generation model should pushes the Pareto frontier of both metrics, main- taining high harmony and diversity when encountering novel melodies or jamming partners.
8

---

Published as a conference paper at ICLR 2026
DAPy Oee
Figure Harmony and diversity evaluated with learned melody jamming agent (a) and in live user sessions (b). CAPT preserves har- monic coherence while restoring progression diversity compared to ReaLchords, yielding a better har- mony and diversity tradeoff and higher perceived control and adap- tation speed.
1.4
Rea colochoss
Coine 10
Diventty
(a) Melody Jamming Agent
(b) User Interaction Trajectory
5 RESULTS
Fixed melody simulation Figure 4a and Figure 4b report online accompaniment on the test set and a held-out dataset, with detailed numbers in Table 1. The pattern is consistent across both settings. Online MLE yields high diversity but low harmony because it lacks RI. post-training. Real.chords and the ablation without the adversarial reward achieve strong harmony but at the cost of reduced diversity. GAPT excels on both diversity and accompaniment harmony, indicating that the adversarial training specifically recovers diversity without sacrificing coherence. To visualize coverage, we embed test set generations with the reward model's chord encoder and plot t-SNE projections (Figure 4c); GAPT spans broader region than the non-adversarial ablation, indicating more varied accompaniment. We further experiment with applying perturbation to input melody. GAPT adapts quickly while maintaining harmony, shown in Figure 7.
Co-adaptation with Melody Jamming Agent Figure 4 summarizes co-adaptive interaction, with full results in Table 2. GAPT consistently outperforms ReaLchords and the non-adversarial ablation on both harmony and diversity, supporting that the adversarial reward acts an explicit diversity regulator during mutual adaptation. notable exception Online MLE, which appears strongest when paired with partner trained accommodate it; this setting reduces the need for recovery or adaptation and keeps interaction states close to the curated distribution. This effect does not generalize to human partners, shown later in the user study.
Real-time user study Figure 3 shows participant ratings. GAPT achieves the highest mean all three questions and significantly exceeds ReaLchords on adaptation speed and on perceived control and agency (p 0.05). In complementary analyses on the human interaction trajectories, Figure 5b and Table 2 place GAPT on the empirical Pareto frontier of harmony and diversity, consistent with the simulation results.
Qualitative feedback Participants evaluated three anonymized systems corresponding to Online MLE, ReaLchords, and GAPT. Comments highlight the tradeoffs we target:
On GAPT, "catches my key and chord changes faster.. it would prompt the right chord to resolve the suspension... it definitely adapts faster." (P10)
On ReaLchords, "harmony was fine but was really dumb... it just keeps giving me the same two chords... was a little boring." " (P7)
On Online MLE, "took some time to adapt... chords not really matching... but sometimes tried novel progressions." (P11)
Direct comparison favored GAPT over ReaLchords: "I liked the first one better." (P6, GAPT vs ReaLchords). Another noted GAPT "was folliin the kind of thing expected." (P3)
Together with Fig. 3, these remarks indicate that the additional diversity from the GAPT yields quicker, more responsive collaboration without collapsing to trivial repetitions.
Across fixed melodies, co-adaptive simulation, and live sessions with musicians, GAPT mitigates reward hacking by preventing diversity collapse while preserving harmony. GAPT attains higher adaptation quality than MLE, higher diversity than coherence-fouue RL, and improved perceived speed and agency in human studies.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
We studied real-time melody-to-chord accompaniment where RL post-training on coherence re- wards tends to collapse output diversity. We introduced Generative Adversarial Post-training that trains discriminator to provide data-likeness reward signal during policy optimization, together with a simple two-phase update schedule that stabilizes learning. Across fixed-melody simulation, model-to-model co-adaptation, and user study with recruited musicians, our method improves harmonic adaptation while restoring diversity to near dataset levels, and yields higher perceived adaptation speed and user ageney. These results show that lightweight adversarial training is an effective and practical mitigation for reward hacking in RL post-training of generative sequence models. Future work includes extending the adversarial training to multi-agent co-adaptive training and integrating personalized preference models.
ACKNOWLEDGMENT
We thank Kimaya Lecamwasam, Ke Chen, and Heidi Lei for their valuable discussions related to this project. We are also grateful to Kunal Jha, Carrie Yuan, Yamming Wan and Gaurav Sahu for their helpful advice on paper formatting and writing.
REFERENCES
James Allwright, Eric Foxley, and Shlien. Abe version of the nottingham music database. URL: http://abb.sourrfforgeeee///i/ 2003.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017. 2, 6
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:21 12.00861, 2021.
Gérard Assayag, Georges Bloch, Mare Chemillier, Arshia Cont, and Shlomo Dubnov. Omax broth- dynamic topology of agents for improvization learning. In Proceedings of the 1st ACM workshop on Audio and music computing multimedia, 2006. 4
Christodoulos Benetatos, Joseph VanderStel. and Zhiyao Duan. Bachduet: A deep learning system for human- machine counterpoint improvisation. Proceedings of the International Conference New Interfaces for Musical Expression, 2020.
Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksüi Kuchaiev, Olivier Delalleau, and Tuo Zhao. Adversarial training of reward models. arXiv preprint arXiv:2504.06144, 2025.
Arshia Cont. Antescofo: Anticipatory synchronization and control of interactive parameters in computer music. In International Computer Music Conference (ICMC), 2008.
Roger B Dannenberg. An on-line algorithm for real-time accompaniment. In ICMC, volume 84, pp- 193-198, 1984.
Chris Donahue, John Thickstun, and Percy Liang. Melody transcription via generative pre-training. In ISMIR, 2022. 7,
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shap- ing to mitigate reward hacking in rlhf. arXiv preprint arXiv:2502.18770 2025. 4
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv: 1248 2017.
10

---

Published as a conference paper at ICLR 2026
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023. 3
lan J Goodfellow, Jean Pouget-Abadie. Mehdi Mirza, Bing Xu, David Warde- Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. 2,
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016. 2, 4,4,
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hemmández-Lobato, Richard Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation mod- cls with kl-control. International Conference on Machine Learning, pp. 1645-1654, PMLR, 2017. 3,4
Nan Jiang. Sheng Jin, Zhiyao Duan, and Changshui Zhang. Rl-duet: Online music accompaniment generation using deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 710-718, 2020. 2,
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 8110-8119, 2020.
Peter Keller. Joint action music performance. In Enacting Intersubjectivily; A Cognitive and Social Perspective the Study of Interactions, pp. 205-221. 25--221.208 2008. 2
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 16, 17
Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 2023. 4
Jason Lee, Kyunghyun Cho, and Douwe Kiela. Countering language drift via visual grounding. arXiv preprint arXiv: 1111999904999 2019. 4
George E Lewis. Too many notes: Computers, complexity, and culture in voyager. In New Media, pp. 93-106. Routledge, 2003. 4
Mike Lewis, Denis Yarats, Yamn N Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end- to-end learning for negotiation dialogues. arXiv preprint arXiv: 1706.0570055222 125, 2017. 1,3
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv preprint
arXiv:2503.20788 2025. 20
Jérôme Nika and Mare Chemillier. Improtek: integrating harmonic controls into improvisation in the filiation of omax. In International Computer Music Conference (ICMC), pp. 180-187, 2012.
Jérôme Nika, Mare Chemillier, and Gérard Assayag. Improtek: introducing scenarios into human-
computer music improvisation. Computers in Entertainment (CIE), 14(2). 2017. 4
Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron C Courville. Language model alignment with clastic reset. Advances in Neural Information Processing Systems, 36:3439-3461, 2023. 4
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1807.v1807.0333 2018.

---

Published as a conference paper at ICLR 2026
OpenAl. How people using chatgpt, September 2025. URL https: /tttss:/openaaii /openaie...... com/ http://wwwwpppppaeeeeeeeeseeseee cha.gggts...ht...p.....a..... Reports "700M weekly active users; ac- cessed 2025-09-24.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022. 3,4
Amey P Pasarkar and Adji Bousso Dieng. Cousins of the vendi score: A family of similarity- based diversity metrics for science and machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 3808-3816. PMLR, 2024. 8
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics- -base character control. ACM Transactions on Graphics (ToG), 102, 2021.
Sarah Perez. Google's ai overviews have 2b monthly users, ai mode 100m in the us and india. TechCrunch, July 2025. URL https://techeruncee wwtpe/rruuccccoco/e/e/e/eeeee com.2225////23//////////// googles-ail wwaiovvevviieeee ews have- seree-2-mmonn 2b-monthly-ussrrse
moode-ooeeel--- 1eeeee100pm 100m in-the-us-ande andle-uus-nddidda india/.
States Gemini app has "450M monthly active users; accessed 2025-09-24.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp-
PMLR, 2021. 6, 16
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020. 16
Christopher Raphael. Music plus one and machine learning. JCML, 2010. 4
Alexander Scarlatos, Yusong Wu, Ian Simon, Adam Roberts, Tim Cooijmans, Natasha Jaques, Cassie Tarakajian, and Cheng-Zhi Anna Huang. Realjam: Real-time human-ai music jamming with reinforcement learning- cearning-tuned transformers. arXiv preprint arXiv:2502.21267. 2025. 4, 8
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017. 5, 17, 20
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 20
Federico Simonetta, Filippo Carnovalini, Nicola Orio, and Antonio Rodà. Symbolic music similarity through graph-based representation. In Proceedings of the Audio Mostly 2018 on Sound in
Immersion and Emotion, pp. 1-7. 2018. 7
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022. 1,
Jianlin Su. Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: en- hanced transformer with rotary position embedding. arxiv. arXiv preprint arXiv:2104.09864, 2021. 16
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016. 6
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 16
12

---

Published as a conference paper at ICLR 2026
Wiebke Trost, Caitlyn Trevor, Natalia Fernandez, et al. Live music stimulates the affective brain and emotionally entrains listeners in real time. Proceedings of the National Academy of Sciences, 2024. 2
Yanming Wan, Jiaxing Wu, Marwa Abdulhai, Lior Shani, and Natasha Jaques. Enhancing per- sonalized multi-turn dialogue with curiosity reward. arXiv preprint arXiv:2504.05206, 2025. 2,
Yifeng Wang, Zhouhong Gu, Siwei Zhang, Suhang Zheng, Tao Wang, Tianyu Li, Hongwei Feng, and Yanghua Xiao. Llm-gan: construct generative adversarial network through large language models for explainable fake news detection. arXiv preprint (eXi::2409.01700 2024.
Zihao Wang, Kejun Zhang, Yuxing Wang, Chen Zhang, Qihao Liang, Pengfei Yu, Yongsheng Feng, Wenbo Liu, Yikai Wang, Yuntao Bao, al. Songdriver: Real-time music accompaniment gen- eration without logical latency nor exposure bias. In Proceedings of the 30th ACM International
Conference on Mullimedia, pp. 1057-1067, 2022. 4
Ziyu Wang", Ke Chen Junyan Jiang. Yiyi Zhang, Maoran Xu, Shuqi Dai, Guxian Bin, and Gus Xia. Pop909: A pop-song dataset for music arrangement generation. In Proceedings of s International Conference on Music Information Retrieval, ISMIR, 2020.
William J Wrigley and Stephen Emmerson. The experience of the flow state in live music perfor- mance. Psychology of Music, 2013.
Yusong Wu, Tim Cooijmans, Kyle Kastner, Adam Roberts, Ian Simon, Alexander Scarlatos, Chris Donahue, Cassie Tarakajian, Shayegan Omidshafici, Aaron Courville, et al. Adaptive accompa- niment with realchords. In Forty-first International Conference on Machine Learning, 2024. 2, 4, 5, 6, 7, 15, 16, 20
Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. Reward-robust rlhf in 1lms. arXiv preprint arXiv:2409. 15360, 2024. 4
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Zhang Ze Yu, Lau Jia Jaw, Zhang Hui, and Bryan Kian Hsiang Low. Fine-tuning language models with generative adversarial reward modelling. arXiv preprint rrXiv:3305.06166 2023. 4
Zhiqiang Zhang, Liqiang Wen, and Wen Zhao. A gail fine-tuned IIm enhanced framework for low-
resource knowledge graph question answering. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 1300-3309,, 2024.
13

---

Published as a conference paper at ICLR 2026
Online MLt
ReaLchords
GAPT wio Adv.
GAPT(our))
00
Figure 7: Live accompaniment harmony (note-in-chord ratio) across time. We transpose the test set melody input up 6 semitones at 16-th beat introduce perturbation. GAPT adapts quickly to the perturbation and maintains good harmony afterward.
Algorithm Generative Adversarial Post-Training
Require: Dataset D, policy To. discriminator Dys warmup Twarm interval Km==5, window m=3,
threshold = 1.0, label smoothing a =00 0.1
1: Initialize step k: O, recent adversarial reward Q 0
2: while training do
3: Sample melody D
4:
Generate online rollout y ~ To(. x) using Eq. 2
5: Compute reward R(x, y) Resh (3r, y) + Reutes (I. v) Ruav (2r, v)
6: Update policy Ty with PPO on Eq. 4
7: Push Tadv into queue keep last values
8: if SIOOULDUJDATEDISCKK Q, Twam: Kint, m, T) then
9:
DISCUPDATE(D D, {y}, a)
see $3.3
10: end if
c-- k 12: end while
function SHOCTDDUPDATTEDISCCCC Q, T Twam: Kint- m, T)
14: if k < Twarm then
15:
return (k mod Kint = 0)
fixed interval during warmup
16: else
return (12)=m) (r
17:
confidence gated after warmup
18: end if
end function
A
USE OF LLM
We used LLMs aid in writing and polishing this paper.
B APPENDIX
B.1
ADDITIONAL. RESULTS
B.2 DATASET DETAILS
We use the Hooktheory dataset released in Donahue et al. (2022), which contains approximately 21,000 melody-chord pairs, and we follow official train, validation, and test split. The POP909 dataset has 909 pairs, the Nottingham dataset has 1,019 pairs, and the Wikifonia dataset has 502 pairs.? 2 For POP909 and Nottingham, we create random splits with 80% train, 10% validation, and 10% test.
For training the chord accompaniment policy, its offline baselines, and its reward models, we sample mini-batches from Hooktheory, POP909, and Nottingham with probabilities [60%, 30%, 10%]. For
We the public-domain Wikifonia subset from https://github. com/ http://a/////////////a//eo
14

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation on model jamming with fixed melodies on the and the held-out test set. We report harmony quality (note-in-chord ratio) and diversity (Vendi Score); higher is better for both. Best system bold, second best underlined.
Test set
Out of distribution dataset
System
Harmony f
Diversity t
Harmony
Diversity t
Online MLE
0.368
29.491
0.362
16.401
ReaLchords (Wu et al., 2024)
0.484
20.968
0.475
8.417
CAPT w/o Adv. Training
0.476
20.814
0.447
8.034
GAPT
0.497
26.645
0.470
11.295
Ground Truth
0.727
27.922
0.784
10.962
Evaluation on model jamming with a learned melody agent (model-to-model interaction) and real-time user interaction. We report harmony quality (note-in-chord ratio) and diversity (Vendi Score); higher is better for both. Best system is bold, second best is underlined.
Table 2:
Learned melody agent
User interaction
System
Harmony
Diversity t
Harmony f Diversity t
Online MLE
0.650
18.071
0.448
12.465
ReaLchords (Wu et al., 2024)
0.626
7.480
0.462
9.786
GAPT w/o Adv. Training
0.540
5.658
N/A
N/A
GAPT
0.648
12.914
0.467
11.794
training the melody jammiin agent, its offline baselines, and its reward models, we sample from Hooktheory, POP909, Nottingham, and Wikifonia with probabilities [50%, 20%, 10%, 20%].
B.3 EXPERIMENT DETAILS
The chord embedding of both the diversity and the t-SNE results are extracted from the chord en- coder in the first full input contrastive reward models. The embeddings are already normalized at embedding extraction.
B.4 USER STUDY DETAILS
The real-time interactive system generates music in chunks with a fixed lookahead of ty beats and a commit horizon of le beats. The frontend maintains buffer of planned outputs to handle network latency. For all user study sessions we set tempo to 80 BPM, sampling temperature to 0.8, ly = beats, 4g 4 beats, and an initial listen only period of 8 beats before accompaniment begins. The frontend runs locally with MIDI keyboard while the model serves from a remote GPU node.
For within-subject comparisons, each participant performs the same three tasks with each model: play the same melody (task one), apply the same key modulation (task two), and start the same co- improvisation prompt (task three). System parameters, including tempo and sampling temperature, are fixed across models.
We report paired t-tests with p-values for statistical significance in Fig. 3. Interaction histories have varied lengths and can exceed the maximum input length of the contrastive reward model. To compute diversity, we apply a sliding window with 50% overlap over each jamming session, embed each window, then aggregate diversity over all windows for given model.
The user study conducted this paper has received IRB approval. All participants signed a written informed consent form before any study activities began. Data were collected using secure, access controlled systems, encrypted in transit and at rest, and stored on institution managed servers. Iden- tifiers were not retained beyond scheduling logistics, and all records were de-identified at ingestion by replacing names and contact information with randomly assigned participant codes. Access to the dataset was limited the research team need to know basis, and any shared materials report
15

---

Published as a conference paper at ICLR 2026
Table The retrieval performance contrastive reward models on the test set.
Note to Chord
Chord to Note
Contrastive Reward Models
R@
R@5
R
mAP@ 10
R
R@05
R 10
mAP@:
model
0.18
0.41
0.54
0.28
0.18
0.41
0.53
0.28
model 2
0.18
0,41
0.53
0.28
0.17
0.41
0.53
0.27
rhythm-only model
0.04
0.14
0.21
0.08
0.05
0.13
0.20
0.08
rhythm-only model 2
0.03
0.12
0.19
0.07
0.03
0.12
0.19
0.07
Table 4: The classification performance of discriminative reward models on the test set.
Discriminative Reward Models
Precision
Recall
F1
discriminative reward model 1 discriminative reward model 2
0.86
0.88
0.87
0.87
0.87
0.87
rhythm-only discriminative reward model I
0.73
0.89
0.80
rhythm-only discriminative reward model 2
0.77
0.82
0.79
only aggregate statistics or anonymized excerpts that cannot be linked to individuals. The study followed institutional policies and applicable privacy regulations throughout.
C MODEL ARCHITECTURE AND TRAINING DETAILS
C.I ARCHITECTURE AND PRE-TRAINING DETAILS OF ONLINE AND OFFLINE MODELS
Online models. The online chord ahoopaancommnnnmennt policy and the melody-jamming agent are decoder-only Transformers in the LLaMA-style family (Touvron et al., 2023). Each has 8 layers, attention heads, and hidden dimension 512. We train with Adam (Kingma & Ba, 2015) at a fixed learning rate 1x10-4 for 11,000 steps, batch size 64, and dropout rate 0.1. Positional encoding uses rotary position embeddings (RoPE) (Su et al., 2021).
Offline models. The offline baselines for accompaniment and jamming use encoder-decoder Transformers with 8 encoder layers and 8 decoder layers, each with 8 attention heads and hid- den dimension 512. We use relative position encodings following T5 (Raffel et al., 2020) instead of RoPE. Training uses Adam with learning rate 1 1x for 13,000 steps, batch size 64, and dropout rate 0.1.
C.2 TRAINING AND ARCHITECTURE DETAILS OF REWARD MODELS
Contrastive reward model. Following Wu al. (2024),
melody encoder and chord encoder, each 6-layer, 6-head Transformer encoder with hidden dimension 512. We normaall the outputs and train with the CLIP-style symmetric contrastive objective (Radford al., 2021). We use Adam (Kingma & Ba, 2015) with learning rate x 10 batch size 196, and dropout rate 0.1. The full-input model is trained for 8,000 steps and the rhythm-only model for 2,500 steps. We train two instances per setting with different random seeds.
Discriminative reward model. We use a 6-layer, 6-head Transformer encoder with hidden di- mension 512. The input is the concatenation of melody and chords. We take the hidden state of the beginning-of-sequeene
token as the logit and train with binary cross-entropy using Adam (Kingma & Ba, 2015) at learning rate 1 x 1x10 and dropout rate 0. The full-input model uses batch size 128 for 3,000 steps; the rhythm-only model uses batch size 64 for 3,000 steps to reduce overfitting.
C.3 TEST-SET PERFORMANCE OF REWARD MODELS
See Tab. 3 for test set performance of contrastive reward models and Tab. 4 for test set performance of discriminative reward models.
16

---

Published as a conference paper at ICLR 2026
Table 5: Simulated interaction results of the melody jamming agent on test set chords.
Learned melody agent
System
Harmony
Diversity
Online MIEE
0.525
29.369
Melody Jamming Agent
0.654
28.124
C.4 PERFORMANCE MELODY JAMMING AGENT
See Tab. for harmony and diversity for the melody jamming agent compared with the melody online MLE model.
C.5 RL POST-TRAINING DETAILS
We fine-tune the online accompaniment policy with PPO (Schulman al., 2017) in an environment where the melody x is given and the policy generates chords y. The total scalar reward for a rollout
R(x,y) v) (2, y) + (rutes( y) + Radv v):
where all three terms have equal weight. Rech is the contrastive coherence reward from the con- trastive model, Rqules is the rule-based musicality reward, and radv the adversarial reward derived from the discriminator. We treat the reward as trajectory-level for PPO updates.
Optimization and schedules. We run 1,000 PPO updates. Adam uses B1 0.9 and 82 = 0.95 for both actor and critic. Actor learning rate 5x10-1 7 and critic learning rate is 9x10 -6 We use linear warmup for the first 100 updates followed by cosine decay for the remaining 900 updates, with floor 10% of the peak learning rate. Each PPO update uses batch size 384 and mini-batch size 48; we iterate over the 8 mini-batches per update. Entropy regularization uses coefficient = 0.01 in Eq. 4. The KI. regularization uses coefficient 3 0.001 Eq. 4. We use standard PPO defaults for gradient clipping and normalization.
C.6 DISCRIMINATOR DETAILS
The discriminator D& is an 8-layer, 8-head Transformer encoder with hidden dimension 512. We take the hidden state of the bfginning-of-sequeeee oe-uueee token as the classification logit. We train with Adam (Kingma & Ba, 2015) using 31 0.9, 82 0.95, dropout rate 0.1, linear warmup for 100 steps, and cosine decay for the next 900 steps. The peak learning rate is 9 x 10-5 and the floor learning rate x 1
C.7 HARMONY AND DIVERSITY ACROSS DIFFERENT RANDOM INITIALIZATION
Tables 6 report Harmony (note-in-chord ratio, %) and Diversity (Vendi Score) for GAPT, GAPT w/o Adv. and Real chords, each trained with 3 different random seeds for RL post training. For each system, we show the mean and standard deviation across seeds, and higher is better for both Harmony and Diversity.
Table Harmony (note-in-chord ratio, %) and Diversity (Vendi Score) on the Test Set and held-out Wikifonia dataset across 3 different random seeds for RL post training. For each model, we report mean standard deviation across seeds. Higher is better for both metrics.
Test Set
Held-out Dataset
Model
Harmony
Diversity
Harmony
Diversity 1
GAPT (ours)
0.497 + 0.017
25.540 + 1.475
0.470 + 0.014
11.092 + 0.277
GAPT w/o Ady.
0.477 0.001
20.942 + 1.057
0.445 + 0.002
8.084 + 0..12
0.486 + 0.005
21.431 + 0.780
0.472 + 0.006
8.519 + 0.253
17

---

Published as a conference paper at ICLR 2026
C.8 TRAINING DYNAMICS GAPT
Figure 8 presents the training dynamics of GAPT, including (a) overall reward, (b) adversarial re- ward from the discriminator, (c) discriminator training accuracy, and (d) discriminator training loss over the course of RL post-training. The discriminator is updated for total of 67 steps, with 40 updates phase (fixed-interval warmup) and 27 updates in phase 2 (adaptive updates). our implementation, each diseriminator update step aggregates gradient descent updates, aligned with the 8 mini-batch gradient descent steps in PPO. The discriminator loss and accuracy reported at each step therefore averages over gradient updates, which makes the logged loss values lower and the logged accuracies higher than if they were recorded before each individual gradient update.
Figure 9 reports the average critic values predicted over the course of training for GAPT and for GAPT w/o Adv. The estimated values for GAPT are consistently slightly higher than those GAPT w/o Adv. Figure 10 further shows the difference in estimated critic values between GAPT and GAPT w/o Adv. alongside the adversarial reward. The difference of critic value closely correspondence with the additional adversarial reward, indicating that the critic effectively estimates the additional contribution of the adversarial reward term.
mnbtinepnn Soany /
Sun of Ronads
Adreoset Reyeed
140
(a) Overall reward
(b) Adversarial reward
100
Traiiing Stops
Troining Stops
(c) Discriminator training accuracy
(d) Discriminator training loss
Figure 8: Training dynamics of overall reward, adversarial reward, and discriminator performance during GAPT RL post-training.
Citlle Veluee
Training Steps
Figure 9: Estimated critic values for GAPT and GAPT w/o Adv. throughout the course of training.
18

---

Published as a conference paper at ICLR 2026
Cricc di GAPT wio Adv. Hefti Adversarin aoversariallie Roward (nig))
Adver sattaattall Reward
10
Lopg
0.8
0.2
Training Steps
Figure 10: The difference of estimated critic values between GAPT and GAPT w/o Adv. compared with adversarial reward.
C.9 ABLATION EXPERIMENTS ON REWARD WEIGHTING
To study the effect of the scalar reward in Equation 6, we vary the coefficients of the reward
R(x,y) acocot + Buukes (2, y) + RRddv(x,)).
where (1. (r. y) denotes the coherence reward, (2, y) the sum of rule based penalties, and Raav (I, y) the adversarial realism reward. In the main experiments, we use a = 3 = = Here we keep the training setup and architecture fixed and only change the coefficients (a, 3, 7). The results on both the test set and the held out Wikifonia dataset are reported in Table 7.
We first observe that upweighting the coherence term to 2, 1, has very limited effect on both harmony and diversity, which indicates that GAPT is relatively robust to moderate changes of the coherence weight. In contrast, when upweight the rule-based term (a 1, 2, 1), both harmony and diversity drop on the test set and Wikifonia. This suggests that overemphasizing rule penalties discourages some harmonically reasonable but slightly rule violating chords, which reduces coverage and harms the balance between coherence and diversity. When we increase the adversarial weight (a = 1, 7 = 2), harmony decreases more noticeably while diversity is also slightly reduced, showing that a too strong adversarial signal encourages the policy to chase discriminator preference at the expense of tight alignment with the input melody.
Removing the rule based rewards entirely (a = 1, = 0, = 1) leads to a clear form of re- ward hacking. The policy learns exploit the remaining rewards by generating structurally in- valid sequences, for example not following the onset plus holds pattern for each chord. In this case the outputs are degenerate and the harmony and diversity scores are not meaningful, which we indicate N/A in the table. If we keep only the invalid sequence penalty Rinvalid inside Riiess (a = = 0, = 1, Riivalid), the model again produces valid chord sequences and achieves rea- sonable harmony and diversity, but both metrics remain slightly worse than the full GAPT setting. Overall, this ablation shows that (i) the equal weighting a 3 = = 1 provides good balance between coherence, adversarial realism, and diversity, and (ii) the rule based term, particularly be- yond the invalidity penalty, plays an important role in preventing reward hacking and in stabilizing the tradeoff between harmonic accuracy and output diversity.
Table Ablation study on reward component weighting in Equation We report Harmony (note- in-chord ratio, %) and Diversity (Vendi Score) on the Test Set and Held-out Dataset (Wikifonia). Higher is better for both metries.
Test Set
Held-out Datasel
System
Harmony
Diversity 1
Harmony' t
Diversity
GAPT
0.497
26.645
0.470
11.295
Upweight Coherence (a 2,9 1,7
0.494
26.742
0.476
11,553
Upweight Rules (a 1,3 2,
25.667
Upweight Adversarial (a 1,8
2) 1.9 0.7
01.466
26.317
0.449
11.184
Exclude Rules
N/A
N/A
N/A
N/A 10.428
Exclude Rules + Invalid Penalty (o = 1. 0, 1, Ramaia)
0L488
25.072
0.461
19

---

Published as a conference paper at ICLR 2026
C.10 ABLATION EXPERIMENTS ON RL OBJECTIVE
We conduct ablation experiments evaluate the effect of adversarial training under different RL objectives, with results shown Table In addition to PPO (Schulman et al., 2017), we experiment with GRPO (Shao al., 2024) for reward maximization. GRPO removes the critic model and instead generates multiple responses for the same input, defining the advantage as the reward normalized by the mean and standard deviation of rewards from responses the same input. Specifically, implement Dr.GRPO (Liu et al., 2025), a variant of GRPO that removes the standard deviation term in the normalization obtain an unbiased estimator based on the mean reward.
As shown in Table 8, GRPO without adversarial training exhibits the same diversity collapse as PPO based RL post training. Adding adversarial training on top of GRPO restores diversity and slightly improves harmony on both the test set and the out of distribution dataset, mirroring the effect observed with GAPT. This supports that the benefit of adversarial training is robust across different RL objectives.
Table 8: Evaluation model jamming with fixed melodies on the test set and the held out test set. We report harmony quality (note in chord ratio) and diversity (Vendi Score); higher is better for both.
Test set
Out of distribution dataset
System
Harmony 1
Diversity t
Harmony
Diversity t
Online MLE
0.368
29.491
0.362
16.401
ReaLchords (Wu et al., 2024)
0.484
20.968
0.475
8.417
GAPT w/o Adv. Training
0.476
20.814
0.447
8.034
GAPT
0.497
26.645
0.470
11.295
GRPO w/o Adv. Training GRPO w/ Adv. Training
0.459
16,872
0.443
6.952
0.478
26.603
0.461
11,592
Ground Truth
0.727
27.922
0.784
10.962
C.11 ABLATION EXPERIMENTS ON ADVERSARIAL DISCRIMINATOR INPUT
We further ablate the input to the adversarial discriminator by comparing discriminator that takes only the chord trajectory as input, (v), with one that conditions on both melody and chord, (x, y). The results are reported in Table 9, When train with D (z, 1), the model still achieves higher diversity than the setting without adversarial training, but its diversity remains lower than original Do (y) variant, and harmony the held out dataset also decreases slightly.
We hypothesize that conditioning the discriminator on both melody and chord makes it easier for De to memorize specific training pairs, which reduces the effective difficulty of the discrimination task. This overfitting can weaken the adversarial signal and limit its ability to regularize the policy toward diverse yet realistic chord trajectories, thereby hurting generalization compared with the chord only discriminator.
Table 9: Evaluation model jamming with fixed melodies the test and the held out test set. We report harmony quality (note in chord ratio) and diversity (Vendi Score); higher is better for both.
Test set
Out of distribution dataset
System
Harmony f
Diversity t
Harmony
Diversity t
Online MLE ReaLchords (Wu et al., 2024)
0.368
29.491
0.362
16.401
0.484
20.968
0.475
8.417
GAPT w/o Adv. Training
0.476
20.814
0.447
8.034
GAPT
0.497
26.645
0.470
11.295
GAPT w/ (z,y)
0.467
23.545
0.443
10.124
Ground Truth
0.727
27.922
0.784
10.962
20

---

Published as a conference paper at ICLR 2026
C.12 AVERAGE NUMBER OF PITCH CLASSES CHORDS
Please refer to the following link for a full vocabulary of 2,821 distinct chord symbols, and this vocabulary is shared across all methods in the paper: https: wtw.eaaaloodsseeeee gapt github.
io/staticn asset s/ chord_names maaees_agmmetedddd jaon.
Table 10 presents average number of pitch classes in chord predicted by each model and ground truth. The averages are very close across the dataset and all models, which indicates that the models do not systematically exploit unusually dense chord symbols inflate the metric.
Table 10: Average number pitch classes chords for different systems on the test and the held out dataset (Wikifonia).
System
Test set
Held out dataset
Dataset (Ground Truth)
3.28
3.54
Online MLE
3.22
3.26
ReaLchords
3,01
3.00
CAPT w/o Adv.
3,00
3.00
GAPT (ours)
3.18
3.18
C.13 ACCOMPANIMENT QUALITY ON LONG SEQUENCES
To evaluate model behavior beyond the RL horizon, evaluate GAPT and the baselines quences of length T 512 in two settings: (i) the held-out test split of the training corpora, (ii) the ofttt--distributtin Wikifonia dataset. We compare two inference strategies:
Naive extension. We increase the transformer context length from 256 512 frames and roll the policy autoregressively without changing the architecture or training procedure. rely on the rotary positional embeddings (RoPE) used our transformer and simply increase the maximum context length from 256 to 512 frames, which in principle allows positional extrapolation beyond the training horizon.
Sliding window. We keep the maximum transformer context 256 frames and maintain moving window over the interaction history. When the sequence length exceeds 256 frames, the input window is incrementally shifted forward while we continue generate online outputs. We use hop size of 8 frames. Concretely, once the current sequence reaches T = 256 frames, we shift the context window by 8 frames to the right, predict the next 8 chord tokens in batch while filling in the next 8 melody tokens as input, and repeat this procedure. This design keeps the effective context bounded while still exposing the model continuously updated history over long sessions.
Figure 11 reports results 137 long sequences from the test set, and Figure 12 shows the same analysis on 300 long sequences from Wikifonia. In both datasets, the note-in-chord ratio remains stable across time, with systematic degradation in harmonic coherence. Compared the naive extension, the sliding window strategy obtains better harmony.
1.0
Note in chard ratlo
0.8
Derasel
0.6
(Naive)
Online MLE (Siding Window)
GAPT(ours) (Nalve)
GAPT(uurs) (Siding Window)
0.0
54
128
382
256
320
384
440
512
Context Length
Figure 1I: Note-in-chord ratio GAPT on the test set for long sequences. We evaluate 137 se- quences with total length T 2 512 sixteenth-note frames and plot the average note-in-chord ratio as a function time.
21

---

Published as a conference paper at ICLR 2026
Note in chard ratlo
Online MLE (Naiv)
(Sldinn Window)
GAPT(pur)) (Naive)
GAPT(curs) (Siding Window))
Context Length
Figure 12: Note-in-chord ratio of GAPT on the held-out Wikifonia dataset for long sequences. We evaluate 300 sequences with total length 2 512 sixteenth-note frames and plot the average note- in-chord ratio function of time.
22


Reviews:
[{"review_id": "MWzkP6rBe8", "reviewer": "Reviewer_pmqm", "summary": "The paper proposes Generative Adversarial Post-Training (GAPT) for real-time melody-to-chord accompaniment. GAPT introduces a discriminator to avoid reward hacking in RL finetuning. Evaluation shows this simple yet effective GAPT framework works to mitigate reward hacking.", "questions": "1. What could be the reason for harmonic incoherence (like in Hooktheory Example 4)? Is it because the melody has higher density of notes in beats comparing to other successful accompaniments? When GAPT underperforms harmony (e.g., slight drop on OOD), what musical patterns cause it? Maybe because of the training dataset?\n2. What are end-to-end latency numbers (ms) and GPU budget during live use? How does lookahead/commit horizon trade off with perceived responsiveness?\n3. Would you mind elaborating a bit more on applying GAPT outside real-time accompaniment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Evaluation metrics are reasonable. Note-in-chord ratio allows flexibility because of the nature of real-time accompaniment, but at the same time evaluates the quality of chords.\n2. Adversarial realism reward is easy to add and conceptually orthogonal to KL/entropy regularization. The confidence-gated update is a neat stability trick.\n3. The RL post training scalar reward not only include discriminative models, but also include music rules which mitigate the bias brought by discriminative model as well.\n4. Appreciate the examples provided to have a straightforward evaluation in ears.", "weaknesses": "1. I would appreciate that if more ablations study would be completed on GAPT itself, especially on the scalar reward in Equation (6), applying different weights to see the behavior of models would be a helpful study to provide more insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WUVvz825O3", "reviewer": "Reviewer_SxDd", "summary": "This work explores the use of adversarial learning to mitigate the reward hacking problem in reinforcement learning for the task of melody-to-chord accompaniment generation.\nReward hacking in RL often arises because (1) human-defined rewards are incomplete and may lead the policy to converge to a local minimum, and (2) RL agents cannot distinguish out-of-distribution (OOD) samples, causing them to over-rely on the reward signal.\n\nWhat makes this paper interesting is how it integrates adversarial learning into the RL framework. Unlike other methods (e.g., [1]), the authors adopt a simple yet direct strategy: they add the discriminator term −log(1−y) directly to the reward. In this way, if the discriminator considers a generated sample to be “real,” the policy receives an additional (and potentially large) reward boost.\n\nExperimentally, the authors demonstrate strong results in both quantitative metrics and perceptual evaluations. As an amateur music enthusiast, I can clearly perceive the improvement brought by the adversarial reward.\n\n[1] Bukharin et al., Adversarial Training of Reward Models, 2025.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Technical Comments**\n\n1. Exploring adversarial learning in the context of melody-to-chord accompaniment is quite novel. Moreover, directly integrating the discriminator’s output into the reward function is a simple yet effective strategy, as demonstrated in the paper.\n\n2. The authors present several insightful analyses and results. For example, in Figure 4, they show that the adversarial reward encourages more diverse generation outputs and expands the representational space. This finding is particularly interesting, since—according to the authors—other components such as entropy regularization remain unchanged compared to the w/o-adversarial baseline.\n\n**Presentation**\n\nThe writing is clear and easy to follow, and the examples used in the comparative experiments are also quite convincing when listened to.\n\n**Experiments**\n\n1. For the main task, the authors demonstrate significant improvements over both the baseline and the w/o-adversarial-reward setting, in terms of quantitative metrics and subjective evaluation.\n2. The ablation study is also comprehensive and effectively supports the paper’s claims.", "weaknesses": "**Technical Comments**\n\n1. Since the adversarial reward takes values in  [0,+∞) and is directly added to the original reward, does this cause any numerical instability during training? Has the author conducted experiments to verify its stability?\n\n2. Adversarial training (GAN) itself is known to be unstable. Did the author perform repeatability experiments to show the variance across different random initializations?\n\n3. What does the reward convergence curve look like during training? It seems that this information is not presented in the paper.\n\n4. Could the author show how the critic value differs between the models with and without adversarial reward? (I notice the authors use PPO instead of GRPO.)\nAdditionally, could they run an experiment using GRPO, since GRPO normalizes rewards using the group/batch average?\nGiven that the adversarial reward is always positive, it might shift the overall value distribution upward, potentially affecting learning stability and normalization.\n\n5. Is the discriminator’s training process stable? What does its learning curve look like over the course of training?\n\n\nI am very interested in this paper and I genuinely like the direction. My current score mainly reflects the fact that I would like to see a clearer and more complete picture from the authors, so that the contribution becomes more solid. If the authors can provide more thorough/deep analysis in the rebuttal or revision, I would be happy to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Al5D22kTX1", "reviewer": "Reviewer_VfzC", "summary": "The paper proposed Generative Adversarial Post-Training (GAPT) for live melody→chord accompaniment. A discriminator is trained to prevent reward hacking, ensuring the realism of the accompaniment. The experiment results show that with GAPT, the model outputs more diverse content.", "questions": "1. What if the discriminator conditioned on melody D(x,y)?\n2. What’s the latency budget and end-to-end timing under live use? Does GAPT change compute/latency vs RealChords?\n3. In the appendix, you mentioned that for all user study sessions, you set the tempo to 80 BPM. Does that mean when participants are improvising, the bpm should be set to 80?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Promise to release training datasets, model checkpoints, and code.\n2. Well written, well structured, clear problem and motivation.\n3. Comprehensive experiments show the improvement of diversity, harmonic coherence, adaptation speed and user agency.", "weaknesses": "1. The chord set of the model is not specified; note-in-chord could be inflated by choosing chords with more pitch classes.\n2. The authors did not benchmark against other RL post-training methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yz1jTamlDC", "reviewer": "Reviewer_wu95", "summary": "The paper proposes GAPT (Generative Adversarial Post-Training) to solve the problem of reward hacking in human-music interaction. The paper argues that current methods for RL post training in music related scenarios hacks the reward but leads to repetitive or low diversity music samples. To solve this, the authors add a discriminator component to distinguish between policy generated vs real-data trajectories. Since GAN based training can be unstable, they also propose training techniques to stabilize the training.", "questions": "1. Could you provide empirical analysis of your method's performance and training stability for sequences longer than 256 frames? This is important because real musical interactions often extend beyond this length, and understanding any degradation in performance or training stability would significantly impact the method's practical utility.\n\n2. In Table 2, Online MLE outperforms GAPT in most metrics except for harmony in user interaction. Could you explain this discrepancy and provide additional analysis to justify why GAPT should be preferred over the simpler Online MLE approach?\n\n3. How does this method work for other musical styles?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses an important problem in live music jamming. The proposed solution of having an adversarial loss is simple and seems to work. Moreover, the authors have good evaluation setting, they evaluate on three scenarios: fixed-melody, model-model and real-time with musicians. The paper is very well written and easy to understand.", "weaknesses": "1. The scope of experimentation is narrow. For example, the proposed algorithm is tested on only melody-chord accompaniment. Similarly, the method is limited to T<=256 frames, where each frame is a sixteenth note. There are no studies in the paper that explains what happens for greater timesteps, is the training still stable?\n\n2. The authors use note-in-chord ratio which is a very simple metric. While the qualitative results suggest that GAPT performs better, from Table 1 and 2, we see minimal improvement compared to other baselines. For instance, Table 2 shows that Online MLE performs better than GAPT in everything except for harmony in user interaction (0.448 vs 0.467 (GAPT)). The quantitative results are not convincing enough to prove the efficacy of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)