Reinforcement Learning with Adaptive Reward Modeling for Expensive-to-Evaluate Systems
Authors: Hongyuan Su
Yu Zheng
Yuan Yuan
Yuming Lin
Depeng Jin
Yong Li
Venue: ICML 2025 poster (2025)
Keywords: decision-making task
computational intensive evaluation
Primary area: reinforcement_learning->deep_rl

Abstract: Training reinforcement learning (RL) agents requires extensive trials and errors, which becomes prohibitively time-consuming in systems with costly reward evaluations.
To address this challenge, we propose adaptive reward modeling (AdaReMo) which accelerates RL training by decomposing the complicated reward function into multiple localized fast reward models approximating direct reward evaluation with neural networks.
These models dynamically adapt to the agent’s evolving policy by fitting the currently explored subspace with the latest trajectories, ensuring accurate reward estimation throughout the entire training process while significantly reducing computational overhead.
We empirically show that AdaReMo not only achieves over 1,000 times speedup but also improves the performance by 14.6% over state-of-the-art approaches across three expensive-to-evaluate systems---molecular generation, epidemic control, and spatial planning.
Code and data for the project are provided at https://github.com/tsinghua-fib-lab/AdaReMo.

Reinforcement Learning with Adaptive Reward Modeling for Expensive-to--valuate Systems
Hongyuan Su 12
Yu Zheng" * 13 Yuan Yuan
Yuming Lin
Depeng Jin
Yong Li 122
Abstract
Agent
Agent
Training reinforcement learning (RI.) agents re- quires extensive trials and errors, which becomes prohibitively time-consuming in systems with costly reward evaluations. To address this chal- lenge, we propose adaptive reward modeling (AdaReMo) which accelerates RL training by decomposing the complicated reward function into multiple localized fast reward models ap- proximating direct reward evaluation with neu- ral networks. These models dynamically adapt to the agent's evolving policy by fitting the cur- rently explored subspace with the latest trajecto- ries, ensuring accurate reward estimation through- out the entire training process while significantly reducing computational overhead. We empirically show that AdaReMo not only achieves over 1,000 times speedup but also improves the performance by 14.6% over state-of-the afrt approaches across three expensive-to-evaauute generation, epidemic control, and spatial plan- ning. Code and data for the project are provided at https://gitub..com//siggguu lita--iii/--aaaaMMa.aa..
Fuskutton
Estimation
g
Reward
Solutonn
Roward
"mile
Reward Calculator
Calculator
Figure I. (a) The out-of-symc RL loop for expensive-to-evalutt systems. (b) Our AdaReMo approach adaptively decouples the loop into separate online and offline systems.
Wan et al., 2021). However, despite tasks with immedi- ate and straightforward feedback such as gaming (Mnih et al., 2013; 2015; Vinyals et al., 2019), many real-world tasks involve rewards that are computationally expensive to evaluate (Zhang et al., 2019; Yuan et al., 2023; Yang et al., 2021; Ding et al., 2024; Liu al., 2024b), creating a loop of fast decision and slow evaluation (Figure (a)). For in- stance, in drug design, reward calculation requires intensive computation of molecular dynamics to determine the abso- lute binding free energy between generated molecule and the target (Yang et al., 2021; ; Lutz et al., 2023), a process that often takes seconds-orde of magnitude slower than the millisecond-seat generation process. Such expensive- lo-evaluate reward functions create a significant efficiency bottleneck, rendering RL training highly impractical infeasible, given the millions of interactions required with the environment.
civeens-moleculrr
1. Introduction
To mitigate the computational burden of costly reward func- tions, model-free reinforcement learning (MFRL) often em- ploys proxy reward (Eckmann et al., 2022) or reduced-scale evaluation (Meirom et al., 2021; Liu al., 2024a). Never- theless, these approaches typically oversimplify the evalua- tion, introducing substantial errors into policy optimization and ultimately resulting in suboptimal solutions (Liu et al., 2021). In contrast, model-based reinforcement learning (MBRL) utilizes a world model to approximate the dynam- ies of the environment, including its reward function (Mo- erland al., 2023; Silver et al., 2016; Ha & Schmidhu- ber, 2018; Lowrey et al., 2019; Wan et al., 2021). Though sidestepping time-consuming reward computation, this ap- proach requires a large amount of high-quality data to train the world model, such as expert demonstrations, which are often sparse in real- world scenarios (Hansen et al., 2023). More importantly, the reward function is notoriously com-
Reinforcement learning (RL) has achieved remarkable suc- cess and emerged as the go-to approach for addressing wide range of tasks (Lowe al., 2017; Chen et al., 2021; Haarnoja al., 2018; Farebrother et al., 2024; Yuan al., 2025). The key to effective RL is a well-defined reward function guiding the agent to update its policy as it nav- igates the vast solution space (Silver et al., 2021; Levine et al., 2020; Lowrey et al., 2019; Kidambi et al., 2020;
'Equal contribution 'eppartment of Electronic Engineering, BNRist, Tsinghua University, Beijing, China 'Zhongguancun Academy, Beijing, China "Massachuselts Institute of Technol- ogy, Cambridge, MA USA. Correspondence Yong Li <liyong07 wwwngggodd@iniguacdu..o..
Yu Zheng
<yuLzheng@mit.ccm
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reinforcement Learning with Adaptive Reward Modeling for Expensiveto--Bvalaae Systems
plicated and exhibits drastic local variations depending on the agent's 's exploration trajectory. Fixed world models strug- gle to capture these complexities and keep pace with the agent's 's progress, leading to increasing prediction errors and degraded policy performance as training proceeds (Janner et al., 2019). The computational and modeling challenges significantly hinders the large-scale application of RL in the real world.
tial efficiency advantages and superior decision perfor- mance of our approach.
2. Related Work
Decision-making under expensive objectives. Traditional approaches often rely on heuristic methods or simplified models to reduce computational overhead, yet they sacri- fice accuracy and fail to make optimal decisions (Eckmann et al., 2022; Jeon & Kim, 2020). Recent studies have ex- plored the use of surrogate models (Wu et al., 2023; Wang & Van Hoof, 2022) and approximation techniques (Elsayed et al., 2024; Shetty et al., 2024) to replace reward computa- tion while maintaining performance. For instance, bayesian optimization (Balakrishnan et al., 2020; Astudillo & Frazier, 2021) and gaussian processes (Lin et al., 2023; Achituve et al., 2021) have been utilized to efficiently navigate high- dimensional search spaces. Additionally, advancements in parallel computing and distributed systems have scaled up these tasks across multiple processors or GPUs, significantly reducing computation time (Lu et al., 2022).
In this paper, we propose a general RL approach to decouple the out-of-syne loop of fast decision and slow evaluation into separate online and offline systems. The agent makes rapid decisions in the online system, while the expensive-to- evaluate reward function is offloaded to the offline system (Figure 1(b). Specifically, we design neural network- based reward model (RM) accurately and quickly approx- imate reward computation, enabling fast interactions with the agent. To address the complexity of the reward func- tion, we introduce adaptive reward modeling (A (AdaReMo) which approximates rewards only within the agent's cur- rently explored subspace, decomposing the complicated reward function into multiple tractable localized functions. AdaReMo continuously updates the RM using offline data to align with the agent's progress, ensuring low prediction errors and preventing outdated evaluations throughout the training process. With RM synchronizing fast decision and slow evaluation, our approach seamlessly integrates these two systems operating on different timescales, delivering efficient and accurate RL for expensive-to-evaluate systems.
Model-based Reinforcement Learning. Recently, MBRL has shown promise in improving sample efficiency by learn- ing model of the environment's dynamics (Yu et al., 202 Janner et al., 2019; Yuan et al., 2022). Early approaches focused on learning explicit models of state transitions and rewards, facilitating planning and policy optimization with- out direct interactions with the environment (Silver et al., 2016; Ha & Schmidhuber, 2018; Hafner et al., 2019). Re- cent advancements have extended MBRL to address real- world complexities, such nigh-dimensional state spaces and complex dynamics, through techniques like ensemble methods (Moerland et al., 2023: Wang et al.. 2024), uncer- tainty estimation (Yu et al., 2020; Xu & Liu, 2023), and offline reinforcement learning (Levine et al., 2020; Luo et al., 2023). However, despite these advancements, their limited ability to consistently provide accurate estimates restricts their application to real-world challenging tasks.
To validate the effectiveness of AdaReMo, we conduct ex- tensive experiments across three challenging real-world scenarios-molecular generation, epidemic control, and spa- tial planning. All these tasks involve expensive-tor toresivv-to-evallute reward functions, typically requiring 1 to 15 seconds per sample, resulting in prohibitively long training times for convergence with traditional methods. Results show that AdaReMo not only achieves sfate-of-the-ar performance with over 14.6% improvements over existing approaches, but more importantly, it enables highly efficient RL training, delivering a remarkable speedup of over 1,000 times.
3. Preliminary
The contributions of this paper are summarized as follows,
We consider Docision-making problems formulated as Markov Decision Processes (MDP) with state space S, action space A, transition probabilities P and rewards R S x A after taking an action in specific state. The agent learns policy he parameterized by θ which outputs an action a = (s)s) and receives a reward r(s, a) from the environment. The objective is to maximize the expected return over the entire decision process, formulated as:
We investigate the critical challenge of synchronizing fast decision with slow evaluation, addressing the efficiency bottleneck in RL for expensive-to- -ovalllutte systems.
We propose adaptive reward modeling which decomposes the complicated and costly reward function into easy-to- capture reward models aligning with the agent's progress, ensuring zero-delay RL training and reliable convergence of decision policies.
T Σ (rtr 1
We conduct extensive experiments across three real-world expensive-to-evaluute systems, demonstrating the substan-
max E.o e
(I)
2

---

Reinforcement Learning with Adaptive Reward Modeling for Expensive-oo-Bvaluate Systems
We first employ GNN encoder to transform state observa- tions into dense node and edge embeddings (the graph is defined according to the specific task, see experiments 5), which are calculated as follows:
Solution
Online Decision System
Reward Model
Agent
Reward
n W mer
(2)
Update
Reward Model
Fine-tune
Fine-tune Pool
n; (+1 = n + tanh
Σ JEN(()
(3)
(4,9977)
c (n) + n2/)/2
(4)
Offline Evaluation System
Calculation f;
where An, denotes input attributes for nodes, (i) denotes the set of neighboring nodes of 71 W,, is learnable pa- rameters, / denotes GNN layers with a maximum of L, eij represents the edge that connects n, and ny, and n, and Cij refer to the node and edge embeddings, respectively. The agent then scores each action using these representations with multi-layer perceptron (MIP), and selects actions based on the probability distribution determined by their scores as follows,
Memory Pool
Reward Calculator
Exploration
Sample
(8,0)
Figure 2. The overall framework of AdaReMo, where green and red lines represent fast and slow processes respectively. (Top) In online decision system, the agent updates its policy with real-time feedback from the RM. (Bottom) In offline evaluation system, the RM is continuously finetuned using direct evaluation on latest exploratory samples of the agent stored in memory pool.
= MMP (a.), Pi = D==="i/ / E
where T denotes the time horizon termination condition and is the the discount factor. Notably, we focus sepensive-to-evalual systems where calculating the reward r(s, a) is time-consuming, rendering trivial RL impractical.
(5)
where s; and pt are the score and probability of taking action on node i or edge i, and denotes the embeddings of action ag which indicates node or edge selection.
4. Method
The agent interacts within the fast online system to collect millions of trajectories for RL training. We optimize its policy Te with PPO (Schulman al., 2017) in an actor- critic fashion, updating its parameters follows:
Our method, adaptive reward modeling (AdaReMo) ad- dresses the efficiency bottleneck by (1) disentangling fast and slow processes into two separate systems, (2) synchro- nizing these systems through a neural network- -ased reward model (RM) dynamically adapting to the agent's 's progress.
VeJ(() = E min (ri(T))A, clip( e, e.). + θ (o)
As illustrated Figure 2, we begin by offloading heavy reward computations into an offline system, and keep the agent in online system updating its policy in real-time with the fast RM. We then bridge the speed gap via adaptive updates to the RM using offline data collected by direct reward evaluation, aligning the RM with the agent to ensure accurate reward estimation throughout the entire training process while introducing no computational overhead. Fi- nally, we design synchronous correction, parallel evaluation, and model warm-up further enhance training efficiency.
(6)
where (s((sa) a) is the advantage of the state-action pair (s, and 7 is the learning rate.
4.2. Reward Model
Feeding the direct evaluations back the agent as rewards for policy optimization is impractical or even infeasible due to the significant computational demands of sophisti- cated reward functions in expensive-to-evaluate systems. To address this challenge, neural networks offer promising solution for accelerating evaluation with their robust fitting capabilities and rapid inference speed. Inspired by RLHF (reinforcement learning from human feedback) in finetuning large language model (Touvron al., 2023), we develop reward model (RM) for rapid reward estimations. RM re- places intensive computations of the evaluation with a deep neural network to estimate the reward for each state-action pair. We employ another MLP with parameters for the RM which shares the same GNN encoder with the agent to
4.1. Online RL Agent
We develop the RL agent based on graph neural networks (GNN) (Kipf& & Welling, 2017) since the graph data struc- ture applies to many real-world tasks, particularly for the three tested scenarios in this work. It is worthwhile to notice that our approach is model-agnostic and can be integrated with other deep learning architectures such as convolutional neural networks (Krizhevsky et al., 2012) and transform- ers (Vaswani, 2017), which we leave for future work.
3

---

Reinforcement Learning with Adaptive Reward Modeling for Expensiveto--Bvalutte Systems
Algorithm Training Process of Online and Offline System
Ey4s
Timestamp
Online Decision System:
Opúrnizz
1
TT
Input: episodelen, policy øe, reward model Rø, transi- tion function T, memory pool M
Expleration
for episode to episodelen do
for / = to T do
RMg
RM
RM
RM
Memory
a, m=(s)) f. = ac),, S141 = T(s az) M.push( (,,
Parallel Reward
end for
Fine-ture Pool
G: = Ek=t 7 k-t kk. A. G. Vo((t)
Calculators
Precee
+ θ + aA, Ve log Th (st, as)
Eeducaionn
(05.0)
end for
Figure 3. The scheme of AdaReMo with asynchronous training framework. Multiple reward calculator constantly evaluate the solution generated by progressively optimized decision policy and for RM fine-tuning. After T policy optimization iterations, RM is fine-tuned to adapt to new decision policy.
Offline Evaluation System:
Input: memory pool M, fine-tune pool F, reward calcu- lator C
repeat
M pop(( ae, fr)), C(st, az)
significantly reduce training parameters. The RM estimates reward as follows,
F. F.push((s,, a2s fr, r2))
until M is None
h2 In', n/, f(s,a) (h, |a),
Adaptive Reward Modeling:
Input: fine-tune pool F, reward model R, fine-tune inter- val, fine-tune epoch while iter % tin-ttuneee interval == 0 do for epoch = 1 to fine-tune epoch do Frrandpop((ss,, az, fr, rx))
where V denotes the set of nodes, h, is the average node representation summarizing the current state, and f(s, a) is the estimated reward. The RM is able to provide immediate feedback to the agent, facilitating policy optimization within the online system in real-time.
L L= - r2)
4.3. Adaptive Reward Modeling
end for end while
While we introduce RM to accelerate RL training, ensuring its accuracy crucial and requires careful examination. On the one hand, neural network-based RM may struggle completely capture the intricacies of the underlying laws of the sophisticated reward function, leading to reward es- timates that not always align with direct evaluations. On the other hand, it is impractical to train RM on every pos- sible state s E S. Consequently, as training proceeds and the agent explores previously unseen or uncommon states, RM may provide erroneous estimates significantly diver- gent from direct evaluations, which mislead the agent, hindering its ability to learn the optimal decision policy.
the full state space S into multiple subspaces (Si, . and fintune the RM using samples within each subspace, thus obtaining [RM1, RM2, ...) accordingly.
As illustrated in Figure 3, fixed-size memory pool is lever- aged to store the recent exploratory samples following the first- frrst-in-first first-out principle, where sate-action pairs are sam- pled to perform direct evaluation for accurate reward r(s, a), which will be added to a fine-tune pool F. As policy op- timization iterates, F is filled with sufficient samples and RM is fine-tuned by the MSE loss as follows,
Though RM may not fully capture the complicated reward function, it still has the ability to accurately predict the re- ward within a reduced and localized state subspace. To ensure accurate reward approximation throughout the en- tire training process, we propose adaptive reward model- ing (AdaReMo) with an asynchronous training framework which updates RM concurrently according to the agent's progress on its policy. The main idea of AdaReMo is to align the definitional domain of RM with the currently ex- plored state subspace by the agent as closely as possible through periodic finetuning. other words, we decompose
L
Σ (f(s, - r(s, (8,0)EF
(8)
Algorithm shows the training process of our online and offline system with AdaReMo. It is worth noting that the PPO algorithm (Schulman et al., 2017) constrains the magni- tude of policy updates to enhance the stability of the agent's 's learning process. Therefore, each subspace S; is compact enough for a neural network RM, to accurately approximate, thus guaranteeing consistently low error reward estima-

---

Reinforcement Learning with Adaptive Reward Modeling for Expensive-to-Evaluate Systems
(a)
(b)
(c)
Table 1. Time spent on evaluating molecular generation (MG),
pandemic control (PC) and urban spatial planning (USP).
Evaluation
MG
PC
USP
RM Simplified
0.02
0.02
0.01
0.03s
0.12s
0.01s
Precise
8.6s
15.2s
10.5s
Figure 4. Three experimental decision-making tasks, molecular generation, (b) pandemic control and (c) urban spatial planning.
programming, and the hyper-parameter K determined by computational resources. Each calculator pulls the lat- est solution from the memory pool, completes its reward evaluation, and then initiates another round of calculation.
tion and aliging RM with the agent throughout the whole process which we empirically show in experiments.
5. Experiments
4.4. Training Acceleration
We investigate three challenging tasks with expensive-to- evaluate reward functions: molecular generation, pandemic control, and urban spatial planning, as shown in Figure Additionally, we summarize the time spent on the corre- sponding evaluations in Table 1, where using the RM results in speedup of approximately 1,000 times compared to employing the precise reward functions.
With AdaReMo integrating the decision and evaluation sys- tems operation on different timescales, we introduce syn- chronous correction and model warm-up to enhance the robustness of reward estimation, as well as parallel compu- tation to further improve the training efficiency.
Synchronous Correction. As RM is updated periodically during policy optimization, there can be samples that reside outside the subspace S; which become outliers to the defini- tion domain of RM, leading to errors in reward estimation. To mitigate the impact of such outliers, we introduce syn- chronous correction mechanism follow:
5.1. Molecular Generation
Molecular generation aims to identify novel molecules that bind most effectively to protein targets, where RL has be- come promising method due to its ability in searching a vast solution space (Yang et al., 2021; Lutz et al., 2023). A molccule can be represented as graph = (V, where atoms are nodes and bonds are edges. The gener- ation process is equivalent to graph expansion, where the agent's action corresponds to adding new fragment (a set of nodes) connected by bond (edge) to the existing molecular structure at specified attachment site.
F(s,a) af(s, a) + - a)),
where re(s, a) represents the correction term and a is trade-off parameter. The correction term obtained from simplified reduce-scale direct evaluations, allowing for reward rectification without slowing down the training.
Model Warm-up. RM's parameters are randomly initial- ized thus cannot offer reliable reward estimates at the he- ginning, which can introduce significant noise into or even disrupt policy optimization. Therefore, is crucial to pre- train the RM before updating the agent's policy. In our implementation, delay policy optimization until the RM has undergone several fine-tuning intervals, ensuring that the agent consistently receives feedback from a warmed- up RM. While the RM warm-up consumes additional time from the start, it substantially enhances subsequent training efficiency and accelerates model convergence.
Evaluating the quality of the molecule presents significant challenges. The molecular docking program is widely used to provide precise measurements of the therapeutic potential of molecules. Through computationally expensive molecu- lar dynamics-based simulations, the program calculates ac- curate binding free energy to identify hit compounds. Here, we employ AutoDock Vina (Trott & Olson, 2010; Eberhardt et al., 2021), a docking engine with outstanding accuracy, to evaluate the effectiveness the generated molecules in targeting proteins, as the reward in the MDP. To compare the performance of each approach, we investigate the average score of the top 5%-oco generated molecules, and hit ratio which is the proportion of docking scores exceeding reference thresholds. Notably, the molecular docking pro- gram requires several seconds to obtain precise results for a single sample, which is unaffordable for traditional RL methods.
Parallel Computation. Increasing the number of samples in the fine-tuning pool F can improve the approximation performance of the RM, as deep learning often benefits from more training data. Therefore, we employ parallel computation to significantly augment the dataset during fine- tuning intervals. Specifically, we implement simultaneous calculation of K reward calculators using multi threaded
5

---

Reinforcement Learning with Adaptive Reward Modeling for Expensive-t--Bvalutt Systems
Table 2.
Performance comparison molecular generation with respect Top 5% Score (T5) and Hit Ratio (HR).
FA7
PARPI
SHTIB
Method
T5t
HR
T51
HR
T51
HR
HierVAE
9.4 0.1
0.06 0.01
12.2 = 0.1
0.25 = 0.01
11.9 - 0.1
0.12 = 0.01
LIMO
9.7
0.11 = 0.02
1.
0.04
10.3 0.8
0.20 0.04
MolDQN
8.2
0.02 = 0.01
1.2
0.02
0.1
0.11
FREED
1.2
0.23 0.04
0.3
0.009
1.2
0.11
MBPO
9.1
0.18 118=0.00 0.07
1..6
0.26
1.5
0.33
GRPO
9.2
0.21 = 0.03
1.5
0.22
11.8
0.38 0.38
RLOO
0.4
0.22 0.02
12.0
0.29 0.06
12.5 + 0.4
0.40
Ours
10.4 = 0.2
0.29 - 0.03
13.1 0.4
0.42 + 0.05
12.7 + 0.4
0.48 +:8+:000
impr%
+1.9%
+19.4%
+1.6%
+20.0%
+3.2%
+17.1%
We experiment on three protein targets, FA7, PARPI, and 5HTIB, which are commonly studied in medical re- search (Yang et al., 2021; Nautiyal et al., 2015). We com- pare our method with state-of-ther of-the-art molecular generation baselines, which are (1) non-RL algorithms including Hi- VAE (Jin et al., 2020) and LIMO (Eckmann et al., 2022), (2) MFRL-based methods including MolDQN (Zhou al., 2019) and FREED (Yang al., 2021), (3) MBRL-based algorithm MBPO (Janner al., 2019), GRPO (Shao al., 2024) and RLOO (Ahmadian et al., 2024). For each metric, we repeat experiments with 10 different random seeds and record the mean and the standard deviation.
Table 3. Pandemic control performance measured by Healthy (H) and Contained (C).
CA-GrQc
SNAP
Method
H
C1
H1
Ct
HSB
31.7+1.8
8.8
22.3+1.1
3.040.3
KED
30.2+2.2
8.7
22.1+2.4
2.340.4
GBP
32.9
9.1-0.3
23.8+0.4
3.140.1
RLGN
35.8
9.8-14
25.6+3.1
3.4+1.1
MBPO
27.2
7.3
15.4+6.0
1.340.6
Ours
39.8
10.4
28.2
3.8
impr%
+7.4%
+7.2%
+10.2%
+11.5%
Table 2 illustrates the performance of different approaches, which confirm the necessity of precise evaluation. Specifi- cally, using the simplified evaluation, MolDQN consistently generates molecules with poor binding quality, showing sharp performance decline of 21.9% and 93.5% in Top 5% Score and Hit Ratio, respectively. In contrast, benefit from the dynamic-based precise evaluation, our approach and FREED always achieve the optimal and suboptimal perfor- mance, with an average improvement of 13.2% and 10.9% over other RL methods.
V represents individuals and E denotes their interpersonal contacts (Meirom et al., 2021).
For accurate evaluation of epidemic control, it is common to employ susceptible-infees iiwlleeiiiffee----eeeeeeeeeeeete recovveed (SIR) model (Ker- mack & McKendrick, 1927) to capture the propagation dy- namics of pandemic. By predicting the health status of individuals at each stage of propagation multiple times, SIR simulation provides more comprehensive decision assess- ment. Performance of epidemic control is measured by Healthy representing the final proportion of healthy individ- uals, and Contained indicating the proportion of simulations where Healthy exceeds 60% (Meirom et al., 2021).
Our method has remarkable advantages over other base- lines. For all the three different target proteins, our method demonstrates the best generation quality. Specifically, the Hit Ratio of our methods improves other MFRL methods by over 8.3%. Meanwhile, by employing AdaReMo, the agent able to capture the localized reward function accurately, reaching an improvement 6.3% over MBRL approach.
We utilize large-scale real-world contact networks CA- GrQe (Rossi & Ahmed, 2015) and SNAP (Leskovec & Krevl, 2014), which are extensively studied in epidemiolog- ical research. Consistent with prior studies (Meirom et al., 2021), the agent isolates 2% of the total population, and we simulates 25 propagation steps with SIR model using 20 different seeds (3 for synchronous correction). Additionally, the parameters of the SIR model are set with an infectious rate 8 0.08 and a recovery rate 7 0.2, informed by real- world pandemic propagation (Yu et al., 2021b). We compare our method against (1) classic approaches includ- ing KED (Tong et al., 2012), GBP (Kimura et al., 2008) and heuristic search approaches based on betweenness (HSB), (2) MFRL approach RLGN (Meirom et al., 2021), and (3) MBRL approach MBPO (Janner et al., 2019).
5.2. Epidemic Control
Mitigating the impact of a pandemic requires strategic al- location of limited resources such as quarantine facilities and vaccine supplies within social networks. The challenge in epidemic control stems not only from the vast and in- tricate social networks but also from the difficulty in accu- rately modeling disease dynamics. Pandemic control can be conceptualized as sequentially selecting nodes on a so- cial network = (V, F.) to be temporarily isolated, where
6

---

Reinforcement Learning with Adaptive Reward Modeling for Expensive-to- Evaallut Systems
(a)
(b)
Table The hasic attributes of the real-world communities.
Community
Location
Area
Round
Grids
HLG DHM
Beijing
3.67km²
km7km
Beijing
3.35km²
8.0km
48
HZ
Guangzhou
2.96km²
6.9km
54
Compurtonna
Figure 6. (a) Training efficiency comparison of four RL-hased methods in molecular generation. (b) The adaptability of RMs. where solutions are evaluated at each policy optimization iteration. Best viewed in color.
passes through green areas per resident.
Figure 5. Urban spatial planning performance measured by Dis- fance (D, the lower the better) and Greenness (G, the higher the better).
We experiment on three real-world communities in China (Zheng et al., 2023), initially bordered by main roads and designated residential areas, with further details pro- vided in Table 4. The task involves partitioning the original community layouts to allocate areas for green spaces, busi- nesses, offices, schools, hospitals, recreation, and residential purposes and ends when all the requirements are satisfied. In the implementation, a complete simulation contains the weekly trajectories of 1,000 virtual residents, while in syn- chronous correction, we decrease to 100 residents. For the performance comparison, we include (1) traditional plan- ning concepts such as centralized (CEN) and decentralized (DCEN), (2) genetic algorithm (GA) (Zheng et al., 2023), (3) MFRL-based method DRL (Zheng al., 2023) and (4) MBRL approach MBPO (Janner et al., 2019).
Table 3 illustrates the results of our approach and baselines. First, classic algorithms cannot address complex epidemic control tasks well. HSB heavily relies on manually designed rules and often fails to capture the essential characteristies of epidemies, resulting in poor performance across networks and metrics. Additionally, KED and GBP also show notice- able declines compared to RL approaches. Specifically, the risk of infection increases by at least 11.7%, while the prob- ability of containing outhreaks decreases by at least 17.2%. Second, our approach achieves the best performance on both Healthy and Contained across different social network, demonstrating its outstanding ability in pandemic control. Specifically, our generated control strategy prevents 3.3% of the population from infection and is the only strategy with containment ratio higher than 10% among all the methods. Moreover, our approach improves by 45% over the MBRL. approach due to the adaptability of AdaReMo.
Figure 5 illustrates the results of our approach and baselines. While MFRL effectively explores the solution space, the imprecise rewards received by the agent limit the efficacy of layout planning. By employing a realistic mobility-base evaluation, our method significantly improves community layouts, reducing average travel distance from 3.06 to 2.78 (-9.2%) and increasing average green space visits from 1.62 to 1.78 (+9.9%). Additionally, within three RL-based ap- proaches, MBRL approach consistently produces layouts with poor accessibility and green space exposure, showing an average performance decline of 18.8% and 14.2% in Dis- lance and Greenness, respectively. Notably, although our method achieves the best planning outcomes overall, MBPO performs the worst among all baselines, underscoring the critical role of the AdaReMo.
5.3. Urban Spatial Planning
Rationalizing the functional division of limited urban land presents a challenge, requiring consideration not only of the city's 's actual development needs but also of the functional interconnections between different types of land uses. The task of urban spatial planning can be formulated sequen- tially selecting edges on city graph G = (V, E), where V represents community lands and road segments, and E indicates their spatial adjacency (Zheng et al., 2023).
To incorporate realistic human mobility into community lay- out evaluation, we utilizes the state-of-the urban mobility simulation method SAND (Yuan et al., 2023) for evaluation. With thousands of reconstructed daily trajectories of resi- dents, each layout for community is measured by Distance and Greenness. Distance indicates the accessibility of the community layout, quantified by the average daily trajec- tory length per resident. Greenness signifies the residents' exposure to green spaces, calculated by the average daily
5.4. Effectiveness and Efficiency of AdaReMo
RM plays pivotal role as the key bridge between online optimization and offline evaluation in our approach. To illustrate the difference on training efficiency between DRL- based methods, we present the evaluation metrics of the generated solutions after the same optimization time in Figure 6(a). While RL using direct evaluation (FREED) initially shows a slight advantage, the performance of our
7

---

Reinforcement Learning with Adaptive Reward Modeling for Expensive-to-Evaluate Systems
(a)
(b)
(a)
Dffiret aereeee
(b)
Moleculer Generction
Pardemic Control
Figure 8. The effectiveness of AdaReMo with different hyper- parameters. (a) Impact of varying fine-tune intervals on training efficiency and convergence. (b) Influence of varying fine-tune epochs on metric performance. Best viewed in color.
Proctse Fanard
Predies flmvadd
Figure 7. The visualization of RMs' estimation errors in (a) molec- ular generation and (b) pandemic control. The element (i.j) represents the error in applying the RM at j iteration to estimate the solution at iteration.
in the distribution of the reward function. These findings underscore the necessity of AdaReMo and the asynchronous training framework, ensuring that RM remains aligned with the decision policy.
approach quickly catches up and surpasses that of FREED, with an increase of more than 17.6%. This observation can be attributed to the insufficient fine-tuning of RM to cap- ture the complex dynamics of the environment during the early training stages, which introduces reward estimation errors. Benefiting from accurate evaluations and fast RM estimations, our approach efficiently optimizes the policy and achieves the fastest convergence to generate quality molecules. In contrast, even after several days of training, the performance of RL without RM improved by only a slight 7.1% 9,, highlighting the necessity of the RM which sidesteps the heavy computational burden of reward calcula- tion with fast estimations using neural networks.
5.5. Empirical Analysis of AdaReMo
Choosing the optimal timing for fine-tuning RM is crucial for enhancing the efficiency of AdaReMo. Here, two critical pypee--prameters for the asynchronous training framework are investigated: fine-tune interval and fine-tune epoch.
We first explored different fine- tinettuning intervals, ranging from 1 to iterations, and trained the RM accordingly. As shown in Figure 8(a), short fine-tuning interval (c.g.. I iteration) impedes effective RM updates, leading policy optimization failure. Specifically, optimization efficiency drops to less than 75% of default settings, and the conver- gence fails even after 300 iterations. Conversely, excessive sampling also hinders agent learning efficiency, resulting in a 5.7% decrease in performance. With a short fine-tuning period, the data in the fine fine-tuning pool F closely track the exploration subspace but may lack sufficient samples due time- tume-consuming evaluation. On the other hand, a longer fine- tuning interval makes F denser and more efficient for R.M fine-tuning. yet risks lagging RM updates significantly be- hind policy optimization, leading to misalignment between the RM and the agent.
In addition to improving overall training efficiency, the adaptability of the model to a continuously optimized de- cision policy is also examined. We randomly select solu- tions with accurate evaluations from the fine-tuning pool F during the training process. Fine-tuned RMs are saved at iterations 100, 300, and 500 (denoted as RM100, RM300, and RM500, respectively) to estimate molecular properties. Figure 6(b) presents the evaluations of the molecules along- side the corresponding RM estimations. Specifically, all RMs accurately estimate molecules generated around their respective iterations, with an average relative loss of less than 0.8% and a relative standard deviation less than 2.1 %. In contrast, RMS exhibit estimations with a relative loss ex- ceeding 37.5% and extremely high variance for molecules outside their trained state subspace.
Similarly, we varied the number of finetuning epochs to examine its influence, as depicted in Figure 8(b). Both excessive and insufficient epochs resulted in significant per- formance decrease of 7.5% and 15. 1,, respectively. Fewer epochs enable quicker RM feedback to the agent but may compromise fine-tuning effectiveness. Conversely, more epochs facilitate thorough RM adaptation but require addi- tional time and risk overfitting. The optimal solution was found with 40 fine-tuning epochs, matching the duration of single policy iteration and demonstrating efficient time utilization. Adequate sampling in F during this period en- sures effective RM fine-tuning, allowing agents to receive accurate feedback in subsequent iterations with updated RM. Insufficient time may diminish fine- tuning quality, while excessive time may force agents to iterate multiple times
To further visualize the consistency of between the RM and the agent, we calculate the errors between RM estima- tions and precise evaluation during the training process. As shown in Figure 7, the elements on the diagonal are ap- proximately O, indicating that the RM can always adapt to the current solutions with accurate estimations. When the iteration differs, the error of estimation rises significantly. Notably, two areas of high variance can be observed: the lower right corner and the middle left. The former is due to the RM's 's initial inability to effectively learn the dynamics of the environment, while the latter indicates a sharp shift
8

---

Reinforcement Learning with Adaptive Reward Modeling for Expensive-to- Evaluluate Systems
FJR
K/ W0.
Wil Mode Warh-up Nh1 Part
Figure 10. Results of the ablation studies on the molecular gener- ation task, where synchronous correction, model warm-up. up, and parallel computation components are removed individually.
Fine-tune mncevalss
Fine-une Ipocha
Figure 9. Best Performance and corresponding iterations of our
framework with different pyper-parameters.
small neighborhood of this estimated value typically yields the optimal setting, as validated in our experiments.
with outdated RM, wasting exploration efforts.
is important to clarify that above discussions do not imply that our framework is highly sensitive to hyper-parameters. We extended the number of optimization iterations from 300 to 500 and recorded the model's best performance along with the corresponding iteration number across vari- ous tyyer-paramette settings. The results are presented in Figure 9. Under suboptimal hyperparameter configurations, training efficiency is substantially reduced, as the number of iterations required to reach peak performance increases from 216 to 463, resulting in more than twice the optimiza- tion time. Nevertheless, the final performance of the model remains remarkably stable, with less than a 4% difference compared to the optimal one. These results demonstrate that while appropriate typer-paramete selection can signifi- cantly improve training efficiency, the framework itself is robust and maintains competitive performance even in the presence of suboptimal papperr-rrmmters...
5.6. Ablation Study
We conduct series of ablation studies to evaluate the indi- vidual contributions of key components within our proposed framework. The performance of each variant which remove the corresponding component are presented in Figure 10.
The exclusion of any single component leads to a measur- able degradation in performance. In particular, the removal of synchronous correction introduces heightened stochastic- ity and systematic bias, resulting in an average performance decline of approximately 17%, with the most severe degra- dation reaching 45.2%. Meanwhile, excluding the model warm-up phase yielded consistent performance drop of around 7% across evaluation metrics. This degradation can be attributed to the policy network being initially exposed to randomly generated rewards, which injects noise into the training process and impairs early-stage optimization. Like- wise, the removal of parallel computation capabilities dimin- ished training efficiency by restricting the reward model's capacity to process a sufficient volume of samples for ac- curate mee-tuning. This constraint led to an estimated 10% reduction in overall performance.
Additionally, we would like provide practical guide- lines that enable users efficiently estimate an approxi- mate range for the optimal syyer-parameete ensuring the method's 's reliability and reproducibility without exhaustive tuning. First, the optimal number of fine-tuning interval can be approximated with the complexity of the reward function, positively correlated with its evaluation time, For example, Table 1 indicates that reward computation in pandemic con- trol is more time-consuming than molecular design, and grid search identified an optimal fine-tuning interval of 9 and 7 (see Figure 8(a)) for the two scenarios, respectively, suggesting that a more complex reward function requires ad- ditional samples for effective RM fine-tuning Second, the optimal number fine-tuning epoch should balance the fine- tuning duration with the policy optimization. Specifically, we can measure the time f required for one sampling and policy optimization iteration. Then, based on the previously determined fine-tuning interval and the reward function's ss computation time, we can estimate the number of samples needed for ane-ttnnng and compute the time 12 per fine- tuning epoch. The theoretically optimal number of epochs N & providing a robust starting point. Search within 12'
6. Conclusion
This work proposes AdaReMo, general and efficient rein- forcement learning approach for systems involving costly reward functions. We introduce reward model to approx- imate reward calculation, which disentangles the fast de- cision and slow evaluation into distinct online and offline systems, enabling efficient policy training without any de- lays. Meanwhile, the reward model continuously adapts to the agent's progress, ensuring accurate reward approxi- mation throughout the entire training process. AdaReMo displays competitive performance in molecular generation, epidemic control, and spatial planning. Looking ahead, we plan to further explore the universality of our approach across different deep learning architectures besides GNN, as well as broader expensive-to-evaluae tasks.

---

Reinforcement Learning with Adaptive Reward Modeling for Expensive-o--Bvalaat Systems
Acknowledgements
molecule generation. Proceedings of machine learning research, 162:5777, 2022.
This work is supported in part by National Natural Science Foundation of China under grant U23B2030 and Zhong- guancun Academy Project No.20240303. This work is also supported in part by Tsinghua Teiversity-Toyode Research Center and Beijing National Research Center for Informa- tion Science and Technology (BNRist).
Elsayed, M., Farrahi, H., Dangel, F., and Mahmood, R. Revisiting scalable hessian diagonal approximations for applications in reinforcement learning. arXiv preprint arXiv:2406.05270 2024.
Farebrother, J., Orbay, J., Vuong, Q., Taiga, A. A., Chebotar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., et al. Stop regressing: Training value functions via classi- fication for scalable deep rl. In Forty-first International Conference on Machine Learning, 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Ha, D. and Schmidhuber, Recurrent world models facil- itate policy evolution. Advances in neural information processing systems, 31, 2018.
References
Achituve, I., Shamsian, A., Navon, A., Chechik, G., and Fetaya, E. Personalized federated learning with gaussian processes. Advances in Neural Information Processing Systems, 34:8392-8406. 2021.
Haarooja, T., Zhou, A., Abbeel, P., and Levine, Soft actor-critic: Off- maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018
Ahmadian, A., Cremer, C., Gallé, M., Fadace, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402. 14740, 2024.
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565. 2019.
Astudillo, R. and Frazier, P. Bayesian optimization of func- tion networks. Advances in neural information processing systems, 11114446-1444 2021.
Hansen, N., Lin, Y., Su, H., Wang. X., Kumar, V., and Rajeswaran, A. Modem: Accelerating visual model- based reinforcement learning with demonstrations. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: http:///ppeeeiieeeeeeeeeee net
Balakrishnan, S., Nguyen, Q. P., Low, B. K. H., and Soh,
H. Efficient exploration of reward functions in inverse reinforcement learning via bayesian optimization. Ad- vances in Neural Information Processing Systems, 33: 4187-4198, 2020.
Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 24:15504-15097, 2021.
Jeon, W. and Kim, D. Autonomous molecule generation using reinforcement learning and docking to develop po- tential novel inhibitors. Scientific reports, 10(1):22104, 2020.
Ding. J., Liu, C., Zheng, Y., Zhang. Y., Yu, Z., Li, R., Chen, H., Piao, J., Wang. H., Liu, J., al. Artificial intelligence for complex network: Potential, methodology and application. arXiv preprint 2402 16887, 2024.
Jin, W., Barzilay, R., and Jaakkola, T. Hierarchical gen- eration of molecular graphs using structural motifs. In International conference machine learning, pp. 4839- 4848. PMLR, 2020.
Eberhardt, J., Santos- D., Tillack, A. F., and Forli, Autodock vina 1.2. New docking methods, expanded force field, and python bindings. Journal of chemical information and modeling, 61 68881389133888 2021.
Kermack, W. O. and McKendrick, A. G. A contribution to the mathematical theory of epidemics. Proceedings of the royal society of london. Series A, Containing papers of a mathematical and physical character, 115(772):700-721, 1927.
Eckmann, P. Sun, K., Zhao, B., Feng, M., Gilson, M. K., and Yu, R. Limo: Latent inceptionism for targeted
10

---

Reinforcement Learning with Adaptive Reward Modeling for Expensiveto--Baaluate Systems
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning- Advances in neural information processing systems, 33: 21810-21822 2020.
Lu, C., Kuba, J., Letcher, A., Metz, L.,, Schroeder de Witt, C., and Foerster, J. Discovered policy optimisation. Ad- vances in Neural Information Processing Systems, 35: 16455-16466 2022.
Kimura, M., Saito, K., and Motoda, H. Minimizing the spread of contamination by blocking links in network. In Aaai, volume 8, pp. 175--180, 2008.
Luo, F.-M., Xu, T., Cao. X., and Yu, Y. Reward-consisted dynamics models are strongly generalizable for offline reinforcement learning, arXiv preprint 21X0022110.0542., 2023.
Kipf, T. N. and Welling, M. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:// spenreview.net/Foree Si-uuummdddSSSSyyyya
Lutz, D., Wang, S., Norn, C., Courbet, A., Borst, A. J., Zhao, Y. T., Dosey, A., Cao, L., Xu, J., Leaf, M., al. Top-down design of protein architectures with reinforcement learning. Science, 38066442)226-2773 2023.
Toulon, France, April 24-26, 2017,
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Meirom, E., Maron, H., Mannor, S., and Chechik, G. Con- trolling graph dynamics with reinforcement learning and graph neural networks. In International Conference on Machine Learning, pp. 7565-7577. PMLR, 2021.
Leskovec, J. and Krevl, A. SNAP Datasets: Stan- ford large network dataset collection. http://snap. stanford. edu/ dat June 2014.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL http://arxiv.crs abs 0bs/1312.5500
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0m 00000, 2020.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540):
Lin, J. A., Antorán, J., Padhy, S., Janz, D., Hernández- Lobato, M., and Terenin, Sampling from gaussian process posteriors using stochastic gradient descent. Ad- vances in Neural Information Processing Systems, 36: 36886-36912, 2023.
529-533, 2015.
Liu, C., Ding. J., Song. Y., and Li, Y. Tdnetgen: Empower- ing complex network resilience prediction with generative augmentation of topology and dynamics. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1875-1886, 2024a.
Moerland, T. M., Broekens, J., Plaat, A., Jonker, C. M., al. Model-baid basel reinforcement learning: A survey, Foundations and Trends@ in Machine Learning, 16(1): 1-118, 2023.
Nautiyal, K. M., Tanaka, K. F., Barr, M. M., Tritschler, L.,
Liu, C., Xu, F., Gao, C., Wang, Z., Li, Y., and Gao, Deep learning resilience inference for complex networked systems. Nature Communications, 15(1):9203, 2024b.
Le Dantec, Y., David, D. J., Gardier, A. M., Blanco, C., Hen, R., and Ahmari, S. E. Distinct circuits underlie the effects of 5-htlb receptors on aggression and impulsivity. Neuron, 86(3):813 826, 2015.
Liu, Q., Yu, T., Bai, Y., and Jin, C. A sharp analysis of model-based reinforcement learning with self-play. In International Conference Machine Learning, pp. 7001- 7010. PMLR, 2021.
Rossi, R. A. and Ahmed, N. K. The network data repos- itory with interactive graph analytics and visualization. In Bonet, B. and Koenig, S. (eds.), Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pp. 4292- 4293. AAAI Press, 2015. doi: 10.1609/AAAI 9277. URL https: /tt::///oo..e http::/://oio.og1...1111111 wtt.g///...11...1
Lowe, R., WU, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed environments. Advances in neu- ral information processing systems, 30, 2017.
v2911. 9277.
Lowrey, K., Rajeswaran, A., Kakade, S., Todorov, E., and Mordatch, Plan online, learn offline: Efficient learning and exploration via model- model-based control. In International Conference on Learning Representations, 2019.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 170.00770 2017.

---

Reinforcement Learning with Adaptive Reward Modeling for Expensive-to-Bvalutt Systems
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J. Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint wrXiv:2402.03300 2024.
level generation. In The Twelfth International Conference on Learning Representations, 2024.
Wu, T., Maruyama, T., Zhao, Q. Wetzstein, G., and Leskovec, J. Learning controllable adaptive simu- lation for multi-resolution physics. arXiv preprint arXiv:2305.00 02230000.12,,, 2023.
Shetty, S., Xue, T., and Calinon, S. Generalized policy iteration using tensor approximation for hybrid control. In The Twelfth International Conference on Learning Representations, 2024.
Xu, S. and Liu, G. Uncertainty aware constraint inference in inverse constrained reinforcement learning. In The Twelfth International Conference on Learning Represen- tations, 2023.
Silver, D., Huang,
A., Maddison, C. J., Guez, A., Sifre, L.., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam,, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nalure, 529(7587):444-489, 2016.
Yang, S., Hwang, D., Lee, S., Ryu, S., and Hwang, S. J. Hit and lead discovery with explorative rl and fragment- based molecule generation. Advances Neural Information Processing Systems, 34:7924-7936, 2021.
Silver, D., Singh, S., Precup, D., and Sutton, R. S. Reward is enough. Artificial Intelligence, 199:103535 2021.
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 11114199-14442 2020.
Tong, H., Prakash, B. A., Eliassi-Raa Rad, T., Faloutsos, M., and Faloutsos, C. Gelling, and melting, large graphs by edge manipulation. In Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 245-2544 2012.
Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-289611 2021a.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 009002307.00928, 2023.
Yu, Z., Lu, S., Wang, D., and Li, Z. Modeling and analysis of rumor propagation in social networks. Information Sciences, 580:857-877 873, 2021b.
Trott, O. and Olson, A. J. Autodock vina: improving the speed and accuracy of docking with new scoring fune- tion, efficient optimization, and multithreading. Journal of compulational chemistry, (1(2):155 2010.
Yuan, Y., Ding. J., Wang, H., Jin, D., and Li, Y. Activity tra- jectory generation via modeling spatiotemporal dynamics. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4752-4762, 2022.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systemx, 2017.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M.,, Dudzik, A., Chung. J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in stareraft ii using multi-agent reinforcement learning. nature, 575
Yuan, Y., Wang, H., Ding, J., Jin, D., and Li, Y. Learning to simulate daily activities via modeling dynamic human needs. In Proceedings of the ACM Web Conference 2023, pp. 906-916 2023.
(7782):350-354, 2019.
Yuan, Y., Ding, J., Jin, D., and Li. Y. Learning the com- plexity of urban mobility with deep generative network. PNAS nexus, 4(5):pga)081, 2025.
Wan, R., Zhang, X., and Song, R. Multi-objective model- based reinforcement learning for infectious disease con- trol. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 1634-1644, 2021.
Zhang, H., Feng, S., Liu, C., Ding, Y., Zhu, Y., Zhou, Z., Zhang, W., Yu, Y., Jin, H., and Li, Z. Cityflow: A multi- agent reinforcement learning environment for large scale city traffic scenario. In The world wide web conference, pp. 3620-3624, 2019.
Wang, Q. and Van Hoof, Model-based meta reinforce- ment learning using graph structured surrogate models and amortized policy search. In International Conference PMLR, 2022.
Zheng Y., Lin, Y., Zhao, L., Wu, T., Jin, D., and Li, Y. Spatial planning of urban communities via deep rein- forcement learning. Nature Computational Science, 3(9): 748-762, 2023.
on Machine Learning, pp.
Wang, Z., Hu, C., Liu, J., and Yao, X. Negatively correlated ensemble reinforcement learning for online diverse game
12

---

Reinforcement Learning with Adaptive Reward Modeling for e Systems
Zhou, Z.,, Kearnes, S., Li, L., Zare, R. N., and Riley, P. Op- timization of molecules via deep reinforcement learning. Scientific reports, 9(1):1072, 2019.
13


Reviews:
[{"review_id": "cpFFOmXvhR", "reviewer": "Reviewer_Q9Xy", "summary": "The authors propose **Adaptive Reward Modelling (AdaReMo)**, an approach that accelerates Reinforcement Learning (RL) by fine-tuning a reward model (RM) multiple times during training. This reduces the need for expensive reward evaluations. In AdaReMo, the RL agent interacts only with the learned RM, while a parallel process computes ground-truth rewards (which are expensive to obtain) to gradually build a fine-tuning pool. Once this pool reaches a threshold (after a fixed number of RL iterations), the RM is fine-tuned for a pre-defined number of epochs and then redeployed in the RL loop. To ensure reliability, the RM is pre-trained for several epochs before any policy optimization begins. PPO is used as the underlying RL algorithm.\n\nThrough evaluations on tasks from **Molecular Generation**, **Epidemic Control**, and **Urban Spatial Planning**, the paper shows that AdaReMo outperforms classical,  model-free and model-based RL baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Q9Xy,\n\nWe would like to express our sincere gratitude for your thorough review and constructive feedback on our paper. We are particularly pleased that you have recognized the effectiveness of our proposed approach, AdaReMo, in solving the efficiency bottleneck between policy optimization and computationally expensive reward calculation. We hope the following responses address your concerns.\n\n**Q1:** Is the warm-up cost included in Figure 6a? A brief discussion of the warm-up cost tradeoff might be valuable.\n\n**A1:** Thank you for your insightful comments. Figure 6a illustrates the training process from when policy optimization begins, and the warm-up cost is not included. Specifically, the model warm-up takes only one fine-tuning interval, accounting for only **3% of the total optimization time**. We also conducted additional ablation experiments by removing the warm-up phase. Without model warm-up, we observed a performance **decrease of 7% across the evaluated metrics**. This decline arises because the policy is initially trained with randomly generated rewards, which introduces additional noise and hinders optimization. These results underscore the critical role of model warm-up in enhancing training efficiency, leveraging a modest (3%) time investment to yield substantial improvements in model performance. We will add the above results and further discussion into a new ablation study section (Section 5.6) of the revised version.\n\n\n**Q2:** Typos and grammatical errors.\n\n**A2:** We sincerely thank the reviewer for their thorough and careful scrutiny. In response to your feedback, we have thoroughly reviewed the manuscript, correcting all identified typos and grammatical mistakes.\n\n\nWe hope the above replies resolve your concerns and welcome any further questions.", "claims_and_evidence": "The claims about the effectiveness of AdaReMo are well-supported by the empirical results.", "methods_and_evaluation_criteria": "The methodology and evaluation criteria are appropriate and well-justified.", "theoretical_claims": "No formal theoretical results are presented, but the approach is grounded in sound reasoning.", "experimental_designs_or_analyses": "The experimental design is clearly described and appears sound.", "relation_to_broader_scientific_literature": "**Key Contribution**:\n- A method for *adaptive and efficient reward modelling* that avoids expensive reward evaluations by strategically fine-tuning a reward model.\n\nThis aligns with ongoing work in sample-efficient RL and learned reward estimation, while targeting real-world domains with costly simulation steps.", "essential_references_not_discussed": "None that are obviously missing.", "other_strengths_and_weaknesses": "**Strengths**:\n- Clear, concise writing.\n- The algorithm design is sensible and addresses a practical bottleneck in real-world RL settings.\n\n**Weaknesses**:\n- No major weaknesses observed.", "other_comments_or_suggestions": "- Line 235: \"reduce-scaled direct evaluations\" → *reduced-scale direct evaluations*\n- Line 301: \"records\" → *record*\n- Line 313: \"have\" → *has*\n- Line 319: \"KDE\" → *KED*\n- Line 404: \"molecular\" → *molecules*", "questions_for_authors": "1. **Is the warm-up cost included in the plots in Figure 6a?**  \n   Clarifying this would help interpret the reported efficiency gains of AdaReMo compared to the baselines. If it is excluded, a brief discussion of the warm-up cost tradeoff might be valuable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fedgRniSm9", "reviewer": "Reviewer_pywE", "summary": "The paper proposes **AdaReMo (Adaptive Reward Modeling)**, a reinforcement learning (RL) framework designed specifically for **expensive-to-evaluate reward systems**, such as molecular generation, epidemic control, and spatial planning. The key innovation is the adaptive decomposition of complex, computationally expensive reward functions into localized, neural network-based reward models (RMs). AdaReMo dynamically updates these RMs by fine-tuning them on recent agent trajectories, thus ensuring accurate reward estimations aligned with the agent’s evolving policy. This approach effectively separates fast policy decisions from slow, costly reward evaluations by handling them asynchronously through online and offline systems. Empirical results demonstrate that AdaReMo achieves significant speedups (over 1,000x faster than baselines) and performance improvements (around 14.6%) across the tested tasks, confirming its efficacy in addressing the computational bottleneck present in RL for real-world systems.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer pywE,\n\nWe express our sincere gratitude for your thorough review and valuable feedback on our paper. Regarding the potential alternative algorithms and the issue of online and offline splits you mentioned, we hope the following replies can address your concerns.\n\n\n**Q1:** REINFORCE/REINFORCE++, RLOO and GRPO serve as substitutes for the proposed AdaReMo.\n \n**A1:** Thanks for your constructive comments. We would like to provide more clarification on the necessity of the proposed AdaReMo compared to substitutes like RLOO and GRPO with additional experiments.\nIn fact, although the introduction of the reward model in AdaReMo draws inspiration from RLHF, its primary focus diverges from the RL optimization algorithms mentioned, such as REINFORCE, RLOO, and GRPO. For instance, GRPO avoids using a critic model to compute advantages due to the typically large size of value networks in RLHF. In contrast, the value network in AdaReMo is significantly smaller, rendering the removal of the critic model less critical. Instead, **AdaReMo targets the efficiency imbalance caused by a computationally expensive reward function**, which sets it apart from these alternatives.\n\nTo evaluate the optimization efficiency of various algorithms, we conducted experiments by replacing PPO with RLOO and GRPO. The results are summarized in the table below:\n\n\n||Molecular Generation(FA7)|Pandemic Control(CA-GrQc)|Urban Spatial Planning(HLG)\n|-|-|-|-|\n||T5$\\uparrow$ \\| HR$\\uparrow$|H$\\uparrow$ \\| C$\\uparrow$|D$\\downarrow$ \\| G$\\uparrow$\n| RLOO | 10.2$\\pm$ 0.4 \\| 0.25$\\pm$ 0.02 | 34.6$\\pm$ 3.8 \\| 9.9$\\pm$ 1.5 | 3.66$\\pm$ 0.38 \\| 2.56$\\pm$ 0.36|\n| GRPO | 9.6$\\pm$ 0.2 \\| 0.27$\\pm$ 0.03|33.1$\\pm$ 4.7 \\| 8.7$\\pm$ 2.1|3.31$\\pm$ 0.24 \\| 2.28$\\pm$ 0.39|\n|PPO| **10.5**$\\pm$ 0.6 \\| **0.31**$\\pm$ 0.06 | **39.4**$\\pm$ 5.7 \\| **10.6**$\\pm$ 2.9 | **2.88**$\\pm$ 0.23 \\| **2.80**$\\pm$ 0.42 |\n\n\nAs demonstrated, **AdaReMo combined with PPO consistently outperforms RLOO and GRPO** across multiple metrics and domains. While RLOO and GRPO eliminate the need for a critic model, **they require multiple sampling**, which increases computational complexity and hinders training efficiency. Furthermore, **the training time per iteration for RLOO and GRPO does not significantly differ from that of PPO**, suggesting that optimizing the critic model is not the primary bottleneck in training efficiency. Additionally, GRPO’s method of calculating advantages **can amplify minor differences**, potentially destabilizing the policy optimization process.\n\nIn summary, these results validate the effectiveness of AdaReMo in addressing reward function complexity. Thank you again for your valuable feedback, and we will incorporate the above discussion and experiments into the appendix of the revised manuscript.\n\n\n**Q2:** Why detach the reward modeling to online/offline splits.  \n\n**A2:** Thank you for your comment. The online/offline splits are proposed to **address the computationally expensive reward functions**, rather than reducing the computational cost of solution generation. Specifically, **the online system** leverages the reward model to deliver rapid reward estimations, enabling the agent to receive **real-time feedback**. Without this mechanism, the agent would face significant slowdowns due to the time-intensive reward calculations, as exemplified by applications like RLGN (Meirom et al., 2021) for pandemic control. Conversely, **the offline system** continuously refines the reward model using the latest exploratory samples **to ensure its accuracy**. In the absence of this offline phase, the reward model would fail to provide reliable evaluations for samples generated by the evolving policy, leading to significant errors throughout the optimization process, as illustrated in Figure 6a. In summary, the online/offline splits **effectively balance real-time efficiency with long-term accuracy**.\n\nThank you again for your valuable comment, and we will include the above discussion in Sections 4.3 and 5.4 of the revised manuscript.\n\n\n**Q3:** Typo in Eq4 and Eq7.\n\n**A3:** Thanks for pointing out this issue. We have corrected it in the revised manuscript. Specifically, Eq4 should be $s_i = \\text{MLP}_p(\\bf{a}_i)$, where $\\bf{a}_i$ denotes the embeddings of action ${a}_i$.\nWe have checked the entire paper and added following corrections as Reviewer kWsg suggested,\n- Eq1. $\\max_{\\Theta} E_{\\pi_\\Theta} \\left[ \\sum_{t=0}^{T} \\gamma^t r(s_t, a_t) \\right]$\n- Eq5. $\\nabla_{\\Theta} J(\\Theta) = E \\left[ \\min \\left( r_t(\\pi_\\Theta) \\hat A_t, \\text{clip}(r_t(\\pi_\\Theta), 1 - \\epsilon, 1 + \\epsilon) \\hat A_t \\right) \\right]$ \n\nWe hope these responses fully address your concerns and welcome any further questions or suggestions you may have.", "claims_and_evidence": "Training reinforcement learning (RL) agents requires extensive trials and errors, which becomes prohibitively time-consuming in systems with costly reward evaluations. \n\nThe major claim that this paper makes is that the proposed adaptive reward modeling (AdaReMo) paradigm surpasses existing methods in significantly reducing computational overhead and optimality. \n\nI am not fully convinced by this claim since there are several RL optimization algorithms that serve as substitutes for the proposed AdaReMo as well as the compared baselines in the paper. Those RL algorithms include REINFORCE/REINFORCE++, RLOO, GRPO, that circumvents the need for a critic model and brings efficiency.", "methods_and_evaluation_criteria": "In Eq (7), all the node embeddings are averaged and sent to the MLP layer to calculate the reward, which is inconsistent with Eq (4) where each edge is regarded as an action.", "theoretical_claims": "I did check all the equations.", "experimental_designs_or_analyses": "I checked all the experimental designs and analyses. The major concern I have is for including other RL optimization algorithms for comparison.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "[+] The paper is well organized and carefully written, making it quite easy to follow.\n[+] The design seems overall reasonable to me. Detaching the reward model to online split and offline split should improve the efficiency of the RL online learning process.\n\n[-] The motivation is still not quite clear to me. If the reward model is of the same size as the policy model (same as the case in RLHF for LLMs), then it is acceptable to obtain the reward. In this paper, the GNN policy and reward model seems even more tiny than most of the open-source LLMs. It is not clear to me why detach the reward modeling to online/offline splits.", "other_comments_or_suggestions": "Typo:\n\nIn Eq (7), the h and a should be flipped?", "questions_for_authors": "See other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dgmJ7bYG7p", "reviewer": "Reviewer_kWsg", "summary": "This work deals with the RL setting in which we have access to a\nreasonably fast simulator of the system, but the reward evaluations\nare slow. They employ the idea of modeling the reward to enable taking\nadvantage of the fast simulation to update the policy.  In particular,\nthey use an asynchronous scheme where they are running multiple true\nreward evaluations in parallel that are added to a buffer for training\nthe reward model. The states for reward evaluation are sampled from a\nbuffer of recent states encountered in policy optimization.  They\noptimize the policy using PPO and their trained reward model.  The\ntraining process also employs a warmup phase for initial reward model\ntraining, as in the beginning the reward model is inaccurate and can\npotentially lead to detrimental policy learning.  Another part of\ntheir proposal is adding a correction term to the reward during policy\noptimization; in particular, even though computing the precise reward\nis slow, there exists a faster surrogate $r_c$ for the tasks that they\nconsider. As their learned reward model may be wrong, they blend this\nsurrogate into their reward prediction to aid with dealing with\noutlier data $\\alpha r + (1-\\alpha) r_c$.  They consider the tasks of\nmolecular generation, pandemic control and urban spatial planning. As\nall of these tasks involve a graph-based structure, they employ Graph\nNeural Networks for the policy.  The performance improved across\nseveral baselines (by around ~10%) including classic approaches as\nwell as the model-based RL algorithm MBPO. The ablation study showed\nthat the method is sensitive to the number of fine-tuning intervals\nand epochs for training the reward model.\n\n\n-------------------------------------------------------\nUpdate:\nThank you for the response; this addresses most of my concerns, so I increased the score.\n\nOne point that still caught my eye was that the results you currently reported were based on the value at the best iteration; however, in the paper some of the learning curves were quite erratic, oscillating up and down, so even if the best iteration result is good, the learning may still be unstable (at least from the presented result, I can't rule out this possibility). However, perhaps in terms of your application, you are mainly interested in the best result, so I can also see it being argued the other way.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Experiments with more seeds should be reported.\n\n**A1:** In our submission, we reported results based on 3 seeds. To address your concern, we have now conducted experiments with **10 seeds**. The results (see table in comments for details) confirm that our method remains **robust across a larger number of seeds** and consistently **outperforms the SOTA baselines** across all evaluated scenarios. Notably, the improvements is more evident with 10 seeds, as the reduced standard deviations indicate greater statistical reliability. \n\nWe will update the results in Section 5 to further validates the effectiveness and robustness of our approach.\n\n**Q2:** More ablation study.\n\n**A2:** We have conducted additional ablation studies on the module described in Section 4.4 and the results are as follows: \n\n| Full | w/o Synchronous Correction | w/o Model Warm-up | w/o Parallel Computation |\n|-|-|-|-|\n|10.5$\\pm$ 0.6 \\| 0.31$\\pm$ 0.06 | 10.1$\\pm$ 0.8 \\| 0.29$\\pm$ 0.07 | 9.8$\\pm$ 0.2 \\| 0.24$\\pm$ 0.04 | 9.5$\\pm$ 0.5 \\| 0.29$\\pm$ 0.03 |\n\nRemoving any component results in a performance decline. Notably, excluding **model warm-up** leads to random rewards in the initial iterations, causing suboptimal policy optimization and a 7% drop in performance. Similarly, removing **parallel computation** reduces training efficiency by limiting the reward model’s access to sufficient samples for accurate fine-tuning, resulting in an approximate 10% performance decrease. These results show that each component contributes to the final performance. We will include these results into a new ablation study section (Section 5.6) of the revised version.\n\n**Q3:** Only provides result for the selected hyperparameter, which is unreliable.\n\n**A3:** Besides the grid search results in Figure 8, we would like to provide **practical guidelines** that enable users to efficiently estimate an **approximate range for the optimal hyperparameters**, ensuring the method’s reliability and reproducibility without exhaustive tuning.\n\nFirst, the optimal number of **fine-tuning interval** can be approximated by the complexity of the reward function, which is **positively correlated with its evaluation time**. For example, Table 2 indicates that reward computation in pandemic control is **more time-consuming** than molecular design, and grid search identified an optimal fine-tuning interval of 9 and 7 (Figure 8a) for the two scenarios, respectively, suggesting that a more complex reward function requires additional samples for effective RM fine-tuning. \n\nSecond, the optimal number of **fine-tuning epoch** should balance the fine-tuning duration with the policy optimization (see Section 5.5). Specifically, we can measure the time $t_1$ required for one sampling and policy optimization iteration. Then, based on the previously determined fine-tuning interval and the reward function’s computation time, we can estimate the number of samples needed for fine-tuning and compute the time $t_2$ per fine-tuning epoch. **The theoretically optimal number of epochs is $N \\approx \\frac {t_1}{t_2}$**, providing a robust starting point. Search within a small neighborhood of this estimated value typically yields the optimal setting, as validated in our experiments.\n\nWe will add the above guidelines on hyper-parameters as Section 4.5 of the revised version.\n\n**Q4:** The RM (reward model) at late-stage becomes worse at predicting the early rewards, using early reward to fine-tune may help.\n\n**A4:** Thanks for your suggestion. We would like to first clarify that the reward prediction error you have observed **does not undermine the performance of our method**, as the RM’s role is to provide accurate evaluations for the current policy rather than historical ones. Moreover, it is worth emphasizing that Figure 7 showcases our AdaReMo enables the RM to adapt to the current policy’s outputs, as evidenced by **near-zero error values along the diagonal elements**.\n\nTo further address this, we include samples from previous fine-tuning cycles into RM updates. The updated results for Figure 7 are shown below:\n\n| RM\\Precise Reward | 50 | 100 | 150 | 200 |\n|-|-|-|-|-|\n| 50|**0.87%/1.92%**|3.81%/4.47%|8.13%/5.67%|13.16%/10.41%|\n| 100|9.47%/9.12%|**1.32%/1.59%**|8.61%/3.55%|26.78%/16.71%|\n| 150|12.35%/8.21%|4.13%/5.19%|**1.31%/2.10%**|7.25%/6.33%|\n| 200|4.21%/7.92%|5.04%/6.30%|2.74%/1.52%|**0.04%/0.38%**|\n\nThe first value uses only current-cycle samples; the second includes previous-cycle samples. Though off-diagonal errors decreased after using historical samples, **diagonal errors increased significantly**, reducing the RM’s ability to accurately evaluate the current policy and **resulting in a 5% performance drop**. This confirms that focusing RM updates on current samples enhances overall optimization effectiveness.\nWe have included the above experiments into Section 5.4 of the revised manuscript.\n\n**Q5:** Equation expressions.\n\n**A5:** Please see our response to Reviewer pywE’s Q3.", "claims_and_evidence": "There are many claims about improved performance, and they also introduce\nmethods such as blending the reward value estimates, the GNN model, etc.\n\nCurrently, I am not convinced by the claims as the number of seeds is\neither low, or not specified. For the molecular task, they said they\nrepeated the experiment with 3 seeds, which is typically considered\nvery low (e.g., see https://arxiv.org/abs/2108.13264 Deep\nReinforcement Learning at the Edge of the Statistical Precipice).\n\nMoreover, some ablations are missing, e.g., an ablation for the\nblended reward in equation 9 is missing.\n\nA major concern for me is the sensitivity study in Figure 8 about the\nnumber of fine-tuning intervals (tested in [1, 3, 5, 7, 9]) and the\nnumber of epochs (tested in [10, 20, 30, 40, 50]). The method only\nprovides reasonable performance for the selected hyperparameter. This\nindicates to me that the method is unreliable. Moreover, if for example\nthe results in the other figures were based on selecting the max results\nacross such a sweep, then we would expect a maximization bias leading to\nunreliable results.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria make sense. The choice of tasks\nto tackle is a strong point of this paper as it shows interesting applications\nof reinforcement learning.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes, I checked them.\nSee the Claims and Evidence section for details for the issues.\nThere are issues with too few seeds used in the experiments and\nmissing ablations.", "relation_to_broader_scientific_literature": "The idea of creating a reward model for tasks where the reward evaluation\nis slow is a simple one, and it exists in prior work as well. What I found\ninteresting in the current work was the choice of tasks. The approach\nitself is a sensible engineering solution to the tasks that they consider.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Other strengths:\nThe ideas are reasonable from a practical point of view.\nI liked the choice of tasks.\n\nOther weaknesses:\nThe clarity could be improved in some places, and there are some\ngrammatical errors.\n\nRegarding clarity, a few examples are: when you first introduce synchronous\ncorrection, it's not clear what the correction term is (one has to read\nthe experimental sections to understand, and it may be better to add early\npointers or keep it self-contained), the number of seeds is not clear.", "other_comments_or_suggestions": "Grammatical errors:\nPage 1: \"hinders\" -> \"hinder\"\n\nAround Line 301: \"records\" -> \"record\"\n\nPage 6: \"simulates\" -> \"simulate\"\n\n\"To further visualize the consistency of between the RM\nand the agent, we calculate the errors between RM estima-\ntions and precise evaluation during the training process.\"\nFix the grammar.\n\nEquation 1 for the objective seems incorrect, as it does not take into\naccount the randomness (you define transition probabilities P, so it\nseems you assume randomness in the dynamics).\n\nEquations 5 and 6 are not PPO, they are REINFORCE.\n\n\nSuggestions:\n\nIf you look at Figure 7, it seems that as the training progresses,\nin the middle, the model becomes worse at predicting the early rewards.\nPerhaps this could be fixed by training the reward model on the stored\nrewards from the early stage as well? The model predictive performance could\nprobably be made to not drop much on the early data.", "questions_for_authors": "How many seeds did you use for all experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UuIOtbzDfv", "reviewer": "Reviewer_joeE", "summary": "The paper presents AdaReMo, an approach designed to accelerate reinforcement learning (RL) in systems where reward evaluations are computationally expensive. The key idea is to decouple the RL loop—where decisions are made quickly—from the reward evaluation process that is slow and costly. AdaReMo achieves this by introducing a neural network–based reward model (RM) that approximates the true reward function. To handle the complexity and variability of real-world reward functions, the approach adaptively decomposes the overall reward function into multiple localized reward models that are trained on the agent’s most recent exploratory data. This adaptive reward modeling is integrated into an asynchronous training framework where the online decision system rapidly collects trajectories using fast RM predictions, while an offline evaluation system periodically updates the RM using precise but expensive reward calculations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer joeE,\n\nThank you for your thoughtful and constructive feedback. We appreciate your recognition of the core idea behind AdaReMo, particularly the decoupling of the fast decision-making loop from the slow, expensive reward evaluations. We hope the following responses can address your concerns.\n\n**Q1:** Hyperparameters settings.\n\n**A1:** Thanks for your constructive comments. \nBesides the grid search results in Figure 8, we would like to provide **practical guidelines** that enable users to efficiently estimate an **approximate range for the optimal hyperparameters**, ensuring the method’s reliability and reproducibility without exhaustive tuning.\n\nFirst, the optimal number of **fine-tuning interval** can be approximated based on the complexity of the reward function, which is **positively correlated with its evaluation time**. For example, Table 2 indicates that reward computation in pandemic control is **more time-consuming** than molecular design, and grid search identified an optimal fine-tuning interval of 9 and 7 (Figure 8a) for the two scenarios, respectively, suggesting that a more complex reward function requires additional samples for effective RM fine-tuning. \n\n\nSecond, the optimal number of **fine-tuning epoch** should balance the fine-tuning duration with the policy optimization (Section 5.5). Specifically, we can measure the time $t_1$ required for one policy optimization iteration. Then, based on the previously determined fine-tuning interval and the reward function’s computation time, we can estimate the number of samples needed for fine-tuning and compute the time $t_2$ per fine-tuning epoch. **The theoretically optimal number of epochs is $N \\approx \\frac {t_1}{t_2}$**, providing a robust starting point. Search within a small neighborhood of this estimated value typically yields the optimal setting, as validated in our experiments.\n\nWe sincerely appreciate your constructive comment, and we have added the above guidelines on hyperparameters in Section 4.5 of the revised manuscript.\n\n\n**Q2:** Are the improvements robust across a wider variety of expensive-to-evaluate systems?\n\n**A2:** Thank you for your insightful question. The proposed AdaReMo is designed as a general solution for expensive-to-evaluate systems. To demonstrate its versatility, we conducted extensive experiments across three quite diverse scenarios: molecular design, pandemic control, and urban planning. The choice of tasks has also been acknowledged by Reviewer kWsg.\nWe sincerely appreciate your suggestion and consider the inclusion of a broader range of expensive-to-evaluate tasks an important direction for extending our approach. In particular, we plan to investigate the effectiveness of AdaReMo in additional domains such as robotics and autonomous driving.\n\nThank you once again for your valuable comments, please let us know if you have additional questions.", "claims_and_evidence": "The paper claims that AdaReMo can decouple the fast decision-making loop from the slow, expensive reward evaluations, achieving over 1,000× speedup and about 14.6% performance improvement across three distinct real-world tasks. The extensive experiments in molecular generation, epidemic control, and urban spatial planning provide strong empirical support for these claims. However, one might ask whether the reported gains depend heavily on specific task formulations or the particular settings (e.g., the chosen hyperparameters and the design of the reward model) used in the experiments. Are the improvements robust across a wider variety of expensive-to-evaluate systems?", "methods_and_evaluation_criteria": "The idea of decoupling the online decision system from the offline evaluation system is a natural fit for scenarios where reward computations are very expensive. Using an adaptive neural reward model that fine-tunes based on recent exploratory data is an inventive solution to keep the online loop fast. The evaluation criteria—such as Top 5% Score and Hit Ratio for molecular generation, Healthy and Contained metrics for epidemic control, and accessibility/greenness metrics for spatial planning—are well chosen to reflect the quality of decisions under high computational cost.", "theoretical_claims": "The paper primarily emphasizes an empirical demonstration of its ideas rather than formal theoretical proofs. The conceptual justification for adaptive reward modeling is well articulated, yet there isn’t a formal mathematical proof guaranteeing, for instance, convergence or error bounds for the reward model approximation.", "experimental_designs_or_analyses": "The experiments span three different domains, each chosen to represent a class of expensive-to-evaluate tasks. The paper performs ablation studies on key hyperparameters (e.g., fine-tuning interval and epochs), which helps to understand the sensitivity of the adaptive reward model.", "relation_to_broader_scientific_literature": "The paper builds on ideas from surrogate modeling and model-based RL, and it resonates with recent work in reinforcement learning from human feedback (RLHF) by adapting techniques to approximate expensive evaluations.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- The decoupling of the RL loop into online and offline components using an adaptive reward model is an interesting solution to a long-standing computational bottleneck in RL.\n- By demonstrating significant speedup and performance gains on challenging real-world tasks, the paper makes a strong case for its practical relevance.\n- The paper is clear in describing the architecture, algorithm (with pseudocode in Algorithm 1), and the adaptive training framework. Detailed experimental results and visualizations (e.g., Figures 3–8) further aid understanding.", "other_comments_or_suggestions": "No comments", "questions_for_authors": "No questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission proposes AdaReMo, a framework for reinforcement learning in domains where reward evaluations are computationally expensive. The key contribution is a modular design that decouples the slow reward evaluation process from the faster policy optimization loop via adaptive, fine-tuned reward modeling. The authors demonstrate the effectiveness of their approach across diverse and challenging domains—molecular generation, epidemic control, and spatial planning—with substantial empirical gains in both training efficiency and policy performance.

The reviews were overall positive. Three reviewers (Q9Xy, kWsg, and joeE) recommended acceptance, citing the paper's well-motivated problem, clear design, and promising empirical results. Reviewer pywE initially expressed concerns regarding baseline comparisons and the motivation for splitting the reward system into online/offline modules. However, the authors’ rebuttal was comprehensive and addressed these concerns convincingly by providing additional experimental baselines (RLOO and GRPO), detailed ablation studies, and fine-grained sensitivity analyses. In addition, Reviewer kWsg subsequently raised their score and acknowledged that the concerns regarding hyperparameter sensitivity had been sufficiently addressed.

While some lingering concerns remain (e.g., training stability and robustness over reward model parameters), the authors have provided detailed empirical evidence (including results from ten random seeds and additional parameter sweeps), which demonstrates robustness and convergence. The learning curves are at times non-monotonic, but the performance at convergence is mostly consistent, and in practice, the best-iteration criterion is often a reasonable standard in domains with expensive evaluations. 

This paper provides contributions to the reinforcement learning community. Its modular and adaptive treatment of reward modeling offers a pragmatic solution to a growing class of real-world problems where reward signals are sparse, delayed, or expensive to compute. As a result, I recommend acceptance.
Decision: Accept (poster)