Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference
Authors: Matthew Riemer
Gopeshh Subbaraj
Glen Berseth
Irina Rish
Venue: ICLR 2025 Poster (2025)
Keywords: Realtime Environments
Asynchronous Algorithms
Time Discretization
Real World Deployment
Deep Reinforcement Learning
Primary area: reinforcement learning

Abstract: Realtime environments change even as agents perform action inference and learning, thus requiring high interaction frequencies to effectively minimize regret. However, recent advances in machine learning involve larger neural networks with longer inference times, raising questions about their applicability in realtime systems where reaction time is crucial. We present an analysis of lower bounds on regret in realtime reinforcement learning (RL) environments to show that minimizing long-term regret is generally impossible within the typical sequential interaction and learning paradigm, but often becomes possible when sufficient asynchronous compute is available. We propose novel algorithms for staggering asynchronous inference processes to ensure that actions are taken at consistent time intervals, and demonstrate that use of models with high action inference times is only constrained by the environment's effective stochasticity over the inference horizon, and not by action frequency. Our analysis shows that the number of inference processes needed scales linearly with increasing inference times while enabling use of models that are multiple orders of magnitude larger than existing approaches when learning from a realtime simulation of Game Boy games such as Pokemon and Tetris.

Published as a conference paper at ICLR 2025
ENABLING REALTIME REINFORCEMENT LEARNING AT SCALE WITH STAGGERED ASYNCHRONOUS INFERENCE
Matthew Riemer* Gopeshh Subbaraj* Glen Berseth Irina Rish 1.2 Mila, Universiiéé de Montréal, and 'IBM Research
ABSTRACT
Realtime environments change even as agents perform action inference and learn- ing, thus requiring high interaction frequencies to effectively minimize regret. However, recent advances in machine learning involve larger neural networks with longer inference times, raising questions about their applicability in realtime systems where reaction time is crucial. We present an analysis lower bounds on regret in realtime reinforcement learning (RL) environments to show that minimiz- ing long-tenn term regret is generally impossible within the typical sequential interaction and learning paradigm, but often becomes possible when sufficient asynchronous compute is available. We propose novel algorithms for staggering asynchronous inference processes ensure that actions are taken at consistent time intervals, and demonstrate that use of models with high action inference times is only constrained by the environment's 's effective stochasticity over the inference horizon, and not by action frequency. Our analysis shows that the number of inference processes needed scales linearly with increasing inference times while enabling use of models that are multiple orders of magnitude larger than existing approaches when learning from a realtime simulation of Game Boy games such as Pokémon and Tetris
INTRODUCTION
An often ignored discrepancy between the discrete-time RL framework and the real-world the fact that the world continues to evolve even while agents are computing their actions. As result, agents are limited in the types of problems that they can solve because the speed at which they can compute actions dictates a particular stochastic deterministic time diseretization rate. Agents that take infrequent actions require some lower-level program to manage behavior between actions, often through simple policies like remaining still or repeating the last action. Ideally, intelligent agents would exert more control over their environment, but this conflicts with the trend of using larger models, which have high action inference and learning times. Consequently, deployed with sequential interaction, large models, which are often found to be essential for complex tasks, increasingly rely on low-level automation, reducing their control over realtime environments. This paper examines this discrepancy and explores alternative asynchronous interaction paradigms, enabling large models to act quickly and maintain greater control high-frequency environments.
typically
Figure la shows the standard sequential interaction paradigm of RL. In this setup, the agent receives a state from the environment, learns from the state transition, and then infers an action. Each process must be completed before the agent can process new state, limiting the action frequency and increasing reliance on low-level automation as the model size grows. In contrast, Figure 1b illustrates the asynchronous multi-process interaction paradigm we propose. Our key insight is that even models with high inference times can act at every step using sufficienlly many staggered inference processes. Similarly, sufficiently many asynchronous learning processes maintain rapid updates without blocking progress, despite high learning times. This work formalizes and empirically tests the benefits and limitations of this approach, making the following contributions:
1. We formalize how the choice of a particular time discretization induces a new learning problem and how that problem relates to the original problem in Definition
'Equal contribution. Direct correspondences (matthew. ri riemer, subbara 8mi queebe..

---

Published as a conference paper at ICLR 2025
Apest
Agent
"G
Infermace
Sax
LOuTTER
Infrome
(a) Sequential Interaction and Learning
(b) Asynchronous Multi-process Interaction and Learning
Figure 1: Frameworks for Environment Interaction in RL. a) The typical sequential interaction paradigm where both learning and action inference block the environment from moving forward. b) The more realistic setting considered in this work where the environment, the agent's inference process, and agent's learning process all proceed at their own rate and interact asynchronously. Multiple self-loops are depicted to denote multiple asynchronous processes. TM denotes the frequency of the environment, Ty denotes the frequency of each inference process, and Tc denotes the frequency of each learning process. Sequential interaction and learning has a frequency of TM + To + Tg.
2. We derive worst-case lower bounds on regret for solving the new problem rather than the original in Theorem 1, leading to conclude in Remark I that typical sequential interaction (Figure la) scales poorly with model size.
3. We propose novel methods for staggering asynchronous inference in Algorithms 1 and 2, addressing the poor scaling properties of sequential interaction (Remark 2).
4. We conduct comprehensive experiments to verify our theory, demonstrating the use of models that are orders of magnitude larger for realtime games like Pokémon and Tetris.
2 REGRET DECOMPOSITION IN REALTIME REINFORCEMENT LEARNING
Background- Sequential Interaction: Most RL research focuses on agents interacting sequentially with Markov Decision Process (MDP) [71; 94] Mssg (S, A. p. r). where S is set of states, A a set of actions, r(s, a) is a reward function with outputs bounded by Tmax. and p(s' [s, a) a state transition probability function. Agents take actions based on policy (als) that maps states to action probabilities parameterized by 0. An unrealistic implicit assumption of this setting is that the time between decisions is fixed and only depends on the MDP. It is also unrealistically assumed that the environment can be paused while the policy generates an action a from state s.
Asynchronous Interaction Environments:
The standard MDP formalism lacks a crucial element for realtime settings where the environment cannot be "paused," " and the agent interacts with asynchronously, described by Travnik et al. [102]. In this case, it is necessary to define the environment's 's behavior when the agent has not selected an action. We believe the most general solution is to preset default behavior if there is available action a, by the agent л at time- step f. This behavior follows a ~ {(s), where a E Ag is possibly from a different action space than A. requiring p and r to be defined over A U Ag. Now we can define an asynchronous MDP Masyee = (S, A, p, 3) as an extension of a sequential MDP Masq with the addition of the default behavior policy 3. Note that 33 does not need to be non- Mar-aaaooiann, because the state space should be defined to include any intermediate computations needed to generate the actions of B. Defining the default behavior as a policy is no more than a useful interpretation of what happens and is equivalent to saying the environment follows a Markov chain (55 when action is available where pd (s'[s) == EucA, p(s'|s, a) ()a(())) with expected reward 8 EneAs B(a ((lls)((s,))) a).
Time Discretization Rates:
The real environment evolves in continuous time, so we must define time discretization rates describe each component of the grent-environment interface in discrete steps. We treat the environment step time as a random variable TM with sampled values TM ~ TM and expected value 7M == E M) Similarly, the inference time of the policy for single action is another random variable T, with sampled values T ~ T and expected value Fe := ET(T)) We can
See Appendix B.5 for a comparison with action chunking methods that produce multiple actions at a time. "While policies in general could have adaptive computation times based on the state, this is relatively uncommon in the literature and will he left to future work for simplicity of the discourse.
2

---

Published as a conference paper at ICLR 2025
now introduce yet another random variable Ty with sampled values TI ~ 'I and expected value fr := E[TT] that is of particular importance to our work, representing the time elapsed between actions taken by the agent gy (rather than 3) in Masyne. This has been called variety of names in the literature including the interaction time, the action cycle time, and the inverse of the interaction frequency. What is very important to note for our purposes is that Tz need not be equal to MM Te. with the precise relation between these variables depending on the particular method of agent deployment. Establishing these three random variables now allows to define the decision problem induced by these choices related to the agent-environment boundary (see Figure for an illustration).
Definition 1 (Induced Delayed Semi-MDP) Any choice of random variables TM, TI, and Tg applied to an asynchronous MDP Masyne induces a delayed semi-MDP Maelay == (S,A, A.P. p.v. 3, TM, Tz, To) where the semi-MDP decision making steps associated with the actual decisions of the agent happen after [TI/ steps L in the ground asynchronous MDP Masyne- The semi-MDP is delayed with respect to Masyne because semi-MDP actions äj E A generated by x are equivalent actions that are delayed by [Ta/TM]] I in Masyee such that To so(āgls;) FT((aetTT//<x) (96) where si St- If [Ta/TM]] 1 the transition dynamics are & and reward dynamics are 8 for [Ta/TM] steps Masyne until 014[7o/7<d is applied.
In general, the optimal policy and optimal re- ward rate will not be the same for Masyne and Maelay. with Maelay incurring additional sub- optimality because of the coarse nature of the decision problem. That said, we have direct control over Tr and To, so it is of interest to understand how design decisions relate to the sub-optimality experienced. Chiefly, we are interested in understanding under what scenar- ios the optimal reward rate Masyne can still be achieved even when Fg >> FM- To do this, we focus on worst case lower bounds on regret iee. the unavoidable regret incurred because of the interaction defined by Maetay in the worst case scenario where 3 is always a suboptimal choice. The realtime regret Aeaatime (T) is the accumulated suboptimality in seconds relative following the optimal policy every discrete steps occurring after TM seconds in Masyne-
Time
MDP
State
Delayed Semi-MDP
Figure 2: Induced Delayed Semi-MDP. We il-
lustrate the semi-MDP described in Definition 1 following the style Figure 1 from Sutton et al. [95]. Masyne is depicted in purple and Mastay is depicted in blue. Actions delayed by the infer- ence time of the policy æ and the default policy B followed between selections.
Theorem 1 (Realtime Regret Decomposition) The accumulated realtime regret Amariime (T) over time of a delayed semi-MDP Maelay relative to the oracle policy in the underlying asynchronous MDP Masyne can be decomposed into three independent terms.
Amatimme (T) Atrorn (T) + A Dinaction (T) + Aella((T)
(1)
Djeam is the regret experienced even sequential environments as result of learning and exploration. The lower bound in the worst case is:"
Dteam E Qi ((VT/TI)
Dinaction ( expresses the regret as result following ( rather than optimal actions in Maryne- The lower bound and upper bound in the worst case is:
Amocttonn (T) (((T/Tz) x (fz - FM)/
Adetay (7) expresses the regret as a result of the delay of actions by i in the underlying asyn- chronous Masyne- The lower bound in the worst case is:
Dactay (T) ( Fx) E (Pminimax) TTa (oo/TM]]
(4)
3

---

Published as a conference paper at ICLR 2025
where Pminimax mingeS a max, ES p(s' s. a) is a measure of environment stochastic- ity and [T//TM] is the number of environment steps elapsed during action inference.
See Appendix B for formal proof of Theorem and our other findings. We believe this work is the first to formally state the regret decomposition in Equation 1. Note though that previous studies on real-world RL have highlighted the challenges of learning from limited samples, realtime inference, and managing system delays in scaling methods to realtime settings [19]. Equation 2 extends known lower bounds learning time [31], using the notation from Definition 1 to explicitly connect with continuous time. Notably, this bound depends on îz (not fg) and assumes learning can keep pace with the environment to learn from every interaction. Equation 3 provides a novel regret bound, formalizing the known suboptimality of interacting with realtime environments at a slower pace [102; 28; 73; 107; 106; 21J. This result highlights limitations of the sequential interaction paradigm.
Remark (Inaction of Sequential Interaction) When π and Masyne interact sequentially, = Ta such that in the worst case A iinection (T) E SM(T/Ta x (Tg - Fmm)/Tm. This implies that even as r + 0G, in the worst case A Dreattime (T)/T E imanticioonn (T)/ E (i(F FMM)/FA5F)
This means a realtime framework with sequential interaction cannot ensure that regret will eventually dissipate. Thus, explore asynchronous alternatives in the next section. Finally, Equation 4 highlights the key limitation in minimizing regret using asynchronous compute. Previous work established that suboptimality from delay in MDPs relates to the stochasticity in the underlying undelayed MDP [17; 64], focusing on communication delays inherent to the environment. Our focus, however, on delays caused by the agent's 's computations, which we can control. Thus, the emphasis on regret associated with the decision that leads to a particular value of To is novel. Since this term the only part of regret that depends on To. it helps identify which environments are manageable when To >> TM- In deterministic environments, there is no regret due to To as Pminimax 1, but in stochastic environments, the degree and temporal horizon of stochasticity determine what values of are tolerable. For simplicity, we present a looser bound here; a tighter bound is available in Appendix B. In summary, stochasticity with respect to actual rewards is what really matters.
3 ASYNCHRONOUS INTERACTION & LEARNING METHODS
Figure 3 highlights key differences between the standard sequential RL framework and the asyn- chronous multi-process process framework we propose. In the sequential framework, interaction and learning delay each other. In contrast, in the asynchronous framework that we propose, actions and learning can occur at every step with enough processes. However, actions are delayed and reflect past states, which may limit performance in some environments. Note that staggering processes to maintain regular intervals is essential. For example, if all inference processes took a deterministic amount of time with no offset between them, all additional actions in the environment would be overwritten with no benefit from increasing compute. Mcanwhile, with staggering we can experience linear speedups.
3.1 BACKGROUND: STAGGERED ASYNCHRONOUS LEARNING
Parallel vs. Asynchronous Updates: Learning from a transition, i.c., computing gradients, usually takes longer than inference. Thus. performing learning in separate processes is crucial to avoid blocking inference [106], especially for models with a large number of parameters. For this use case, one might be tempted to consider parallel learning processes increase the effective batch size without increasing wall-clock time per batch as this avoids wasted computation. Indeed, parallel updates are better for training large language models when final performance and compute efficiency are most important. In contrast, asynchronous learning can produce updates even faster than learning from a single transition, making the model more responsive to exploration. However, lock-free asynchronous approaches risk overwriting updates, potentially wasting computation that does not contribute to final performance. That said, our focus is on maximizing responsiveness in large models, not necessarily compute efficiency, and overwritten updates are not wasted with respect to regret.
Known algorithms achieve regret upper bounds within a logarithmic factor of this lower bound [68]. Whenn the environment is deterministic, Pminimax 1 and Deery (T) = O. When the environment is uniformly random, Painima 1/|S| and as |S| oc, A (T) E ((7).
4

---

Published as a conference paper at ICLR 2025
Environment
Sequential Inference and Learning
Staggered Asynchronous Inference
Staggered Asynchronous Learning
Figure 3: Realtime Interaction Frequency. We illustrate the comparative interaction frequency of methods that sequence learning and inference and those that maintain multiple staggered asynchronous processes. Even when inference times are greater than the environment step time, it is possible to use asynchronous compute to eliminate inaction and learn from every step.
Round-Robin Asynchronous Learning: Langford et al. [50] laid the foundation for addressing asynchronous update staggering for large neural network models using variants of stochastic gradient descent (SGD). They showed that applying updates in delayed, orderly fashion avoids wasted compute on overwritten gradients. Their approach demonstrated convergence for delayed SGD, with linear scaling limited only by the time taken update parameters relative to computing gradients. This method allows significant linear scaling with minimal compute waste for large models and the delay in the updates will not be a persistent source of regret Theorem 1. While not our novel contribution, this strategy is underexplored. We investigate its scaling properties in our experiments.
3.2 OUR NOVELTY: STAGGERED ASYNCHRONOUS INFERENCE
In Remark highlighted that is fundamentally limited by for sequential interaction, which results in persistent regret even as time goes on when Ty > FM- We will now highlight two novel algorithms for staggering inference processes that lead to reduction in 7z when the number of inference processes Nz are increased. Algorithm I is capable of scaling the expected interaction time with the number of processes by πI S min(TTuu /Nz, FM) where T8 the maximum encountered value of ro. Meanwhile, Algorithm 2 is capable of scaling the expected interaction time with the number of processes by 7r min /Nz, f.). Both algorithms can eliminate inaction. 5
Remark (Inaction of Asynchronous Interaction) For any Fa when т and Masyne interact asynchronously with staggering algorithms I or 2, there is value of the number of inference processes N such that for all N₂ 2 Ni, imactiion ( (T)/T -> 0 as time goes to T + CO.
Algorithm always ensures each processes waits for the current estimate of max amount of seconds before an action is taken by that process to preserve the spacing between actions. Adjustments are made to the waiting time in each process considering dist(r.y), the distance process is behind process in the cycle of processes, until the estimate converges to the true T --ax value. The benefit of this algorithm is that the spacing between actions stays very consistent with no variance once the maximum value estimate has stabilized. This makes Maelay easier learn from. The downside is that the amount of necessary compute to eliminate inaction may be relatively high.
On the other hand, Algorithm stops all waiting all processes as time goes on, so that the expected interaction time of each process is Fg. An estimate of Fg is maintained and when the estimate changes after an action taken, processes wait for an amount of time designed to adjust the average spacing between processes to 7o/ NN. The law of large numbers ensures that the estimate converges to 7a in the limit as - co and that the waiting time diminishes zero. Algorithm 2 has strietly smaller compute requirement than Algorithm 1, but experiences variance in Ty driven by the variance in To, which makes Maelay harder to learn from. The compute advantage becomes more significant for
Appenddi B.6 descrihes how this lead to achieving sublinear regret in deterministic environments.
5

---

Published as a conference paper at ICLR 2025
distributions that have variance in To such that r8th Fg is large. In our experiments, consider Algorithm because we found the variance in Ty is small for the models and hardware we consider.
Algorithm 1 Maximum Time Inference Staggering
Initialize:
ig 0 and (prclesssppocssmmm = 0 V processnum E (1, Nz) Run: INFEAENCCiioocessnuu) V processnum E [1,... Nz.
function INFERRENNC(((Proeessnu))
while alive do dleep(dcla(pprocessnum) delay[processnum) <
b Sleep for any delays accumulated by other processes
a, To To(ss)
Have the policy sample action and inference time
Tg 2 F6 then
Other processes sleep for the difference with the maximum for num o processnum E [l,..., Nr] do
ôT 4- Ta Tg
delay[num] delay[num] dist(num /at//uum.processum)) x87 /Nz
amax + T8
> Set new global maximum
0
else
sleep( (Toa - TD)
>> Sleep for the remaining time Register action in environment
Oc+ 14g /TM]
Hardware Optimization: In this paper, focus is the possibility of achieving speedups when adequate hardware is available to facilitate it. As such, we focus on what can be achieved with an ideal set of hardware rather than the most efficient way utilize a given constrained In our experiments, we run each process on its own dedicated CPU such that resource constraints like memory capacity, and memory bandwidth do not present significant issues. However, if, for example, we aimed to implement multiple simultancous processes on a single GPU, we would have to consider tradeoffs between memory capacity, memory bandwidth, and latency that will jointly serve to limit the possible speedups. We leave analysis of these practical tradeoffs to future work.
of hardware.
4 RELATED WORK
Previous work such as Travnik et al. [102] has considered the asynchronous nature of realtime environments. However, we are not aware of any prior paper that has formalized the connection between asynchronous and sequential versions of the same environment we have. Travnik et al. [102] highlight the reaction time benefit of acting before you learn, and Ramstedt & Pal [73] highlight the reaction time benefit of interacting based on one-step lag. Meanwhile, the interaction frequency of both of these approaches are limited by sequential interaction paradigm and thus the drawback highlighted in Remark also applies to them.
Realtime interaction:
Designing the interaction rate: Farrahi & Mahmood [21] examined how the choice of Tz affects the learning performance of deep RL algorithms in roboties. They found that low TI complicates credit assignment, while high 7r complicates learning reactive policies. Karimi et al. [36] proposed policy that executes multi-step actions with learned TI within the options framework. which may aid in slow problems where credit assignment is challenging. However, this approach does not address the action delay issue we focus on and may worsen by committing to multiple actions based on delayed state. Our policy, defined in the semi-M -DPP framework (Definition 1), relies on low-level policy B, similar the options framework [95]. The key difference is that 3 cannot be modified, preventing intra-option learning and thus making it impossible to improve & even when it sub-optimal. Thus we would rather minimize the use of ( i.e. minimize inaction.
Reinforcement learning with delays: Reinforcement learning in environments with delayed states, observations, and actions is well-studied. Typically, delays are treated as communication delays inherent to the environment [103; 9]. In contrast, we focus on delays resulting from our computations, which are under our control and part of agent design. Our formulation of delay as part of regret is novel due to this unique focus. Common methods address delay by augmenting the state space with all actions taken since the delayed state or observation [8; 37; 63], but this is infeasible for us since these actions are available when computation begins. Instead, approach aligns more with

---

Published as a conference paper at ICLR 2025
methods addressing delay without state augmentation [89; 9; 16; 2; 35]. However, these methods limited by the environment's 's stochasticity [17; 64], as highlighted by Equation 4 of Theorem 1.
Asynchronous learning: Most work on asynchronous RL involves multiple environment simulators learned from asynchronously or in parallel [62; 20; 90]. We explore a more challenging real-world setup with single environment, limiting exploration opportunities. Unlike typical asynchronous setups where each process interacts sequentially with the environment and then learn from that interaction [62], our setting benefits from making interaction and learning asynchronous (Remark 2). Indeed, our paper introduces the concept of staggering asynchronous interaction, which is an innovation with benefits unique to the non- non-pausing environment setting we explore. Similarly ours, some prior work has considered asynchronous learning to avoid blocking inference [106], focusing on model-based learning [28; 29; 27; 107] and auxiliary value functions [96; 14). The novelty of our approach is in its use of multiple asynchronous staggered inference processes instead of single process, critical contribution for deploying large models (see Remark and Remark 2).
5 EMPIRICAL RESULTS
To show that our proposed method does indeed provide practical benefits for minimizing regret per second with large neural networks in realtime environments, we perform a suite of experiments to validate the theoretical claims made in the previous sections. Our experiments include:
evaluation of the speed of progress through a realtime game strategy game where constant learning is necessary to move forward when action inference times are large.
Question 1:
an evaluation of episodic reward in games where reaction time must be fast to demonstrate that asynchronous interaction can maintain performance with models that are multiple orders of magnitude larger than those using sequential interaction.
Question 2:
Question 3: an evaluation of the scaling properties of Algorithms and 2 to demonstrate that the needed number of processes to eliminate inaction, Ni, scales linearly with increasing inference times Fy and parameter counts (
Question 4: an evaluation of the scaling properties of round-robin asynchronous learning [50] demonstrate that the number of processes needed to learn from every transition also scales linearly with increasing learning times and parameter counts (0].
Implementation Details: For our experiments, we implemented the Deep Q-Network (DQN) learning algorithm [61] within our asynchronous
framework, using a discount factor of y = 0.99, a batch size of 16, and an Adam learning rate of 0.001 for the sparse reward Game Boy games (with 0.00001 used for the comparatively dense reward Atari games). A shared experience replay buffer stores the million most recent environment transitions. For preprocessing, we down- sampled monochromatic Game Boy images to 84x84x1, similar to the preprocessing that is standard for Atari [61]. Following the scaling procedure previously established by [13] and [91], we used a 15-layer ResNet model [20] while scaling the number of filters by factor & to grow the network. The model sizes correspond to: = (IM parameters). k = 7 (10M). i = 29 (100M), and & = 98 (1B). Models were deployed multi-process CPUs using Pytorch multiprocessing library on Intel Gold 6148 Skylake cores at 2. 2.4GHz, with one core per process and multiple machines for models using 40 processes. See Appendix for further detail regarding our experimental setup and a discussion of limitations. As large models are known to be difficult to optimize in an online RL context, we also tried training common IM models with an equivalent amount of added delay as the larger models to perform a sanity check. Our initial experiments indicated performance was similar.
Realtime Environment Simulation: To run a comprehensive set of scaling experiments that would not be feasible with real-world deployment, we need a simulation of realistic realtime scenario. Towards this end, we considered two games from the Game Boy that are made available for simulation as RL environments through the Gymnasium Retro project [65]. We implemented realtime version of the Game Boy where it is run at 59. 7275 frames per second such that TM = 1/59.7277 79777755 and with "noop" actions executed as the default behavior B. This exactly mimics the way that humans would interact with the Game Boy as a handheld console [104] and matches the setting in which humans compete over speed runs for these games. We also leveraged three Atari environments [7] run at 60 frames per second with "noop" actions executed as 3 to mimic the way that humans interact with the Atari console in the real-world. These are ideal settings for addressing our core empirical questions.
7

---

Published as a conference paper at ICLR 2025
100
150
Aoandhronss
eeaming
Asynchrooius lsfe and Leaming
-.- Segareri a leaming and infeeree
Sequenial inference and Leaminn
Pokemon Caught
1335
120
Sequentiil Leorning and
Scquential Inference Learning
Battles Won
70
Asyichronzio brferance and car Geamtigg
105
Asynchfronuu
Learning
60
90
50
75
40
30
30
15
%
12 Wall Clock Time (Hrs)
18
24
0
12
18
24
Wall Clock Time (Hrs)
(a) 100M: Pokémon Battles Won vs. Time
T
(b) 100M: Wild Pokémon Caught vs. Time 7
Figure 4: Realtime Pokémon Performance. Battles in Pokémon Blue time for [ = 100M. b) Wild Pokémon caught in Pokémon Blue over time for (0] = 100M. The parallel learning baseline considers effective batch size that is 33x larger, incurring 33x fewer updates.
5.1 FASTER PROGRESS THROUGH A REALTIME GAME WITH CONSTANT NOVELTY
Pokémon Blue: Pokémon Blue is valuable environment for our study due to its long play through time and constant novelty over many hours of play. Acting quickly is not necessity to complete this game as it lets the agent dictate the pace of play, but better players still differentiated based on their speed of completing the game. Indeed, the game has a large community of "speed runners" aiming to complete milestones in record times, with even the fastest milestones taking multiple hours [93]. It an interesting domain for our study because acting quickly is only beneficial to the extent that the agent displays competent behavior, so action throughput alone will not lead to better results when the quality of play correspondingly suffers. Because Pokémon Blue is known as a challenging exploration problem that perhaps even exceeds the scope of previous deep RL achievements [30], we divided the game into two settings based on expert human play: 295 battle encounters (Figure 9a) and 93 catching encounters (Figure 9b). Agents are deployed in these settings and must complete each encounter (by winning battle or catching a Pokémon) before progressing.
Question Can asynchronous approaches achieve faster progress in a realtime strategy game where constant learning is necessary to move forward even when action inference times are large?
For Pokémon Blue we leverage Nr = N2 and enough learning threads to learn every 5 environment steps. We did not find benefit from learning every step given that the underlying game is not responsive to every action taken at the frame level. For all models, the explo- ration rate is annealed from 1.0 to 0.05 over the course of the first 100,000 steps of learning. We compare to the standard RL interaction paradigm where inference and learning are performed sequentially [94] and when the order is flipped for realtime settings [102]. Our sults in Figures 4a and 4b showcase that asynchronous inference and learning combine for superior scaling of realtime performance as models grow. See Appendix A for results for the 1M and 10M models. The improve- ment over sequential interaction corresponds with our expectations given Remarks and 2.
Figure 4:
10
Episodic Reward Rate
10'
Asyndtavooss Intcraction
Sequential Interaction
Ronlom Policy
Noops
10
10
Figure 5: Realtime Tetris Performance
vs. 10]. The average episodic return over 2,000 episodes of learning. We compare models with single inference process to those that perform staggered asynchronous inference following Algorithm 1.
5.2 MAINTAINING PERFORMANCE IN GAMES THAT PRIORITIZE REACTION TIME
Tetris:
We also explore the game Tetris (Figure 9c) that presents a different kind of challenge for our agents where more of a premium is put on reaction time. In Tetris, the player will lose the game they wait indefinitely and do not act time. While slow policy can eventually win in Pokémon, despite taking longer than necessary, policy that does not act timely cannot progress in Tetris as new pieces must be moved correctly before they fall on existing pieces.
Question 2: Can asynchronous interaction help for games that prioritize reaction time as |0] grows?
8

---

Published as a conference paper at ICLR 2025
Figure 5: To aid with exploration and jump-start learning, single episode of human play is provided to each agent to learn from. The agent continues to learn from a total of 2,000 episodes with an exploration rate of 0.05. We that sequential interaction scales quite poorly for games that prioritize a high frequency of actions and cannot surpass random performance for [0] 2 1M as we would expect based on Remark 1. Meanwhile, staggered asynchronous inference following Algorithm 1 can achieve a much higher reward rate for [0] > 1B as we would anticipate based on Remark 2.
Figure 6: We also provide supplemental results for three Atari games with 2,000 training episodes in the same setting (see Figure 10), demonstrating superior performance as [0] scales up. Here we additionally compare
deeper ResNet architecture with 30 convolutional layers rather than 15. The deeper architecture brings up inference times at the same number of total parameters and thus leads to an even bigger gap between staggered asynchronous and sequential approaches. We find that the ability for performance to be maintained with asynchronous methods as the model size grows depends on the game. For example, Boxing includes a very stochastic opponent policy, making very hard to maintain human-level performance in the presence of delay, while the Krull environment predictable enough for agents to surpass human performance even when delay is significant. In Figure 14 we see similar overall trend while using Rainbow learning [26] rather than DQN learning.
5.3 COMPUTATIONAL SCALING OF ASYNCHRONOUS INTERACTION AND LEARNING
Question 3: How does N7 from Remark scale with To and the number of parameters |@|?
Figure 7: We measure N for Algorithms 1 and 2 when the Game Boy is run at the standard frequency using an e-greedy DQN policy at c = and e = 0.5. Figure 7a shows that Nj scales roughly linearly with Fg in all cases, as expected for effective staggering (Remark 2). Figure 7b also demonstrates that Ni scales roughly linearly with 10). When e = O, the variance in To is very small and Algorithms and 2 thus perform the same. When e 0.5, the variance in To is high because sampling random actions is very fast, showcasing predictably superior performance for Algorithm 2. unaffected by stochasticity in Te, but the values of Fa are.
The performance of Algorithm
Notation for Asynchronous Learning: We also would like to consider the compute scaling properties of round-robin asynchronous learning [50]. We now assume that the time learn from an environment transition can be treated as a random variable Te with sampled values Tc ~ T and expected value Tc := E[[2), c]. N will denote the number of learning processes such that all Ne 2 N% include at least one transition learned from for each environment transition.
Question 4: How does N scale with ic and the number of parameters 0/?
Figure 8: In Figure 8a we demonstrate that Nž grows approximately linearly with 7z- This scaling is line with what we would expect for the round-robin algorithm with large networks [50]. Additionally, our results in Figure 8b appear to also showcase linear scaling of N 2 I with 0). We plot three different batch sizes per learning process each figure to highlight the tradeoff between efficiency in achieving particular amortized learning throughput and the number of changes made to
8000
8000
10
Asguchronons Interacticn: Ixilee
7000
7000
Asyuchrsouus Interaction: 2rDeup
Sequential Interaction: IxDrep
Eptoodie Rewond ltate
6000
Uptoodic Reward (taie
6000
Sequential Internction: 2xDeop
-10-
5000
5000
Haman Perfermance
All Noops
20-
2000
4000
Fpitedii
-30
2000
3000
2000
2000
-0
1000
1000
50
10'
10
10
10
10
10
10
10
10
16
30
10
16
10
[el
(a) Boxing
(b) Krull
(c) Name This Game
Figure 6: Realtime Atari Performance vs. (0). The average episodic return over 2,000 simulated learning episodes. We compare models with single inference process to those that perform staggered asynchronous inference following Algorithm in a) Boxing. b) Krull, and c) Name This Game (see
Figure 10). Human performance reported by [61]. Small confidence intervals may be hard to see.
9

---

Published as conference paper at ICLR 2025
10a
102
Alg 1:
Alg 1:e
10a
2: -
10a
2:
Alg 2: = 0.5
Alg 2:4 0.5
10
10-
10
10"
10e
[0]
(a) N7 VS. Ty Algorithms and
(b) N2 vs. 10 for Algorithms 2
Figure 7: a) We plot the scaling behavior of the inference compute requirement Ni as the expected action inference time Fg increases for ResNet polices across CPUs Us in the Game Boy environment. b) We plot the scaling behavior of N2 instead function of the model size (0]
5x10
5x10
Batch Size
Batch Size of
Batch Size of 16
10
Batch Size of 16
Batch
10a
10a
10a
10
10
10-1
10
10
10a
10
10'
10"
10a
TL
|0]
(a) N2 vs. Fe for Round-Robin Robin Learning
(b) NE
[6] for Round Robin Learning
Figure 8: a) We plot the scaling behavior of the learning compute requirement N 2 as the expected transition learning time fr. increases for ResNet polices across CPUs in the Game Boy environment. b) We plot the scaling behavior of N instead as function of the model size |0].
the parameters in the inference process. Higher batch sizes result in longer and increasingly parallel updates with better amortized learning throughput using the same number of processes and fewer changes to the parameters used for inference (as highlighted by fc). Meanwhile, smaller batch size result in shorter and increasingly sequential updates that lead to worse amortized learning throughput using the same number processes and more changes to the parameters used for inference.
6 DISCUSSION
Implications for Software: One difficulty in deploying RI environments within realtime settings is limitations in current APIs such as Open gym [10]. In realtime settings, the environment needs to run in its own separate process so that it is not blocked by the agent (just like the real world). We hope that the contribution our code and environments can thus help jump-start the community's 's ability to conduct research on this important setting. A new paradigm of agent-environment interaction is needed where 1 process is used for the environment with Nz processes dedicated for inference and Ne processes dedicated for learning, depending specific resource constraints.
Implications for Hardware: Our paper demonstrates the benefits of asynchronous computation within realtime settings and thus advances in hardware that enable this will serve to amplify the impact of our findings. Memory bandwidth is primary bottleneck in allowing for asynchronous computation with current hardware. So any improvements i.e. in GPU memory bandwidth, bandwidth across nodes, or the number of GPUs or CPUs per node will make the scalability of asynchronous approaches increasingly viable. Taking a longer-term perspective, hardware architectures that move beyond the von Neumann seperation of memory and compute, such as so called "neuromorphic" computing, will also serve to enable larger scale asynchronous computation like we see in the brain.
In this paper, we have taken deeper look at RL in realtime settings and the viability of increasing the neural network model size in these environments. Our theoretical analysis of regret bounds has demonstrated the downfall of models that implement single action inference process as model sizes grow (Remark I) and we have proposed staggering algorithms that address this limitation for environments that are sufficiently deterministic (Remark 2). Our empirical results playing realtime games corroborate these findings and demonstrate the ability to perform well with models that are orders of magnitude larger. While conventional wisdom often leads researchers to think that smaller models are necessary for realtime settings, our work indicates that this is not necessarily the case and takes step towards making realtime foundation model deployment realistic.
Bigger Models:
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In this paper, we fully disclose all the necessary information to reproduce the main experimental results, as detailed in both Section 5 and Appendix A. We have also released all the necessary code along with detailed instructions to reproduce our results at https://github. wwatthubb.ommCIIC www.//om////eeeeeco AAT/relltime rl. Experimental settings, including hyperparameters needed to reproduce our results are clearly outlined in the paper and further elaborated significance measures in our plots when applicable, such as shading in Figures 4 and 5 to communicate run-to-run variance in terms of 95% confidence intervals. We also include comprehensive information on the compute resources, such as the type of hardware, memory, and time required for execution, as outlined in the "Implementation Details" paragraph of Section 5, This ensures that our results are reproducible, and necessary computational resources are well well-speeified for accurate replication.
in the appendix. Moreover,
provide appropriate statistical
ACKNOWLEDGEMENTS
We would like to acknowledge our support from the Canada CIFAR AI Chair Program and from the Canada Excellence Research Chairs (CERC) Program. We also thank the IBM Cognitive Compute Cluster, the Mila cluster, and Compute Canada for providing computational resources. We are grateful to Yoshua Bengio for his support during initial diseussions that helped shape the direction of this paper. Moreover, we would like to thank Johan Sbando-Ceron, Khurram Javed, Mark Ring, and Rich Sutton for valuable feedback that helped shape the framing of our contribution. We also thank Olexa Bilaniuk for his assistance with questions related to computational resources and multiprocessing.
REFERENCES
[1] Marwa Abdulhai, Dong-K Ki Kim, Matthew Riemer, Miao Liu, Gerald Tesauro, and Jonathan How. Copcext-specific representation abstraction for deep option learning. arXiv preprint arXiv:2109.09876 2021.
[2] Mridul Agarwal and Vaneet Aggarwal. Blind decision making: Reinforcement learning with delayed observations. Pattern Recognition Letters, 150:176-182, 2021.
[3] Cameron Allen, Michael Katz, Tim Klinger, George Konidaris, Matthew Riemer, and Gerald Tesauro. Efficient black-box planning using macro-actions with focused effects. arXiv preprint arXiv:2004.13242, 2020.
[4] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty- AAAI Conference on Artificial Intelligence, 2017.
[5] Andrew Barto and Thomas G Dietterich. Reinforcement learning and its relationship to supervised learning. Handbook of Learning and Approximate Dynamic Programming, pp. 45-63, 2004.
[6] Pouya Bashivan, Martin Schrimpf, Robert Ajemian, Irina Rish, Matthew Riemer, and Yuhai Tu. Continual learning with self-organizing maps. arXiv preprint arXiv:1904.099: 2019.
[7] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-277 2013.
[8] Dimitri P Bertsekas. Dynamic programming and optimal control. Journal of the Operational Research Society, 476))333, 1996.
[9] Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Bi- nas. Reinforcement learning with random delays. In International conference on learning representationss, 2020.
[10] G Brockman. Openai gym. arXiv preprint arXiv: 2016.

---

Published as a conference paper at ICLR 2025
[11] Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D Greene, Dan Jurafsky, et al. Recursive routing networks: Learning to compose modules for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume / (Long and Short Papers), pp. 3631- 3648, 2019.
[12] Michael B Chang, Abhishek Gupta, Sergey Levine, and Thomas L Griffiths. Automatically composing representation transformations means for generalization. 2019.
[13] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. PMLR, 2020.
[14] Gheorghe Comanici, Amelia Glaese, Anita Gergely, Daniel Toyama, Zafarali Ahmed, Tyler Jackson, Philippe Hamel, and Doina Precup. Learning how to interact with a complex interface using hierarchical reinforcement learning. arXiv preprint arXiv:2204. 10374, 2022.
[15] Payel Das, Kahini Wadhawan, Oscar Chang, Tom Sercu, Cicero Dos Santos, Matthew Riemer, Vijil Chenthamarakshan, Inkit Padhi, and Aleksandra Mojsilovic. Pepevae: Semi-supervised targeted design of antimicrobial peptide sequences. arXiv preprint arXiv:1810.0774. 2018.
[16] Esther Derman, Gal Dalal, and Shie Mannor. Acting in delayed environments with non-
stationary markov policies. International Conference on Learning Representations, 2021.
[17] Esther Derman, Gal Dalal, and Shie Mannor. Acting in delayed environments with non- stationary markov policies. arXiv preprint arXiv:2101. 1992, 2021.
[18] Pierre Dognin, Jesus Rios, Ronny Luss, Inkit Padhi, Matthew D Riemer, Miao Liu, Prasanna Sattigeri, Manish Nagireddy, Kush R Varshney, and Djallel Bouneffout. Contextual moral value alignment through context- based aggregation. arXiv preprint arXiv:2403. 12805, 2024.
[19] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world rein- forcement learning. arXiv preprini arXiv: I 1904. /2001, 2019.
[20] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-leamer architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018.
[21] Homayoon Farrahi and A Rupam Mahmood. Reducing the cost of cycle-time tuning for real-world policy optimization. In 2023 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2023.
[22] Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Picter Abbeel, and Igor Mordatch. Learning with opponent-lcarning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 122-130. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
[23] Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte carlo estimator. In International Conference on Machine Learning, pp. 1529-1538. PMLR, 2018.
[24] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson. Pierre Cérard-Marchan Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):335-362 September 2020. doi: 10.
URL https:///oo.org// wtt..//oo/o.1111//411111111 11100//////811111 2649-2.
[25] Fenno F Heath, Richard Hull, Elham Khabiri, Matthew Riemer, Noi Sukaviriya, and Roman Vaculin. Alexandria: Extensible framework for rapid exploration of social media. In 2015 IEEE International Congress on Big Data, pp. 483 IEFE, 2015.
12

---

Published as a conference paper at ICLR 2025
[26] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot. Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[27] Todd Hester and Peter Stone. Texplore: real-time sample-efficien reinforcement learning for robots. Machine learning, 90:385-429, 2013.
[28] Todd Hester, Michael Quinlan, and Peter Stone. A real-time model-based reinforcement learning architecture for robot control. arXiv preprint arXiv: 101.100..9... 2011.
[29] Todd Hester, Michael Quinlan, and Peter Stone. Rtmba: A real-time model-based reinforce- ment learning architecture for robot control. In 2012 IEEE International Conference on Robotics and Automation, pp. 85-90. IEEE, 2012.
[30] Shayaan Jagtap. Why can a machine beat mario but not pokemon?, 2018. URL. https: weeat-mrrio-but-no---heeee
[31] Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(4), 2010.
[32] Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael Jordan. Is q-learning provably efficient? Advances in neural information processing systems, 31, 2018.
[33] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rI? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021.
[34] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 10Xii:20001083611, 2020.
[35] Armin Karamzade, Kyungmin Kim, Montek Kalsi, and Roy Fox. Reinforcement learning from delayed observations via world models. arXiv preprint w:Xi::2403112309 2024.
[36] Amirmohammad Karimi, Jun Jin, Jun Luo, A Rupam Mahmood, Martin Jagersand, and Samuele Tosatto. Dynamic decision frequency with continuous options. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7545-7557 IEEE, 2023.
[37] Konstantinos V Katsikopoulos and Sascha Engelbrecht. Markov decision processes with delays and asynchronous cost collection. IEEE transactions automatic control, 48(4): 568-574, 2003.
[38] Elham Khabiri, Matthew Riemer, Fenno F Heath III, and Richard Hull. Domain scoping for subject matter experts. 2015 AAAI Fall Symposium Series, 2015.
[39] Khimya Khetarpal, Matthe Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: review and perspectives. arXiv preprint arXiv:2012.13490 2020.
[40] Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Gol- naz Habibi, Gerald Tesauro, Sami Mourad, Murray Campbell, and Jonathan How. Learning hierarchical teaching policies for cooperative agents. arXiv preprint 1903.03200 2019.
[41] Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Gerald Tesauro, Murray Campbell, Sami Mourad, Golnaz Habibi, and Jonathan P How. Hetero- geneous knowledge transfer via hierarchical teaching in cooperative multiagent reinforcement learning. 2019.
[42] Dong Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan How. A policy gradient algorithm for learning to learn in multiagent reinforcement learning. In International Conference on Machine Learning, pp. 5541-5550. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
[43] Dong-Ki Kim, Matthew Riemer, Miao Liu, Jakob Foerster, Michael Everett, Chuangchuang Sun, Gerald Tesauro, and Jonathan P How. Inftuencing long-term behavior in multiagent reinforcement learning, Advances in Neural Information Processing Systems, 35:18808-1882, 2022.
[44] Dong- Kim, Matthew Riemer, Miao Liu, Jakob N Foerster, Gerald Tesauro, and Jonathan How. Came-theoretical perspectives on active equilibria: A preferred solution concept over nash equilibria. arXiv preprint arXiv:22 10. 16/75, 2022.
[45] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13)3321-3352, 2017.
[46] Tim Klinger, Dhaval Adjodah, Vincent Marois, Josh Joseph, Matthew Riemer, Alex 'Sandy' Pentland, and Murray Campbell. A study of compositional generalization in neural models, 2020. URL https://arxiv.or//aa///00000 09437.
[47] Sachin Konan, Esmaeil Seraj, and Matthew Gombolay. Iterated reasoning with mutual infor- mation in cooperative and byzantine decentralized teaming. arXiv preprint arXiv:2201.088 008484, 2022.
[48] James Kostas, Chris Nota, and Philip Thomas. Asynchronous coagent networks. In Hal Daumé IIII and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5426-5435. PMLR, 13-18 Jul 2020. URL https://proceedings.mm mlr. prreeess//.mm.r.rr.///...... 1 19/kostas20a.htin html.
[49] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1 1179-1191, 2020.
[50] John Langford, Alexander J Smola, and Martin Zinkevich. Slow learners are fast. In Pro- ceedings of the 22nd International Conference on Neural Information Processing Systems, pp. 2331-2339, 2009.
[51] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint S0t::220000224199 2020.
[52] Lihong Li, Thomas J Walsh, and Michael I. Littman. Towards a unified theory of state abstraction for mdps.
[53] Zhizhong Li and Derek Hoiem. Learning without forgetting. JEEE transactions pullern analysis and machine intelligence, 40(12):2935 2017.
[54] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments, Advances in neural information processing systems, 30, 2017.
[55] Tyler Malloy, Tim Klinger, Miao Liu, Matthew Riemer, Gerald Tesauro, and Chris Sims. Consolidation via policy information regularization in deep rl for multi-agent games. arXiv preprint arXiv:201 1,100,1.111...1 2020.
[56] Tyler Malloy, Chris R Sims, Tim Klinger, Miao Liu, Matthew Riemer, and Gerald Tesauro. Deep rl with information constrained policies: Generalization in continuous control. arXiv preprint arXiv:2010.0400 046466 2020.
[57] Tyler Malloy, Tim Klinger, Miao Liu, Gerald Tesauro, Matthew Riemer, and Chris R Sims. RI generalization in theory of mind game through sleep metaphor (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 15841-15842, 2021.
[58] Tyler Malloy, Chris R Sims, Tim Klinger, Miao Liu, Matthew Riemer, and Gerald Tesauro. Kapacity-limitee decentralized actor-critic for multi-agent games. In 2021 IEEE Conference on Games (CoG), pp. 1-8. IEEE, 2021.
14

---

Published as a conference paper at ICLR 2025
[59] Tyler Malloy, Miao Liu, Matthew D Riemer, Tim Klinger, Gerald Tesauro, and Chris R Sims. Learning in factored domains with anformation-constaan visual representations. arXiv preprint arXiv:2303. 2023.
[60] Amin Memarian, Maximilian Puelma Touzel, Matthew Riemer, Rupali Bhati, and Irina Rish. Summarizing societies: Agent abstraction in multi-agent reinforcement learning. In From Collective Learning across Scales, 2022.
Cells to Societies:
[61] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[62] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. International conference on machine learning, pp. 1928-1937. PMLR, 2016.
[63] Somjit Nath, Mayank Baranwal, and Harshad Khadilkar. Revisiting state augmentation methods for reinforcement learning with stochastic delays. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1346-1355, 2021.
[64] Somjit Nath, Mayank Baranwal, and Harshad Khadilkar. Revisiting state augmentation methods for reinforcement learning with stochastic delays. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1346-1355, 2021.
[65] Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv: /804.03720 2018.
[66] Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko, Pau Rodriguez. Matthew D Riemer, Julio Hurtado, Khimya Khetarpal, Ryan Lindeborg, Lucas Cecchi, Timothée Lesort, et al. Sequoia: A software framework unify continual learning research. arXiv preprins arXiv:2108. (00i:00500000005, 2021.
[67] Shayegan Omidshafici. Dong-K; Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christo- pher Amato, Murray Campbell, and Jonathan P How. Learning to teach in cooperative multiagent reinforcement learning. intelligence, volume 33, pp. 6128 6136, 2019.
Proceedings of the AAAI conference on artificial
[68] Ronald Ortner. Regret bounds for reinforcement learning via markov chain concentration. Journal of Arlificial Intelligence Research, 15-115, 2020.
[69] Inkit Padhi, Pierre Dognin, Jesus Rios, Ronny Luss, Swapnaja Achintalwar, Matthew Riemer, Miao Liu, Prasanna Sattigeri, Manish Nagireddy, Kush R Varshney, et al. Comvas: Contextual moral values alignment system. Proceedings of the International Joint Conference on Artificial Intelligence, 2024.
[70] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URI. http://papers. http:://papersnneurss..c/// heurips.cc/paper9001. intti-a-aaamaaiiitttllleeeeeeee high- performance-dese deeee-ddeeeeeeeeeeeeee -ibraryi.bbbbr
[71] ML Puterman. Markov decision processes, 1994. Jhon Wiley & Sons, New Jersey, 1994.
[72] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference on machine learning, pp. 4218-4227. PMLR, 2018.
[73] Simon Ramstedt and Chris Pal. Real-time reinforcement learning. Advances in neural information processing systems, 32, 2019.
15

---

Published as a conference paper at ICLR 2025
[74] Matthew Riemer. Distributed computing in social media analyties. Distributed Computing in Big Data Analytics: Concepts. Technologies and Applications, pp. 121-135, 2017.
[75] Matthew Riemer, Sophia Krasikov, and Harini Srinivasan. A deep learning and knowledge transfer based architecture for social media user characteristic determination. In Proceedings of the third International Workshop on Natural Language Processing for Social Media, pp.
39-47, 2015.
[76] Matthew Riemer, Aditya Vempaty, Flavio Calmon, Fenno Heath, Richard Hull, and Elham Khabiri. Correcting forecasts with multifactor neural attention. In International Conference on Machine Learning, pp. 3010-3019. PMLR, 2016.
[77] Matthew Riemer, Michele Franceschini, Djallel Bouneffouf, and Tim Klinger. Generative knowledge distillation for general purpose function compression. NIPS 2017 Workshop on Teaching Machines, Robots, and Humans, 5:30, 2017.
[78] Matthew Riemer, Elham Khabiri, and Richard Goodwin. Representation stability regular- izer for improved text analytics transfer learning. arXiv preprint 1eX.v:7704.03611 2017
[79] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv: 1910, 2018.
[80] Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. Advances in neural information processing systems, 31, 2018.
[81] Matthew Riemer, Tim Klinger, Djallel Bouneffouf. and Michele Franceschini. Scalable recollections for continual lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1352-1359, 2019.
[82] Matthew Riemer, Ignacio Cases, Clemens Rosenbaum, Miao Liu, and Gerald Tesauro. On the role of weight sharing during deep option learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5519-5526, 2020.
[83] Matthew Riemer, Sharath Chandra Raparthy, Ignacio Cases, Gopeshh Subbaraj, Maximil- ian Puelma Touzel, and Irina Rish. Continual learning in environments with polynomial mixing times. Advances Neural Information Processing Systems, 2022.
[84] Matthew Riemer, Khimya Khetarpal, Janarthanan Rajendran, and Sarath Chandar. Balancing context length and mixing times for reinforcement learning scale. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[85] Matthew Riemer, Gopeshh Subbaraj, Glen Berseth, and Irina Rish. Realtime reinforcement learning: Towards rapid asynchronous deployment of large models. ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024.
[86] Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In International Conference on Learning Representations. International Conference on Learning Representations, ICLR. 2018.
[87] Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, Atticus Geiger, Lauri Karttunen, Joshua Greene, Dan Jurafsky, and Christopher Potts. Dispatched routing networks. Technical report, 2019.
[88] Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint arXiv:1904.12774 2019.
[89] Erik Schuitema, Lucian Buçomiu, Robert Babuška, and Pieter Jonker. Control delay in reinforcement learning for real-time dynamic systems: A memoryless approach. In 2010 IEEE/RSJ international conference on intelligent robots and systems, pp. 0226-3231. IEEE,
2010.
16

---

Published as a conference paper at ICLR 2025
[90] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[91] Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-leved atari with human-level efficiency. In International Conference Machine Learning, pp. 30365-30380 PMLR, 2023.
[92] speedrun.com com. Manipulation FAQ.
Pokémon red/blue/mg manipulation faq, 2020. URL hup://http:
Red/Blue/RRNG
[93] speedrun.com Pokemon red/blue leaderboard, 2024. URL https//wwww.peednm.coom// pkmmredblue.
[94] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
[95] Richard Sutton, Doina Precup. and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning, Artificial intelligence, 112(1-2):
181-211, 1999.
[96] Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: scalable real-time architecture learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Voluue
pp. 761-768, 2011.
[97] Megh Thakkar, Quentin Fournier, Matthew D Riemer, Pin- Yu Chen, Amal Zouaq, Payel Das, and Sarath Chandar. A deep dive into the trade-offs of parameter-effictee preference alignment techniques, 2024. URL https://arxxv.or//bb//// 04879.
[98] Megh Thakkar, Yash More, Quentin Fournier, Matthew Riemer, Pin- Chen, Amal Zouaq, Payel Das, and Sarath Chandar. Combining domain and alignment vectors to achieve better knowledge-safcty trade-offs in 1lms. arXiv preprint arXiv:2411.06824 2024.
[99] Philip S. Thomas. Policy gradient coagent networks. In J. Shawe-Tayleee R. Zemel, P. Bartlett, F. Percira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL paper/201 /tte///ee00444000000744442266669
files/
[100] Maximilian Puelma Touzel, Amin Memarian, Matthew Riemer, Andrei Mircea, Andrew Robert Williams, Elin Ahlstrand, Lucas Lehnert, Rupali Bhati, Guillaume Dumas, and Irina Rish. Scalable approaches for theory of many minds. In Agentic Markets Workshop ICML 2024, 2024.
[101] Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032. 2024.
[102] Jaden B Travnik, Kory W Mathewson, Richard S Sutton, and Patrick M Pilarski. Reactive reinforcement learning in asynchronous environments. Frontiers in Robotics and AI, 5:79, 2018.
[103] Thomas Walsh, Ali Nouri, Lihong Li, and Michael Littman. Learning and planning in environments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18: 83-105, 2009.
[104] Wikipedia. Game Boy Wikipedia, the free encyclopedia. hwtp://iiiiiiedii.rr////eeeeeee 2024. [Online; accessed 20-May-2024).
[105] Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. In Conference on robot learning, pp. 575-588. PMLR, 2021.
17

---

Published as a conference paper at ICLR 2025
PIDGEOT .61
MENTWO
SLP
SCORE
HP:
LEVEL
RAICHU
LOLA
L44
145
TYPE ELECTRIC
NP:
L LINES
1 128 122
41/131
8/15
THUNDERBOLT
BODY SLAM AM THUNDER WAVE QUICK ATTACK
MEWTWO was
caught
(a) Pokémon Battling
(b) Pokémon Catching
(c) Tetris
Figure 9: a) A frame from the final battle of Pokémon Blue when the agent is deciding on the next move. b) frame from the final catching encounter of Pokémon Blue when the agent has just successfully caught Mewtwo. C) A frame from Tetris right before the agent completes its first line.
[106] Yufeng Yuan and A Rupam Mahmood. Asynchronous reinforcement learning for real-time control of physical robots. In 2022 International Conference on Robotics and Automation (ICRA), pp. 5546-5552. IEEE, 2022.
[107] Yunzhi Zhang, Ignasi Clavera, Boren Tsai, and Pieter Abbeel. Asynchronous methods for model-based reinforcement learning, arXiv preprint arXiv:1910.12453 2019.
[108] Modjtaba Shokrian Zini, Mohammad Pedramfar, Matthew Riemer, Ahmadreza Moradipari, and Miao Liu. Coagent networks revisited. arXiv preprint 10X44 2020.
A FURTHER DETAILS SUPPORTING THE MAIN TEXT
Software Libraries: Our experiments leverage Numpy [24], which is publicly available following a BSD license. Neural network models were developed using Pytorch [70], which is publicly available following modified BSD license. The Gym Retro project [65] used to simulate the Game Boy in RL environment is made available following MIT license. We are not liberty to distribute the proprietary ROMs associated with Pokémon or Tetris and each person that deploys our provided code must separately obtain their own copy.
We depict the environments considered in our paper in Figure 9. We consider six discrete actions for both Pokémon Blue and Tetris including the A button, the button, the left directional button, the up directional button, the right directional button, and the down directional button. the Battling Environment when the opponent Pokémon knocked out by the agent's Pokémon a reward of 1 is received and reward of - is received when users Pokémon is knocked out. Battles include 1-6 Pokémon for the agent and 1-6 Pokémon for the opponent AI. In the Catching Environment reward of 1 is received by the agent when wild Pokémon is captured and -1 when the encounter is terminated unsuccessfully. In Tetris we provide changes in the in-game score as reward for the agent learn from. We leverage Atari environments provided by Gymnasium [101], specifically using the "NoFrameSkip-v4" variants with action 0 taken for "noops" to simulate real-world play console.
Environment Details:
Training Procedure for Tetris: The one episode of human play provided for Tetris included 16,106 steps where non-noop actions were taken. We used our experiments from Figure 7 to calculate the amount of action delay per step for each model and populated the replay buffer with 16, 16.106 transitions corresponding to these actions with observations delayed by the expected amount for each model. We the trained each model for 16,000 steps before tuning the model in a simulation of the environment with the corresponding amount of delay and inaction for 2,000 episodes with one update on the replay buffer for each action taken in the environment. The episodic reward from Figure 5 corresponds to the average episodic reward achieved during that training period. To ensure differences in learning are only the results of delay and inaction while ignoring differences based on different learning speeds resulting from the model size itself, we use the delay and inaction calculated for models of that size, but then always learn model of the standard 15 convolutional layer ResNet depth with A = 1 filter
18

---

Published as a conference paper at ICLR 2025
1 1.:44 4.8
AinVSiiN
(a) Boxing
(b) Krull
(c) Name This Game
Figure 10: Realtime Atari Environments. Frames from the Atari environments Boxing, b) Krull, and c) Name This Game. These particular games were selected because DQN learning is known to achieve comparable performance to humans within 2,000 episodes when trained without delay.
size. We also tried using the same model size, which followed the same general trend, but report it this way to address concern mentioned by a reviewer of this paper.
Training Procedure for Atari: We followed similar training procedure for Atari. We used experiments from Figure 7 to calculate the amount of action delay per step for each model. The number of delay and inaction steps were normalized for the slightly faster 60 frame per second environment. This only has an impact for the 1B model size, which now has a delay of 599 steps rather than 596 steps for the standard depth ResNet model. The episodic reward from Figure 6 corresponds to the average episodic reward achieved during that training period. For the 2xDeep architecture we leveraged a ResNet model with 6 blocks rather than blocks, which corresponds 30 convolutional layers rather than 15 convolutional layers. In the final three blocks, each layers has 32k: filters where 0.0625 corresponds 70k parameters, h 0.75 corresponds to 1M parameters, k = 4.75 corresponds to 10M parameters, k 18.5 corresponds to 100M parameters, and k = 63 corresponds to IB parameters. As standard practice for Atari, we now learn every environment steps. However, we do not skip frames and allow models to act every step if it can.
Statistical Significance: We also note that error bar shading throughout our paper reflects 95% confidence intervals computed with three random seeds: 0, 1, and 2.
Limitations: both our experiments on Pokémon Blue and Tetris, performance well below human-level. This is because both of these games pose significant exploration problems and train our models from scratch for limited amount of time. We believe that these experiments are more than sufficient to showcase the benefits of staggered asynchronous inference in comparison to the sequential interaction framework by showcasing when the latter framework breaks down in realtime settings. However, speculate that the results showing that game play does not suffer despite significant action delay will likely not generalize to more intricate human-level policies. We also corroborate our findings
setting where agents can perform training episodes for the Atari games Boxing, Krull, and Name This Game
human-level with limited
Algorithm Pseudocode: Here we provide detailed pseudocode for Algorithm 2, which could be included in the main text due to space constraints.
A.1 POKÉMON RESULTS WITH SMALLER MODEL SIZES
Due to space restrictions we were also not able to include our experiments for the Pokémon battling and catching domains with |e| < 100M in the main text. We provide these results for |0] 1M in Figure and |8] = 10M in Figure 12. As expected by our theory, the difference between asynchronous interaction and sequential interaction expected to be less when the action inference
19

---

Published as a conference paper at ICLR 2025
Algorithm 2 Expected Time Inference Staggering
Initialize:
i, Tiot 0, Cuo O, and delay/processnumm] 0 processnum [I,..., Nzl
Run: INFERENCC(pproessnum)) v processnum E [1,..., Nz.
function INFERENCE((POOeSSTum)
while alive do
sleep(delay|pnooessuum))
b Sleep for any delays accumulated by other processes
delay(processnum) <-
a, TB ~ TT (8+)
Have the policy sample an action and inference time
Ouo f Uwt +
Tipe + Tio + TB
F8 Twt Aco
ôT < To to
if ôT 2 then
Wait more further from the current process
for num + processnum E [I,..., I....,,,, Nz] do
delay[num] < delay[num] + dist(num, processnum) abs(ôr)/Nz
else
Wait more closer to the current process
for num &
processnum E [1. .t......M N2] do delay[num] < delay[num] + (Nz Jjdis(nuumproocessuum abs(őr)
G++ TT/ a
> Register action in environment
40
36
Royndrrarou rnre and learerg
Sequanbial Learing
Asynstronnia Inlurune and Laaanity
Sequettiel Invenenee learning
32
Pokemon Caught
28
Learning
Battles Won
28
24
20
20
16
12
8
Wall Clock Time (Hrs)
Wall Clock Time (Hrs)
(a) 1M: Pokémon Battles Won vs. Time 7
(b) 1M: Wild Pokémon Caught vs. Time
Figure 11: Realtime Pokémon Performance for Staggered Asynchronous Interaction & Learning. a) Battles won in Pokémon Blue as function of time for [0] = 1.. b) Wild Pokémon caught in Pokémon Blue function of time for [0] 1M.
time is low. In this experiment, the sequential interaction baseline acts every steps at 1M parameters, steps 10M parameters, and 70 steps at 100M parameters, For the Pokemon game is expected that there is not much improvement acting at every step it is well known to speed runners that in common circumstances the game could take as many as 17 frames to respond to non-noop actions in which time intermediate actions "buffered" and not yet registered in the environment [92]. This is a fact commonly exploited by speed-rume speed-runners to allow for them to manipulate the RNG the game and execute actions in "frame perfect" manner despite natural human imprecision when it comes to action timing. As a result, the overall performance in this domain is in line with expectations differences most significant when inference times greater than this 17 environment step threshold. The results reported in Figure and in Figure 12 are mean performances along with 95% confidence intervals, calculated from 10 with different random seeds.
A.2 PERFORMANCE FUNCTION OF NoN-NoOP ACTIONS
In Figures 13a and 13b we observe that the difference between progress through the game for sequential and asynchronous algorithms per non-noop action is not statistically significant in the Pokémon battling and catching environments. As such, our asynchronous algorithm takes full advantage of the increased throughput of non-noop actions taken in the environment.
20

---

Published as a conference paper at ICLR 2025
45
20
Asynidronoug Inforence and Learning
Agynchrerous Inference Leaming
40
Sequencil Leaming
Sequenti:
Pokemon Caught
16
18
bcarning
Battles Won
30
14
12
25
10
8
15
5
0
24
24
Wall Clock Time (Hrs)
Wall Clock Time (Hrs)
(a) 10M: Pokémon Battles Won Time
(b) 10M: Wild Pokémon Caught Time
Figure 12: Realtime Pokémon Performance for Staggered Asynchronous Interaction & Learning. Battles in Pokémon Blue as a function of time for [0] 10M. b) Wild Pokémon caught in Pokémon Blue as function of time for |0] 10M.
160
Asynchronous inference and Learning
Asynchmnous inference and warning
Sequential Inference Learning
Sequential Leaminng
Sequential Leaming and Inference
Caught
Sequential Leaming Inference
Battles Won
50
200
40
30
40
20
Num Noop Actions
x10
Num Noop Actions
x10*
(a) 100M: Pokémon Battles Won vs. Number of Non- (b) 100M: Wild Pokémon Caught vs. Number of Non- Noop Actions
Noop Actions
Figure 13: Realtime Pokémon Performance for Staggered Asynchronous Interaction & Learning. a) Battles in Pokémon Blue function of non-noop actions taken in the environment for [0] = 100M. b) Wild Pokémon caught Pokémon Blue as function of non-noop actions taken in the environment for [0] 100M.
21

---

Published as a conference paper at ICLR 2025
A.3 RAINBOW EXPERIMENTS
We now also include results with Rainbow [26] used for learning rather than the DQN on Atari We again use a standard k filter size at the standard 15 convolutional layer ResNet depth now with the Rainbow model for learning based on model inference times that correspond to those used for the DQN. Note that Rainbow performing worse than the DQN in the Boxing environment, even without delay, was reported in Table 5 of the original paper [26]. Our implementation of Rainbow draws heavily from the following repository hyperparameters taken from that repository other than the 0.0001 learning rate corresponding to other Atari experiments. These experiments yet again showcase the superior ability of staggered asynchronous inference to maintain performance larger model sizes and inference times.
with
B PROOFS FOR EACH THEORETICAL STATEMENT
Our proofs rely the following core assumptions, restated from the main text:
The environment step time can be treated as an independent random variable TM with sampled values TM Ty and expected value FM TT.
2. The environment interaction time can treated an independent random variable Tz with sampled values 7I ~ Tz and expected value 7r == TT[
3. The action inference time of the policy can be treated an independent random variable Te with sampled values Ty ~ Ta and expected value Fa =- [[[T]
4. Asynchronous learning can learn from every interaction with Maelay-
Issues with the proof in an earlier workshop version of the paper [85] have been corrected here.
B. DEFINITION
Most of Definition just recaps the dynamics of how the agent interacts with an asynchronous ground MDP following assumptions 1-3 about the nature of that interaction. All that is left to show is that this can be viewed as a delayed MDP and that it can be viewed as a semi-MDP. The interaction process highlighted in Definition matches that of Random Delay Markov Decision Process (RDMDP) [9] where the action delay distribution defined by the random variable [ Tre/Tyll 1. To show it is semi-MDP as well, we consider the same proof style of Theorem in Sutton al. [95]:
8000
5000
Asynchrunous Interaction
Sequeniaa Interaction
Human Perfortance All Noops
6000
6000
Unbondic Reward Ralc
Untonds: Reward Rati
Eqieedic Roward Ruie
-10
3000
3000
4000
-20
-30
2000
2000
40
1000
1000
10'
10'
[9]
10'
10
[0]
10'
10
[0]
10
(a) Boxing
(b) Krull
(c) Name This Game
Figure 14: Realtime Atari Rainbow Performance vs. |0) The average episodic return over 2,000 simulated learning episodes. We compare Rainbow models with a single inference process to staggered asynchronous inference following Algorithm in a) Boxing, b) Krull, and e) Name This Game (see Figure 10), Human performance was originally reported by [61]. Small confidence intervals may be hard to see.
22

---

Published as a conference paper at ICLR 2025
A semi-MDP consists of (1) set of states, (2) set of actions, (3) for each pair of state and action, an expected cumulative discounted reward, and (4) a well-defined joint distribution of the next state and transit time. We now demonstrate each of these properties. The set of states is S and the set of actions is A. The expected reward and the next-state and transit-time distributions are well defined for every state and delayed action. These expectations and distributions are well defined because Masyne is Markov, thus the next state, reward, and time are dependent only on the delayed action chosen and the state in which was initiated. Transit times with arbitrary real intervals are permitted in semi-MDPP
B.2 THBBRREM
To prove Theorem I we will demonstrate the validity of each equation of the theorem following the order of presentation in the main text.
By definition (T) and innation (7) must be independent contribution to the total regret because learning regret is only incurred when acting in the environment following π and inaction regret is only incurred when not acting in the environment and thus following the default behavior policy 3. The interaction frequency does not depend on the parameter values of π as they change following from the independent random variable assumption. Even when regret from learning is eliminated and regret from inaction is eliminated there is still another independent source of regret that persists Adday (T) reflecting the lower reward rate of the best possible policy in acting over Maelny in comparison to the best possible policy acting over Masyne-
Equation I:
The worst case lower bound for the standard notion of regret arising from the need for learning and exploration has been established as Ateam E ((vT) where T denotes the number of discrete learning steps taken in the environment. Given that we learn from every asynchronous interaction with the enviromment following assumption 4, then the regret as a function of r scales with the expected number of discrete environment steps as a function of T i.c. [T]( 7/Fz because T'M and T7 are independent. Therefor, (Team(T) E ((VT/Tz). As noted in the footnote in the main text, this analysis equally applies to the known Acemm (T) E Ö(VT) minimum upper bound [68].
Equation 2:
Equation 3: The worst case lower bound on Ainactio(T)) is derived by considering a worst case environment with two actions an and 02 and two states $1 and 82 where the default behavior takes its own action a3 at every state. The reward provided is 1 at and 0 at The next state $1 regardless of the state if either a1 or a2 taken and $2 if 03 is taken. In this environment the optimal reward rate is and the reward rate when following / is 0.0. The expected number of times 03 is taken during seconds in Masyne is then T T/rr x (Fz - FMm/FM because ,3 is used for (Fz 12-F- -FM) Fz percent of ground MDP actions and the expected number of actions taken over is T/ T/FM when T'M and Ty are independent. Therefor, Ainection (T) E [(((T/TZ) x (Fz for this particular environment. Meanwhile, the expected inaction regret is also upper bounded by Amnaction (T) S Fmax x (T/1z) x (fz FMM/FM where I Tmax is the maximum possible reward per step because by definition the agent cannot incur regret from inaction when it is acting in the environment. Therefor, we have demonstrated that Equation 3 holds.
The worst case lower bound ( is derived by considering worst case environment with n states S = {$1,..., .... and n actions A = [a1,...,(n) an where the default behavior 3 takes its own action ag every state. If the agent takes the action a; corresponding to the state 8; for any i E {l,...,?] } the agent receives a reward of 1, otherwise it receives a reward of O. The agent stays in the current state s; regardless of the action with probability Pminimax and goes to the next state in the cycle 8;41 with probability 1 - Pmniimax where Pminimax 2 1/n because the sum over next state probabilities must equal 1. After state 8, the agent returns to state 81. The probability of staying in the current state for & consecutive environment steps is (Puninimxx )a and in the limit as n - oo this is also the probability that the agent in the current state steps later. the agent is not in the current state l steps later, the agent will apply sub-optimal action based on the old state because no state will overtake the current state as more likely than the current state with the state likelihoods converging to uniform in the limit k - Do when the Markov chain fully mixes. So the regret per ground environment step incurred by the optimal policy that acts with step lag is 1 (Pminimaxx Thus, the best expected regret rate per step that can be ensured with actions delayed by k = [TO/TM] is 2 E[1 (Pminimax [ro/Tm]]. Moreover, the expected number of steps
Equation 4:
23

---

Published as a conference paper at ICLR 2025
taken in the asynchronous MDP over seconds is T/ /fz, so we can lower bound the expected regret
as Adclyy (7) 2 (T/Fz) x E - (Pminimax) (Tonmmmaa((TTTia)) FTp// This then leads to the conclusion of Equation 4 from the main text that Adetoy (T) E (((T/Fz) E[[ (Pminimaxx [re/rM]]]
Tighter Version of Equation 4: While the definition of Pmninimax in the main text holds for the counter example above, this example relies the fact that transitioning states actually has impact on the reward of a policy and its value function. In general, it does not matter if the state changes the change does not impact the optimal policy, So a tighter version of Pminimax would be defined over a π r"-irrelevance state abstraction Øx of the state space rather than the ground state space, following the terminology of [52]. In a Φ abstraction every abstract state in ΦT (S) has an action a" that optimal for all the ground states and in that abstract state. As result, Φ-- (8;) = Φx (sz) implies that m Q= (s;, a) max Q ($),a). For example, tighter version of Equation 4 could be written with Pminimax = min,es, m (s))Ed. (S) Ed (e,)= 1y| a).
B.3 REMARK
We restate the derivation of the remark from the main text, filling in a bit more detail for clarity. When π and Masne interact sequentially, we must have TI 2 To, so Amsction E Q(T/Fr x (iz FFM) E M(TTTO Ta X (Fu - FM) /M) This implies that even as T - oc, the worst case regret rate realtime (T)/T E Q(Dimmction (T)/T) (a(T - FMM)/TTFF) following from Theorem 1.
B.4 REMARK 2
Interaction Time of Algorithm 1: At any point in time, by definition im < T/pax as the estimated maximum must be less than or equal the current maximum. Therefore, with Nz equally spaced processes, the interaction time must be tz [ /Nz/TM] X TM and Algorithm I ensures equal spacing between processes whenever ig does not change. Regardless of the changing value of 4gaax, we can also bound the individual interaction time steps TI I [T /Nz/TM] x TM and correspondingly the global average `I S T8 /Nz when fI > TM and FI = FM otherwise.
Interaction Time of Algorithm 2: As r 00, fg + Fa due to the law of large numbers and thus 8r + O, which implies that additional waiting times become zero in the limit. Therefore, with Nz. equally spaced processes, the interaction time must be тΣ Fo/ /T/NN//Mll x TM and Algorithm 2 ensures equal spacing between processes whenever fgg does not change and ôT = O. Thus, as T + oo the global average is îz Fo/Nz when Fz FM and :I FM otherwise.
Bringing it Together: Algorithm is capable of scaling the expected interaction time with the number of processes by Fz S. min(r /Nz, FM) where T8 max is the maximum encountered value of To as 7 - oc. This then implies that for Nz 2 N = [T max /Fml. Fr = FM- Algorithm 2 is capable of scaling the expected interaction time with the number of processes by Fं min /N FM) as T - oo following the law of large numbers. This then correspondingly implies that for Nr. 2 No [To/TM], Fz TM-
B.5 COMPARISON WITH ACTION CHUNKING APPROACHES
The action chunking approach learns policy that produces multiple actions at time with single inference step i.c. a policy ne Ta, as+a/s;) that produces i actions at a time. Clcarly action chunking does not address the root cause of regret from delay Adelay (T) that our asynchronous inference framework also suffers from as k steps of delay is built directly into the policy. However, it less clear on the surface how the action chunking approach relates Ateam (T) and Ainection (T).
Does action chunking eliminate Ainaction(()) (T)? Action chunking could eliminate regret from inaction +mpmi /k: = TM- But this is not possible for any value of To max with sufficiently large A because T max must depend on k as the size of the outputs produced scales linearly with k. So action chunking can eliminate inaction for some policy classes, but not in general as policy inference times become large. Our asynchronous inference approach provides more general result in this regard (see Remark 2).
Does action chunking impact Ateam (T)? A hidden term excluded from our bound of Aleam (T) in the main text is a dependence on the size of the action space |A] such that (r) E 2 /T[[A//F [31]. We suppressed this dependence for clarity in the main text, but it is important keep in mind
24

---

Published as a conference paper at ICLR 2025
when considering action chunking approaches this implies that the regret lower bound scales with / |A and thus can lead to an exponentially worse regret from learning Atemm (T) even in cases where Aimuttion (7) is eliminated with sufficiently large k.
B.6 ACHIEVING SUBLINEAR REGRET WITH STAGGERED ASYNCHRONOUS INFERENCE
As we have demonstrated in Remark 2, when enough asynchronous threads are provided, model of any size can eliminate regret from inaction. Regret from delay is not addressed by this approach, but goes to zero in deterministic environments. As a result, to demonstrate that total regret is sublinear for deterministic environments following Equation 1, we must only show that A (T) can be upper bounded by a quantity sublinear in r. For example, we use Q-Learning as we do in our experiments, there are known upper bounds for the case of utilizing optimistic exploration with carefully designed learning rate of Akam (T) E (VTT) where T is the number of steps in the environment [32]. In expectation E[T]= = 7 T/FM. the upper bound the expected regret will be Ateam (T) E ((VT) as FM is a constant. As a result, our approach can achieve sublinear regret whenever Adelay (T) is sublinear, which includes, but is not limited to, all deterministic environments.
C TOWARDS ENABLING REAL-WORLD DEPLOYMENT OF AGENTS IN EVEN MORE COMPLEX SCENARIOS
Designing Safe Default Policies 3: In our work, we assume 3 performs poorly such that it is unsafe and thus our goal with asynchronous interaction is avoid using it altogether. However, an alternate research direction motivated by our formulation in Definition would be to work on refining the B function itself such that relying on it is guaranteed be safe and not lead to irrevertible consequences, This is, for example, major difference between the implementation of /3 in the games Pokémon and Tetris. In Pokémon inaction does not prohibit the agent from eventually achieving success in the game, whereas in Tetris inaction leads to missed opportunities that can never be overcome. This materializes in our empirical results as we see a slower degradation of performance for the sequential interaction baseline as the model size grows for Pokémon. That said, the limitation of our asynchronous inference approach is that policies are based on previous states, which can itself be unsafe in stochastic environments. In this case, it would be logical to also consider beneficial and safe ways to modify 3 itself, which can then just be viewed as another model that happens to have a faster inference time that should also be provided resources for asynchronous learning. This case closely mirrors the options framework [95; 4] and variants of it that consider deep hierarchies of policies [80; 82; 1]. As we do not want ,3 itself to lead the agent to OOD states, probably would make sense to train this policy with pessimism bias [49; 33] or a bias towards actions that make focused effects in the environment [3]. Moreover, our work is complimentary alignment of large deep models [97], particularly for enhanced safety [98] and alignment with moral values [69; 18]. It is also worth noting that while neural scaling laws demonstrate generalization improvements with increased model size in the regime of having access to as much data as needed [34], capacity limits have often been found to improve generalization for RL [56; 59] and multi aalti-agent RL [55; 58; 57]. As such, it may not make sense scale model size indefinitely, as suggested by the general discourse of our paper, in data limited settings where safety is critical.
efficient methods for performing
Broader Domains with Diverse Subtasks or Nonstationarity: When it comes to both learning a policy over the delayed semi-MDP and learning a default behavior policy, a key challenge is to adapt these policies to changing environment dynamics and new scenarios. Broadly, this is referred to as Continual RL (see [39] for a comprehensive survey). On the optimization end, these settings require consideration of the stability-plastiiiyy dilemma. For example, we used recency based replay buffers in this work that have been standard in the RL literature since [61]. However, this recency based buffer maintenance strategy only makes sense when we are purely focused on plasticity. A reasonable alternative for continual learning would be buffers based on reservoir sampling [79] to emphasize stability or scalable memory efficient approximate buffers [77; 81; 6]. Another way to promote stability during learning is with regularization based approaches. For example, approaches that leverage knowledge distillation from old versions of models [53; 78] or approaches that penalize movement in parameter space [45]. In settings where subtasks are diverse it is also important to consider that the optimal interaction frequency may depend on the context. In this case, it may be beneficial to consider algorithms that adaptively select which layers to process at inference time
25

---

Published as a conference paper at ICLR 2025
[86; 11; 12; 87], see [88] for survey of approaches. More generally, this adaptive computation problem can be formalized within the coagent networks framework [99; 48; 108]. That said, the compositional generalization needed for utilizing these models in practice makes achieving real-world success very challenging [46]. Moreover, in the case of POMDPs, especially when they are large scale [83], it may make sense adaptively change the interaction history context length sent to policy to tradcoff the extent of modeling non-Markovian dependencies with inference and evaluation
times [84].
Interacting in Multi-agent Settings: Another key consideration when generalizing our approach to more realistic settings interacting with other agents in the environment. While asynchronous interaction can definitely help eliminate inaction, the amount of tolerable delay depends not just on the environment, but also on the behavior of other other agents in the environment. For example, the Boxing environment considered in our experiments from Figure 6 can also be played as a two agent game. We find that it is very difficult to perform on human-level in this environment even with delay low as 3 environment steps, which ties back to the stochasticity of the policy used by the Al in single player mode, Another key challenge in multi-agent environments is when the policies of the other agents change i.e. due learning, which makes the environment nonstationary from the perspective of each agent. When the agents are trained in a centralized fashion, they can be considered in each others updates mitigate this nonstationarity [54]. However, in the more typical decentralized setting agents must build theory of mind models to speculate about the (changing) behavior of other agents [72] which must be carefully constructed for scalability in the presence of many other agents [60; 100J. Effective approaches have been developed to address this challenge by meta-learning with respect to the updating policies of other agents [22; 23; 42; 105; 43] or by learning to perform effective communication between agents [51; 67; 40; 47]. In order to generalize our theoretical results in Theorem to environments with multiple learning agents, we would have to consider regret with respect to complex game-theoretic solution concepts [44]. As such, we leave consideration of these approaches and exploration of formulation for realtime regret this setting to future work.
Applicability Beyond RL: In this paper, we focused on RL environments because of our use of the formalism of RL to establish our theoretical contributions. However, the general principles of this paper should apply to many practical domains that are not typically modeled using RL such as biomedical applications like peptide design [15], monitoring conversation topics across the internet [75; 25; 74; 38], and time series prediction based on incoming internet data [76]. Indeed, we chose to study the framework of RL as these problems can generally be considered special case of the RL formalism [5]. For example, one broad application area of particular relevance to our paper is continual supervised learning. There are many potential settings of deployment related to this general topic that can all be considered special cases of the formulation we consider in this paper [66].
26


Reviews:
[{"review_id": "v3oxeXWKiz", "reviewer": "Reviewer_a7T1", "summary": "This paper addresses the challenges of employing large models for real-time reinforcement learning, where high inference latency prevents conventional sequential inference-and-learning processes from converging. The authors first formulate asynchronous interactive learning as a delayed semi-Markov Decision Process (semi-MDP) and analyze regret lower bounds by decomposing them into several theoretical components. Their analysis demonstrates that staggering multiple processes can enable effective learning while sequential learning cannot.\n\nThe paper then proposes two algorithms that stagger multiple processes for asynchronous RL by maintaining a constant action interval. On the evaluated game benchmarks of Pokemon Game Boy and Tetris, the proposed methods significantly outperform sequential methods, especially when the model size is large. Additionally, this paper analyzes the scaling properties of the number of inference processes needed.", "questions": "Please see the above section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Originality:** To the best of my knowledge, the decomposition of regret bounds into several components under real-time settings is interesting and novel. The proposed algorithms are well-motivated by this theoretical analysis.\n- **Clarity:** The paper is well-written and easy to follow. The presentation is clear, including comprehensive details on problem formulation, theoretical proofs, and experimental settings with results.", "weaknesses": "- The paper's impact could be broadened by evaluating more experimental settings and environment domains to demonstrate the generality of the theory-motivated algorithms. For instance, including locomotion/control environments with continuous action spaces would provide stronger evidence for applying these methods to real-time robotics applications.\n- On question 2: this paper could be more convincing, with more detailed discussions explaining why sequential learning methods fail as network size increases, for instance, including concrete examples demonstrating how large networks prioritize high-frequency actions could provide evidence to support the theoretical analysis empirically in a more grounded way.\n- When comparing models of different scales, one potential extension is to consider variations in the number of layers, in addition to the number of channels, would provide a more comprehensive evaluation since both factors significantly affect inference time.\n- Algorithm 2 would benefit from experiments/results in the settings with relatively high inference time variance. Though this is a minor point, such experiments would better justify the choice between two algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8cODUINLsZ", "reviewer": "Reviewer_NeQu", "summary": "This paper considers real-time reinforcement learning where the environment continually evolves as the agent performs inference, thereby introducing potential delays on various aspects. The paper proposes a regret decomposition into three terms---learning time, inaction time (i.e. default behaviour while policy is performing inference), and irreducible regret due to stochasticity of the environment with delays. The paper focuses on reducing the two former terms through asynchronous learning and inference. The paper conducts experiments on three environments with varying model sizes. The experiments demonstrate a positive (negative) correlation with model size and delays (performance) with the sequential formulation, and further demonstrate that the proposed asynchronous variant can reduce the performance degradation.", "questions": "- I believe the following questions are captured by $\\Delta_{delay}$, please correct me if I am wrong:\n\t- How does technique like action chunking play into this? In those situations I believe $\\beta$ will be the actions taken by the policy from state $s$ which will be sufficient with more deterministic environments?\n\t- Furthermore, how meaningful is the asynchronous inference if the inference time is too coarse such that $\\pi(s_{t-1}) \\neq \\pi(s_t)$ assuming no inference delays?\n- I am unsure how to interpret Figure 8, particularly why is $N^*_\\mathcal{L}$ larger with smaller batch size? I would expect the inverse as it should be less time-consuming to update smaller batch sizes?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is mostly well written and the decomposition seems reasonable.\n- The proposed method is simple and easily applicable to real-life systems.\n- The experiments seem mostly convincing", "weaknesses": "- I would appreciate if the paper considers $\\Delta_{delay}$ by artificially injecting noise to the transitions to see the theoretically described bound.\n- While the decomposition is nice, I am also interested in seeing whether the regret bound is sublinear. I was wondering if the authors have considered theoretically quantifying these algorithms under this framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DmZjPUZJpl", "reviewer": "Reviewer_Pk39", "summary": "This paper addresses a key challenge in applying RL models in real-time environments: the need for rapid, continuous responses in settings where even minor delays can accumulate into significant regret. The authors analyze regret bounds and show that minimizing long-term regret is often impractical in conventional sequential RL settings but becomes achievable with sufficient asynchronous computing resources. They introduce new algorithms to stagger inference processes asynchronously, ensuring that actions can still be made consistently, even with larger models. Empirical results on Game Boy games like Pokémon and Tetris support these theoretical insights, showing that the proposed methods support much larger models than previously feasible in real-time environments. The findings suggest that performance relies more on the environment's predictability over the inference period than on rapid action frequency, making it possible to deploy large-scale models in real-time.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an advancement in asynchronous interaction and learning within real-time RL, addressing the challenge of deploying large neural network models in environments where timing is crucial.\n\nThe authors define an induced delayed semi-MDP which connects the asynchronous nature of real-time environments to the traditional sequential interaction paradigm. This framework allows for a deeper analysis of the sub-optimality that arises from time discretization and computational delays. Furthermore, they propose a novel decomposition of real-time regret into three components: regret from learning, regret from inaction, and regret from delay. This decomposition sheds light on the limitations of sequential interaction and the advantages offered by asynchronous approaches. A notable innovation in the paper is the introduction of algorithms for staggered asynchronous inference, which enable the use of models with longer inference times without compromising the frequency of actions. This key development facilitates the deployment of larger models in time-sensitive settings.\n\nIn summary, this paper makes some contributions to the field of real-time RL. The authors present a novel problem formulation, and provide some empirical evidence to support their findings. The presentation and detailed information enhance reproducibility. Overall, the work has important implications for the deployment of large models in real-time settings and may lead to advancements in RL software and hardware design.", "weaknesses": "I have some questions and concerns about this work that I provide in the following.\n\n- In Section 2, it is mentioned that the time between decisions only depends on the MDP. How does it **only** depend on the MDP? Since it denotes the time between decisions, doesn’t it depend on policy instead? The respective sentence doesn’t make sense.\n- About taking actions from the behavior policy:\n    - How to select the default behavior that is safe and performant at different time steps? What if the default behavior policy leads the MDP_async into OOD states wrt the learned policy?\n    - How can we make sure that the behavior policy mimics the real world as it evolves, considering factors such as changing environmental dynamics, varying agent interactions, and the need for continual adaptation to new scenarios?\n    - If we change noop with the worst or the best action, how the results and conclusions would change? How important it is to the core idea of the paper?\n\n- In the Empirical Results Section, Questions 1 and 2 are not hyperlinked while Questions 3 and 4 are. I’d suggest making them consistent.\n- The experimental section raises two slightly different questions about the performance of asynchronous interaction and learning in a real-time strategy (RTS) game with constant novelty (lines 346 and 424). The first question, stated in the general description, emphasizes evaluating the “speed of progress,” “action throughput,” and “maintaining learning performance.” In contrast, the experiment-specific question appears to narrow the focus to determining whether asynchronous methods can indeed enable faster progress despite longer model and inference times. While both questions relate to asynchronous methods' impact on learning and progression, their phrasing suggests slightly different evaluation criteria, which may confuse readers about the primary aim of the experiments. The first question frames the experiments more broadly, implying a focus on learning stability and throughput, whereas the second centers specifically on asynchronous methods' ability to handle extended inference times in a constantly learning environment.\n\n- About lines 433 and 434: Does the performance improvement over sequential interaction need to be statistically significant? If yes, then how are the results in Figs. 9 and 10 justified when the improvement is only significant for |\\theta|=100M? If no, then it would be helpful to include a discussion explaining why statistical significance is not required to interpret these results as meaningful. For instance, if the improvement trends observed with smaller |\\theta| values still support the theoretical advantages outlined in Remarks 1 and 2, an explanation should clarify why these trends are practically relevant, even if not statistically significant. Such clarification would strengthen the interpretation of the scaling benefits claimed for asynchronous methods.\n- Why are confidence intervals omitted from Figs. 7 and 8?\n- The empirical evaluation is only conducted with DQN as the baseline method. To strengthen the evaluation, consider including more advanced baselines like QR-DQN as well. Comparing with such variants would provide a more comprehensive view of the asynchronous framework's performance across different reinforcement learning strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mZohC1xqSL", "reviewer": "Reviewer_VvDT", "summary": "This paper highlights an often ignored discrepancy between the discrete-time RL framework and the real world. The world evolves while agents compute actions, limiting the types of problems agents can solve. Large models with high action inference and learning times conflict with intelligent agents' control over the environment. The paper examines this and explores asynchronous interaction paradigms for large models in high-frequency environments. It formalizes and tests the approach's benefits and limitations. Contributions include formalizing how time discretization induces a new learning problem, deriving regret bounds, proposing methods for staggered asynchronous inference, and conducting experiments on realtime games like Pokémon and Tetris.", "questions": "I don't have other questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Regret Decomposition seems to be rigorous in real-time reinforcement learning. This specific choice of time discretization is novel in Definition 1.\n2. The staggered asynchronous learning and inference phase addresses  the poor scaling properties, highlighting that while the latter allows actions and learning at every step, actions are delayed and reflecting past states. It also emphasizes the importance of staggering processes to maintain regular intervals and shows implications for performance such as linear speedups.\n3. Several Experiments clearly verify their theory.\n4. The paper has great organization and is easy to follow.", "weaknesses": "* The experiment is recommended to be based on other base RL strategies rather than just DQN.\n* Parallel updates should be considered as a baseline.\n* Can staggered asynchronous inference be applied to multi-agent systems?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)