Handling Delay in Real-Time Reinforcement Learning
Authors: Ivan Anokhin
Rishav Rishav
Matthew Riemer
Stephen Chung
Irina Rish
Samira Ebrahimi Kahou
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
real-time reinforcement learning
delay
pipelining
parallel computations
Primary area: reinforcement learning
TL;DR: We suggest pipelining to increase throughput in real-time reinforcement learning, and propose methods to effectively handle the associated delays.

Abstract: Real-time reinforcement learning (RL) introduces several challenges. First, policies are constrained to a fixed number of actions per second due to hardware limitations. Second, the environment may change while the network is still computing an action, leading to observational delay. The first issue can partly be addressed with pipelining, leading to higher throughput and potentially better policies. However, the second issue remains: if each neuron operates in parallel with an execution time of $\tau$, an $N$-layer feed-forward network experiences observation delay of $\tau N$.
Reducing the number of layers can decrease this delay, but at the cost of the network's expressivity. In this work, we explore the trade-off between minimizing delay and network's expressivity. We present a theoretically motivated solution that leverages temporal skip connections combined with history-augmented observations.  We evaluate several architectures and show that those incorporating temporal skip connections achieve strong performance across various neuron execution times, reinforcement learning algorithms, and environments, including four Mujoco tasks and all MinAtar games. Moreover, we demonstrate parallel neuron computation can accelerate inference by 6-350\% on standard hardware.  Our investigation into temporal skip connections and parallel computations paves the way for more efficient RL agents in real-time setting.

Published as a conference paper at ICLR 2025
HANDLING DELAY IN REAL-TIME REINFORCEMENT LEARNING
Ivan Anokin Rishav Rishav Matthew Riemer , Stephen Chung
Irina Rish 126 Samira Ebrahimi Kabou 136
Mila Uniierrsité Montréal UUniversity Calgary University of Cambridge *CIAAR AI Chair
IIBM Research
i Lvan. wnwkhinemiien quebec
ABSTRACT
Real-time reinforcement learning (RL) introduces several challenges. First, policies are constrained to a fixed number of actions per second due to hardware limitations. Second, the environment may change while the network is still computing an action, leading to observational delay. The first issue can partly be addressed with pipelining, leading higher throughput and potentially better policies. However, the second issue remains: if each neuron operates in parallel with an execution time of т, N-layer feed-forwant network experiences observation delay of TN, Reducing the number of layers can decrease this delay, but at the cost of the network's expressivity. this work, we explore the trade-off between minimizing delay and network's s expressivity. We present theoretically motivated solution that leverages temporal skip connections combined with augmeeed ob- servations. We evaluate several architectures and show that those incorporating temporal skip connections achieve strong performance across various neuron ex- ecution times, reinforcement learning algorithms, and environments, including four Mujoco tasks and all MinAtar games. Moreover, we demonstrate parallel neuron computation
accelerate inference by 6-350% standard hardware. Our investigation into temporal skip connections and parallel computations paves the way for more efficient RL agents in real-time setting.
INTRODUCTION
Miniörid
(b)
IUP ot Skip mup
Wi Kith aug
CIN oug
CWN LSTM
EXN LST
C3 Wi 301
SUP et
200
MRU
CWN
CNN
Figure 1: (a) Parallel computations layers speed-up inference time. Speed-up on GPU is achieved using default Pytorch software and widely accessible Nvidia GPU. (b) Normalized averaged perfor- mance and standard error of agents in parallel computation framework. Agents with skip connections and distory-augmentee observations exhibit strong performance. Performance is averaged across the following environments: HalfCheetah-v4, Walker2d-v4, Ant-v4 and Hopper-v4 on Mujoco, and DoorKey-5x5- -x055-0 MiniGrid.
all six environments MinAtar, and Empty- Hamptt-Rannom-55x- Performance on Mujoco also averaged across four different neuron execution times.
Neural network inference presents several challenges in real-time reinforcement learning (RL) as the environment can change significantly, even during the networks' inference process. One major challenge is that the inference time directly impacts throughput, the number of actions the agent can produce per second. High throughput is important domains like robotics, algorithmic trading, and real-time gaming, where frequent decision-making can significantly improve policy performance.

---

Published as a conference paper at ICLR 2025
To address this challenge, straightforward approach is to speed up inference by employing pipelining techniques. In a pipelined architecture, instead of waiting for the entire neural network to complete its forward pass input as soon as it produces its output for the current one (Carreira et al. 2018 luzzolino et al. 2021). This approach increases the throughput of neural network (see Fig. Tap. as layers are effectively working in parallel but on different inputs. Throughout this paper, we refer to this approach as the parallel computation framework.
imput before processing the next, each layer begins processing the subsequent
However, even within the parallel computation framework, a traditional N-layer feed-forward neural network still suffers from another issue known observational delay: the agent's action at time step t is based on an observation from time step t - N6 where denotes execution time of each layer. pipelined system, each layer is processing data from different time steps simultaneously layer 1 processes input from time i, layer 2 processes the output of layer 1 from time 6, and so on (as illustrated in the center graph of Fig. 2) This challenge leads us the central question of this paper:
This delay arises because,
If we use parallel computations of layers, how do we address observational delay?
Reducing the number of layers can mitigate delay but limits the network's s expressivity. To overcome this, propose using temporal skip connections. Traditionally, skip connections used to stabilize training and allow gradient flow in deep networks (Ronneherger et al. 2015 He et al. 2016). However, within the parallel computation framework, skip connections offer another advantage: they do not only shortcut between layers along depth, but also along time, by sending activations forward in time (see the rightmost graph in Fig. 2 This temporal application of skip connection in the parallel computation setting reduces the observational delay. Nevertheless, the computational paths through these temporal skip connections are shorter than those without them and thus offer limited expressivity compared to longer paths through more neurons.
Sequential Compatanions
Parallel Computations
With Stip Connections
idic pertod
Arote
1-2
7-A 1+6 1-76
1+S1
Figure 2: Computation flow of agents. Left graph represents sequential computations and the central graph parallel computations of layers. is execution time of each neuron (or layer). All nodes at each column are available at the same time and can be processed further in parallel. The right architecture with skip connections exhibits less delay as it performs shortcuts along time-steps.
We explore the trade-off between delay and network expressivity and investigate various types of architectures to find an optimal balance. Our theoretical analysis quantifies the impact of skip connections on reducing the regret associated with observational delay. Furthermore, we justify the importance of augmenting observations with past data in architectures with temporal skip connections. Experiments confirm the importance of skip connections and history- augmeeted observation (see Fig. lb and our analysis shows that the skip connection offers fast but less refined path for processing inputs, while the main connections provide a slower but more refined path. Our results show that in many environments this allows the policy in parallel computation setting to achieve similar performance to an oracle agent with an instantancous forward pass, provided the inference time of a layer is not large.
While the parallel layer computation approach and temporal skip connections were proposed before to accelerate predictions image (luzzolino et al. 2021 Fischer et al. 2018) and video (Carreiral jet al., 2018) Kugele et al., 2020) domains, this is the first application in RL a domain where one bad action critically impact the entire trajectory due to the agent's influence the environment.
2

---

Published as a conference paper at ICLR 2025
To summarize, we introduce solution to real-time RL: speeding up inference time by parallel computations of layers and addressing associated observational delay. We demonstrate that parallel computations significantly improve throughput modern hardware like GPUs. To address the observational delay, we provide a theoretically justified solution using temporal skip connections and distory-augment observations. Our experiments demonstrated its effectiveness across various cases, paving the way for more efficient RL agents in real-time setting.
2 RELATED WORK
Parallel computations of neurons (or layers). Parallel processing of information is consistent with
popular mathematical models of the human cortex (Tomita et al., 1999 Betti & Gori 2019 Kubilius et al.) 2018 Larkum 2013). where neurons operate asynchronously y. Inspired by this, several attempts have been made to parallelize neural networks, aiming to maximize processing resource utilization and reduce latency. CCrreira et al. (2018) introduced parallel video networks that employ parallel layer computations and temporal skip connections, significantly boosting throughput (or frame rate) during inference. Similarly, Juzzolio et al. (2021) explored this approach for still images, enabling fast "anytime predictions" that improve over time. Additionally, Fischer al. (2018) provided a theoretical framework for these ideas, and Kugele et al. (2020) applied them for Spiking Neural Networks on image and video domains. Unlike these approaches, we apply these ideas in RL.
Several studies have proposed techniques to handle parallel computations of layers not only during the forward pass but also during the backward pass (by modifying or replacing backpropagation) in
both training and inference. Sideways (Malinowski et al., 2020 2021) achieved this with approximate backpropagation in the video domain. Asynchronous Coagent Networks (Kostas et al., (0200 and Chung (222)) introduced methods where each neural network unit operates independently to maximize its own reward, enabling asynchronous inference and training of neurons. However, Sideways focuses on video data, and both Coagent Networks and Chung (2022) are limited to small number of neurons, making scalability to larger networks challenging compared to our approach.
Delay in RL. Early works on handling delays in traditional RL settings include (Walsh al. 2007]
Bander & White 1999; Katsikopoulos & Engelbrecht 2003 Altman & Nain 1992). Notably, (Katsikopoulos & Engelbrecht| (003) was the first to introduce the notion of Delayed Markov Decision Process (DMDP). However, their results have not been fully translated into Deep RL.
Recent efforts have addressed delays in Deep RL. Firoiu et al. (2018) tackled delay by predicting future observations, while Wang et al. (2023) trained the critic without delay, augmented state information with historical data, and used sulf-supervised losses to improve performance on DMDPs. The RLRD method (Bouteiller et al. (2021) further enhanced the critic by augmenting its input with future on-policy actions available due to delay, resulting in more accurate value estimations.
These approaches consider delay as external factor to the agent. However, our agent inherently introduces delays due to parallel computations, resulting in additional interplay between the agent's architecture and these inherent delays. This allows us to introduce more inductive biases, such as temporal skip connections, into the neural network architecture to effectively mitigate such delays.
PROBLEM SETTING AND NOTATION
2018) is defined as a tuple AS Po. P,)), where S and A are the state and action spaces, respectively. Po specifies the initial state distribution such that Po(s) is the probability of a state s S being an initial state. P specifies the state transition probability such that P(s', llss,, r|s, a) is the probability of reaching to new state s' E S with an immediate reward r E after taking an action a E AA at a state & E S. 7 E [0, 1) is the discount factor, which weights the importance of rewards at future steps. It is typically assumed that MDPs are "pausable" i.c. that the agent and environment proceed in a turn-based interaction framework where each waits for each other before proceeding. In realtime environments, however, the agent and environment each proceed at their own pace (Travnik et al. 2018).
A Markov Decision Process (MDP) (Puterman) 1994 Sutton & Barto
Then define asynchronous delayed MDP as a tuple (S, (S,A,P5 Po. Pd 7, (,d), which extends the standard notion of an MDP by defining 3 - the default behavior policy of the system between actions taken by the agent, E N is amount of delay, and fixed interaction frequency, indicating the number of
3

---

Published as a conference paper at ICLR 2025
environment steps between the agent's actions the same time. P is the delayed transition probabil- ity function, which we will define to model the environment's 's dynamics under the influence of both the agent and the default policy. (s R 8, a) Es
I P(sk. Tx 8k-1, sk-1,ak-1) 8o S, ag = a
where 80 = 8, 40 a, Sd s' and R = Σi= k= Th is the cumulative reward over d steps. The delayed transition probability function pd captures the probability of transitioning from state to state s' over steps, starting with the agent's s action a and followed by the default policy 3.
We extend asynchronous delayed MDP further to asynchronous delayed observation MDP (asyn- chronous DOMDP) to define that agent observes history of past states (#t-dN; Bt-d(N-1)- 88 where N will define number of layers in neural network later.
3.1 FORMALIZING THE PARALLEL COMPUTATION FRAMEWORK
We execute layers of our neural network in parallel to speed-up inference in realtime settings. Thus, need to incorporate computational constraints related to the parallel computations. We define 5 as neural execution time i.c. the number of environment steps that pass during the computation of single neural network layer. 8 > 1, a default policy takes control for 8 steps. For 8 1, we either accelerate the environment or group [1/6] layers together to form new macro layer. The agent's s policy, x, represented with N-layer neural network, observes a history of past states, his intervals of ö: π (hs) π (+--N6] . 82-7 St- (87). The policy must respect the computational constraint that it cannot process past state S through more than [ layers before producing action at As such, our goal to find a policy I (hs) that maximizes cumulative rewards in a usynchronous DOMDP with delay of [o], subject to the constraint that T.(s a) < [ Ak {[ō],-. [No]} where (s; a) number of layers between 8t-k and a. We can view a neural network as a directed acyclic graph (DAG), where the nodes represent input data intermediate computational results, and the edges represent the computational operations. L(s, a) is the path this graph from to a. We define neural network that only consists of the longest paths in π (hs) as vanilla feed-forward neural network. All other paths will be referred to as temporal skip connections.
3.2 SOURCES OF REALTIME REGRET
We define delay regret as the difference in performance of the optimal policy in the original MDP and the optimal policy in asynchronous DOMDP. Similarly, an inaction regret is defined as the difference in performance of the optimal policy in the original MDP and performance of default policy 3 in asynchronous DOMDP. We give the formal definitions in Appendix
4 METHOD
We show benefits of temporal skip connections for minimization delay regret bound Adelay in Proposition [ and benefits of temporal skip connections combined with the state augmented with recent actions in Proposition 2
4.1 ADDRESSING DELAY
In a vanilla feedforward neural network deployed to address realtime RL, actions aj are based on states 86-N8 delayed by N6 steps. Temporal skip connections directly alleviate this issue as actions a₂ are now based on a set of N states (st (St-N6;-., 81-83- J. As a result, skip connections lead to a tighter lower bound on delay regret, Adcty (t) in worst case environment.
Proposition 1 (Tighter Delay Regret Bound): For any vanilla N layer neural network without tem- poral skip connections in parallel computation framework, the regret resulting from delay A delay (1) after L steps in a worst case environment can be lower bounded by:
vanilla delay
(1) E & -Ppninimox) [N6]])
where Pminimax == mil.ee..... m ((s'(s, a) is measure of environment stochasticity. How- ever, a network with temporal skip connections achieves fighter bound on delay regret A dkip
delay (t):
4

---

Published as a conference paper at ICLR 2025
delay delay (t)
E 12(t(1 - (Pminimax) ()1))
which is less sensitive to the environment stochasticity measured by Pminimuxx-
Following the lower bound on delay regret established in (Riemer et al., 2024). the delay regret depends on the number of stochastic environment steps between an action and the input used to produce it. Temporal skip connections enable the policy to incorporate the state from [ō] steps ago, whereas a vanilla feedforward network can only condition on steps [N68] in the past. This can lead to an exponential reduction in the policy's inaccuracy caused by the stochasticity in the environment, which becomes especially prominent for environments that are highly stochastic or neural networks with large number of layers.
4.2 ADDRESSING TRAINING STABILITY
Another difficulty with vanilla feedforward neural networks, even with parallel inference, is that the effective delayed decision process where actions az are taken based on the delayed state 8t-Mi -a is non-Markovian. This fact will lead to unstable learning in many environments as typical RL algorithms are not expected to converge in this regime. Meanwhile, this is another key issue that can be addressed with temporal skip connections and augmenting state with recent actions. With this architecture, we have access to all previous actions when computing a and thus can consider stable augmented state space &, (8 N---8tt-11 1) that the decision process is Markovian with respect to as p(r,, 814 [8,, a+) is stationary and stable over time.
Proposition 2 (Markovian Property): A vanilla layer neural network without skip connections in parallel computation framework bases its actions on the delayed state S:-N8 and experiences non-Markovian environment transitions p(re, St+1|st-N6- as) without having access to Ot-N6:t-1 = dt-N:;-.. -... 0t-1- These actions available when using temporal skip connections, making environ- ment Markovian based on the augmented delayed state space sn = =8t-NN at (4:No:t-1).
The vanilla network is non-Markow as it depends on past actions from changing policy. Skip con- nections and past actions remove this non-stationary dependency. This property can be illustrated with the following example: If the action a at time is based on the state st-1, the transition probability function becomes (( 0z) ((s'||s st at) P(sr ) (t22). While P(s'Is (ss'|s,. az) is stationary, the term x -84 is non-stationary because the policy changes throughout learning. However, by augmenting the state with a--1, the policy term disappears, and the transition function becomes stationary. Proposition?? to explain optimization issues related to delay that may be present even when the environment is deterministic within the parallel computation framework. When using earlier state to generate policy, the effect of the actions of that policy also depend on the actions taken between action com- putations because of the non non-Markon Markovian nature of that input representation. As such, the transition dynamics appear nonstationary as the policy itself changes and appear stochastic when the policy is stochastic. This serves slow down learning and leads to instability that hurts sample efficiency as we demonstrate in our experiments.
2 is an important point to emphasize as it extends Proposition []
When using temporal skip connections, our policy conditions on N previous states and actions while only the most recent one 8:-8 and a;-s are needed in our derivations of Propositions and However, utilizing these previous states is still helpful within the framework of parallel layer computation because we are able to consider more neural network layers for states that are more outdated. This way the policy can be more expressive with respect to previous states than is the most recent state. This is useful feature in environments that are relatively stable and predictable across each step while requiring complex high-level reasoning. For example, the overall structure of the maze may stay constant across steps, processing a higher level plan of action with more recent steps being used to encode the representation of the agent's current location. Indeed, our experiments validate the value of adding more layers with outdated states in delayed variants of popular environments within the deep RL community.
maze environment distant steps can be useful in
Performance gap. Propositions I. highlight the performance gap between agents with and without skip connections and last-action augmentation, in terms of delay regret. Besides, Propositions T and 3 (Appendix ()) provide insights into the performance gap between the instantaneous and real-time
5

---

Published as a conference paper at ICLR 2025
actors in a parallel computation framework under worst-case environments. Notably, even with skip connections, the delay 8 remains. In contrast, the instantancous actor does not experience any delay or inaction regrets. The closer the environment is worst-case scenario, the more pronounced the performance gap becomes.
However, when using skip connections, state-augmentttion with last actions, deterministic environ- ment, and neural execution time less than 1, Propositions [ falls short to differentiate between instantaneous and real-time actors. In this case, the real-time actor also exhibits zero delay and inaction regrets. Nevertheless, we anticipate the real-time actor to perform worse than the instanta- neous actor. Skip connections may lack the expressivity needed to efficiently differentiate between distinct environment states, effectively perceiving the environment as stochastic. This limitation makes Proposition relevant again.
4.3 ALGORITHM
We apply Soft Actor Critic (SAC) (Haarnoja et al. 2018)) for continuous action-space environments or PPO for discrete]] ". We train critic without delay following suggestions from (Wang al. 2023) and an actor with appropriate delay and restriction following Subsection with vanilla backpropagation. We employ last action repetition as default policy, 3, if 1.
The basic structure of our SAC algorithm is presented in Algorithm 1 To begin collecting experience, we initialize the first observation from the environment and set initial hidden activations, depicted in Fig. 2 to zero While the critic is trained online without delay, actor is trained within the parallel computation framework by unrolling on sub-trajectories sampled from the buffer (with hidden activation set to at the first state of sub-trajectory), allowing all weights to be available for backpropagation. For details on the PPO variant of the algorithm, refer to Appendix A
Algorithm 1 Soft Actor-Critic Algorithm with parallel neuron computation.
1: Init an actor and critic with random parameters.
2: Set initial state to be so, hg hi where h/ is activations for layer j at a time step 0.
3: Wrap the environment with sticky actions or repeating observations wrapper needed based on neural execution time.
4: for E 0....., do
5: Oe, 14 hi Actor(st. ....... h hg (Query current policy for the next action and next Actor's hidden activations given current observation and hidden activations)
6: Take the action a and receive {re, $141} from the environment.
7:
Put {st, dz, Tt, 1 } to the buffer.
8:
Sample transition {si, di, Fi, 81+1 from the buffer and update the critic it.
9:
Sample sub-trajectory from the buffer {&t, ag Fsy 81+1; -ettk 85+2]
Init hg- ..., f and simulate the actor dynamic forward on given sub-trajectory.
11: Update the actor on the last transition of the sub-trajectory (via tuck-propagation through time if needed)
12: end for
5 EXPERIMENTS
We perform main experiments on Mujoco Todorov al.) 2012), MiniAtar (Young & Tian 2019) and MiniGrid (Chevalier- Bievsllie-Boiiveert et al. 2023) environments. Mujoco has a continuous action space, while MiniAtar and MiniGrid have discrete action spaces. We train our agents using SAC for Mujoco and PPO (Schulman et al., (017) for MiniGrid and MinAtar. We report mean and standard error (SE) in all our plots and experiments unless stated otherwise. We normalize return for every environment and neuron execution time with respect to vanilla SAC or PPO performance without delay, Additional architectural and training details can be found in Appendix E
Full code is available at https: / / thub. com/ avecpl ez rea me me-ageen
SSnnee initialize the hidden activations to zero, the first delayed actions can be ineffective. We also tried initialize them by performing instantaneous forward pass the first ohservation, but saw improvement.

---

Published as a conference paper at ICLR 2025
5.1 MAIN RESULTS
We aim to validate our theoretical predictions that architectures with skip connections outperform those without, and that sastory-augmented observations will further enhance the performance of agents using skip connections according to Propositions [ and
We explore the following architectures within the parallel computation framework:
1. Default architectures: three-layer MLP or five-layer Convolutional Neural Network (CNN);
2. Augmenting observations with historical states and/or actions in the default architectures (see Appendix
C for details):
3. Replacing the second last fully connected layer with an LSTM the default architectures;
4. Adding skip connections to the default architectures;
5. Augmenting observations with historical states and/or actions in the architectures with skip connections.
6. The RLRD (Bouteiller et al. 2021 with neural execution time of one in Mujocd
We tested these architectures on four Mujoco environments (HalfCheetah-Vee v4, Walker2d-v4, Ant- v4, and Hopper-v4) and across four different neuron execution times (ranging from one four). Additionally, we also tested these architectures on all six MinAtar environments and two toy MiniGrid environments Random-5x55 applied to both MinAtar and MiniGrid. A summary of results across Mujoco, MinAtar and minigrid presented in Fig. [6 and detailed quantitative results can be found in Appendix !
and DoorKey-5x5
where a neuron execution time of one is
Our findings show that adding skip connections to default MLP/CNN architectures significantly enhances performance. Additionally, augmenting observations with historical states and/or actions further improves performance, aligning with Propositions [ and 2 Fig. presents more detailed results for the Mujoco environments with varying neuron execution times. It demonstrates that agents with skip connections and state augmentation consistently match exceed the performance of agents without skip connections and RLRD across nearly all tested environments and neuron exccution times.
Moreover, as expected, in Fig. lb we observe that neither LSTMs nor tiistor--ugmeente observations offer much benefit to the default architecture without skip connections in the Mujoco MinAtar environments. In contrast, history augmentation and LSTMs significantly improve performance in the MiniGrid environments, likely due to their underlying POMDP structure, where historical information essential for better decision-making. We conjecture that temporal skip connection is also helpful for POMDP, it allows agents integrate historical data from different time steps.
HalfCheetah-vi
Walker2d-v4
Ant-v4
Hopper-vi
Normalsed Retum
0.6
1.0
0.0
0.8
1.0
0.5
0.5
0.6
Neuron Execution Time
Neuron Execution Time
Neuron Execution Time
Neuron Execution Time
three
RLRD
Figure 3: The performance of different agents and RLRD method on Mujoco. The agent with skip connections performs as well as, or better than, other agents in general. SAC without delay, which has normalized performance of one, is omitted from the plots. The shaded area indicates SE across 3 seeds.
Performance drop. We aim to quantify the performance gap between an agent in a standard MDP without delay and our best agent in the parallel computation framework. Additionally,
RR.DD addresses DOMDP rather than policy-constrained DOMDP, making it not directly comparable to other choices. We use the publicly available RLRD code to obtain the results.
7

---

Published as a conference paper at ICLR 2025
interested in identifying scenarios where it may be possible close this gap between the agent in these two settings.
Fig. indicates that, in many cases, there is no drop in performance when compared to the vanilla SAC without any delay. For example, this holds true for Hopper across all neuron execution times, as well as for Walker and Ant with neuron execution times of one and two.
HalfCheetah is the only Mujoco environment where a significant performance drop occurs compared to the agent without delay. To address this, accelerated the environment making time between consecutive observation twice shorter. This adjustment resulted in a normalized performance of 0.87 + 0.06 for the agent with skip connections, bringing it closer the performance of the vanilla SAC with instantaneous actions.
In MiniGrid, the performance drop caused by parallel computations is relatively minor, whereas in MinAtar, the drop is more pronounced (refer to Appendix Tables [0 and IT). We conjecture that rendering skip connections alone insufficient to close the performance gap in MinAtar with considered neural execution time.
One potential solution could involve increasing the neural network's expressivity or reducing the neural execution time. However, if the architecture and its associated delay are fixed, the optimal solution achievable with this architecture may be strictly worse compared an instantaneous actor, as discussed in Section 422
Overall, the results show that in most environments, an agent with skip connections operating in the parallel regime can achieve performance comparable to an agent without delay, while significantly improving inference time. However, in more complex cases, skip connections alone may not be sufficient match the performance of an agent without delay.
5.2 ABLATION STUDY
To identify the most effective type of skip connection, we conducted ablation study comparing three options: projection to action, projection from observations, and combination of projection action with residual connections, shown in Fig. 4 For simplicity, we refer to these sometimes as proj-to-action, proj-from-obs, and proj-to-action & res, respectively. Additionally we tested all possible forward skip connections between layers in Mujoco, denoting this option as All Skips. The results of the ablation study are summarized in Table I The findings help guide our selection of the default skip connection type for each environment. Based on the results, we adopt prej-from-obs for Mujoco environments and proj-tu-action & res for MinAtar and MiniGrid, referring to these configurations as "skip connections" throughout the rest of the paper.
A detailed ablation study other architectural choices, including the number of layers and augmen- tation strategies, is provided in Appendix(
Residiat Connections
Projections
Projections Frev Obuervation
Drys
1-28 Tiwe
7e8 14:38
1-6 246 24
7ow
Figure 4: Illustration of different skip connections. 8 represents execution time of each neuron.
Disentangling architectural benefits of skip connections. To disentangle the architectural ben- efits of temporal skip connections from their impact on reducing delays in parallel computation framework, we report the performance of a vanilla SAC pipeline, both with and without traditional skip connections and without any computational delay, in Table 2 The results show significant
8

---

Published as a conference paper at ICLR 2025
Table 1:
Comparison between different skip-connection. Normalized averaged performance and standard error of agents are reported. For each task mean and SE is computed based on three seeds.
MuJoCo
MinAtar
MiniGrid
Projections from Observation
0.79 0.04
0.45 + 0.05
0.91 + .006
Projections to Action
0.78 + 0.04
0.46 + 0.04
0.95 + .002
Projections to Action Residual All Skips
0.77 0.05
0.52 = 0.06
0.96 + .002
0.75 + 0.05
performance improvement from traditional skip connections only in the Ant environment. Therefore, we believe that the performance gain from temporal skip connections may be due factors beyond just reducing computational delay in only the Ant environment in parallel computation framework.
Table 2: Vanilla (without delay) SAC with and without skip connections.
Halfcheetah-v4
Walker2d-v4
Ant-v4
Hopper-v4
SAC
11739 283
4415 + 227
3595 1027
2672 + 463
SAC w/ skip connections
11250 + 32
4597 + 100
5719 + 176
2451 + 52
5.3 ANALYSIS
Distillation. We aimed to determine whether performance limitations were due to the RL algorithm or the expressivity of our architecture. To investigate this, we used a distillation approach (employing DAgger (Ross et al. 2011)) to transfer highly-performing vanilla SAC HalfCheetah policy (return of 11, 000) into our agent with skip connections and a neuron execution time one. However, the distilled agent achieved return only 7590 + 93, which was comparable to training the same architecture directly with SAC (7892 + 378). This suggests that the performance bottleneck is not algorithm-spe speciiic but rather consequence of the reduced expressivity of the agent's architecture in capturing the true state.
Analyzing skip connections. We hypothesize that skip con- nections enable the generation of fast & effective actions, while subsequent layers refine these actions. removed various projection and connection pathways in a three- layer proj-to-action agent in Ant-v4 environment with a neuron execution time of four (Fig. 5). Specifically, we removed projections from observations, projections from the first-layer representations, and connections from the second-laye repre- sentations to the action space. The agent performed poorly with- the first projections, but still achieved some non-zero return when the connections from the last layer were removed, supporting our hypothesis.
validate this, we
USEE
Figure 5: Removing different con- nections in the proj-to-action agent. Mean and one SD across 100 episodes are reported.
5.4 INFERENCE TIME SPEED-UP
We evaluated the speed-up caused by parallel computations of neurons on various hardware platforms, observing significant improvements inference time when utilizing a GPU. Fig. Ta illustrates the percentage improvement in inference speed as the number of layers increases across different hardware configurations.
GPU. For GPU setting we measured performance speed-up single A A100SXM4 GPU with 40 GB memory. The tests were conducted on a deep Multilayer Perceptron (MLP) with batch size of and hidden layer size of 256 for all layers. For parallel computation concatenated all inputs to the layers and combined all layer weights into one large sparse matrix. For agents without skip connections, this matrix has block-diagonal form. We then used either regular
the GPU, naively

---

Published as a conference paper at ICLR 2025
sparse matrix multiplication to compute the output for each layer. In Fig. la these approaches are labeled as GPU and GPU (sparse weights), respectively. The MLP was implemented in Py Torch, utilizing Py PyTorch's 's sparse tensor representations and sparse matrix multiplication for the GPU (sparse weights) approach.
Fig. shows that the parallel computations the GPU accelerate inference time considerably for deep neural networks. Regular matrix multiplication reached its peak performance speed-up around 30 layers, after which the speed-up started decline; sparse matrix multiplication surpassed regular matrix multiplication at around 30 layers and continued to increase almost linearly with the number of layers achieving 350% speed-up for 100-layers MLP in our test setting.
CPU. We evaluated the benefits of parallelizing layers using C++ mulli-threading on a CPU with 32 cores and 32 GB of RAM. Our tests showed 6% speed-up for 10-layer network, but gains dropped to 0.111% for networks with over 20 layers due to thread synchronization overhead. We used a batch size of 10,000 and hidden dimensions of 10,000 in MLP, with similar trends observed across other configurations. The limited speed-up can be attributed to the Eigen C++ library, which optimizes matrix multiplications through multi-threaddin, reducing the impact of further parallelization. In contrast, parallelizing naive matrix multiplications (without Eigen's 's optimizations) scales linearly with the number of layers, doubling for 2 layers, tripling for 3, and so on, until performance plateaus around 40 layers.
LIMITATIONS
One important assumption make in our experiments is that we have fixed neuron execution time (d) which is not the case in real world environments where can be stochastic. We propose this future line of work where methods can explore handling stochastic 8. Additionally, limit our experiments to at most 5-layer neural network, as scaling vanilla RL methods to deeper architectures is non-trivial and often requires additional losses training tricks (see OObando-Ceron ee al.] (2024)). Finally, we believe neuromorphic computing will benefit from our approach the most due to parallel nature of our approach. However, since neuromorphic chips are widely available our immediate impact on the field may be limited.
7 CONCLUSION
Our work addresses the challenge of delays in reinforcement learning caused by parallel computations of neurons. We theoretically and experimentally show the advantages of architectures with temporal skip connections and history augmentation. These architectures demonstrate robust performance across various environments and neuron execution times. Furthermore, we demonstrate that when neuron execution time is sufficiently small, agents in the parallel regime can achieve similar perfor- mance to agents in the instantaneous regime, while significantly accelerating inference time on GPUs. This property is particularly beneficial in dynamic settings requiring rapid decision decision-making, However, when neuron execution times are bigger, or environments are more complex (c.g., MinAtar), the performance gap between the instantaneous and parallel regimes widens. Further research is needed to either mitigate this gap or identify cases where may be unavoidable. Future studies could also explore asynchronous neuron computation and leverage hardware optimizations to further enhance speed-up.
8 ACKNOWLEDGMENT
We acknowledge the support from the Canada CIFAR AI Chair Program and from the Canada Excellence Research Chairs Program. The research was enabled in part by computational resources provided by the Digital Research Alliance of Canada and Mila Quebec AI Institute. IA thanks Nishanth Anand and Arsenii Kuznetsov for helpful discussions and comments, and Serge Zakharoy for his consultation on Eigen C++
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Eitan Altman and Philippe Nain. Closed-loop control with delayed information. ACM sigmetrics performance evaluation review, 20(1):193-204 1992.
James L Bander and CC White. Markov decision processes with noise-corrupted and delayed state observations. Journal of the Operational Research Society, 50:660-668, 1999.
Alessandro Betti and Marco Gori. Backprop diffusion is biologically plausible. arXiv preprint (9Xiv:9912.4440 2019.
Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Binas. Rein- forcement learning with random delays. In International Conference on Learning Representations, 2021. httl/ttts///opeereeeee http:///wweaaaaa/e//e//////aaaaa wtw..eeet.orrmmm1.oaFFnnnn.....
João Carreira, Viorica Päträucean, Laurent Mazare, Andrew Zisserman, and Simon Osindero. Mas- sively parallel video networks. European Conference on Computer Vision (ECCV). DeepMind and Department of Engineering Science, University of Oxford, 2018.
Maxime Choisei Bolun Dai, Mark Towers, Rodrigo Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.113811 2023.
Stephen Chung. Learning by competition of self-interested reinforcement learning agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6384-6393, 2022.
Vlad Firoiu, Tina Ju, and Josh Tenenbaum. At human speed: Deep reinforcement learning with action delay, 2018
Volker Fischer, Jan Köhler, and Thomas Pfeil. The streaming rollout of deep networks-towards fully model-parallel execution. Advances in Neural Information Processing Systems, 31, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905. 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Michael L. luzzolino, Michael C. Mozer, and Samy Bengio. Improving anytime prediction with parallel cascaded networks and a temporal-difference loss. In Proceedings of the 35th Conference Neural Information Processing Systems (NeurIPS). NeurIPS, 2021.
K.V. V. Katsikopoulos and S.E. Engelbrecht. Markov decision processes with delays and asynchronous cost collection. IEEE Transactions Automatic Control, 48(4):568-53 574, 2003. doi: 10. 109/ TAC.2003.809799.
Michael Kearns and Satinder Singh. Ncar-optimal reinforcement learning in polynomial time. Machine learning, 49(2), 2002.
James Kostas, Chris Nota, and Philip Thomas. Asynchronous coagent networks. In International Conference on Machine Learning, pp. 5426554264450 5435. PMLR, 2020.
Jonas Kubilius. Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel LK Yamins, and James DiCarlo. Cornet: Modeling the neural mechanisms of core object recognition. BioRxiv, pp. 408385, 2018.
Kugele, T Pfeil, M Pfeiffer, and E Chicca. Efficient processing of spatio-temporal data streams with spiking neural networks front, 2020.
Matthew Larkum. cellular mechanism for cortical associations: an organizing principle for the cerebral cortex. Trends in neurosciences, 36(3): 36(3): 2013.
11

---

Published as a conference paper at ICLR 2025
Mateusz Malinowski, Grzegorz. Swirszez, Joao Carreira, and Viorica Patraucean. Sideways: Depth- parallel training of video models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Mateusz Malinowski, Dimitrios Vytiniotis, Grzegorz Swirszez, Viorica Patraucean, and Joao Carreira. Gradient Borward-propagaae for large-scale temporal video modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9249-9259, 2021.
Johan Obando-Ceron Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep rl. arXiv preprint urXiv:2402.086600 2024.
ML Puterman. Markov decision processes. 1994. Jhon Wiley & Sons, New Jersey, 1994.
Matthew Riemer, Gopeshh Subbaraj, Glen Berseth, and Irina Rish. Enabling realtime reinforcement learning at scale with staggered asynchronous inference. arXiv preprint arXiv:2412.14355 2024.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention (MICCAI), pp. 234 241. Springer, 2015.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. Workshop and Conference Proceedings, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: an introduction, 2nd edn. adaptive computation and machine learning, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033 IEEE, 2012. ISBN 978- 778-1-4673-1737-5. URL http: uni trritrreee..... http:///ee..oooooooooooooo///ttt
Hyoe Tomita, Machiko Ohbayashi, Kiyoshi Nakahara, Isao Hasegawa, and Yasushi Miyashita. Top- down signal from prefrontal cortex in executive control of memory retrieval. Nature, (67544) 699-703, 1999.
Jaden B Travnik, Kory W Mathewson, Richard S Sutton, and Patrick M Pilarski. Reactive reinforce- ment learning in asynchronous environments. Frontiers in Robotics and AI, 5:79, 2018
Thomas J. Walsh, Ali Nouri, Lihong Li, and Michael L. Littman. Planning and learning in environ- ments with delayed feedback. In Joost N. Kok, Jacek Koronacki, Raomon Lopez de Mantaras, Stan Matwin, Dunja Mladenič, and Andrzej Skowron (eds.), Machine Learning: ECML 2007, pp. 442-453, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540-74958-5.
W. Wang, Han, X. Luo, and Li. Addressing signal delay deep reinforcement learning. In The Twelfth International Conference Learning Representations, Virtual Event, October 2023. ICLR.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for more efficient reinforcement learning experiments. CoRR, 03176
2019. URL /ttt:://axxi..... wwtp://wwwwwgraaaaa// 903
12

---

Published as a conference paper at ICLR 2025
A PPO ALGORITHM WITH PARALLEL NEURON COMPUTATION
The basic structure of the PPO algorithm is presented in Algorithm To begin collecting experience, we initialize the first observation from the environment and set initial hidden activations, depicted in Fig. 2 by performing instantaneous forward pass on the first observation. While the critic is trained online without delay, our actor is trained within the in-parallel computation framework by unrolling recent sub tajetrajcctories stored in the buffer (with hidden activation reset with instantaneous forward pass), allowing all weights to be available for backpropagation.
Typically, in PPO, the critic and actor share common backbone. However, to enable online training of the critic without delay, we employ separate neural networks for the critic and actor.
Algorithm 2 PPO with parallel neuron computation.
1: Init an actor and critic with random parameters.
2: Set initial state to be so,
where hy is activations for layer j at a time step 0.
3: Wrap the environment with sticky actions repeating observations wrapper if needed based neural execution time.
4: for E 0,,.... do
5: de.
h Actor(st. h!...., h (Query current policy for the next action and next Actor's hidden activations given current observation and hidden activations)
6: Take the action a, and receive {re, 8 from the environment and put {st, at, r't to the buffer.
7: if buffer is full then
8:
Compute gae return on the buffer. for / E 0.
9:
n_epochs do
10:
Init hg. .... hg and simulate the actor dynamic forward and get critic output without delay the collected buffer.
11:
Compute the PPO loss
12:
Update the actor and the critic (via back-propagation through time if needed) end for
13:
Empty the buffer
15: end if
end for
B SUPPLEMENTARY EXPERIMENTAL RESULTS
Atari games. We present preliminary results small subset of Atari environment in Table We use the same architectures and hyper-parameters as we used for MiniGrid experiments. As standard choice in Atari we augment the state with 4 past observations, grayscaled observations, for all actors and bin reward to be +1,0, -1 by its sign, and repeat each action four times for agents.
Table 3: Subset of Atari games average returns after training on I mln observations. The results are averaged across three seeds, mean and standard deviation are reported. PPO denotes vanilla PPO without inference delay. CNN and CNN with skip connections denote architecture executed withing parallel computation framework with neural exccution time of one.
PPO
CNN w/ aug
CNN w/ skip & aug
Boxing-v5
21.4
2.8 1.4
2.6 - 3.6
Breakout-v5
7.6
6.3
6.9 - 5.5
BattleZone-vS
3683 + 375
5336 + 1366
4943 + 984
SpaceInvaders-v5
386
456 + 31
Assault-v5
186
660
712 37
Bowling-v5
37.2
37.9 4.9
40.9 + 9.6
Freeway-v5
22.18 + 0.34
22.18 0.34
22.18 0.34
13

---

Published as a conference paper at ICLR 2025
Stochastic environments. We also conducted experiments in stochastic environments by introduc- ing "sticky actions" MinAtar, where the agent's 's last action was repeated with a probability of 0.25 (Table 4. This modification led to decline in performance across all agents; however, the relative trends remained consistent.
Table 4: Results after training on 10 million samples in MinAtar games with a sticky action probability of 0.25. The mean and standard error across three seeds are reported. The neuron execution time is 1. PPO refers the standard implementation PPO without inference delay.
Breakout-v0
Seaquest-v0
Freeway-v0
Asterix-v0
SpaceInv-v0
PPO CNN w/ aug
8.29 1.16
21.48 + 8.55
60.15 + 1.53
25.38 2.40
91.38 = 12.33
3.38 + 0.32
2.72 + 1.15
27.59 + 2.49
4.25 + 1.86
25.08 + 0.43
CNN w/ skip aug
6.29 0.18
5.38 + 0.83
53.51 + 0.84
9.94 + 2.03
40.47 + 1.04
Sequential baseline. Throughout the paper, use an agent with pipelining (parallel computation of layers) but without skip connections as our simplest baseline. In Table [ we also present a baseline for an agent that computes layers sequentially (see the leftmost graph in Fig. - To construct this bascline, we needed to estimate how much slower an agent would be without parallel computations. To highlight the potential benefits of pipelining, we assumed an ideal speed-up scenario for parallelization, representing the performance gain at its theoretical limit. Specifically, for
three-layer neural network, we assumed threefold slowdown when abandoning parallelization.
It is important to note that Table 5 reinterprets the information available in Fig. We report the sequential agent for MinAtar in Table 10
Table 5: Mujoco average normilized returns after Imln states of training for the four selected environments. Neural execution time is one.
Halfcheetah-v4
Walker2d-v4
Ant-v4
Hopper-v4
sequential three layers w/ aug three layers aug three layers w/ skip & aug
0.246
0.651
0.516
0.456
0.574
0.888
0.974
0.998
0.685
0.807
0.828
1.309
C SUPPLEMENTARY ABLATION RESULTS
Varying number of layers. We are interested how the number of layers impacts the performance of the agents. Our hypothesis that performance will be highly sensitive to the number of layers in default MIP, as it directly influences the amount of delay. In contrast, we expect the sensitivity to be lower for MLPs with skip connections. Additionally, we aim investigate whether an architecture with well-tuned number of layers, but without skip connections, could outperform one that includes skip connections.
Figure shows that the optimal number of layers without skip connections for Mujoco environments in the parallel pipeline two. However, this configuration does not outperform the MLP with skip connections. In fact, the results are even stronger: as shown in Figure 3 the three-layer MIP with skip connections consistently outperforms both two- and three-layer ML.Ps without skip connections across nearly all environments and neural exccution times.
Additionally, we varied the number of layers in the augmented agent with skip connections (having neuron execution time of four) across three Mujoco environments, as illustrated in Figure I0 We found that increasing the number of layers from two to three improved performance in all environments, trend also supported by the last two bars in Figure 6 However, beyond three layers, the performance trends diverged and stopped being statistically significant, leading us to adopt three layers the default choice for skip-connected MI.P. Notably, performance does significantly
14

---

Published as a conference paper at ICLR 2025
drop when exceeding three layers, suggesting that the architecture with skip connections adapts the effective number of layers to manage the delay.
MujoCo
MinAtar
Normalzee performacc
0.8
0.5
CNN w/ LSTM & skip & aug
0.6
layera wi skip & aug
3 layers w/ skip & aug
CNN w/ LSTM & aug
CNN w/ skip aug
CNN w/ LSTM & skip
1 layer w/ aug
layers w aug
3 layers wi aug
03-
CNN Wr LSTT
CNN w/ skip
2 layers
3 layers
0.2
layer
0.0
0.0
W/o
skip connections
with LSTM
Figure 6: Ablating number of layers in Mujoco.
Figure 7: Recurrence with skip connections.
Combining recurrent and skip connections. One way make agents more expressive without increasing delay is to add recurrent connections. We investigated whether this result in better perfor- mance. We experimented with combining recurrent (LSTM) and skip connections. However, this combination degraded performance on MinAtar (see Fig. 7 or failed to provide notable improvements on MiniGrid (see Table IT]. We believe that combination of LSTM and skip connection may require additional tuning of hyperparameters.
Action repetition. We included SAC-repeat-2 and SAC-repea--3, which are variants of the vanilla instantaneous SAC algorithm with action repetition, as part of our ablation study (Table 6). In these versions, the same action is repeated in the environment two three times, respectively. This can improve overall performance in MuJoCo environments by simplifying the credit assignment problem. Our findings show that action repetition significantly enhance performance on the Ant and Hopper tasks with two repetitions, and on Hopper with three repetitions. We believe this makes action repetition particularly responsible for the good results in parallel computations setting, when the neural execution time is set to two for Ant and Hopper, and three for Hopper.
Table 6: Average returns after 1mln states of training in the four selected environments for SAC and SAC with sticky actions. The results are averaged across 3 seeds. Mean and standard error are reported.
Halfcheetah-v4
Walker2d-y4
Ant-v4
Hopper-v4
SAC
11739 + 283
4415 + 227
3595 + 1027
2672 + 463
SAC-repeat-2
8626 523
4670 + 221
4102 + 1228
3520 + 140
SAC-repeat-3
8168 + 618
3763 + 582
2625 + 796
3517 + 94
Ablating observation augmentation strategies. Following common practices, we augment obser- vations with four past frames in MiniGrid to account for its original partial observability, two recent actions in Mujoco, and one recent action in MinAtar, based on Proposition 2 and the ablation study results presented here.
We conducted an ablation study to investigate different observation augmentation strategies by varying the number of recent available actions included in state augmentation for Mujoco, ranging number of actions from one to three. Fig. 8 presents the results for two architectures: three-layer MLP and three-layer MLP with skip connections. While there no significant difference in performance for the standard three-layer MLP, the MLP with skip connections shows a slight performance improvement when augmenting the state with the two most recent available actions. Based on these findings, we state augmentation with actions as the default choice for Mujoco environments.
15

---

Published as a conference paper at ICLR 2025
Similarly, Fig. shows the results for MinAtar. We experimented with three augmentation strategies: using the four most recent available states, adding the most recent available action to the hidden representations of the last two fully connected layers, and combination of both. Interestingly, all these strategies resulted approximately the same performance improvement for the CNN with skip connections. Therefore, chose the simpler and theoretically supported approach of augmenting with the last available action as the default strategy in MinAtar environments.
MujoCo
MinAtar
CNNN skip & last action
Normalized performanc
0.5-
Jayers w/ skip & 1 action
3 layers w/ skip & actions
layers skip & 3 actions
CNN w/ skip last action
0.4
3 layers w/ action
3 layers w/ 2 actions
layers w/ Faction
CNN skip 4 obs
0.3
CNN w/ skip
w/o
w/ skip connections
Figure Ablating augmentation choices Mu- joco.
Figure Ablating augmentation choices in Mi- nAtar.
D ADDITIONAL ANALYSIS
Noisy computations. In simulations, while we model parallel neuron computations during both inference and training, the processes were globally synchronized, meaning that all neurons completed and initiated new computations simultaneously. As step towards introducing asynchronous neuron computations, we tested a noisier version of parallel computation by applying dropout in every hidden layer during the training and inference stages in agent with skip connections. In Fig. II one can see that the agent is quite robust to a large amount of dropout, and the performance starts to than 40%. The motivation behind this approach
deteriorate if dropout probability becomes comes from the fact that when each neuron updates asynchronously, we can track the time elapsed since the last update and if this time exceeds predefined threshold, we can zero the activation, mimicking the effect of dropout to some extent.
4000
5000
Return
Return
4000
HalfCheetahvve
2000
Walker2d-v4 Happerv4
3000
3
0.0
0.1
0.2
0.3
0.4
0.5
Number layers
Probablity dropout
Figure 10: Varying number of layers the agent with skip connections having neuron execution time of four. The shaded area indicates the stan- dard emror.
Figure 11: Average return with SE across 3 seeds vs different amounts of dropout for the agent with skip connections in HalfCheetah with neuron exe- cution time of two.
Qualitative analysis. Trajectories rollouts of the CNN agent with skip connections and the CNN agent (without skip connections and with history augmentation) is presented in Fig. 12 for MiniGrid-
16

---

Published as a conference paper at ICLR 2025
DoorKey-5x5-v0. The objective in the game is find the key, toggle the door and reach the destination. The trajectories show the agent with skip connections demonstrates less "roaming around" behaviour compared to the agent without skip connections. inal location while the agent without skip connections is fairly indecisive. Notably, took the agent without skip connections 2x more steps average reach the goal compared to the with skip connections. We present multiple trajectory samples in Appendix
12
Figure 12: Behaviour of agents with (top row) and without (bottom row) skip connections HaniGrid-DDorKeyy
For comparison, we pick sub-trajectory from the full episode for each agent. The arrows in each figure indicate the observations that influence the decision-making process. For the sake of brevity, we have shown only one set of temporal connection in both the cases. However, they exist throughout. The heatmap below each figure denotes the action probabilities, the actions in this case are (in the same sequence in the heatmap): 1:turn left, r turn right, move forward, p:pickup ckup an object, t: toggle an object. The agent with skip connections show less "wandering" behaviour. For instance, the agent with skip connections reaches the door and continues to take random actions while the agent with skip connections toggles the door much earlier. Interestingly, the agent without skip connection is quite confident in its decisions. We hypothesize that because an agent's 's own policy makes the environment appear statttttton,,, high confidence may help it cope with this.
E IMPLEMENTATION DETAILS AND HYPERPARAMETERS USED IN EXPERIMENTS
R2
PC
Plttten
h2
(3 2. lul-, stride=1)
h2
h2
Pletenn
Conn(25, L66, leenell=3, stri stride-2)
Max Ponl
Cuno(16, harnalm) stral=m))
Convt3, 16, h stride=4)
Gane(3s bernel sirde-s)
hg-1
Figure 13: Minigrid agent without skip connec- tions
Figure 14: Minigrid agent with skip connec- tions. Residual connections are emitted from the figure for simplicity.
We use ReLu activation function. Our SAC actor employees three-layer MLP if not stated otherwise with hidden dimensions of 256.
17

---

Published as a conference paper at ICLR 2025
Our PPO actor employs 3-layer Convolutional Neural Network (CNN) followed by two fully connected layer with hidden dimension of 512. All CNN layers have kernel size of 3 and {32, 64, 64) channels, maintaining the same resolution throughout the CNN for MinAtar and using strides (4, 2, } for MiniGrid. The feature volume is then flattened and fed into the fully connected layer for action prediction. For architectures with skip connections, the feature volumes from previous layers are maxpooled, concatenated and then flattened and subsequenty fed to the fully connected layers. Our Q-network shares the same architecture the actor.
Notably, when working with networks incorporating skip connections, we observed performance drop when attempting to combine all convolutional features by flattening and concatenating them into a single feature volume. To address this, we experimented with various methods for feature com- bination and found that max-pooling all features fixed size before flattening and concatenating yielded the best results.
We present the architecture with skip connections used for the MiniGrid experiments in Figure [4] similar architecture was also employed for MinAtar. Specifically, given an input æt time step and a set of hidden activations (h 1i........... if these are convolutional features, they are max-pooled using the formula:
size current surrent-spatial.sizz
last spatial size
Here, current.spatiall.ize refers to the spatial size of the current convolutional feature, and last spatial size refers the spatial size of the final convolutional feature the network (the third convolutional block in Fig. 14). After max-pooling, the features are flattened, concatenated, and passed through linear layers for further processing.
The hyperparameters used in the main experiments on SAC Mujoco and PPO MinAtar/MiniGri can be found in Table 7 For training LSTM in Mujoco, we used learning rate of le-4 instead of default value specified in the table, as we observed slight improvement in performance with this adjustment.
F DEFINING REGRETS
We define regret with respect to cumulative undiscounted rewards:
G (t,x') =
π
iCL)ED((")
where i E D(T') indicates the time steps where the policy n' is used. The expectation is over trajectories generated by following policy æ, rewards are accumulated only at times i D(x').
We denote π as an optimal policy in original MDP and T paarallet as an optimal policy in parallel computation framework. Then delay regret, Atelay (l), and inaction regret, A Aincction (1), are defined as:
decly (t) = T (t, TTaaillee Ginna (t, π. (parnille)
Ainectoon (t) T (1, G
(4)
where 3 is default policy we employ during inaction period.
18

---

Published as a conference paper at ICLR 2025
Table 7: Hyperparameters used in experiments.
Parameter
Value
SAC Mujoco
Discount rate 7
0.99
Policy frequency
2
Target network frequency
Target smoothing coefficient
0.005
Policy learning rate
3e-4
Q-function learning rate
1c-3
Optimizer
Adam
Adam beta
(0.9, 0.999)
Adam epsilon
1e-8
Replay buffer size
1,000,000
Batch size
256
Learning starts
10,000
Entropy regularization
Auto-tuned
Target entropy scale
PPO Min and MiniGrid
Discount rate ?
0.99
Lambda for general advantage estimation
0.95
Entropy coefficient
0.01
Value function coefficient
0.5
Normalize advantages
True
Number of steps unroll policy
32
Number of cavironments
32
Update epochs
4
Learning rate
2.5c-4
Anneal Ir
True
Optimizer
Adam
Adam beta
(0.9, 0.999)
Adam epsilon
le-5
Maximum gradient norm for clipping
0.5
19

---

Published as a conference paper at ICLR 2025
G DISCUSSION OF REALTIME SETTING AND INACTION REGRET
In realtime RL settings agents and environments interact asynchronously at their own pace. As discussed by Travnik al. (2018), this setting is more realistic of real world deployment than the typical sequential interaction paradigm of RL where the agent and environment are both assumed wait for each other in a turn- based manner. Sources of regret as function of time were recently analyzed for this setting by Riemer et al. (2024) where it was concluded that the total accumulated realtime regret realtime can be decomposed into three different sources such that realtime A inaction A delay + Aeem Here Aeam is the typical kind of regret analyzed in RI. (Kearns & Singh 2002 that arises from the need for the algorithm to explore and learn from environment. TTiis kind of regret present even in standard turn-based environments that can pause. However, inaction and dolay are new notions of regret specific to the realtime setting. Ainaction is regret incurred when an agent does not act frequently enough in the environment. Meanwhile, Aedaa is the regret incurred because actions are produced based on delayed observations.
Proposition 3 (Inaction Regref): In parallel computation framework for any N layer neural network constrained such that 8 S inaction regret zero. Otherwise, if 2 1 the regret from inaction is independent of N and bounded by:
A inaction (t) E e - 1))
(5)
This proposition follows from the fact that inference time speedups within the parallel computation framework allow a network length N achieved an times increased action throughput. This result then follows when considering the worst case environment from (Riemer et al. 2024) Theorem I where the default behavior is always sub-optimal when the agent does not act with its own policy. This bound with the parallel computation framework is significantly better for large networks than the bound Aimetioo (t) E (t([N6 - 1) with a single standard sequential layer inference process.
H TRAINING & HARDWARE
We used A A100SXM4 GPU for training all our methods. We use the same GPU for testing our methods well. It took approximately 2 hours per seed to train a MinAtar experiment for 10 million steps, 4 hours per seed MiniGrid experiment, and took 7 hours per seed to train one MuJoCo experiment for 1 million steps.
20

---

Published as a conference paper at ICLR 2025
I DETAILED UNNORMALIZED RESULTS
Detailed unnormalized results for all main and ablation study experiments are provided in Tables 10 and for Mujoco, MinAtar, and MiniGrid, respectively.
For MinAtar, we additionally include results for neural execution time of 0.4 and 2. For MiniGrid, also provide results with neural execution time of 4. The neural execution time of 0.4 for the MinAtar five-layer CNN agent is achieved by treating the computation of 2.5 layers as a new basic block within the parallel computation pipeline.
Table Mujoco average returns after 1min states of training for the four selected environments. The results are averaged across ten seeds for two layers, three layers and w/ projections from observation agents without augmentations, across seeds for RLRD and the rest of results use seeds. Mean and standard error are reported. We take mean action policy during evaluation stage as we notice it may significantly boost the performance of the delayed actor. RLRD (Bouteiller et al. 2021) is baseline that addresses DOMDP rather than parallel computations.
Halfcheetah-v4
Walker2d-v4
Ant-v4
Hopper-v4
SAC
11739 + 283
4415 + 227
3595 + 1027
2672 + 463
neuron exccution time of
RLRD for delay of
3147 1044
3714 547
2924 568
3314 157
one layer
5086 + 662
1209 - 652
1043 = 526
759 65
two layers
6660 + 360
4271 164
1938 = 274
3001 + 293
three layers
7814 130
4459 + 176
2792 - 620
3115 173
LSTM
7096 +
3764 410
2847 587
2462 162
three layers w/ proj-to-action & res
8295 + 522
4567 + 123
4425
3019 # 432
three layers w/ proj-to-an proj-to-action
7690 + 480
4048 681
4599
2871 552
three layers w/ proj-from--t -roj-from-obs
7892 = 379
4497 = 140
3728 4
3187 + 195
three layers w/ all skips
8102 + 285
4934
4170
3381 114
two layers w/ aug
6881 + 467
3516 681
3347 + 532
3454 + 76
three layers w/ aug
6735 + 387
3920 + 33
3502 + 252
2666 + 422
LSTT w/ aug
7980 + 168
3289 + 207
2167 335
2948 + 317
two layers w/ proj-from-obs & aug
7082 + 266
4596 376
2969 48
3389 + 148
three layers w/ proj-from-obs & aug
8037 + 201
3561 682
2976 972
3499 + 30
three layers w/ all skips & aug
8165 + 196
4490 + 174
4729 + 267
3253 + 229
neuron execution time 2
layer two layers
1846 + 440
1535 + 329
1783 + 243
1980 + 439
3173 +: 399
3791 + 368
2061 + 108
3317 31
three layers
3027 + 329
3277 + 180
1780 + 276
2538 + 281
LSTM
2413 363
3084 92
2078
2764 150
three layers w/ proj-to-action & res
3330
4226 258
2049
3531 = 61
three layers w/ proj-to-action
4729
3197 + 396
2685
3597 = 14
three layers w/ rroj-from-obs
492
4137 291
2669 224
3569 =
three layers w/ all skips
8
3068 415
1472
3607 = 17
two layers w/ aug
2142 598
2656 69
2467
3544
three layers w/ aug
1298 1.88666
3499
2231 224
2297 345
LSTM w/ aug
3027 636
277
1459 278
1794 467
two layers w/ proj-from-obs & aug
1156 116
214
2255 145
3628 30
three layers w/ proj-from-ote & aug
4937 + 351
4362 + 194
3180
3665 +9
three layers w/ all skips & aug
3649 + 416
4419
3172 1772+10
3580 + 22
21

---

Published as a conference paper at ICLR 2025
Table 9: Continuation of the Table
Halfcheetah-v4
Walker2d-v4
Ant-v4
Hopper-v4
neuron execution time of
one layer
1293 + 481
2157 189
1597 110
1150 347
two layers
2299 + 350
3037 + 180
1884 + 125
2098 +
three layers
3145 + 158
3054 106
1865 - 79
1171 + 79
LSTM
2816 254
2843 324
9
1144 293
three layers w/ proj-to-action & res
2897 + 679
2711 245
1205
3647
three layers w/ proj-to-action
2886 - 147
2391 1
2160 66
3633 = 33
three layers w/ proj-from rej-frommobss
3224 424
3043 = 182
2097
3505 - 43
three layers w/ all skips
2745 + 733
2583 8
+
3290 + 153
two layers w/ aug three layers w/ aug LSTM w/ aug
2
3074 261
182
3294 + 42
2886 204
2
1856 59
1218 - 80
2212 - 509
2700 299
1
1291 + 35
two layers w/ proj-from-obs aug
1729 616
3006 133
566 699
3603 + 24
three layers w/ proj mooj-from-oe from-obs & aug
3214 + 417
4415 174
2139 1
3504 + 90
three layers w/ all skips & aug
3459 + 440
3237 179
1927 191
3647 =
neuron execution time of
one layer two layers
1284 + 251
1975 + 159
1132 + 561
1372 + 151
1681 + 281
2355 + 291
1427 + 467
1263 + 91
three layers
2421 + 133
2532 109
735 756
1041 21
LSTM
2496 + 303
2353 + 58
617
724 176
three layers w/ proj-to-action & res
1748 + 414
2886 + 186
1909
3472 + 78
three layers w/ proj-to-as proj-to-action
2330 594
3079 319
1642 = 29
3310
three layers w/ proj-from-obs three layers all skips
2674 288
2898 + 134
1959 + 78
2990 = 209
1733 + 408
3031 + 86
1898 159
3016 128
two layers w/ aug three layers w/ aug
2378 177
2762 1 69
1785 = 36
1190 = 136
2813 206
2716 1 139
1512 56
1005 11
LSTM w/ aug
2206 - 168
2792
1702 = 62
892 - 55
two layers w/ proj-from-obs aug three layers w/ proj-from-ole & aug three layers w/ all skips & aug
1738 + 393
2726 147
1862 126
3215 + 26
3375 + 193
3206 + 197
1911 + 66
3369 + 53
2879 + 437
3074 + 134
1906 + 53
2711 + 293
J FULL TRAJECTORY ROLLOUTS FOR MINIGRID
We present a few full trajectories of agents with and without skip-connsttions on Fig.
22

---

Published as a conference paper at ICLR 2025
Table 10: Full results for PPO MinAtar. Average returns after training on 10 million samples on MinAtar games. Results are averaged across three seeds and standard error is reported. Sequential CNN w/ aug represents an agent that computes layers sequentially and we assume the speed of the inference of this five-layers neural network is five times slower.
Breakout-v0
Scaquest-v0
Freeway-v0
Asterix-v0
SpaceInv-v0
PPO
20.81 = 0.15
25.94 + 14.52
64.68 + 0.93
42.01 + 0.52
297.49 + 69.86
neuron execution time of 0.4 25.75
CNN
16.88 = 0.81
3.65
57.28 = 0.51
9.89 = 3.82
78.55 + 2.24
neuron execution time of
Sequential CNN w/ aug
0.79 + 0.06
1.51 + 0.32
28.68 + 0.18
2.24 + 0.20
14.29 + 0.93
CNN
7.92 + 0.84
8.41 0.05
28.865 + 4.31
7.81 + 2.27
35.39+1.55
CNN w/ aug
6.67 + 0.27
7.10 1.75
10.41+0.50
41.95
LSTM
6.49 0.43
5.63 1.01
2.46
34.31
LSTM w/ aug
4.20 = 1.31
4.45
0.77
1.68
37.26
CNN w/ skip
14.46 = 1.81
15.59 4.26
52.11
11.74 1.20
09.80 1.42
CNN w/ skip & aug
16.96 = 0.58
16.01 3.34
50.79 + 2.06
13.24 = 1.09
73.58 = 3.72
CNN w/ skip & aug & Istm
11.69 + 0.56
4.16 + 2.61
16.80 + 0.72
1.23 + 0.54
52.37 + 1.41
neuron execution time of 2
CNN
2.53 + 0.18
3.22 + 1.10
31.29 + 0.30
8.61 + 0.55
29.35 + 2.23
LSTM
2.68 = 0.04
4.14 1.50
29.74 1.22
5 1.96
34.84 2.20
CNN w/ skip
5.41 = 0.27
9.65 + 1.09
40.84 1.70
0.2811001
53.50 = 3.67
Table 11: MiniGrid average returns after training on 10 min states for the two toy environments. The agent receives reward of one only when reaching the target location. The results are averaged across three seeds, mean and standard error are reported. PPO denotes vanilla PPO without inference delay.
Empty-Random--555wW
DoorKey-5x5-v0
PPO
0.963 + 0.0013
0.961 0.0015
neuron execution time of
CNN
0.812 = 0.0137
0.613 + 0.0053
CNN w/ aug
0.894 +: 0.0036
0.859 + 0.006:
LSTM
0.922 + 0.0038
0.904 + 0.0074
LSTM w/ aug
0.855 = 0.0343
0.895 + 0.0134
CNN w/ skip
0.924 = 0.0025
0.930 + 0.0020
CNN w/ skip & aug
0.932 + 0.0031
0.922 + 0.0047
CNN w/ skip Istm
0.926 + 0.0055
0.920 0.0097
CNN w/ skip & aug & 1stm
0.933 + 0.0024
0.932 +: 0.0018
neuron execution time of
CNN
0.810 +:2 0.0043
0.607 + 0.0118
CNN w/ aug
0.894 = 0.0075
0.872 + 0.0019
LSTM
0.919 = 0.0027
0.899 + 0.0085
LSTM w/ aug
0.916 + 0.0059
0.788 0.1088
CNN w/ skip
0.923 + 0.0044
0.933 0.0010
CNN w/ skip & aug
0.921 + 0.0025
0.919
CNN w/ skip 1stm
0.933 1 0.0011
0.567 0.2333
CNN w/ skip & aug & Istm
0.923 + 0.0074
0.927 + 0.0033
23

---

Published as a conference paper at ICLR 2025
(a) Rollout I: Doorkey, Agent without skip connections.
(b) Rollout I: Doorkey, Agent with skip connections.
(c) Rollout 2: Doorkey, Agent without skip connections.
(d) Rollout 2: Doorkey, Agent with skip connections.
Figure 15: Trajectory rollouts (best viewed in zig-zag fashion starting from top left) comparing agents with and without skip connections in the Doorkey environment. Each sequence shows one full trajectory.
24


Reviews:
[{"review_id": "G6XmEw7O0i", "reviewer": "Reviewer_aysZ", "summary": "The paper proposes to use parallel pipeline computation and temporal skip connections to tackle real-time RL. It presents a formalization of the problem and proposed method as well as empirical results on Mujoco, MinAtar, and Minigrid.", "questions": "- proposition 1 includes a relation between environment stochasticity and observational delay, i.e., the bound breaks down for deterministic environments; how do the empirical results relate to this?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- the paper is well structured and clearly written\n- evaluation for both discrete and continuous action spaces\n- error bars for quantitative analyses", "weaknesses": "- the novelty of the proposed method is limited to the (rather narrow) problem setting of real-time RL \n- the empirical evaluation is limited to toy tasks (not really real-time RL problems); the paper would greatly benefit from a real-world application (or any application where the real-time constraint arises naturally)\n- no reproducibility due to missing code\n\nEspecially the combination of the first two points restricts the contribution of this paper as both novelty and significance are severely compromised.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yYJlw2x1FR", "reviewer": "Reviewer_3i3j", "summary": "This paper mentions two main challenges in real-time reinforcement learning (RL): (1) limited action frequency due to neural network inference time, and (2) observational delay where actions are actually based on previous observations due to significant inference latency. While pipelining the policy network provides a framework to partially address these challenges, it is still not enough. The paper proposes two methods to overcome these challenges:\n- Building on policy network pipelining, the paper introduces temporal skip connections which cache hidden representations from previous timestamps to compute the current action within a single layer's execution time. My understanding is that similar to residual connections, this approach can be viewed as an ensemble of multiple actors with different capabilities.\n- The paper leverages history augmentation by incorporating previous actions to better maintain the Markovian property in real-time RL setting.\nTo validate their approach, this paper conducts experiments across various environments. The results demonstrate that when neuron execution time is small, the proposed method successfully accelerates inference without hurting performance. However, with longer neuron execution times or more complex environments, there is still a more noticeable degradation in performance.", "questions": "- It would be helpful to provide clearer motivation and reasoning for how the neural execution times were chosen in different experiments. For example, providing both the environment step time and layer execution time would give readers a better sense of the magnitude of neural execution time in practice.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper did a good job presenting clear explanations of the motivation, research questions and statements, experimental methodology, and results. The writing is well-structured and easy to follow.\n- The discussion of the actor expressive capacity vs observational delay is an interesting and important task in real-time reinforcement learning.\n- The proposed methods could be motivated and connected with some theoretical understanding.", "weaknesses": "- The paper could potentially benefit from establishing a stronger connection between the theoretical analysis/motivation and experimental results. Specifically, including a concrete example that demonstrates how incorporating previous actions helps maintain the Markovian property which is broken before, and enhances training performance. This would strengthen the paper's arguments and make them more compelling.\n\n- The proposed methods have suboptimal performance in certain environments, such as in MinAtar games. In my opinion, the paper could benefit from a more detailed/thorough discussion on those scenarios and possibly explore potential solutions, such as enhancing the action network's expressiveness to better handle the rapid changing observations (just a thought, not limit to this). These discussions/explorations would provide very valuable insights into current method's limitations and motivate future research.\n\n- (Minor) The experimental evaluation could be further improved by including a broader range of testing environments, such as (could be a subset of) vanilla pixel-based Atari games. My understanding is that these experiments would further demonstrate the proposed methods are general across even more domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PrrroMIPFa", "reviewer": "Reviewer_hZ2b", "summary": "This paper presents a novel approach to addressing delays in real-time RL caused by neural network computation. The authors introduce a skip connection architecture that enables faster observation-to-action response by creating direct pathways in the network. This architecture is theoretically justified, and empirical results show its effectiveness over baseline models across multiple simulated environments. Through extensive ablation studies, the authors demonstrate that the observed performance improvements are due specifically to the skip connections, demonstrating the potential of this approach for real-time RL applications.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors introduce skip connections into neural networks to address inference delay in real-time RL. This approach is novel in the context of RL and represents the first application of skip connections specifically to reduce inference delay.\n2. Extensive experiments validate the effectiveness of the proposed method. Results show that, with skip connections, agents operating under delay significantly close the performance gap compared to agents without delay and substantially outperform agents with delay but without skip connections.\n3. The method is theoretically justified, and the theoretical findings are consistent with the experimental results.\n4. The paper is clearly written and well-structured.", "weaknesses": "1. Experimental repetition is inconsistent across settings (10, 5, or 3 times), which could impact result reliability. Standardization or explanation is needed.\n2. The neural network used is simple, resulting in minimal inference delay. Testing with more sophisticated models, such as computer vision-based architectures for processing observations, would better demonstrate real-world applicability. Additionally, the authors note that skip connections do not fully mitigate the performance drop from delay in complex tasks, raising the question: is this approach only effective for simple tasks where delay is less critical?\n3. Scalability is limited, as larger architectures may encounter GPU memory constraints, reducing the benefits of parallel computation in complex environments.\n4. Although targeting real-world delay issues, the method is only tested in simulations, leaving its real-world effectiveness unvalidated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vKyUJDovQX", "reviewer": "Reviewer_AMk5", "summary": "The authors explain the negative impacts of execution time and observation delay on real-time reinforcement learning performance. They describe a model architecture which includes “temporal skip connections” and “history augmentation”, explaining the theory behind it, and describing its impact on model expressiveness vs minimizing execution delay. Experimentally, they compare this new model architecture to existing architectures in several “game” environments, showing improved performance.", "questions": "I think the conclusion sentence “These architectures demonstrate robust performance across various environments and neuron execution times” undersells the paper results somewhat. Consider adding a slightly stronger follow-up statement “Experimentally, these architectures often perform better than traditional architectures” or similar.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Very interesting paper:\n- Great introduction to the problem space.\n- Great theoretical motivation section.\n- Architecture is well presented & Figure 2 is excellent.", "weaknesses": "Most of the paper speaks to the novel skip-connection architecture with history augmentation. As I understand the paper, some sections measure how existing parallelism approaches improve inference performance. I think these “mundane” sections take focus away from the novel content. Consider moving 1) the abstract sentence “accelerate inference by 6-350% on standard hardware” 2) Figure 1a, and 3) Section 5.4 to the appendix. This will help focus the paper on just the novel content. Apologies if the above is based on a misunderstanding of the paper content.\n \nTable 2 would be easier to absorb if it included % deltas.\n\nErata:\n- Typo: number of layes\n- Inconsistent speech marks: As ”skip connections”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)