Title: Belief Projection-Based Reinforcement Learning for Environments with Delayed Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: time-delay system

reinforcement learning
Abstract: We present a novel actor-critic algorithm for an environment with delayed feedback, which addresses the state-space explosion problem of conventional approaches. Conventional approaches use an augmented state constructed from the last observed state and actions executed since visiting the last observed state. Using the augmented state space, the correct Markov decision process for delayed environments can be constructed; however, this causes the state space to explode as the number of delayed timesteps increases, leading to slow convergence. Our proposed algorithm, called Belief-Projection-Based Q-learning (BPQL), addresses the state-space explosion problem by evaluating the values of the critic for which the input state size is equal to the original state-space size rather than that of the augmented one. We compare BPQL to traditional approaches in continuous control tasks and demonstrate that it significantly outperforms other algorithms in terms of asymptotic performance and sample efficiency. We also show that BPQL solves long-delayed environments, which conventional approaches are unable to do.

Belief Projection-Based Reinforcement Learning for Environments with Delayed Feedback

Abstract

We present a novel actor-critic algorithm for an environment with delayed feedback, which addresses the state-space explosion problem of conventional approaches. Conventional approaches use an augmented state constructed from the last observed state and actions executed since visiting the last observed state Using the augmented state space, the correct Markov decision process for delayed environments can be constructed; however, this causes the state space to explode as the number of delayed timesteps increases, leading to slow convergence. Our proposed algorithm, called Belief-Projection-Based Q -learning (BPQL), addresses the state-space explosion problem by evaluating the values of the critic for which the input state size is equal to the original state-space size rather than that of the augmented one. We compare BPQL to traditional approaches in continuous control tasks and demonstrate that it significantly outperforms other algorithms in terms of asymptotic performance and sample efficiency. We also show that BPQL solves long-delayed environments, which conventional approaches are unable to do.

1 Introduction

Deep reinforcement learning (RL) methods have been successfully applied to diverse domains, such as decision-making tasks and robotic control problems [23, 2]. The deep RL framework has shown its potential by mastering the highly complex board game Go and defeating the professional Go player Lee [30]. It has also demonstrated superhuman performance, even in games where a perfect model is not provided, such as Atari video games [24]. Furthermore, modern deep RL algorithms have achieved significant performance improvements in continuous control domains, such as robotic locomotion [28, 29, 14]. Recently, many attempts have been made to apply RL-based control in not only simulations but also the real-world domain [13, 16, 22, 26]. Remarkable examples include applying the RL method to a quadrupedal robot to learn walking gaits and plan the motion of robotic manipulation in the real world [14, 17].

Despite recent progress in RL methods, adapting RL algorithms to the real world remains challenging for many reasons. Delayed feedback from the environment is a particular challenge. For example, latency may occur when the controller attempts to communicate with an agent if the agent is located far from the controller or if a large quantity of data, such as high-resolution images, must be transmitted. Furthermore, hardware issues can cause sensing or actuator delays. This delayed feedback may hinder achieving the control objectives.

Therefore, controlling signal delay is essential for applying RL algorithms to real-world domains. In this study, we propose a novel approach called belief projection-based Q -learning (BPQL) to overcome a constant delayed environment in which feedback is delayed by fixed timesteps. BPQL addresses the state-space explosion problem, which commonly occurs in conventional approaches, and achieves better performance than conventional approaches in terms of asymptotic performance and sample efficiency.

2 Related Work

2.1 Standard Reinforcement Learning

A Markov decision process (MDP) is defined as a 5-tuple ( X , A , R, P, γ ) , where X is the state space, A is the action space, R : X × A ↦→ R is the reward function, P : X × A × X ↦→ [0 , 1] is the transition kernel, and γ ∈ (0 , 1) is a discount factor. The policy π ( ·| s ) maps the state-to-action distribution. In the standard RL framework, the agent chooses an action to maximize the discounted cumulative rewards by interacting with an environment defined as an MDP.

Let the distribution of the initial state be ρ 0 . Then, the expected discounted cumulative reward from policy π is given as:

where initial state s 0 ∼ ρ 0 , a t ∼ π ( ·| s t ) , and s t +1 ∼ P ( ·| s t , a t )

From the standard definitions, the value and Q -value functions are obtained as:

where ∀ k ≥ 0 , a t + k ∼ π ( ·| s t + k ) and s t + k +1 ∼ P ( ·| s t + k , a t + k ) .

Both value functions satisfy the Bellman equation [3]:

In the standard RL framework, we assume that the reward and next state only depend on the immediately previous state and action based on the Markov assumption. However, in an environment where the observation is delayed, the next state does not depend on the immediately previous state but on an older state and action history. This delay property forces the environment to be a partially observable MDP, not an MDP.

In this study, we consider the observation-delayed environment but not the action-delayed environment. However, observation and action delays are intrinsically linked [18, 34], which allows us to control the action-delay problem in the same manner as the observation-delay problem.

2.2 Constant Observation-Delayed Environment

In a delayed feedback environment, the agent receives time-delayed feedback from the environment, which makes it difficult to choose a timely and correct action at each timestep. In conventional control theory, signal delay is often handled by augmenting the state as a combination of the last observed state and history of actions for the delayed timesteps [20, 25].

This augmented state can also be used in RL frameworks. Considering a constant delayed feedback system, an environment with delayed feedback can be formalized by a constant delayed MDP (CDMDP) [34]. The CDMDP is defined as a 6-tuple ( X , A , R, P, γ, d ) , where d is the number of timesteps between the execution of an action and receipt of feedback from that action (i.e., delayed timesteps).

The CDMDP is known to be reducible to an MDP ( S , A , ¯ R , ¯ P,γ ), where S = X × A d is an augmented state space, ¯ R : S × A ↦→ R is the reward mapping, and ¯ P : S × A × S ↦→ [0 , 1] is the transition kernel [18, 34]. This makes it possible to treat the CDMDP as a regular MDP. From this perspective, some model-free approaches, referred to as augmented state approaches , have been proposed [4, 34, 27, 6]. These use the augmented state, which is constructed by concatenating the last observed state and previous actions since visiting the observed state. However, the augmented state space grows exponentially as the number of delayed timesteps d increases. The curse of dimensionality makes the augmented state approach tractable for environments with only a relatively short delay.

Model-based approaches have also been proposed for handling delayed environments [34, 15, 11, 8, 1]. In these approaches, the agent predicts the current state by simulating the environment model and selects an action based on the predicted state. Increasing the accuracy of the transition model is key to this approach. However, in a complex, stochastic environment with a long time delay, the rapid growth of model error increases the difficulty of training the agent.

To the best of our knowledge, the closest work to ours is the study [1] that proposes a model-based algorithm expectation maximization Q -learning (EMQL). This algorithm, however, can only be used in discrete states and action spaces, whereas our proposed algorithm can be used in continuous state-action spaces without constraints. Additionally, although EMQL can suffer from dynamic model errors, our algorithm avoids this because it is a completely model-free approach.

As aforementioned, delayed environments have been mainly handled with two methods: the complete information method (e.g., the augmented state method) and the state estimation methods (e.g., the model-based method). As inherent drawbacks, the first one suffers from the curse of dimensionality and the second requires accurate model-based state estimation. Therefore, the research objective of this work is to resolve these two drawbacks together, i.e., find a model-free approach that is basically free of dimensionality curses. This is why the proposed algorithm was born.

3 Dynamics in Augmented State Space

In this section, we analyze the augmented state approach for the constant delayed environment. Figure 1 illustrates the interaction between an agent and a delayed environment. At each time t , the agent executes action a t and receives the delayed state s t +1 -d and reward r t -d , where d is the number of delayed timesteps. These feedback values are the response from action a t -d . This misalignment of state, action, and reward makes it difficult to find an optimal policy.

3.1 Notation

We have observed that the CDMDP can be treated as a regular MDP by replacing the state space in the CDMDP with the augmented state space. The augmented state is constructed by concatenating the last observed state and previous actions and is denoted by ¯ s .

More formally, in an environment delayed by d timesteps, the augmented state ¯ s t is constructed by concatenating the last observed state and previous d actions:

The policy that receives the augmented state as input is called the augmented state-based policy , denoted as ¯ π ( ·| ¯ s ) . ρ ¯ π (¯ s ) is the agent's steady-augmented state distribution for augmented state-based policy ¯ π . Furthermore, we refer to the value of the augmented state as the augmented state-based value , denoted as ¯ V ¯ π (¯ s t ) . Similarly, the Q -value of the augmented state is called the augmented state-based Q -value , denoted as ¯ Q ¯ π (¯ s t , a t ) .

3.2 Augmented State Approach

Our objective is to train the agent in a constant delayed environment. In the augmented state approach, a new augmented state is constructed by augmenting the last observed state and previous actions to treat the CDMDP as a regular MDP.

We can apply a modified Bellman operator called the delay Bellman operator ¯ T ¯ π to determine the augmented state-based value-function. The operator ¯ T ¯ π is given as:

where d denotes the number of delayed timesteps in the environment. The expected reward is used in the delay Bellman operator because we assume that all feedback is delayed in a constant delayed environment, including the reward, which implies that the exact reward R ( s t , a t ) cannot be determined at time t .

The augmented state-based values can be computed by repeatedly applying the delay Bellman operator. After computing the augmented state-based values, the policy is updated to increase these values for all augmented states in the policy-improvement stage.

Although the augmented state-based values can be computed using the delay Bellman operator, the state-space explosion issue follows; the augmented state space grows exponentially as the number of delayed timesteps d increases because the augmented state space S in the reduced MDP is a Cartesian product of the original state space and d action space (i.e., S = X × A d ). This indicates that the larger the state space, the larger the set of samples required to compute the value of the entire set of states. This makes the augmented state approach impractical for a long-delayed environment.

3.3 Alternative Method to Represent the Augmented State-Based Value

Because calculating the value of the augmented state space by applying the delay Bellman operator suffers from the state-space explosion issue, we propose an alternative method for representing the augmented value.

The distribution of the probability of visiting the true current state s t (which has not yet been observed) depends on the augmented state ¯ s t . If two successive transition probabilities P ( s t | ¯ s 1 t ) and P ( s t | ¯ s 2 t ) are similar, we can assume that the augmented states ¯ s 1 t and ¯ s 2 t have similar representative meanings. Based on this assumption, we define the belief projection of the augmented values as follows.

Definition 3.1. For a policy ¯ π , state space X = { s 1 , s 2 , · · · , s i } and augmented state space S = { ¯ s 1 , ¯ s 2 , · · · , ¯ s j } , let B , ¯ V and D be:

where W = [ ρ ¯ π (¯ s 1 ) , ρ ¯ π (¯ s 2 ) , ..., ρ ¯ π (¯ s j )] is steady-augmented state probability vector when the agent follows the policy ¯ π . Then, the projected values V proj. are defined as:

where the projection operator Π w is B ( B ⊤ DB ) -1 B ⊤ D .

We refer to matrix B as the belief matrix . If the matrix B ⊤ DB is not invertible, then the inverse operator can be replaced with the Moore-Penrose pseudoinverse operator. We refer to the outcome

from the projection operator Π W as the belief projection . The belief projection operator Π W projects a vector in R j onto the column space of the belief matrix B with respect to the weighted Euclidean norm || · || W .

Because the belief projection lies in the column space of the belief matrix, we can use the vector V β to decompose the projected values as follows:

We refer to the elements of V β as the beta values . The vector V β can also be defined as:

From Equation (9), the augmented state-based value can be decomposed into the sum of the expectation of the beta values over the original states and the residual:

where t > d . Similarly, we can extend Equation (9) to the belief projection for the augmented Q -values (projected Q -values); the beta Q -values for a given action a satisfy the following:

3.4 Linear Function Approximation

The augmented state approach is impractical for a long-delayed environment because its state-space size grows exponentially as the number of delayed timesteps increases. In a long-delayed environment, this indicates that directly calculating the augmented state-based Q -values by applying the delay Bellman operator requires a significantly large set of samples.

To avoid this issue, instead of calculating the augmented state-based values ¯ V directly, we estimate the belief projection Π W ¯ V . Let ¯ T ¯ π be a matrix form of the delay Bellman operator ¯ T ¯ π i.e., ¯ T ¯ π ¯ V := ¯ R + γ ¯ P ¯ V , where ¯ R = [ E P ( s t | ¯ s 1 ) ,a ∼ ¯ π ( ·| ¯ s 1 ) [ R ( s, a )] , · · · , E P ( s t | ¯ s j ) ,a ∼ ¯ π ( ·| ¯ s j ) [ R ( s, a )]] ⊤ is a vector consisting of expected rewards and ¯ P is the transition matrix. In this setting, focusing on finding a solution V ∗ β = arg min V ∈ R i || BV -Π W ¯ T ¯ π ( BV ) || W could be achieved very efficiently, especially when |S| ≫ |X| .

Note that the values are approximated as linear combinations of the belief matrix and the beta-values. In other words, the belief projection can be seen as a linear function approximator where the feature vector for an augmented state ¯ s k ∈ S is [ P ( s 1 | ¯ s k ) , P ( s 2 | ¯ s k ) , · · · , P ( s i | ¯ s k )] and the corresponding parameters are the beta values [ V ¯ π β ( s 1 ) , V ¯ π β ( s 2 ) , · · · , V ¯ π β ( s i )] ⊤ . In that sense, we can guarantee the contraction of the combined operator Π W ¯ T ¯ π by using the well-known contraction property of linear function approximation [32, 5].

Proposition 3.2. Let the projection operator onto the column-space of the belief matrix B with respect to the weighted Euclidean norm || · || W be Π W , where W = [ ρ ¯ π (¯ s 1 ) , ρ ¯ π (¯ s 2 ) , ..., ρ ¯ π (¯ s j )] is a steady-augmented state probability vector, and the Markov chain be irreducible i,e, ρ ¯ π (¯ s k ) > 0 for all k ∈ { 1 , 2 , ..., j } . Then the combined operator Π W ¯ T ¯ π is γ -contraction with respect to || · || W .

Proof.

This contraction property of Π W ¯ T ¯ π guarantees to find a fixed unique solution ˆ V ∗ β = [ ˆ V ¯ π β ( s 1 ) , · · · , ˆ V ¯ π β ( s i )] by repeatedly applying the combined operator Π W ¯ T ¯ π . This ˆ V ∗ β can be used as an estimator for the true beta values V β , providing a direction for policy update. For example, a new policy ¯ π new can be obtained by greedily choosing an action, i.e., ¯ π new (¯ s t ) = arg max a t ∈A ( E P ( s t | ¯ s t ) [ R ( s t , a t )] + E ¯ P (¯ s t +1 | ¯ s t ,a t ) [ E P ( s t +1 | ¯ s t +1 ) [ ˆ V ¯ π old β ( s t +1 ) ]]) , where the ¯ π old is the policy before improved.

4 Actor-Critic Algorithm for Constant Delayed Environment

In the previous section, we presented an iterative method that computes the value function in the smaller state space created by belief projection, rather than in the larger augmented state space. However, despite the favorable convergence property of value function approximation through belief projection, explicitly calculating the belief matrix can be challenging, especially in cases where the augmented space is large or continuous. To address this challenge, we introduce a practical sampling-based reinforcement learning algorithm based on the theoretical insights from the previous section. In this algorithm, the agent learns the beta values without the explicit computation of the belief matrix.

First, we define the delay soft Bellman operator ¯ T ¯ π soft , which is the soft Bellman operator [14] for the delayed environment setting:

where α is the temperature parameter. We can compute the augmented state-based soft Q -values by repeatedly applying the delay soft Bellman operator.

In the policy improvement stage, the policy is updated towards the exponential of the evaluated augmented state-based soft Q -function. The improved policy ¯ π new can be obtained as [14]:

where ¯ Z ¯ π old ( s t ) is a normalizing function. This updated policy ¯ π new guarantees ¯ Q ¯ π new (¯ s t , a t ) ≥ ¯ Q ¯ π old (¯ s t , a t ) for all (¯ s t , a t ) ∈ S × A .

The beta Q -values are approximately computed by minimizing the average of weighted squared residual error i.e, δ ¯ π residual in Equation (12) as:

Let the augmented Q -value be a soft Q -value; then, substituting ¯ Q ¯ π ( s t , a t ) into (15) with the target value in (13), Equation (15) can be expanded to the following equation:

Subsequently, we replace the augmented state-based soft Q -value with the expectation of beta Q -values:

by fitting the augmented Q -value to the belief projection. Then, the objective can be rewritten as:

Now, our objective has changed to finding the beta Q -values, where their expectation best represents the augmented state-based Q -values. We can also rewrite Equation (14) using the belief projection

instead of using the augmented Q -values:

where Z ¯ π old ( s t ) is a normalizing function for the distribution Q ¯ π old ( s t , · ) . Equation (21) holds because log ¯ Z ¯ π old (¯ s t ) and E P ( s t | ¯ s t ) [log ¯ Z ¯ π old ( s t )] are independent of ¯ π ′ .

Using Equation (23), we can update the policy by minimizing the expectation of the Kullback Leibler-divergence of the policy and exponential of the beta Q -value. Notably, we need not evaluate the augmented state-based soft Q -value, which causes the state-space explosion problem when the delayed timestep is large. By contrast, we evaluate the beta Q -values and improve the policy using these values.

The beta Q -value is expected to converge more stably and quicker than the augmented state-based Q -value because its input state space is considerably smaller than the augmented space, which also helps to obtain an improved policy with higher quality.

In the continuous state setting, the parameterized beta Q -function Q ¯ π θ,β ( s t , a t ) can be approximately computed by minimizing the squared-residual in Equation (18) denoted by J Q β ( θ ) with the aid of replay memory D [24]:

where r t = R ( s t , a t ) , Q tar θ,β is a target network for Q ¯ π θ,β [24], and ¯ π ϕ is a parameterized policy.

In the policy improvement stage, the policy can be trained by minimizing the objective J ¯ π ( ϕ ) , which can be formalized from Equation (23) as:

By iteratively minimizing J Q β ( θ ) and J ¯ π ( ϕ ) , the actor and critic networks can be trained for a delayed environment. This is the BPQL algorithm. Note that the first and second subscripts of Q θ,β refer to the neural network weights and the beta Q -values, respectively. Actually, the beta Q -values depend on the neural network weights. Nevertheless, θ and β are written together for clearer notation. Only θ is the variable set to be optimally taken through learning.

In the BPQL algorithm, expanded transition tuples that include augmented states are stored in the replay memory D . To construct the augmented state, we use a temporary buffer B in which the original states and action history are stored. The details of BPQL are summarized in Appendix A.

5 Experiments

We compared the performance of the BPQL algorithm with the following three baselines 4 :

Delayed Timesteps=3

Delayed Timesteps=6

Delayed Timesteps=9

5000

Normal SAC

4000

5000

4000

4000

BPQL (ours)

Delayed-SAC

Augmented SAC

3000

3000

3000

2000

2000

2000

1000

1000

1000

0-

0.0

2'0

0.4

step (1e6)

0.6

0.8

1.0

0.0

0.2

0.4

step (1e6)

0.6

0.8

1.0

0.0

0.2

0.4

step (1e6)

0.6

0.8

1.0

Augmented approach: Augmented SAC

Model-based approach: Delayed-SAC

Normal SAC

In the augmented approach, we solve the control problem by adopting an augmented state space, which facilitates treating the CDMDP as a regular MDP. This is a popular and widely used method for an agent learning in a delayed environment [34, 27]. The policy and critic in the augmented state approach are trained by minimizing the objectives ¯ J ¯ Q ( θ ) and ¯ J ¯ π ( ϕ ) :

We named this augmented-based approach Augmented SAC .

In the model-based approach, we use the delayed-Q algorithm [10], which uses a forward model to estimate the timely proper state. In this approach, the agent takes an action based on the predicted current state from the dynamic model using the last observed state and action history, i.e., ( s t -d , a t -d , a t -d +1 , · · · , a t -1 ) . The model repeats the one-step transition prediction d times to

(a)NoisyInvertedPendulum-v2

(b)Augmented SAC with different Q-Network Size

Delayed Timesteps=6,std=0.1

Delayed Timesteps=6,std=0.15

HalfCheetah-v3 (d=9)

1000

500

Normal SAC

Delayed-SAC

6000

Augmented SAC (256,256,256)

Augmented SAC (256,256)

Augmented SAC

4000

Augmented SAC (512,512)

008

400

BPQL

5000

BPQL

Augmented SAC (1024,1024)

4000

3000

wnay 6

600

nay 6

300

wnay 56

3000

400

200

2000

2000

200

100

1000

1000

0

0

50000

100000

150000

step

200000

250000

50000

100000

150000

step

200000

250000

0.0

0.2

0.4

step (1e6)

0.6

0.8

1.0

0.0

0.2

0.4

step (1e6)

0.6

0.8

1.0

predict the current state. In the learning stage, we train the critic and policy using true transition tuples, i.e., ( s t , a t , r t , s t +1 ) . The original study used double deep Q -networks (DDQN) [33] as the base learning algorithm. However, because DDQN is only applicable to discrete action spaces, we have used SAC for our base learning algorithm, so named this model-based approach Delayed-SAC .

Lastly, in the Normal SAC approach, the agent naively uses delayed feedback for training without any modification.

5.1 Performance Comparison

We tested the algorithms on several tasks using the MuJoCo benchmark [31] and evaluated their performances in environments with different numbers of delayed timesteps. 5 Figure 2 shows that the augmented and model-based approaches are inappropriate for environments in which the delayed timestep is large, whereas the proposed BPQL algorithm exhibits significantly better performance in a long-delayed environment. Table 1 lists the overall experimental results, confirming that BPQL outperforms the conventional approaches by a wide margin in environments ranging from a short ( d =3) to a long delay ( d =9).

5.2 Stochastic Environment

Weevaluated BPQL and other baselines on the noisy version of the InvertedPendulum-v2 environment, where Gaussian noises were added to the actuator such that the acting becomes stochastic for the same input for the actuator. The added noises to the actuator are sampled from normal distribution with zero mean and standard deviations of 0.1 and 0.15. In this stochastic environment, BQPL has also shown better performance than conventional algorithms, and the difference gap in performance increased as the level of stochasticity grew.

5.3 Performance comparison of augmented approach with various capacity of Q -network

We conducted additional ablation study to investigate whether the size of Q -network was too small to learn and extract important features from the augmented state. We tested the performance of Augmented SAC with different hidden size of the Q -network, including (256,256), (512,512), (1024,1024), and (256,256,256). The results of this additional experiment show that the size of the Q -network does not significantly affect the performance of the augmented SAC as shown in Figure 3-(b). This confirms that addressing the state space explosion problem is a crucial factor in training the agent in a delayed environment.

6 Conclusion

In this study, we proposed a novel model-free algorithm BPQL for a constant delayed environment. BPQL evaluates the beta Q -function based on the original state-space size rather than evaluating the

augmented Q -function, which helps the parameterized Q -function learn more stably and converge faster. In regards to the MuJoCo benchmark for continuous control tasks, BPQL achieves significantly better performance than the augmented state approach, which is a popular and widely used algorithm for delayed environments. Our results show that BPQL provides a promising avenue for handling delayed environments, such as real-world robot control environments where sensing and actuator delay exist, or a communication system with a narrow bandwidth.

As BPQL cannot be applied to a random delayed environment, it would be meaningful to extend our work to an environment with randomly delayed feedback. Ensemble learning of the beta Q -functions and augmented state-based policies that cover the entire delay range could be one possible approach. Furthermore, real-world applications using the proposed algorithm is an exciting direction for future work.

References

Mridul Agarwal and Vaneet Aggarwal. Blind decision making: Reinforcement learning with delayed observations. Pattern Recognition Letters , 150:176-182, 2021. doi: 10.1016/j.patrec. 2021.06.022.

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine , 34(6):26-38, 2017. doi: 10.1109/MSP.2017.2743240.

Richard Bellman. A markovian decision process. Journal of mathematics and mechanics , pages 679-684, 1957.

Dimitri P Bertsekas. Dynamic Programming and Optimal Control, 2nd Edn, Vols. 1 and 2 . Athena Scientific, Belmont, MA, 2001.

Dimitri P Bertsekas. Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications , 9(3):310-335, 2011.

Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Binas. Reinforcement learning with random delays. In Proceedings of the Ninth International Conference on Learning Representations , Online, 2021.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Baiming Chen, Mengdi Xu, Liang Li, and Ding Zhao. Delay-aware model-based reinforcement learning for continuous control. Neurocomputing , 450:119-128, 2021. doi: 10.1016/j.neucom. 2021.04.015.

Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07207 , 2019.

Esther Derman, Gal Dalal, and Shie Mannor. Acting in delayed environments with nonstationary Markov policies. arXiv preprint arXiv:2101.11992 , 2021.

Vlad Firoiu, Tina Ju, and Josh Tenenbaum. At human speed: Deep reinforcement learning with action delay. arXiv preprint arXiv:1810.07286 , 2018.

Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning , pages 1587-1596, Stockholm, Sweden, 2018. PMLR.

Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 3389-3396, Singapore, Singapore, 2017. IEEE. doi: 10.1109/ICRA.2017.7989385.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Todd Hester and Peter Stone. TEXPLORE: real-time sample-efficient reinforcement learning for robots. Machine learning , 90(3):385-429, 2013. doi: 10.1007/s10994-012-5322-7.

Jemin Hwangbo, Inkyu Sa, Roland Siegwart, and Marco Hutter. Control of a quadrotor with reinforcement learning. IEEE Robotics and Automation Letters , 2(4):2096-2103, 2017. doi: 10.1109/LRA.2017.2720851.

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In Proceedings of the 2nd Conference on Robot Learning , pages 651-673. PMLR, 2018.

Konstantinos V Katsikopoulos and Sascha E Engelbrecht. Markov decision processes with delays and asynchronous cost collection. IEEE Transactions on Automatic Control , 48(4): 568-574, 2003. doi: 10.1109/TAC.2003.809799.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Woosuk Kwon and A Pearson. Feedback stabilization of linear systems with delayed control. IEEE Transactions on Automatic Control , 25(2):266-269, 1980. doi: 10.1109/TAC.1980. 1102288.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Jan Matas, Stephen James, and Andrew J Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Proceedings of the 2nd Conference on Robot Learning , pages 734-743, Z ürich, Switzerland, 2018. PMLR.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Peterson, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015. doi: 10.1038/nature14236.

Jung Hun Park, Han Woong Yoo, Soohee Han, and Wook Hyun Kwon. Receding horizon controls for input-delayed systems. IEEE Transactions on Automatic Control , 53(7):1746-1752, 2008. doi: 10.1109/TAC.2008.928320.

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. DeepMimic: Exampleguided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics , 37(4):1-14, 2018. doi: 10.1145/3197517.3201311.

Simon Ramstedt and Christopher Pal. Real-time reinforcement learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , pages 3073-3082, Vancouver, Canada, 2019.

John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In Proceedings of the 32nd Iternational Conference on Machine Learning , pages 1889-1897, Lille, France, 2015. JMLR.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature , 550(7676):354-359, 2017. doi: 10.1038/nature24270.

Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, Vilamoura-Algarve, Portugal, 2012. IEEE. doi: 10.1109/IROS.2012.6386109.

John N Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffference learning with function approximation. Advances in Neural Information Processing Systems , 9:1075-1081, 1996.

Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI conference on Artificial intelligence , volume 30(1), pages 2094-2100, Phoenix, AZ, 2016.

Thomas J Walsh, Ali Nouri, Lihong Li, and Michael L Littman. Learning and planning in environments with delayed feedback. Autonomous Agents and Multi-Agent Systems , 18(1): 83-105, 2009. doi: 10.1007/s10458-008-9056-7.

A Architecture and Pseudo Code of BPQL

St-dSt-dat-drt-dSt+1-dSt+1-d

④st+1

TemporaryBuffer

ReplayMemory

Delayed Env

MinimizeJQand Jπ

Agent/Policy

s①

②at

Algorithm 1 Belief-Projection-Based Q -learning (BPQL)

1: initialize a policy ¯ π ϕ ( a | ¯ s ) , beta critics Q θ,β ( s, a ) , target beta critics Q tar θ,β ( s, a ) , an empty replay memory D , a temporary buffer B , delayed timesteps d , learning rate for the beta critic λ Q β , learning rate for the policy λ ¯ π , and soft update ratio τ

B Proof of Proposition 3.2

Proposition 3.2 Let the projection operator on the column-space of the belief matrix B with respect to the weighted Euclidean norm || · || W be Π W , where W = [ ρ ¯ π (¯ s 1 ) , ρ ¯ π (¯ s 2 ) , ..., ρ ¯ π (¯ s j )] is a steadyaugmented state probability vector, and the Markov chain be irreducible i,e, ρ ¯ π (¯ s k ) > 0 for all k ∈ { 1 , 2 , ..., j } . Then the combined operator Π W ¯ T ¯ π is γ -contraction with respect to || · || W .

Proof. Let V 1 proj. and V 2 proj. be an arbitrary vector in R j , ¯ P ¯ π (¯ s ′ | ¯ s ) be a transition probability of ¯ s → ¯ s ′ when the agent follow the policy ¯ π , and V 1 , proj. k , V 2 , proj. k be the k-th element of the V 1 proj. and V 2 proj. respectively.

∵

Σ

j

k

=1

ρ

(¯

s

k

) ¯

P

(¯

s

l

|

¯

s

k

) =

ρ

(¯

s

l

)

by the definition of

ρ

where Equality (31) holds by the Pythagorean Theorem, and Inequality (34) follows from the Jensen's inequality.

¯

π

¯

π

¯

π

¯

π

C Environment Details

(a)

(b)

(c)

(d)

(e)

(f)

D Implementation Details

In this section, we provide the implementation details of the algorithms used in this study. All methods presented in the experiment used an SAC as their base learning algorithm with the following characteristics:

Stochastic Gaussian policy approaches.

Automated entropy adjustment [14].

Clipped-double Q -learning, which was introduced in the TD3 algorithm to prevent overestimating the Q -value [12].

Soft target update, which changes the target values slowly and improves the learning stability[21].

Adam optimizer, a variant of the stochastic gradient descent method [19].

The details of the hyperparameters are presented in Table 3.

E Plots of Performance Comparison

In this section, we present several plots of the performance curves of BPQL and other baselines in the environments with delayed timesteps of 3, 6 and 9.

Delayed timesteps=3

HalfCheetah-v3

5000

8000

3500

4000

3000

6000

2500

piemag

000E

2000

4000

N2000-

1500

2000

1000

1000

500

0.0

0.2

0.4

0.6

0.8

16

00

0.2

0.4

0.6

8'0

16

0.0

0.2

0.4

90

0.8

1.0

Swimmer-v3

step

InvertedPendulum-v2

step

Reacher-v2

step

1e6

120

1000

100

008

10

puemag 6n

80

puemag 6n

600

puema

Delayed-SAC

Normal SAC

400

BPQL (ours)

Augmented SAC

40

25

200

20

30

0.0

0.2

0.4

0.6

0.8

16

50000

100000

150000

200000

250000

50000

100000

150000

200000

250000

step

step

step

Delayed timesteps=6

HalfCheetah-v3

6000

5000

0OSE

5000

4000

3000

4000

puemay t

3000

2000

2500

3000

N2000-

1500

2000

1000

1000

1000

500

0

0.0

0.2

Swimmer-v3

0.4

step

90

0.8

1.6

0.0

0.2

InvertedPendulum-v2

0.4

step

0.6

8'0

1.6

00

0.2

Reacher-v2

0.4

step

90

0.8

1e6

1.0

120

1000

100

10

puemag 6ng

80

600

800

puemag

15

-25十

Delayed-SAC

Augmented SAC

Normal SAC

400

BPQL (ours)

20

200

35

40

0.0

0.2

0.4

9'0

0.8

1.46

50000

100000

150000

200000

250000

50000

100000

150000

200000

250000

step

step

step

Delayed timesteps=9

HalfCheetah-v3

6000

3500

4000

5000

3000

4000

3000

2500

2000

Avg

2000

1500

2000

1000

1000

1000

500

0.0

0.2

Swimmer-v3

0.4

step

0.6

0.8

16

0.0

0.2

InvertedPendulum-v2

0.4

step

0.6

0.8

1.6

0.0

0.2

Reacher-v2

0.4

step

0.6

0.8

1e6

1.0

120

1000

100

800

10

pemat

80

600

pemag

20

15

Normal SAC

Delayed-SAC

60

40

400

25

30

Augmented SAC

BPQL (ours)

20

200

35

40 +

0.0

0.2

0.4

0.6

8'0

1.0

50000

100000

150000

200000

250000

50000

100000

150000

200000

250000

F Additional Experiments

F.1 Extremely Long Delayed Environment

We conducted additional experiments to determine how well BPQL could solve the control problem even in very long delayed environments. Figure 7 shows that in BPQL, the policy is improved through interaction with the environment in a very long delayed environment, but the conventional methods find it difficult improve their policy.

HalfCheetah-v3(d=20)

InvertedPendulum-v2(d=25)

6000

Normal SAC

Delayed-SAC

5000

1000

BPQL

AugmentedSAC

4000

800

Return

Return

3000

600

9

2000

Avg

400

1000

200

0

0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

step (1e6)

1e6

step (1e6)

1e6

F.2 Action delay

We also evaluated BPQL and other baselines in action delayed environments. Figure 8 illustrates the interaction between an agent and action delayed environment.

Agent

Environment

①at

2

at-d

at

at-1

at-2

at-d+1

Delayed action

Action queue

F.2.1 Augmented State in an Action Delayed Environment

In an action delayed environment, the augmented state at time t + d is defined as:

where the d is the number of delayed timesteps for an action. The CDMDP for an action delayed environment is defined as a tuple ( X , A , R, P, γ, d a ) , where X is the original state space, A is the action space, R : X × A ↦→ R is the reward function, P : X × A × X ↦→ [0 , 1] is the transition kernel, γ ∈ (0 , 1) is a discount factor, and d a is the number of delayed timesteps for an action.

This CDMDP can be reduced to a MDP ( S , A , ¯ R , ¯ P,γ ), where S = X × A d a is an augmented state space, ¯ R : S × A ↦→ R is the reward mapping, and ¯ P : S × A × S ↦→ [0 , 1] is the transition kernel as in the case of observation delayed environments [18]. BPQL for action delayed environments uses the augmented state defined in Equation (38) instead of the one defined in Equation (6).

F.2.2 Results

Delayed timesteps=3

HalfCheetah-v3

InvertedPendulum-v2

0008

5000

3500

1000

4000

3000

0009

008

Return

4000

3000

2500

2000

pmay Sn

600

Augmented SAC

Normal SAC

Delayed-SAC

2000

1500

400

BPQL

1000

1000

500

D

0

0.0

0.2

0.4

step (M)

0.6

0.8

1.0

0.0

0.2

0.4

step (M)

0.6

0.8

1.0

0.0

0.2

0.4

step (M)

0.6

0.8

1.0

50000

100000

150000

step

200000

250000

Delayed timesteps=6

HalfCheetah-v3

InvertedPendulum-v2

6000

4000

3500

1000

5000

3000

Return

4000

3000

2500

pmay 5

008

Normal SAC

3000

6

2000

1500

2000

600

Augmented SAC

BPQL

Delayed-SAC

2000

400

1000

1000

1000

500

200

0

0.0

0.2

0.4

step (M)

0.6

0.8

1.0

0.0

0.2

0.4

step (M)

0.6

0.8

1.0

0.0

0.2

0.4

step (M)

0.6

0.8

1.0

50000

100000

150000

step

200000

250000

Delayed timesteps=9

HalfCheetah-v3

InvertedPendulum-v2

0009

4000

3500

1000

5000

3000

008

4000

3000

2500

wnay 6n

puemay 6n

009

Normal SAC

3000

2000

Delayed-SAC

2000

2000

1500

400

Augmented SAC

BPQL

1000

1000

1000

200

500

D

0.0

0.2

0.4

step (M)

0.6

0.8

1.0

0.0

0.2

0.4

step (M)

0.6

0.8

1.0

0.0

0.2

0.4

step (M)

0.6

0.8

1.0

50000

100000

150000

step

200000

250000

F.3 Discrete Domain

Weconducted additional experiments on the classical discrete control OpenAI gym [7] tasks: CartPolev1 and LunarLander-v2. In the CartPole-v1 environment, there are two available actions, while in the LunarLander-v2 environment, four actions are available. We compared the performance of BPQL with the Delayed-Q algorithm [10] and Delayed-SAC in various action delay settings. To adapt BPQL and Delayed-SAC to the discrete setting, we implemented a discrete version of SAC [9] for both of them, instead of the original continuous version. We trained the agent for 1 million timesteps in all environments (except for Delayed-SAC and BPQL in the CartPole-v1 task, which was trained for 0.5 million timesteps), and obtained results using 10 random seeds.

The results showed that all algorithms performed similarly in the CartPole-v1 environment, which is considered relatively easier. However, in the LunarLander-v2 environment, as the delay increased, both Delayed-Q and Delayed-SAC showed a notable deterioration in performance. They even failed to control the agent in many episodes under the 10 timesteps delay setting. In contrast, the proposed BPQL algorithm consistently maintained high performance across all tested delay settings (3,5, and 10).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a solution to the delayed feedback problem, in this case a fixed timestep delay. There are approaches to solving this problem in the literature, often associated with cumulative error or un-traceability given longer delays. One common approach — using augmented states by concatenating observed states & actions taken since visiting the delayed state — which grows exponentially the augmented state space as the delay increases. The authors’ algorithm (Belief Projection Based Q-Learning, BPQL) is formally shown to equally represent augmented state-based (Q-)Values. Further, the introduced belief-projection of those values is shown to be decomposable over the original states + residuals, therefore working on the original state-dimensionality rather than the exponential one of augmented state learning. The values are shown to be linearly approximable, an insight, which the authors then use to reformulate the discrete version into a continuous control version of a Belief-based Actor-Critic Algorithm, which outperforms naive/ augmented (-model) approaches of the soft-actor-critic in an delayed version of the MuJoCo benchmark.\n\nSTRENGTHS:\n- The approach is formulated on a quite technical level and appears to be a novel application of belief-based ML to the delayed feedback problem. There is related work in different directions cited for solving delayed environments (augmented approaches, model-based predictions, expectation minimization Q-Learning), although I am not very familiar with the augmented techniques, with are directly compared against here. The reasoning towards the belief based Q-Learning Actor-Critic seems to differentiate enough from Agarwal&Aggarwal2021, which they cite as closest related work.\n- The mathematical reasoning seems reasonable as far as I can follow. The underlying constant delayed MDP and the assumptions are stated clearly. Furthermore, MuJoCo represents an appropriately difficult evaluation domain and the evaluation is sound, although there is no compute resource or training-time mentioned (as far as I can tell).\n\nWEAKNESSES:\n- The paper seems to be missing a discussion on the related concept of belief-state architectures, which may be considered related to the model-based approaches mentioned here but with more relevance to this chosen approach.\n- One point of concern I have with the claimed novelty compared to Agarwal&Aggarwal2021 in the difference of developing a continuous control algorithm, rather than a discrete model. I find it confusing that much of the definition of the BPQL / Actor-Critic model revolves around Q- Values and Q-Learning (the Q-Function, as it is called here) which are an inherently discrete concept The authors mention a generalization to continuous control problems, but the transfer l.228-224 reads more like an afterthought to position the algorithm in a niche without direct comparison benchmarks. If most of the work covers the Q-Learning approach it would be informative to proof performance in this approach compared to e.g. Agarwal&Aggarwal2021 or the DDQN Model-based approach of van Hasselt. et. al. 2016 before evading into the novelty of continuous problems.\n- Furthermore, the paper is lacking a motivation on why belief-based projection is an appropriate technique and was chosen here (aside from not dealing with exponentially growing state-spaces, which is repeated ... often enough). The reader is presented with a very final formulation of the belief matrix and belief projection operator initially, with the following pages re-contextualizing the known MDP / Bellman Equation into this framework. Without a background in control theory or kernel operators (and given the lack of references) I assume this part to be proposed by the authors or established theory, but without any descriptive intuition behind this formalization, much Chapter 3 reads like a proof that would be usually found in the Appendix. For a clearly readable publication I find this submission is written without the reasoning to convey its key-ideas behind the formalities to a broader audience.\n- Given that the evaluation is based on the AC method, I would have preferred to have the AC technicalities in the paper itself rather than the Appendix. Some of the implementation details I found helpful to reason about the evaluation results were found in the Appendix and only the code itself. However, given the technical nature of the paper I would understand why they were cut.\n- \nI feel conflicted about claiming significant performance gains against the mostly self-constructed SAC baselines with ‘only’ 1M training steps on a complex domain like MuJoCo. The evaluation does seem cleanly run and favorable for the BPQL approach (at least in terms of sample efficiency), however, I would argue that only 1M steps is a debatable length for some of these problems to be solvable by the intentionally slow SAC algorithm (regardless of hidden-dim size, not even accounting for delay complexity). Out of the three baselines, the naive SAC is akin to the random baseline and the adjusted model-DDQN baseline was not originally designed for a SAC and is therefore hard to judge in terms of comparability. As mentioned above, it seems as if this work is introducing a Q-Value approach and evaluating on continuous control problems that are a short transfer away, but is not comparing to established discrete solutions to this problem. If better benchmarks for this specific environment niche (continuous control delayed feedback) are not available, it would be nice to see the theoretical maximum reward per domain or other relational metrics that would indicate how the BPQL is performing in general, not just compared to the benchmarks. Apart from the baseline evaluation, the belief-projection application seems like an improvement in terms of computability and therefore, sample efficient training for delayed environments.\n- While the BPQL approach may be employed to any fixed step delay environment, with the initial setting of ‘uncertain latency’ that is motivated in the introduction, the authors are correct in listing this fixed-step assumption as a notable limitation. Especially in the continuous domain case (which is the differing focus of this work), having one (small) continuous, fixed delay signal is quite the niche. Nevertheless, given the defined setting the proposed algorithm shows promise in it’s intended purpose.\n\nQUESTIONS:\nAside from concerns mentioned above, the following questions arise: \n- Why is the belief-projection chosen here and how would you motivate this decision?\n- Why was the SAC algorithm chosen as the baseline, how was it parameterized and how would common baselines like A2C or PPO compare in this setting?\n- What is the runtime overhead of computing the belief-projection compared to the (technically simpler) augmented state techniques?\n- Would it be possible to show e.g. memory-complexity of the baselines and of BPQL as a metric in the specific case of MuJoCo domains here? L.225ff mentions that the algorithm also stores augmented states (original states and action history) in an extra temporary buffer, could you please contextualize this to augmented state learning? (As mentioned above, this would also be information to include in the compute-resource / runtime statement.)\n\nLIMITATIONS:\nAssumptions and limitations are stated in the context of the formalization. No potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a projection approach for more compactly representing the value function of a delayed system in reinforcement learning. A complete algorithm based on SAC with neural network approximations is presented and benchmarked on four Mujoco environments where it outperforms competing approaches.\n\nSTRENGTHS:\n- The paper addresses an important problem when applying RL on real-world systems: latency. This is a common issue in robotics in particular. As such, a good solution would be of considerable value to the robot learning community.\n\n- The paper is well-written and mostly easy to read\n\n- The approach demonstrates impressive improvements on delayed systems in Mujoco benchmarks\n\nWEAKNESSES:\n- The experiments are only on four rather simple Mujoco control benchmarks. It would have been nice to have something more realistic, but I think the results were clear enough to establish the promise of the approach.\n\n- A few important details could be more clear (see questions below)\n\nSide note: Code is included but not publicly released. As the RL community has struggled with reproducibility due to small discrepancies in implementations, this would have been nice.\n\nMinor:\n- Not sure what you mean by \"agnostic random delayed environment\" in the conclusions? If you just mean randomly delayed environments, that sentence can be simplified a lot.\n\n------------ \nAfter rebuttal:\n\nThe authors addressed all of my concerns and therefore I raise my score to a 7. This paper could be very useful to better solve delayed control problems, which I can attest to being common in robotics. That said I agree with Reviewer 3qfv that the method sections were needlessly difficult to read. However, the authors promised to add some intuitions earlier in the final manuscript, which I think will be good enough. I thank the authors for their work and encourage them to open source their code so there is less room for misunderstandings by future work on this important problem.\n\nQUESTIONS:\n1) The text is a bit vague on what added assumptions / complexity there is of the beta q-function representation. Initially it seemed like the state transitions/beta matrix would have to be estimated separately in (10) where a linear projection was used.  Later a \"practical approximation\" is introduced where it seems the beta Q-function can be updated iteratively via sampling as normal in Q-learning? Does this preserve optimality or is it an approximation (further approximation on top of regular NN SAC). If not optimal, can you clarify what added assumptions there actually are?\n\n2) Why is it performing better even on the noisy environment without delay? This is not obvious from the claimed contributions.\n\nLIMITATIONS:\nTo be clarified, the method seems very general.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper aims to tackle the delayed feedback RL problem. In such a problem setting, usually there is some fixed/constant delay in the environment, so that the observed rewards are delayed by $d$ timesteps. Traditionally, augmented state spaces have been used to solve such an issue, however, for large delays this leads to an exponentially large dimension for learning the value function. Thus, the proposed approach (BPQL) leverages the deconstruction of the augmented value function via the belief matrix into a lower dimensional beta q value function, and shows that this approximates the augmented q function (derived from the soft bellman update). This outperforms baselines on a variety of control tasks.

STRENGTHS:
- I think there is good technical novelty and soundness in this paper
- The problem space is very interesting and relevant to real world applications
- BPQL drastically outperforms basleines 
- This approach is general (as it applies to continuous settings)

WEAKNESSES:
I think it would be interesting to explore this idea on different types of environment delay (for example delays coming from different sources like actuators, sensors, etc. It would also be good to see more analysis and ablations on different levels of delays (value of $d$). 

Finally, it would be great if this apporach applied to more realistic robotics settings (even simulated versions).

QUESTIONS:
See weaknesses section

LIMITATIONS:
This has been sufficiently addressed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of delayed feedback in reinforcement learning,\nwhere the observations and rewards received by the agent are those generated by\nthe environment multiple (`d`) time steps ago, most commonly caused by latency\nin hardware. This problem is typically solved by augmenting the policy and    \nvalue functions with the `d` previous actions, recovering the Markov property                                                  \nof the problem. This paper argues that this augmented 'state space' is                                                         \nprohibitively large and proposes Belief-Projection-Based Q-learning (BPQL) as a                                                \nfix to the issue.                                                                                                              \n                                                                                                                               \nBPQL modifies SAC such that the value function (critic) that is being learned\nonly takes the observation (and not the `d` actions) as input (though the                                                      \npolicy (actor) still depends on the augmented space). The result is an       \napproximation of the original problem, where the critic learns the expected\nvalue of the policy based on just the last observation, as opposed to the    \nobservation and last `d` actions.                                                                                              \n                                                                                                                               \nThis approach is motivated by a substantial theoretical section that derives  \nand discusses the (approximated) loss function. The method is compared against                                                 \naugmented SAC, a model-based approach, and regular SAC on MuJoCo for different\ndelays.\n\nSTRENGTHS:\nThis paper proposes a high-performance solution to the complex problem of    \ndelayed feedback. It does so in a fairly simple (in the positive sense) way                                                                                                                                                                                   \nthat should be easily implementable and testable.                                                                              \n                                                                                                                               \nAnother advantage is the theoretical support of what otherwise would be a\nfairly ad-hoc solution. This is particularly useful because, in my opinion, the\nintuition of BPQL is fairly nice: the critic can ignore the additional action\nhistory input because this depends only on the policy it is evaluating: the\naction sequence of the policy is relatively 'stable', or determined, so not\nparticularly variable or something that needs to be given to the critic in    \norder to evaluate the expected return. \n\n======== post rebuttal ==========\n\nThe authors brought up reasonable arguments, context, and experiments to defend their work. I would personally still learn towards rejection due to the fact that, in its current form, I believe that the combination of the presentation, algorithmic contribution, and empirical evaluation are in my opinion relatively weak. However, I do not feel strong enough about this to champion for a rejection when others disagree.\n\nWEAKNESSES:\nDespite these strengths, there are a few concerns that makes me lean towards\nrejection.\n\nWhile the experiments show otherwise, I am slightly surprised that the action\nspace is big enough to cause issues in terms of policy evaluation and would\nneed more data to be truly convinced. Clearly augmented SAC is suffering but,\nunless there is something particular about these problems (of which I have no\ndetailed knowledge), generally the observation/state space is so much larger\nthan the action space that the additional input of `d` actions should not be\nthat significant.\n\nRegarding clarity, the presentation of the paper was in my opinion confusing:\nFirst, the method section contained a substantial amount of information that\nwould fit better in the background (e.g. 3.1, 3.2, and the start of 4). Second,\nthere was a very limited amount of information on the idea behind the paper or\nits overview until you start digging into the details. For example, until the\nlast paragraph before the experiment section I had no idea where this was\ngoing. Additionally, the theoretical section seems to attempt to justify the\napproximations made by BPQL, but that was not clear at all until diving into\nthe section afterwards.\n\nMost importantly, i still struggle to understand the theoretical support. I\nbelieve it eventually it provides a theoretical explanation for BPQL's loss\nfunction, but I only truly understand the loss function from a intuitive\nperspective and had to gloss over section 3 altogether. The main reason I lean\ntowards rejection is because it was not presented in a way that the theoretical\nsection was a contribution to me. This mostly because it is fairly 'easy' to\npropose to 'just drop' some input to reduce the space and have it work well for\nsome problems. So the theory is important here but, in my perspective,\ndifficult to understand.\n\nLastly, some ablations that would be very informative are missing. For example,\nit is clearly important that the policy accepts the augmented space (because\nthat seems to be the key difference between BPQL and 'normal SAC'), but the\nobvious alternative is to do some variant where the policy takes in only the\nlast observation but the critic gets the augmented input. I think more in depth\nexperiments would further motivate the approach. In particular, I would want to\nsee why it is so important that the policy does accept the last `d` actions as\ninput: is this a property of the problems, or is there some theory or intuition\nthat explains this discrepancy?\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-08-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new belief projection-based method for approximating delayed states in reinforcement learning. The method is derived from basic principles with properties of the projection demonstrated. The method is then transformed into a practical, model-free approach, that is extensively evaluated and bench-marked, with promising results.\n\nSTRENGTHS:\n- The paper is easy to follow, the derivations are cleanly done and well explained.\n- The idea is interesting and sound\n- I really liked the logical progress of deriving the method with the assumption of having access to all the probabilities etc. and then step by step turning it into a practical method\n- The experiments show a good number of baselines, and sufficiently many standard benchmarks. The ablation with the delay steps, network capacity, and stochasticity are well done. The number of repetitions is adequate.\n- The method shows clear experimental benefits\n\nWEAKNESSES:\n- I found Sect 2.1 and 2.2 a tiny bit confusing in terms of notation initially. Especially in Sect 2.1 it was initially not clear why X is used and initially it wasn't clear to me that the reward is also delayed (Sect 2.2). Just some minor writing tweaks\n- Sect. 2.2: in conventional control having a system-model-based observer (e.g. Kalman filter) is also quite common - not just the augmented state - similar to the methods described a bit further down\n- In Sect. 1 it almost sounds like the method also considers (or should consider) varying delays for e.g. hardware issues. I'd suggest making more clear from the start that you consider constant delays.\n- I would have appreciated some little outline/forward pointer. On the first read I was a bit puzzled by the apparent assumption e.g. in Eq (7) of having access to the full model while initially a model-free method was promised. Briefly explaining the structure of the paper in the beginning would avoid that confusion.\n- missing limitations (see below)\n\nQUESTIONS:\n- See small writing suggestions above\n- The belief projection method comes a bit out of the blue, can you better motivate why this is a good idea?\n- The setting is connected to POMDPs where there is quite a bit of literature but which is only mentioned in passing. Can you discuss the relation a bit better?\n- Please detail the limitations better, you mention random delayed environments which are an obvious limitation.\n  * Do you think the method would still perform reasonably if there is some small randomness in the delay (even if that's not backed by the theory)?\n  * What are the implications of the linear approximation?\n  * More generally, this seems to be a lossy compression, bringing everything back down to the original state-space size. In which cases will having the full augmented state (and infinite data etc.) still result in better performance?\n  * Can you do something in between, i.e., project to a slightly larger state-space?\n\nLIMITATIONS:\n- societal impact is not really applicable\n- an explicit discussion on limitations is a bit short (see above)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-08-09"}