Reinforcement Learning via Lazy-Agent for Environments with Random Delays

Authors: Jongsoo Lee, Jangwon Kim, Jiseok Jeong, Soohee Han
Primary area: reinforcement learning
Keywords: Delayed system, Markov decision process, reinforcement learning

Abstract:
Real-world reinforcement learning applications are often hampered by delayed feedback from environments, which violates the fundamental assumption of the Markovian property and introduces significant challenges. While numerous methods have been proposed for handling environments with constant delays, those with random delays remain largely unexplored owing to their inherent complexity and variability. In this study, we explored environments with random delays and proposed a novel strategy to transform them into their equivalent constant-delay counterparts by introducing a simple agent called the *lazy-agent*. This approach naturally overcomes the challenges posed by the variability of random delays, enabling the application of state-of-the-art methods, originally designed for handling constant delays, to random-delay environments without any modification. Empirical results demonstrate that our lazy-agent significantly outperformed other baseline algorithms in terms of asymptotic performance and sample efficiency in random-delay environments.

Decision: reject
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729354300329

thread_type:
official_review

<thread_object>
creator: reviewer_mdvd

<content>
summary:
The paper introduces Lazy-Agent for handling reinforcement learning in environments with random delays.
Traditionally, state-of-the-art RL techniques assume constant delays, simplifying the learning process but failing to reflect real-world scenarios' complexity.
A Lazy-Agent operates under the assumption that all feedback, such as states or rewards, is delayed by the maximum possible time, even though the actual delay is random.
This method allows the transformation of random-delay environments into constant-delay ones, making it possible to apply conventional RL techniques designed for constant delays.
Rather than reacting immediately, the Lazy-Agent waits for the maximum delay before making decisions.
This simplifies the management of random and unpredictable delays by treating them as constant delays, thereby facilitating the use of existing reinforcement learning methods for delayed feedback.
The paper extends the belief projection-based Q-learning (BPQL) framework, introducing Lazy-BPQL.

strengths:
- The authors demonstrate that using Lazy-Agent can convert random-delay Markov decision processes (RDMDPs) into constant-delay MDPs (CDMDPs). This transformation allows established constant-delay solutions to be applied to random-delay environments.
- The proposed method is simple and easy to implement.

weaknesses:
- For certain tasks (such as Ant, Hopper, Humanoid, and Pendulum), the performance gains from using Lazy-BPQL are minimal.
The reported improvements in performance, when considering standard deviation, are negligible.
- There is no significant advantage of Lazy Agents over traditional methods in all benchmarks, which raises concerns about the broader applicability of the Lazy Agent model.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730345112235

thread_type:
official_review

<thread_object>
creator: reviewer_pxdk

<content>
summary:
This paper proposes a way to transform random delay environments into constant delay environments, by imposing the specified maximum delay onto all observed states, regardless of their actual delay. Experiments show that agents trained under this transformation achieve similar results to agents trained in constant delay environments.

strengths:
1. Nice visual representations e.g. Figure 1 help with understanding
2. The paper proposes a straightforward way to apply an algorithm for constant-delay environments to environments with random delays
3. Convincing number of environments and comparison algorithms used in experiments

weaknesses:
1. Still requires a maximum number of delay time steps to be specified, which may be unrealistic in some environments (e.g. if observations must be sent over a high-latency network, if the observations require lengthy processing). This would admittedly be a challenging setting to address.
2. Only addresses the issue of large augmented state dimensions if the specified max delay is small.
3. I found the explanation of BPQL a bit confusing, as I don’t see how it avoids state space explosion if you are still training a policy on the augmented state.

questions:
1. Unsure why you need to wait for $t>2_{o_{max}}$ in line 19 of Algorithm 1, instead of just $t>o_{max+2}$, if the purpose is just to wait until you have the next state/augmented state?
2. In section 3.2 it is stated that the state dimension could become infinite in infinite-horizon MDPs, wouldn’t this only occur if $O_{max}$ was infinite? 
3. Could you provide more details on how BPQL mitigates the state space explosion problem, particularly given that the policy is still trained on the augmented state?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717653298

thread_type:
official_review

<thread_object>
creator: reviewer_amsf

<content>
summary:
This paper presents a derivation and a small-scale empirical analysis that shows random-delay MDPs can be transformed to nearly equivalent constant-delay MDPs by employing a lazy-agent that assumes that all states are (constantly) delayed by the maximum delayed times.

strengths:
Soundness
======
 The approach is sound, and, albeit a small increment over existing approaches, well demonstrated empirically. The paper lacks an in-depth discussion of the benefits and limitations of the approach. 
	
Significance & Related work
=========
There is no related work section in the paper and thus the significance of the work is not clarified.  

Experimentation
=========
The experimental analysis shows the equivalence of RDMDPs to CDMDPs in BPQL in the MuJoCo environments. 


Presentation
=========
The paper is well written.

weaknesses:
Soundness
======
The paper lacks an in-depth discussion of the benefits and limitations of the approach. 
	
Significance & Related work
=========
There is no related work section in the paper and thus the significance of the work is not clarified.  

Experimentation
=========

There is a need, however, for more in-depth ablation experiments that evaluate the impact of assuming states are processed in order, and of highly varied random delays.

questions:
* What is the performance of the approach in the other MuJoCo environments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733146803941

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to express our sincere gratitude to all the reviewers for taking the time to review our manuscript, and providing valuable feedback. The reviewers' comments have greatly contributed to enhancing the overall quality and clarity of our study.

We would like to : 1) summarize and highlight the key contributions of our study, and 2) outline the major modifications made to the revised manuscript in response to the reviewers' comments. 

---

## **Contributions**

This study makes a primary contribution by introducing the *lazy-agent* that straightforwardly transforms random-delay environments into their equivalent constant-delay environments. This transformation enables the application of state-of-the-art constant-delay approaches to random-delay environments without any modifications. 

Empirically, we demonstrate that the BPQL framework [1], originally designed to manage constant delays, achieves exceptional performance in random-delay environments when integrated with our lazy-agents, termed *lazy-BPQL*. This lazy-agent-based approach outperforms other baseline algorithms that directly operate in random-delay environments. 

These results offer valuable insights that there is no need to devise new methods for handling random delays, as the lazy-agent naturally facilitates the application of conventional constant-delay approaches to random-delay environments.

---

## **Major Updates in the Revised Manuscript**

The followings contents have been updated or newly included in the revised manuscript:

- Experiments investigating 1) the impact of assuming ordered states and 2) the performance of lazy-BPQL in random-delay environments with higher randomness $o_{\text{max}} \in$ {25, 30} are included.
- A section for related works is included to provide a comprehensive understanding of conventional approaches and to highlight our contributions.
- A section for limitations is included to clarify the strengths and weaknesses of our study.
- To clarify the performance achieved by lazy-BPQL compared to other baseline algorithms, we include a delay-free baseline, *delay-free SAC*, and report delay-free normalized scores [2].
- Figure 3 from the original manuscript has been moved to Appendix B.2 in order to avoid any points of confusion and improve readability.

The modified parts are highlighted in red in the revised version of our manuscript.

---

Lastly, we would like to thank the reviewers once again for their efforts and thoughtful comments.

Best regards,

Authors

---

**Reference**

[1] Kim, Jangwon, et al. "Belief projection-based reinforcement learning for environments with delayed feedback." Advances in Neural Information Processing Systems 36 (2023): 678-696.

[2] Wu, Qingyuan, et al. "Boosting reinforcement learning with strongly delayed feedback through auxiliary short delays." Forty-first International Conference on Machine Learning
</content>
</thread_object>

</thread>


Metareview:
metareview:
Reinforcement Learning via Lazy-Agent for Environments with Random Delays 

Summary: The paper addresses the challenge of handling random delays in reinforcement learning (RL), which disrupt the Markovian property and complicate policy optimization. The authors propose a new framework utilizing a "lazy-agent" to transform random-delay environments into their constant-delay equivalents, enabling the application of existing RL algorithms without modifications. The approach uses a belief projection-based Q-learning (BPQL) framework to mitigate state-space explosion issues caused by delay variability. Empirical evaluations on MuJoCo benchmarks demonstrate that the lazy-agent approach outperforms existing methods.

Comments: We received three reviews, with the scores  5, 5, 6, and the average score is 5.33.

Reviewers have given positive comments about some aspects of the paper. In particular, the reviewers appreciated the conceptually simple and clear algorithm to address this challenging problem. The reviewers also mentioned that the paper is well-written. However, the reviewers have raised multiple concerns about the paper which outweigh the positive aspects. Reviewer aMSF has mentioned the lack of a clear literature review that makes it difficult to evaluate the original contributions of this paper. Reviewer PXdK has commented that the paper does not really avoid the state space explosion problem which the authors acknowledge in the rebuttal. Reviewer mDVd mentioned that for certain tasks, the performance gains from using Lazy-BPQL are minimal. Moreover, there is no significant advantage of Lazy Agents over traditional methods in all benchmarks. 

Overall, while the lazy-agent framework is conceptually interesting and the paper is well-presented, it still has significance weaknesses as pointed out above. Addressing these weaknesses and improving the experimental and theoretical rigor would enhance the paper’s contribution.