Dual Critic Reinforcement Learning under Partial Observability
Authors: Jinqiu Li
Enmin Zhao
Tong Wei
Junliang Xing
Shiming Xiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Partial Observability
POMDP
Primary area: reinforcement_learning
TL;DR: This paper introduces DCRL, a framework for efficiency improvement and variance reduction under partial observability.

Abstract: Partial observability in environments poses significant challenges that impede the formation of effective policies in reinforcement learning. Prior research has shown that borrowing the complete state information can enhance sample efficiency. This strategy, however, frequently encounters unstable learning with high variance in practical applications due to the over-reliance on complete information. This paper introduces DCRL, a Dual Critic Reinforcement Learning framework designed to adaptively harness full-state information during training to reduce variance for optimized online performance. In particular, DCRL incorporates two distinct critics: an oracle critic with access to complete state information and a standard critic functioning within the partially observable context. It innovates a synergistic strategy to meld the strengths of the oracle critic for efficiency improvement and the standard critic for variance reduction, featuring a novel mechanism for seamless transition and weighting between them. We theoretically prove that DCRL mitigates the learning variance while maintaining unbiasedness. Extensive experimental analyses across the Box2D and Box3D environments have verified DCRL's superior performance. The source code is available in the supplementary.

Dual Critic Reinforcement Learning under Partial Observability
Jinqiu Li Enmin Zhao 1.2, Tong Wei', Junliang Xing", Shiming Xiang 1.2 'Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences DDepartment of Computer Science and Technology, Tsinghua University (lijinqiu2221, zhacenmin2018] @ia.ac.en, wt22 @ maiiss.iinggua.ee. edu.en smxiang @aaaggg..pir....cn
Abstract
Partial observability in environments poses significant challenges that impede the formation of effective policies in reinforcement learning. Prior research has shown that borrowing the complete state information can enhance sample efficiency. This strategy, however, frequently encounters unstable learning with high variance in practical applications due to the over-reliance on complete information. This paper introduces DCRL, Dual Critic Reinforcement Learning framework designed to adaptively harness full-state information during training to reduce variance for optimized online performance. In particular, DCRL incorporates two distinct critics: an oracle critic with access to complete state information and a standard critic functioning within the partially observable context. It innovates synergistic strategy to meld the strengths of the oracle critic for efficiency improvement and the standard critic for variance reduction, featuring novel mechanism for seamless transition and weighting between them. We theoretically prove that DCRL mitigates the learning variance while maintaining unbiasedness. Extensive experimental analyses across the Box2D and Box3D environments have verified DCRL's superior performance.
1 Introduction
Real-world issues commonly involve a degree partial observability, where agents lack access to the full sys- tem state and must base decisions on partial observations. These problems are typically defined as partially observ- able Markov decision processes (POMDPs) [I. Despite the progress in reinforcement learning (RL) for address- ing complex tasks (26)), learning effective strategies scenarios with partial observations remains challenging. Traditional RL methods in POMDP contexts often en- counter impractical computations in processing the com- plete history, leading to suboptimal performance. Luckily, full states are accessible during training in many training scenarios. For example, in applications like autonomous driving[ 7 or robot control tasks [8) where computational and time limitations exist, only low-cost RGB images are available for deployment. At the same time, high-precision sensors can record more accurate and
State
Observation
Figure I: The WallGap environment, procedurally generated navigation task in MiniWorld. The agent's 's goal (red tri- reach the target (red box) within as few steps as possible under par- tial observability.
angle)
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

detailed data during training. Similarly. in card games 19. the private hands of other players are typically hidden during deployment but revealed during training. This distinction of separating available information during training and deployment stages has driven the adoption of an onfline
learning and online execution paradigm POMDP settings [00
This study focuses on the asymmetric actor-critic framework 8 14 15 where the actor and critic rely on different inputs. The critic, serving as an auxiliary tool for training, can be transformed into an oracle critic to align with the offline learning and online execution paradigm. Specifically, the oracle critic is empowered to access the complete states while the actor remains constrained to partial observations. Using an oracle critic conditioned on the state and observation history promises to enhance performance and theoretically achieve unbiasedness [1O]. However, the practical perfor- mance of such approaches raises doubts, particularly in domains characterized by high uncertainty and extensive state spaces. For instance, in the illustrated first-person 3D navigation task depicted in Figure | the agent is tasked with reaching a dynamic target that varies across episodes within as few steps as possible. This task involves a first-person view observation and an aerial view state. Introducing oracle critic incorporating state information in this complex environment may increase variance, potentially undermining learning efficiency and overall performance.
To tackle the challenges mentioned above, we argue that the indiscriminate utilization of state information inadvisable. We introduce a Dual Critic Reinforcement Learning framework (DCRL). DCRL consists of two critics: the oracle critic conditioned on the <history, state> tuple, and the standard critic conditioned solely on history. It introduces synergistic strategy to meld the strengths of the oracle critic for efficiency improvement and the standard critic for variance reduction. In summary, the main contributions of this work are as follows:
We propose DCRL framework to address partially observable tasks. This framework features a novel mechanism for seamless transition and weighting between the dual critics.
We theoretically prove that DCRL mitigates learning variance while maintaining unbiasedness.
The DCRL framework is simple to implement. While our emphasis lies on the Advantage Actor-Critic (A2C) [16,, DCRL can also be integrated into other actor-critic architectures, such as Proximal Policy Optimization (PPO) 177.
We extensively validate the DCRL framework based on A2C and PPO. Experimental results conducted on both Box2l) and Box3D environments confirm its effectiveness. The source code is available in the supplementary material.
2 Related Work
Most methods can be adapted for partial observability by training memory-based policies to condition past observations. Recurrent Neural Networks (RNNs) [8] are commonly used for their simplicity and strong performance. Deep Recurrent Q-Networks (DRQN) 19 integrates Long Short- Term Memory (LSTM) [20] cells into the Deep Q-Network (DQN) 2 architecture, which demonstrated promising results. In addition to LSTM, other memory mechanisms such as Gated Recurrent Units (GRUs) 21 22), Transformers
23 24 and Language Models 225 26 have been adopted in RL algorithms for memory learning. However, POMDPs are hard to solve when relying solely on memory-based policies because the size of the history grows linearly with the horizon length.
crucial that many environments, complete states are observable during the training phase. Utilizing full observable states during training has emerged as popular paradigm in RL. We discuss prior works on the utilization of full observability in RL algorithms. Additionally, we provide further discussion about training the world model, applications in multi-agent reinforcement learning, and other double structures in Appendix
Using complete states to train the critic. This topic is closely related to our paper and has been applied in many large-scale scenarios, such as DouDizhu [ and Mahjong (7]. The critic in the actor-critic framework is a training tool not required during deployment, making its structure naturally suitable to exploit the state. This is achieved by critic asymmetry, where the actor and critic receive history and state, respectively. Asymmetric Actor-Critic [ uses the critic conditioned on ground- truth states to address image-based -aaed robot control tasks. AACC [15] employs the critic conditioned on both states and observations to solve contextual Markov decision processes. Theoretically, using the

---

state as the critic input is biased, whereas using the <history, state> tuple is unbiased in expectation [O]. However, introduces high variance in practice.
Using complete states to train the policy. These methods typically involve training a teacher policy using complete state information and then utilizing this policy to guide the learning of student policy under partial observability. The forms of guidance vary, such as maintaining consistency of the Q- function [28], the encoder [II. setton-distibbutionns [29], and others. In addition introducing extra loss terms to enforce consistency between the student policy and the teacher policy, the transition from the teacher policy the student policy can also be facilitated through dropout of state information from the input [30 32]. Common issues include the imitation gap and suboptimal teacher, which can limit the potential capabilittes of the student policy.
Using complete states to learn the belief representation. Belief state methods I 33 are classical in POMDPS. These methods often necessitate knowledge of the dynamics and reward functions, thus rendering them more applicable to scenarios with well-defined system dynamics. Consequently, they are typically constrained to simpler settings characterized by small state spaces, where the complexities of the environment are more tractable. Several recent studies [34H36] focus on learning belief representations by predicting future observations and rewards. VRM 34 utilizes latent stochastic variable to model sequential observations. However, learning latent states that can capture long-term dependencies challenging. Incorporating the complete states can help alleviate this issue. DouZero+ [37] introduces auxiliary tasks for predicting states while concurrently training policies conditioned on observations and the predicted states. Believer [3] decouples belief state modeling from policy optimization. They first capture state features through representation learning and then train the policy conditioned on observations and the predicted states.
3 Preliminaries
This section introduces the background topics required to understand our work: partially observable Markov decision processes, the recurrent Advantage Actor-Critic algorithm, and the Unbiased Asymmetric Actor-Critic algorithm.
3.1 Partially Observable Markov Decision Processes
A partially observable Markov decision process (POMDP) 1 is formally identified as 7-tuple, namely P - A, Ωt R, O, 7)), where S. A. and Ω represent the state space, action space, and observation space, respectively. The transition function T' : S A + AS is denoted T(s, a, s') = p( a). describing the probability of transitioning from state s to state s' after taking action a. The agent receives observations rather than states, and the observation function O S defined as O(s, a, o) representing the probability of observing o conditioned on state s and action The reward function R: S A + E. is expressed as R(s, a). representing the reward received by acting in state s, and 7 E [o. 1] is the discount factor. In POMDPs, the optimal policy may depend on the entire history of observations and actions, denoted as H = (hylht (0o, 0p....0-11, at o:))- The belief-state b H + AS is the probability distribution over states given history h. The history reward function is defined as R(h, a) = Eajo. [R(, a)). The agent aims to maximize the expected cumulative discounted rewards EE E 7 '(R(s,, a:)] through interactions with the environment.
+ AΩ is
This paper assumes that both states and observations are accessible during training, while only the observations or are accessible during deployment.
3.2 Recurrent Advantage Actor-Critic
Advantage Actor-Critic (A2C) [6 is primarily designed for fully observable control problems. It can be casily adapted to handle partially observable control problems by replacing all occurrences of the state with the respective history h. To clearly distinguish between different methods, denote this method as AC. It trains an actor model π H + AA parameterized by 0, and a critic model V H -> RR parameterized by o. The policy loss is defined as:
CAC policy (0) = -E Σ 0
(h,, a.) log To [o(olhh.))
(1)
3

---

where Q" (h,, ar) denotes the history-action value of the policy т. In practice, (he, as) is replaced by the n-step bootstrapped advantage reduce variance:
n-
5s a,)
?"t R(hi+dd: ac+d) + y" VV - Vo(he).
(2)
d=0
Correspondingly, the policy loss is policc (0) -B[ do at) log To(as) Ita]] In this work,
the value loss defined as vallue (0) E
-i= 5o(he, at) ). The entropy loss is Centropy (0)
-E Ei=o EGEa To ma(aelly, (1x) log To (a (+a)
The total loss is LAC value + LAC palicy (0) + Contropy (0),
where 1) is weight for entropy regularization.
3.3 Unbiased Asymmetric Actor-Critic
Unbiased Asymmetric Actor-Critic [00] a variant of recurrent A2C that can exploit state information during training. It considers an oracle critic model V H. x S -> I& parameterized by 1 instead of the standard critic model. To clearly distinguish between different methods, we denote this method as UAAC. The n-step bootstrapped advantage for UAAC is defined as:
8y S1, (4)
Σ R((ss4 acta) Vo Sctn) (he,
(3)
d=0
Correspondingly, the policy model is trained to minimize the following objective function:
UAAC (0) =
Σ
de (he, St, az) To (h+)
1=
The value loss is defined as CUAAC actor-critic method, the state s provides additional context to determine the agent's true underlying situation, which may facilitate the learning of the oracle critic. Importantly, it has been proven that (T s) provides Monte Carlo estimates of the respective v=(h) for arbitrary policies, which is theoretically unbiased, i.e.,
= de (he, 8t, )+) Compared with the standard
v*(h) = Esja (h, s)],
(5)
Q" a) = EEja 10" (h, a)).
(6)
Therefore, the equality Vo (0) = V Vo (0) holds, indicating that the policy gradient
remains unchanged in expectation.
4 Dual Critic Reinforcement Learning
Critic models that learn accurate values slowly often become a bottleneck for policy performance in many scenarios. When state information is incorporated, may be easier learn (* with neural network compared with learning V (h), leading to faster policy learning. However, // (*( 8) has nonzero variance. Taking into account the variance arising from V
V/ (h, s): Var, [[V* (h, s)]
Eaja [I]V=(h, 8) - V (1)] with a practical batch size that is not too large, the variance tends to be high, especially in high- high-uncertainty belief states and large state spaces. In games like DouDizhu, when the initial information collection does not accurately infer the opponents' hands, training with VT (h, s) incorporating complete state information increases variance, thus limiting model generalization. Conversely, using V ("(h). which relics solely on observations, enhances robustness in such cases. This concept is akin to replacing the reward-to-ge term with the Q-value reduce variance in the vanilla policy gradient method, as discussed in 38 39
4.1 Theories of the Dual Critic
Definition 1. Define the dual value function as V dual (h, 8) = -1- (1-3))V²(h, (b),,), where /3 is a weight that balances the contributions of the / (h, 8) and (*(h), and B E [0, 1). Correspondingly; the dual Q function is defined as Q (h, s, a) = (1 (h, + 3Q" ah,

---

Theorem 1. For arbitrary control problems and policies, (h, s) is an unbiased estimate of (h) and Qduai 8, a) an unbiased estimate of Q" (h, a).
Proof. Following from Equation ( we have:
Eg h [dual (h, s)] = (1) 5v (h) 3)V (h,
= (1 - -)B) [h, + 3V
(h))
Similarly, as inferred from Equation (6 Raj (h 8, a)] = Q (h, a) by replacing the with Q using the same approach outlined above.
Therefore, the variance the dual value is defined as Var gn [ dual (h, s)] Eaja p [ (h, 8) V=()]]??, and that of Unbiased Asymmetric Actor-Critic defined as [ s)] = Eajh [IIV= - (h)]l]]3]
Theorem 2. The variance of the dual value is smaller than that of the Unbiased Asymmetric Actor-Critic, i.e., Var,jn[Väus (h, s)] S Var,ja [V [V" s)].
Proof. According to the definitions and | E [0, 11. it holds that
Var a/a paraaaVVduu (Vdual (h, s)] Var, 1/ [11 3)V* (h, 3V (h))
(1 B) Var, A (h) S Var, (TT) (h,
(8)
Similarly, Var, A (QTua (h, s, a)] & Var.n (h, 8, a)] by replacing the with Q using the same approach outlined above.
Considering the unbiasedness demonstrated above, we contemplate employing the dual function for policy updates. The policy gradient based on the dual Q function can be formulated as:
V a FaCoual (0) E Σ Qauat(he, St, ax) Va log no (axlht)).
(9)
This formulation maintains the policy gradient unchanged in expectation.
Theorem 3. The dual Q function maintains the policy gradient unchanged in expectation.
Proof. Following Theorem I therefore,
V CoCpolicy (0)
Σ (he loggm))a
Σ Ennaa (he at) log To (ailte)))
Σ Ene Bailh (hima Se, at) ax) log (a (ho)
(10)
Σ Bhinna Qāama (hi, St, at) V log To (a(a((h)) lh+)
Σ Qanathee St, at aaaaaVooo log (a [ha)
V
ddual policy (0).
Similarly, the dual Q function decreases the variance of the policy gradient according to Theorem
5

---

Available: training
State
Oracle Critic
Advantage 8h, 8,
1
B /
B
+
Advantage
clip
sum
daull (h, 8, a)
Standard Critic
B
Observation
Advantage 8(h, a)
Policy
Actor
Loss
Available:
E
ddual (h, 8, a) Volo (a(h))
training + deployment
To
Input
Data Flow
Policy Gradient Flow
Figure 2: Implementation of DCRL framework. DCRL innovates synergistic strategy to meld
strengths of the oracle critic for efficiency improvement and the standard critic for variance reduction.
4.2 Learning with Dual Crities
Building upon the abovementioned theory, we incorporate the dual Q function to balance variance and efficiency. This integration introduces the Dual Critic Reinforcement Learning framework (DCRL). As shown in Figure 2 the DCRL consists of three main components: 1) standard critic network (hh) parameterized by é, 2) oracle critic network V(h, s) parameterized by th, and 3) actor model raahh parameterized by 0. Inspired by the Equation (9) and Theorem we propose the policy and value losses of DCRL as the following objective functions:
CDCRL (0) = E
[(1 - B) (he, Se, at (a))ogra(ae(hh) + (8e (a)logro
(11)
t=
DDCRL
(Ø, 1/)
E Σ
5s a,
E
Σ
(h Sc. a,)
(12)
where (h,, ax) and (he, 8t, az) follow the definitions from Equation and Equation
3 The
dual advantage formulated as dana 8t, 0x) (1 33) St, 35 (hr,
We propose novel weighting mechanism to synergistically meld the two critics and thoroughly exploit the complete state information. When the received return does not exceed the agent's value estimate, the advantage is non-positive (ft,, a,) S 0), 3 is clipped to 0. In cases with (h,, at) > 0, 3 takes effect. is a parameter associated with but not with s. Hence, this mechanism maintains the theory mentioned above. This mechanism helps stabilize training and prevent large updates in cases where the advantage is non-positive. Similar
Learning where only
40) this clipping mechanism can be viewed as form of samples with positive advantage provide valuable insights into the optimal soft Q-value. From an empirical perspective, this approach embodies the notion that if there is still room for policy enhancement after being updated by the oracle critic (h, s). the standard critic Vhh) should be engaged. With full-state information from the oracle critic, DCRL avoids falling into vicious cycle of poor policy and bad value estimates. The standard critic in DCRL can effectively mitigate variance, whereas recurrent Actor-Critic methods preclude such opportunities due to entanglement in vicious cycle. These two critics establish a mutually supportive relationship. facilitating variance reduction while effectively using full-state information. The proof regarding lower- boudd--ooddooo--oooo Sooundsoft-Q-leeriing Q-leannin is provided in Appendix C
In addition to the mechanism mentioned above, there is another natural way to leverage the dual-critic framework by interchanging the roles of the oracle and standard crities. Specifically, we shift the weighting factor to the forefront of the (h St, ax) term which denote G', and the inside term

---

DistShift2
LavaCrossingS 9992
LavaGap56
MultiRoom N2-54
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
1
0
2
1c6
Empty-16x16 16x16
RedBlueDcors-8Ke 8x8
SimpleCrossingsRNS
Unlock
1.0
1.0
1.0
0.8
0.5
AuA
0.5
0.5
0.5
0.2
0.0
0.0
0.0
0.0
0.5
1.0
3
Frames
1e6
Frames
1e6
Frames
1e6
Frames
le5
DCRL (Ours)
Recurrent Actor-Critic
Asymmetric Actor-Critic
Oracle Guiding
Unblased Asymmetric Actor-Critic
Figure 3: Learning curves on MiniGrid. The x-axis and y-axis represent the training frames and average returns, respectively. Shaded bars illustrate the standard error. All curves are trained based on A2C over random seeds.
of the policy loss is defined as (1 - 8 8')6 ae (ae + (ly St, a (e)logoo(ae(hh)) 3 is clipped to O when &e (hy, St, (t) S O and otherwise takes positive value. 3r is parameter associated with both and s. Unlike DCRL, this alternative technique lacks the characteristic of unbiasedness and fails to thoroughly exploit state information while deviating from the lower-boune soft-Q-learning framework. As result, its performance exhibits instability compared with DCRL. The experimental section provides a detailed discussion of this technique's performance.
Experiments
We conduct series of experiments to evaluate the effectiveness of DCRL in partially observable domains. Our research focuses on the model-free setting where complete states are available during training. We compare our DCRL approach with several baselines.
Baselines. Recurrent Actor-Critic is recurrent version of the standard actor-critic algorithm, namely A2C [16] or PPO 17]. This baseline represents methods that do not utilize state information. Asymmetric Actor-Critic is a representation method based on critic asymmetry. The critic uses the state as input, independent of the history. Oracle Guiding (30) is representation method based on teacher policy. This approach gradually removes state information from the policy input, transitioning the teacher policy to student policy under partial observability. The dropout probability linearly increases with training iterations until only observation remains. We adopt a 50% cutoff for all tasks. Unbiased Asymmetric Actor-Crite [10] is state-of-the-art method for POMDPs. The critic uses the <history, state> tuple as input to exploit state information while introducing no bias into the training process, thus achieving superior performance. To ensure fair comparison, we keep the main network architectures similar across methods. We present the mean and standard deviation of results obtained from five different seeds for each environment, For comprehensive overview of other implementation details and hyperparameters, please refer to Appendix
5.1
MiniGrid
To see how useful DCRL is across various POMDP tasks, we compare various methods in Box2D environments. MiniGrid [11 is procedurally generated environment with goal-oriented tasks. As shown in Figure 4 the triangular agent with discrete action space aims to reach the goal. These tasks involve interacting with different objects and solving diverse maze maps that vary in each episode. The observation comprises a first-person view representing 7 x 7 3 image. At the same time, the
7

---

state offers fully observable perspective, with the image size varying according to the tasks. The image is compact and efficient encoding, rather than raw pixels.
All experiments are conducted based on A2C, and the results are plotted in Figure 3 Recurrent Actor-Critic struggles to learn a robust policy across most environ- ments, primarily due to the agent's need for extensive exploration without state information. Despite leverag- ing states, Oracle Guiding and Asymmetric Actor-Critic performance exhibits instability, attributed to their heuris- tic nature and lack of theoretical guarantees. In contrast, the Unbiased Asymmetric Actor-Critic maintains stability and outperforms the mentioned methods. Nonetheless, it faces challenges in scenarios such as LavaC LovaCnssinggS992 and RedBlueDDoors-Sxe Our DCRL framework surpasses all other methods in eight environments, particularly ex- celling challenging tasks. Taking LavaCrossing59N2 and Empty-16x16 as examples, the map size of the former smaller, but the layout is more complex. In the case of Empty-16x16, the states remain simple and relatively constant, making it easy to infer key information from history. Conversely, in LavaCrossing59N,, objects change between episodes, leading to high uncertainty in the belief state. While leveraging states aids in precise value function learning, the variance poses challenges in policy optimization. DCRL effectively overcomes these limitations and emerges formidable competitor against other baseline methods.
State
Observation
Figure 4: The LavaCrossingS9N2 envi- ronment in MiniGrid. The agent's goal (red triangle) is to reach the target (green box) within few steps as possible un- der partial observability.
Table I; Performance of agents on MiniGrid after le5 frames (27 games) and le6 frames (13 games) of training. We report the mean and standard error of the performance.
A2C
PPO
Methods
1e5 frames 1e6 frames
1e5 frames
le6 frames
Recurrent Actor-Critic
0.
0.28
0.26+10.07
0.52+00.33
Asymmetric Actor- Critic
0.26=
0.32#0.14
0.27
0.60=0.06
Oracle Guiding
0.18
0.27+0.06
0.25+0.10
0.48=10.5
Unbiased Asymmetric Actor-Critis
0.31
0.39
0.28+0.10
0.59-0,08
DCRL (Ours)
0.63:10.07
0.8910.01
0.47
0.84
To further validate the effectiveness of DCRL, we provide the visualization of different critic values in Appendix D.1. The visualization demonstrates that the dual value function Vauaa (h, s) effectively reduces variance compared with the oracle critic. Besides, we evaluate our DCRL in 27 MiniGrid tasks based on A2C and PPO. The average score across all games provided in Table demonstrate the overall performance for different methods. Our DCRL outperforms other baselines in 26 out of the 27 games. The detailed learning curves are available in Appendix D.2
5.2 MiniWorld
This section investigates whether DCRL benefits more complex continuous state space tasks and whether it can be applied to other Actor-Critic algorithms, such as PPO. Unlike A2C, PPO lacks strong theoretical connection DCRL, but we claim they can still complement each other. To empirically verify this, we implement all methods based on PPO and evaluate them on the MiniWorld domain 4T.. The observations and states are represented as 80 60 images rendered as pixels. The results depicted in Figure 5 illustrate that DCRL enhances PPO performance across all four environments. In contrast MiniGrid, the state format in this benchmark is not concise encoding and tends to contain redundancy and noise. Although states assist the agent in effectively discerning information about the environment, the variance introduced by noise poses a challenge to the learning process. Furthermore, it is observed that the Unbiased Asymmetric Actor-Critic performs poorer than the Recurrent Actor Critic across tasks. Nevertheless, DCRL converges superior policies by synergistically melding the two crities.
8

---

TMaze
YMaze
WallGap
Sidewalk
0.8
1.0
0.8
1.0
0.8
0.6
///
0.8
my
0.4
0.4
0.2
0.2
0.0
0.0
0,0
0.0
1
3
Frames
le6
Frames
leb
Frames
le6
Frames
1e6
DCRL (Ours)
Recurrent Actor-Critic
Asymmetric Actor-Critic
Oracle Guiding
Unbiased Asymmetric Actor-Critic
Figure 5: Learning curves on MiniWorld. The x-axis and y-axis represent the training frames and average returns, respectively. All curves are trained based on PPO over random seeds.
DistShift2
LavaCrossingS9M)
LavaGap56
MultiRoom N2:S4
1.0
1.0
1.0
1.0
0.8
0.5
0.5
0.2
0.2
0.0
0.0
0
0.0
0,5 Frames
1.0
2
Frames
1e5
Frames
1e6
Frames
1e5
DCRL (Ours)
DCRL Interchange Version DCRL State Version
Standard Critic Version
Oracle Critic Version
Figure 6: Ablation studies on MiniGrid verify the two key factors of DCRL. All curves are trained based on A2C across 5 random seeds.
5.3 Ablation Studies
Ablation analysis of components in DCRL. We conduct detailed ablation studies to analyze how DCRL improves efficiency and enhances performance, DCRL contains two key factors: the dual-critic structure and the weighting mechanism. We replace the dual-critic structure with the single-critic structure to verify the structure's effectiveness. The Standard Critic Version and Or- acle Critic Version are constructed with the following expressions for the inside terms of the policy loss: (hr,, as) logToo (a, (42) + 350g (he, as llogoo (1a) and d (Ite, 8t, Or logga (aa)aaaaaa/ (az]/te) - 3' 2he, St, az loggTo(aa (a, h4). Next,
evaluate another weighting mechanism baseline (DCRL Interchange Version) discussed in Section 4.2 with the inside terms of the policy loss as (1 (1-8") --, at ao)logTo (4)) + 3' & (h, St. ae logms (aelhh) The adjustment of B and &' fol- lows the same way as discussed in Section [42 Additionally, we introduce an intuitive variant that modifies the standard critic's input from history to state, referred to as the DCRL State Version. This adjustment aligns with previous works that introduce biases; however, it has the potential to enhance efficiency gains. All methods utilize the same architectures and hyperparameters as DCRL.
We select four environments from the MiniGrid domain, where state information plays a crucial role. In these environments, methods that utilize states perform significantly better than those that do not. We compare our DCRL approach with four ablation baselines, and the results are presented in F It can be observed that DCRL converges optimal policies faster than other methods. Intuitively, incorporating the concept of positive advantage (5+) can optimize the utilization of beneficial samples and expedite the training process. Consequently, single-critic structure methods integrating a 6+ term outperform those lacking it, as demonstrated by the superior performance of approaches like the Standard Critic Version compared with the Recurrent Actor-Critic. However, it is crucial that the two critics fulfill distinct and orthogonal roles within the dual-critic structure. DCRL incorporates an oracle critic that utilizes state information compared with the Standard Critic Version. The inclusion of state information provides the agents with additional context to accurately determine the environment's true state, facilitating learning an accurate value function and ultimately improving performance. In comparison to the Oracle Critic Version, DCRL reduces variance. These factors contribute to DCRL's efficiency. As for the DCRL Exchange Version, although it is also a dual-critic structure, its policy gradient is biased, and its weighting mechanism

---

DistShift2
LavaCrossingS9N)
LavaGap56
MultiRoom N2-54
1.0
1.0
1.0
1,0
0.8
0.3
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
1
4
0
1
2
le5
Empty-16x16
RedBlueDoors-88x 8x8
SimpleCrossingsRks
Unlock
1.0
1.0
1.0
0.8
0.5
0.5
0.5
0.5
0.2-
0.0
0.0
0.0
0.0
0.0
0.5
Frames
1e6
Frames
1e6
Frames
1e6
Frames
1e5
Unbiased Asymmetric Actor-Critic (B-0)
DCRL (p-1/5)
DCRL (8-1/3)
DCRL (8-1/2)
DCRL Clip Version (8=1/2)
DCRL (B=2/3)
DCRL (8=4/5)
Figure 7: Ablation studies on MiniGrid analyze the robustness of in DCRL. All curves are trained based on A2C across 5 random seeds.
does not utilize state information as effectively as DCRL. Its performance only shows advantages in limited enviromments. converges to suboptimal policies in DistShift2 and LavaGapS6. The DCRL State Version demonstrates commendable performance among the various DCRL variants. However, the biased characteristics of the DCRL State Version contribute to instability in its effectiveness. Notably, our DCRL achieves superior training performance in three out of four environments. These findings further highlight the significance of our DCRL framework.
Ablation analysis of /3 in DCRL. Our DCRL incorporates dynamic weighting mechanism, which can be interpreted form bowwer-boudd-ofte soft-Q-leaminn, distinguishing it from conventional linear weighting methods. This mechanism substantially improves DCRL's performance while maintaining high stability concerning hyperparameters. To substantiate this claim, we implement simplified weighting method with fixed ratios for weighting the two critics, referred to as the DCRL No Clip Ver- sion. Additionally, we conduct ablation experiments on the parameter / E {1/5, 1/3, 1/2, 2/3, 4/5). The results depicted in Figure indicate that the DCRL No Clip version shows performance improve- contrast,
ments in only some environments compared to the Unbiased Asymmetric Actor-Critic DCRL consistently enhances performance across all environments, underscoring the importance of the dynamic weighting mechanism. Furthermore, the stability of /3 is notably evident. We choose = 1/2 for reporting results in the paper, given its consistently superior performance.
6
Conclusion
This paper introduces novel reinforcement learning framework designed for partially observable tasks. The motivation behind this work is that complete state information may be accessible during training in many scenarios. However, introducing state could potentially lead to bias or variance. The proposed DCRL method leverages two asymmetric crities to use the state to expedite training and mitigate variance in practice. We underscore the benefits of training with two critics and provide thcoretical evidence supporting the unbiasedness and variance reduction achieved through approach. Empirical results obtained across multiple tasks serve validate the efficacy of DCRL.
Limitations. Our framework operates under the assumption that state information is accessible during training. However, this assumption may not always hold, as only part of the state information might be available within a broader context. It crucial to discuss whether this limitation will impact the performance of our framework, necessitating further investigation. Furthermore, our observations suggest that the availability of complete state information can sometimes hinder performance in certain environments. This observation underscores the need for a more nuanced approach to the design and utilization of state information. Addressing these issues could lead enhancements in the performance and applicability of our framework.
10

---

7 Acknowledgement
This work supported in part by the Natural Science Foundation of China under Grant No. 62222606 and 62076238.
References
[1] Leslic Pack Kaelbling, Michael L Littman, and Anthony Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2)99-134, 1998.
[2] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-555 2015.
[3] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
[4] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. Nature, 550(7676):54-359, 2017.
[5] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140
2018.
[6] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):644-09) 2020.
[7] Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Jia Zeng, Zhiqi Li, Jiazhi Yang, Hanming Deng, al. Delving into the devils of bird' s-eee-view percep- tion: A review, evaluation and recipe. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2151-217) 2024.
[8] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. Proceedings of Robotics: Science and Systems, pages 1-10, 2018.
[9] Guan Yang, Minghuan Liu, Weijun Hong, Weinan Zhang, Fei Fang. Guangjun Zeng, and Yue Lin. PerfectDou: Dominating doudizhu with perfect information distillation. In Advances in Neural Information Processing Systems, pages 14954--49955. 2022.
[10] Andrea Baisero and Christopher Amato. Unbiased asymmetric reinforcement learning under partial observability. In International Conference Autonomous Agents and Multiagent Systems, pages 44-52, 2022.
[11] Dongqi Han, Tadashi Kozuno, Xufang Luo, Zhao Yun Chen, Kenji Doya, Yuqing Yang, and Dongsheng Li. Variational oracle guiding for reinforcement learning. In International Confer- ence on Learning Representations, pages 1-13, 2022.
[12] Gaspard Lambrechts, Adrien Bolland, and Damien Emst. Informed pomdp: Leveraging additional information in model-based rl. arXiv preprint arXiv:2306. 1488, 2023.
[13] Andrew Wang, Andrew C Li, Toryn Q Klassen, Rodrigo Toro Iearte, and Sheila A Mcllraith. Learning belief representations for partially observable deep rl. In International Conference Machine Learning, pages 35970-35988 2023.
[14] OpenAl, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafa Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray. Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in hand manipulation. International Journal of Robotics Research, 39(1):3-20, 2020.
11

---

[15] Wangyang Yue, Yuan Zhou, Xiaochuan Zhang, Yuchen Hua, Zhiyuan Wang, and Guang Kou. Aacc: Asymmetric actor-critic in contextual reinforcement learning. arXiv preprint arXiv:2208.02336, 2022.
[16] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. In International Conference on Machine Learning, pages 1928-1937) 2016.
[17] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634/ 2017.
[18] Jeffrey L Elman. Finding structure in time. Cognitive Science, (4(2):179-22)) 1990.
[19] Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. arXiv preprint 150700677 2015.
[20] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, (8):1735-1780, 1997.
[21] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3553 2014.
[22] Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be strong baseline for many pomdps. In International Conference on Machine Learning, pages 16691-16723, 2022.
[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In Advances in Neural Informa- fion Processing Systems, pages 6000 2017.
[24] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayaku- mar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning.
pages 7487-7498, 2020.
[25] Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Annual Meeting of the Association for Computational Linguistics, pages 2978-2988 2019.
[26] Fabian Paischer, Thomas Adler, Vihang Patil, Angela Netto-Nemling, Markus Holzleitner, Sebastian Lehner, Hamid Eghbal-Zadeh, and Sepp Hochreiter. History compression via language models in reinforcement learning. In International Conference Machine Learning, pages 17156-1718 2022.
[27] Jinqiu L.,, Shuang Wu, Haobo Fu, Qiang Fu, Enmin Zhao, and Junliang Xing. Speedup training artificial intelligence for mahjong via reward variance reduction. In IEEE Conference on Games, 2022.
pages
[28] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian. Yong Yu, and Tie- Yan Liu. Universal trading for order execution with oracle policy distillation. In AAAI Conference on Artificial Intelligence, pages 107-115, 2021.
[29] Hai Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, and Robert Platt. Leveraging fully observable policies for learning under partial observability. In Conference on Robot Learning, pages 1673-1683, 2023.
[30] Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang. Li Zhao, Tao Qin, Yan Liu, and Hsiao- Hon. Suphx: Mastering mahjong with deep reinforcement learning. arXiv preprint 200.::20003119900 2020.
[31] Piexre-Alexandre Kamienny, Kai Arulkumaran, Feryal Behbahani, Wendelin Boehmer, and Shimon Whiteson. Privileged information dropout in reinforcement learning. arXiv preprint arXiv:2005.09220. 2020.
12

---

[32] Stephan Weigand, Pascal Klink, Jan Peters, and Joni Pajarinen. Reinforcement learning using guided observability. arXiv preprint 1111::2004.109866 2021.
[33] David Silver and Joel Veness. Monte-carlo planning in large pomdps. In Advances in Neural Information Processing Systems, pages 2164-2172, 2010.
[34] Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable control tasks. In International Conference on Learning Representations, pages 1-12, 2020.
[35] Yuhui Wang and Xiaoyang Tan. Deep recurrent belief propagation network for pomdps. In AAAI Conference on Artificial Intelligence. pages 10236-10244, 2021.
[36] Xiaoyu Chen, Yao Mark Mu, Ping Luo, Shengbo Li, and Jianyu Chen. Flow-based recurrent belief state learning for pomdps. In International Conference on Machine Learning, pages 3444 3468, 2022.
[37] Youpeng Zhao, Jian Zhao, Xunhan Hu, Wengang Zhou, and Houqiang Li. Full douzero+: Improving doudizhu by opponent modeling, coach-guided training and bidding learning. IEEE Transactions on Games, pages 1-13, 2023.
[38] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, pages 1057-1063, 2000.
[39] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems, pages 1008-1014, 1999.
[40] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Interna- tional Conference on Machine Learning, pages 3878-3888 2018.
[41] Maxime Chevalier-Boisvert,, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. arXiv preprint arXiv:2306.13833 2023.
[42] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI conference on artificial intelligence, pages 2974 2982, 2018
[43] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competttve environments. pages 6382 6393, 2017.
[44] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21 (117))::1-51, 2020.
[45] David Henry Mguni, Haojun Chen, Taher Jafferjee, Jianhong Wang, Longfei Yue, Xidong Feng. Stephen Marcus Mcaleer, Feifci Tong. Jun Wang, and Yaodong Yang. Mansa: Learning fast and slow in multi-agent systems. In International Conference on Machine Learning, pages 24631-24655 PMLR, 2023.
[46] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-leamiig. In AAA! Conference on Artificial Intelligence, pages 2094-2100, 2016.
[47] Scott Fujimoto, Herke Hoof, and David Meger. Addressing functiin approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1,587-159.. 2018.
[48] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870, 2018.
13

---

[49] Dongming Wu, Xingping Dong, Jianbing Shen, and Steven CH Hoi. Reducing estimation bias via triplet-average deep determimistic policy gradient. JEEE Transactions on Neural Networks and Learning Systems, 3 (1 ):4933-4945, 2020.
[50] Yen-Chen Wu, Bo-Hsiang Tseng, and Milica Gasic. Actor-double-critic: Incorporating model- based critic for task-oriented dialogue systems. In Conference on Empirical Methods in Natural Language Processing: Findings, pages 354-865, 2020.
[51] Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, and Xiu Li. Efficient continuous control with double actors and regularized critics. In AAAI Conference on Artificial Intelligence, pages 7655-7663, 2022.
[52] Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. In Advances in Neural Information Processing Systems, pages 1-11,
[53] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 .4.00011444 104, 2023.
14

---

A Discussions
Incorporating state information can offer significant advantages for training, as it often encapsulates high-level features that improve the learning of V V(h, s) compared to ((h) [TO]. For example, in an extreme scenario where the state information & represents the true value of (h),, the agent can effortlessly infer this true value through (hh, s). Specifically, in the context of the Empty Navigation task, when the state information indicates the agent's relative distance to the goal, the value function learning process is significantly streamlined. Besides, in most POMDPs, rewards are directly defined based on state information rather than on observations. Consequently, learning the value function V(h, s) with the reward as the target is typically simpler than learning V(h). This characteristic makes state information effective in most practical applications, a conclusion that numerous previous studies have empirically validated.
However, state information does not always enhance training. When the state information contains significant noise, introducing excessive noisy features to the input can hinder learning the value function. In unfamiliar environments, whether state information genuinely supports the learning process remains to be determined. The motivation behind this work is that state information may be accessible during training in many scenarios. However, introducing a state could potentially lead to bias variance. In most cases, state information offers higher-level features that assist agents in decision-making. In these situations, DCRL enhances the process by further reducing variance. Conversely, when state information and partial observations correspond one-to-one, the oracle critic demonstrates no variance; DCRL may become ineffective under these circumstances.
B
Additional Related Work
Using complete states to train the world model. Traditional POMDP solutions rely on accurate environment models. Informed Dreamer [12] improves world modeling by incorporating state information. However, the dynamics models of POMDP environments are complex, posing challenges to modeling even with state information. Model inaccuracies will inevitably limit the performance. Our focus in this paper is model-free methods.
Using complete states in multi-agent reinforcement learning. Centralized Training Decentral- ized Execution (CTDE) 44 is prevalent paradigm for addressing challenges in multi-agent reinforcement learning (MARL). The optimization objective in MARL is to maximize the rewards under the global states, optimizing each agent accordingly. These methods mostly presuppose that the combination of optimized actions under partial observations leads to optimality under global states. Building on CTDE, 455 introduces Global agent to decide when to activate global information usage. However, the policy that maximizes rewards under partial observations does not necessarily align with the policy that maximizes rewards under global states, i.c., Q(h, a) * Q(s, a) 19). Contrary to CTDE's assumptions, the learning objective of DCRL is to derive the optimal policy under partial observations.
Double structure in the actor-critic framework. our work, we employ two asymmetric critics to address partially observable tasks. Previous studies have explored various double structures. In the context of double critics, Double DQN (46) utilizes replicated Q-network as the target network to handle overestimation issues, method later extended to subsequent works such as TD3 [47] and
SAC 48 Considering the opposite properties of single-critic and double-critic methods, TADD 49
incorporates novel triplet critics mechanism to reduce the estimation bias. Additionally, ADC (50]
takes ad advantage of environmental models by training a model-basch critic, which is then combined with the model-free eritic. This approach enables learning from the model-free path despite model noise, enhancing the robustness to environmental models. In the context of double actors, DARC [51]
introduces double actor structure boost the agent's 's exploration ability by providing two paths for policy optimization, thus reducing the risk of the agent getting stuck locally. Moreover, DAC [52]
utilizes a parallel actor-critic architecture learn better options, indicating that the two crities can be effectively consolidated into one.
C Additional Proofs
Theorem 4. In the DCRL Interchange Version, Espo [(1 (b) + (h, # (h)..
15

---

Proof. Since 3' is parameter associated with both h and it cannot shift to the forefront of the expected symbol. Therefore, the dual value of the DCRL Interchange Version is biased. Also, the policy gradient in the DCRL Interchange Version is biased.
IT
Theorem 5. The learning objective (6 (ogg in DCRL can be viewed as a form of lower-bound-sofl-- seaeb-uundsoo------eeriii Q-learning.
Proof. According to Self-Imitation Learning [40] the Lower bound soft Q-learning updates Q(h, a) follows:
|II (R 1 R
(13)
where 1 is a behavior policy, (he, at, Rd) is the trajectory triples from μ4, R 2k=e+1 7 (rx + o.H)) is the regulay-regulariiedd return, represents the weight of entropy bonus, and H = -logı (ax)/t) is the entropy of the policy, According to the form of optimal soft value function and optimal policy the entropy-regularie RL, it is natural to consider the following forms of a value function V and policy w:
(hh = log E. exp (Q(h,a)/o)) Ta h) exp ((Q(h, a) V(h)) /a), Qh,, a) (h) + T(a h).
(14)
(15)
(16)
For convenience, we can define the following:
N R a log (a h),
(17)
A = R - R a) = R- V(h).
(18)
We can derive the gradient estimator of lower-bound-ooff---eaammn for the actor-critic architecture as follows:
V Fp a. R-n
E | (oQ(h,
= E |- V (a ग (a h1) + V(N)) A-]
= E a Volog T h) [oV(H)
E V Clbb
[oV(H)
(19)
policy
=
E
a Policy
VoV(h) (R - a(h,a))
VoV(h) (R-V(N))
=
E
V Pulicy
+ V 1
(R (R-v(h)
|
= E
V 0 Puttic
E
a V Cpolicy
+ V cle
where Clb policy
log (a I)) (R- v(l)) and Clalue = (
As n + 0o, the learning objective in DCRL is 1 - V(h)) (o])(a//)). It can be viewed as form of bound-softt--learmnnn Q-learning when + 0.
In the DCRL Interchange Version, the equation Esja [R. V(h, R V(h) holds according to Equation 5 However, the equation [, V(h, s))+ (f- (hh)) does not hold due to the presence the + operation. The inconsistency in the equation arises from the fact that the + operation does not commute with the expectation operator, leading to difference in the results. Consequently, the learning objective in the DCRL Interchange Version cannot be considered form of bownr-bound soft Q-learning.
16

---

D Additional Experimental Results
D.1 Visualization Critic Values
To illustrate the effecttivnness of DCRL in variance reduction, we visualize the values of different critics in the Sinpty-Randon -666 environment with first-person view size of 3 x 3, as depicted in 8 In this configuration, the same history is linked different states. We collect history samples, each associated with over 300 state samples, meaning that each V* (*(h) is connected to multiple V* (*'h,,, 8) and multiple (h, s). The converged critic models are used predict V (h), V (h, 8), and (duat(h, s), respectively. The 8 violin plots in Figure 9 depict the details of each history sample. These plots show density estimation of the values, with the center line indicating the mean value. It is evident that DCRL (shown in orange) demonstrates lower variance compared with the oracle critic (shown in blue).
Observation
State
State
State
Figure 8: MiniGrid-Emin Empty-Randon -x66.. The dimensions of the fiss--person view are 3 x 3. The left figure illustrates the observation, while the right displays different states that share the same observation.
History Sample
History Sample
History Sample
History Sample 3
0.80
0.82
0.80
0.78
0.78
0.83
0.78
0.80
0.76
0.73
0.74
0.74
0.75
0.70
0.72
0.73
History Sample 4
History Sample 5
History Sample
History Sample 7
0.82
0.85
0.80
0.83
0.78
0.80
0.78
0.76
0.78
0.74
0.75
0.72
V(h)
Wh.
Voun(h),
Figure 9: Visualization of 8 history samples. The violin plots show density estimation of the values, with the center line indicating the mean value.
D.2 Learning Curves of All the Compared Methods
Figures II and 12 present the learning curves of all compared methods across 27 MiniGrid tasks. utilizing A2C and PPO, respectively.
D.3 Learning Curves of Different /
Figure 1 presents the learning curves for different values of 3 across 27 MiniGrid tasks.
17

---

D.4 Comparison with DreamerV3
Although our research focuses on the model-free setting, we also compare DCRL with DreamerV3 [53]. a widely used model-based method in POMDPs. DreamerV3 constructs a model of the environment and enhances its behavior through the imagination of future scenarios. Figure I0 presents the average scores over Le steps across tasks in the MiniGrid environment. The results demonstrate that DCRL consistently outperforms DreamerV3 across all tasks. Notably, DCRL requires fewer computational resources and less training time than DreamerV3.
Distthift2
Doarkey 5x5
Empty 8x8
KeyCorridor3R1
1.0
0.8
W
0.2
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e5
1e5
1e5
LavaGapS6
MultiRoom N2 S4
SimpleCrossingS9N)
Unlock
1.0
0.8
0.8
0.6
0.8
0.5
0.5
0.4
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.5
0.0
0.5
1.0
0.0
0.5 Frames
Frames
145
Frames
1e5
Frames
le5
1e5
DCRL (Ours) Oracle Guiding
Recurrent Actor-Critic Unblased Asymmetric Actor-Critic
Asymmetric Actor-Critic Dreamerv3
Figure 10: Learning curves on MiniGrid, All curves are trained based on A2C over 5 random seeds.
E
Experiment Details
E.1
Environment Descriptions
We describe each task environment used in experiments. More detailed introductions can be found on the official project homepage of each benchmark. The source code is available in the supplementary.
MiniGrid
MiniGrid 411 comprises Box2D environments featuring various room layouts, interactive objects, and goals. A red triangular agent operates within a discrete action space, with actions such as "Move forward", "Turn left", "Tum right", "Pick up object", "Toggle/activate others. Interactive objects include keys, doors, balls, and more. The agent must learn specific action sequences achieve the final goal within its limited forward field of view, with rewards given only upon goal attainment. Each episode samples a different grid layout, and the agent has a first-person partially observable view of the environment.
object", and
We selected different experimental settings. For instance, as depicted in Figure the agent in the CovaaCossingg99N2 environment aims to reach the green goal square in the room's opposite corner while avoiding deadly lava rivers that terminate the episode failure. Each lava stream runs either horizontally or vertically across the room, with single safe crossing point. Fortunately, path the goal is guaranteed to exist.
The agent receives 7 x 7 x 3 dimensional image of the environment as observation. The three channels represent object ID, color ID, and state. The agent is positioned at the observation's bottom center, and the state is encoded similarly to the observation, consistently displaying all grids without adjusting to agent's orientation. The final channel may include 1, 2, 3. or values the agent's current coordinates, indicating the agent's facing direction (east, south, west, and north).
18

---

MiniWorke
Similar MiniGrid, MiniWorld 41 is minimalistic Box3D interior environment simulator consist- ing of connected rooms with objects inside. In MiniWorld, the agent has 3D partial view of the environment and must navigate to the red box. For simplicity, actions are discrete, including "Turn left", "Turn right", "Move forward", "Move back" and others. Turn and move actions rotate move the agent by small fixed interval, simulating the behavior differential drive robot. Rewards are based the number of time steps required to successfully complete the task, with small penalty incurred. the task is not completed within the maximum steps, the current episode is terminated, and reward of 0 is assigned.
We selected four experimental settings. For instance, as depicted in Figure the WallGap environment features two rooms connected by gap in wall. The agent must navigate toward the red box, randomly placed in one of the rooms.
Observations consist of RGB images sized 80 x 60 x 3 captured from the agents perspective, while states RGB images of size 80 60 3. representing an acrial view of the environment, In our experiments, RGB images are converted into grayscale images.
E.2 Hyperparameters
We run all experiments on a single server with 64 Intel(R) Xeon(R) Gold 5218 CPU processors @ 2.30GHz and I Tesla V100 GPU. The hyperparameters for training each method summarized in Table 2
Table 2: Hyperparametees used for training each method.
Hyperparameter
MiniGrid
MiniGrid
MiniWorld
Algorithm
A2C
PPO
PPO
Seeds in experiments
5
5
5
Discount factor ?
0.99
0.99
0.99
for GAE Rollout steps
1
0.95
0.95
5
512
512
Number of workers
16
16
16
Entropy loss coef Optimizer
0.01
0,01
0.01
R.Mpprop
Adam
Adam
learning rate
le-3
3e-4
3e-4
max grad norm
0.5
0.5
0.5
PPO clip range PPO training epochs
0.2
0.2
4
4
PPO mini-batch size dual update iteration dual training epochs dual batch size B
512
512
16
4
4
8
640
2048
2048
0.5
0.5
Best chosen from [0.1, 0.5)
E.3 Network Structures
In this section, we describe the structures of the policy network, the standard critic network, and the oracle critic network. The general architecture for all environments is the same, except for the representation of input images. We utilize the CNN model for representation, and the structures are contingent upon the dimensions (n, m) of the input images. Detailed information each environment is provided below.
Representation Network MiniGrid
(1) n < or m 33 Conv2d(in=3, out=32, kernel=2, stride=1, pad=l), Relu,
Conv2d(in=22, out=64, kernel=2, stride=1, pad=0),
19

---

Relu,
Conv2d(in=64, out=64, kernel=2, stride=1, pad=0),
Relu,
Flatten.
(2) n & 15 or m - 15
Conv2d(in=3, out=32, kernel=2, stride= pad=O),
Relu,
Conv2d(in=32, out=64, kernel-2, stride=1, pad=0),
Relu,
Conv2d(in=64, out=64, kernel22, stride=1, pad=0),
Relu,
Flatten.
(3) n > 15 or 15
Conv2d(in=3, out=32, kemel=2,, stride=1, pad=O), Relu,
MaxPPol22((ernel22))
Conv2d(in=32, out=64, kernel22, stride=1, pad=0),
Relu,
Conv2d(in=64, out=644 out=64, kernel=2, stride=1, pad=0),
Relu,
Flatten.
MiniWorld
Conv2d(in=1, out=32, kernel=8, stride=4, pad=0),
Relu,
Conv2d(in=32, out=64, kernel=4, stride=2, pad=0),
Relu,
Conv2d(in=64, out=64, kernel=3, stride=1, pad=0),
Relu,
Flatten.
Policy Network
Representation Network, FC(inzinput_ding _iim out=64), Relu,
LSTM(in=64, out=64). FC(in=128, out=128).
Relu,
FC(in=128, out=number of actions). Standrad Critic Network
The structure the same as that of the policy network, except for the dimension of the final MLP output head, which is set to 1.
Oracle Critic Network
The structure is the same that of the policy network, except for the dimension of the final MLP output head, which is
20

---

DistShift1
DistShif2
Doorkey 5x5
Door Key 6x6
1.0
1.0
0.5
0.5
0.0
0.0
0.0
2
0.0
0.5
1.0
1e5
1e5
le5
1e6
DoorKey-8xt 8x8
Emply-5x5
Emply 6x6
Emply 8x8
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
1
1
2
Empty-16x16
Emsty-Random-555
Hmppty-anddom-6w
FourRooms
1.0
0.6
Return
0.4
0.5
0.5
0.5
0.2
0.0
0.0
0.0
0.0
0.0
0.5
1.0
1e6
1e5
1e5
1e6
KeyComMoorS3R1
LavaCrossing59Ni
LavaCrossing59Na
1.0
1.0
0.8
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.2
0.0
0.5
1.0
1.5
LavaGapS5
LavaGapS6
LavaGapS7
MultiRoom-N22 N2-54
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.0
0.0
0.0
0.0
le5
1e5
1e5
l65
RedBlueDoors-6x6
RedBlueDoors-Bx0
SimpleCrossingS3N1
SimpleCrossing5oNz
1.0
1.0
1.0
0.8
0.5
0.5
0.2
0.2
0.2
2
1
2
1e6
1e6
1e6
Frames
1e6
SimpleCCresinggSne
SimpleCrossingS1Ng
Unlock
1.0
1.0
DCRL (Ours)
Recurrent Actor-Critie
Retum
0.8
Asymmetric Actor Critic
0.5
0.5
Guiding
0.2
0.2
Unblased Asymmetric
Actor-Critic
0.0
0.0
0.0
Frames
1e6
1e6
le5
Figure 11: Learning curves on MiniGrid. All curves are trained based on A2C over 5 random seeds.
21

---

DistShift11
DistShift2
DearKey 5x5
DoorKey 6x6
0.0
0.0
2
1e5
1e5
le5
1e5
DoorKey Bx8
Emply-55x5
Emply 6x6
Emply 8x8
1.0
0.8
0.8
0.8
Return
0.5
0.5
0.0
0.2
0.0
2
1
2
1
1
2
1e5
1e5
Empty-16x16
Empty-Random-545
Emppy-Randdo-666
FourRooms
1.0
0.8
0.8
0.4
Return
0.6
0.6
0.2
0.4
0.0
0.2
0.0
0.5
1.0
2
1e6
le5
1e5
1e6
KeyCorrdorS3R1
LawaCrossing
LavaCrossing
LavaCrossingS 9N3
1.0
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.2
0
2
0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e7
1e7
LavaGapSS
LavaGapS6
LavaGapS7
MultiRoom--225
1.0
1.0
0.8
0.8
0.8
0.6
0.2
0.2
0.2
0.2
0.0
0.0
0.0
le5
le5
1e5
1e5
RedBlueDoors-6x6
Doors880
SimpleCrossingS9N1
SimpleCrossing59ne
1.0
0.8
0.8
0.8
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.5
1.0
1.5
0
1
0.0
0.5
1.0
1.5
0
1 Frames
2
1e6
1e6
1e6
1e6
SirmpleCrossingseN
SimpleCrresingS1115
Unlock
1.0
1.0
DCRL (Ours)
0.8
Recurrent Actor-Critic
0.5
Asymmetric Actor Critic
D.5
Guiding
0.2
0.2
Unbiased Asymmetric
ActoFCritic
0.0
0.0
0.0
0.5
1.0
0.0
2.5
5.0
7.5
Frames
1e6
1e7
Frames
1e5
Figure 12: Learning curves on MiniGrid. All curves are trained based on PPO over 5 random seeds.
22

---

DistShift1
DistShift2
Doorkey-5x5
DoorKey-6x6
1.0
0.8
0.5
0.5
0.5
0.5
0.2
0.0
0.0
0.0
0.0
0.5
1.0
DoorKey-8x8
Emply-5x5
Emply-6xx
Emply-8xa
1.0
1.0
1.0
Reaurn
0.5
0.5
0.0
0.0
0.0
0.0
1e6
1e5
1e5
1e5
Empty-16x16
Empty-Random-55
Empty-Random-
1.0
1.0
0.4
0.5
0.5
0.5
0.2
0.0
0.5
1.0
1e6
1e5
1e5
1e6
KeyCorriderS3R1
LavaCrossingS9N1
LavaCrossingS9N2
LavaCrossingSoN3
1.0
1.0
1.0
0.8
0.8
0.5
0.5
0.5
0.2
0.0
0.5
1.0
1.5
0
2
1e6
1e6
LavaGapSS
LavaGapS6
LavaGapS7
MultiRoem N2 54
1.0
1.0
1.0
1.0
0.8
0.8
0.5
0.2
0.2
0.2
0.0
0.0
0.0
0.0
1e5
1e5
le5
RedBlueDoors 6xx6
RedBlucDoors-8xx8
SimpleCrossingSNN1
SirmlleCrosinggeN2
1.0
1.0
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.5
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
1e6
1e6
1e6
Frames
1e6
Simple Frossingnn
SimpleCroscingS11N5
DCRL (8=12)
1.0
1.0
1.0
Version (2-1/2)
DCRL No Clip
0.8
Unblased Asymmetric
Actor-Critic (8=0)
0.5
0.5
0.5
DCRL (8=1/5)
0.2
DCRL (8=1/3)
(8-2/3)
2
0
DCRL (8=4/5)
Frames
le6
Frames
le6
Frames
le5
Figure 13: Ablation studies on different values of and DCRL No Clip Version. All curves are trained based on A2C over random seeds.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide a detailed introduction to the proposed method and provide relevant theoretical proof in Section 4 In Section the effectiveness of the proposed method.
[ we present experimental results to demonstrate
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer:
[Yes]
Justification:
We provide a complete proof in Section 4 and Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and theorems that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer:[Yes)
Justification: We provide experimental details in Appendix E and the source code is available the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The source code available in the supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cor
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 /tippe.cc/pblllt/ggiiddee/////// ioaeetteeee/eeSeemmimieem www.iiisssiipoiiiooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yor Yes]
Justification: We provide experimental details in Appendix and the source code is available the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yee
Justification: We present the mean and standard deviation of results obtained from five different seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:[ Yes]
Justification:
We provide experimental details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
The research conducted in the paper conforms with the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited the original paper that produced the code package.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, thcerswithcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the source code and instructions in the supplementary.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "A9OxX05GSy", "reviewer": "Reviewer_Hmpg", "summary": "This paper proposes a dual critic architecture for learning POMDPs. One critic is the standard critic that uses the history information while the other critic is the unbiased asymmetric critic that uses the history and state information. The authors prove that using both critics reduces variance and propose a weighting mechanism between the two critics based on the advantage. Empirical results show that DCRL provides a substantial benefit over the asymmetric critic or unbiased asymmetric critic.", "questions": "See weaknesses", "limitations": "The authors clearly discuss the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The authors motivate the problem clearly and explain why a dual critic approach would be useful\n- The authors prove that variance is reduced\n- Experimental results show significant improvements over baselines on minigrid and miniworld.", "weaknesses": "The primary weakness of this paper is novelty. While I can appreciate that the dual critic architecture reduces variance, the proofs seems fairly obvious. In my opinion, the main novel contribution of this paper then relies on the weighting mechanism and the empirical results showing that DCRL can beat AC, UAAC, and also recurrent A2C. The novelty therefore seems marginal.\n\nAdditionally, asymmetric actor critic (AC) was proposed to utilize additional state information to learn a policy more efficiently over the standard recurrent A2C where the critic only uses history information. UAAC then corrects the bias in AC by incorporating both the state and the history. However this paper proposes going back to the standard recurrent A2C method for one of the critics to achieve unbiasedness. It would have been interesting to compare two dual critic architectures: 1) V(h) and V(h,s) and 2) V(s) and V(h,s). Although the second approach would be biased, I would assume that using V(s) would lead to more efficiency gains. This seems to be an obvious comparison that was not considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments, which have significantly enhanced the quality of our manuscript. Below, we provide a point-by-point response to your feedback.\n\n> Weakness 1:\n>\n> The primary weakness of this paper is novelty. While I can appreciate that the dual critic architecture reduces variance, the proofs seems fairly obvious. In my opinion, the main novel contribution of this paper then relies on the weighting mechanism and the empirical results showing that DCRL can beat AC, UAAC, and also recurrent A2C. The novelty therefore seems marginal.\n\nThank you for your valuable feedback. In this study, we address the critical issue of high variance resulting from an over-reliance on state information, **a topic that has been largely overlooked but is highly relevant to the POMDP community**. To mitigate this issue, we propose a DCRL framework that harnesses the strengths of two critics. We provide theoretical evidence demonstrating that DCRL mitigates the learning variance while maintaining unbiasedness. Moreover, our DCRL incorporates **a dynamic weighting mechanism**, which can be interpreted as a form of **lower-bound-soft-Q-learning**, distinguishing it from conventional linear weighting methods. This mechanism substantially improves DCRL performance while maintaining **high stability concerning hyperparameters**.\n\nWe conducted additional experiments to validate our claims: 1) We implemented a simplified weighting method that uses fixed ratios to weight the two critics (referred to as the DCRL No Clip Version). 2) We performed ablation experiments on the values of the $\\beta$ parameter ($\\beta \\in \\\\{1/5, 1/3, 1/2, 2/3, 4/5\\\\}$). Figure 1 of the rebuttal PDF illustrates that the DCRL No Clip version shows performance improvements in only some environments compared to UAAC. In contrast, DCRL consistently enhances performance across all tested environments, **demonstrating that the dynamic weighting mechanism of DCRL is crucial. Additionally, the choice of $\\beta$ exhibits notable stability**. We selected $\\beta = 1/2$ as the parameter for reporting our results in the paper, as it consistently exhibits superior performance.\n\n> Weakness 2: \n>\n> Additionally, asymmetric actor critic (AC) was proposed to utilize additional state information to learn a policy more efficiently over the standard recurrent A2C where the critic only uses history information. UAAC then corrects the bias in AC by incorporating both the state and the history. However this paper proposes going back to the standard recurrent A2C method for one of the critics to achieve unbiasedness. It would have been interesting to compare two dual critic architectures: 1) V(h) and V(h,s) and 2) V(s) and V(h,s). Although the second approach would be biased, I would assume that using V(s) would lead to more efficiency gains. This seems to be an obvious comparison that was not considered.\n\nWe greatly appreciate your suggestion. Following your suggestion, we conducted **ablation experiments comparing two different dual architectures**: 1) $V(h)$ and $V(h,s)$ (our DCRL), and 2) $V(s)$ and $V(h,s)$ (referred to as the DCRL State Version). Figure 2 of the rebuttal PDF illustrates that the DCRL State Version demonstrates commendable performance among the various DCRL variants. However, the biased characteristics of the DCRL State Version contribute to instability in its effectiveness. **Notably, our DCRL achieves superior training performance in three out of four environments.** These findings further highlight the significance of our DCRL framework. \n\nWe appreciate your constructive feedback. These questions are very inspiring and discussion-worthy. The quality of the manuscript has been greatly enhanced by incorporating these and other reviewers' valuable suggestions. If you find that these concerns have been resolved, we would appreciate it if you would consider reflecting this in your rating of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mIULjn7yst", "reviewer": "Reviewer_gCno", "summary": "In this work, the authors aim to learn policies in the POMDP setting. They make use of two critics - one that has the privileged state information and one that doesn't and uses only history. Creating a dual value function which is a convex combination of these two value functions. They use this dual value function for policy optimization. They perform experiments in box2d and box3d environments.", "questions": "(1) Do you think in most real world settings you described in the introduction, you will have access to the state information for training the oracle critic?", "limitations": "The authors have discussed their limitation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors use the bias-variance tradeoff well. Using standard critic is biased but using only the oracle critic has high variance. The come up with an estimator with low bias as well as low variance and empirically justify this in their experiments.", "weaknesses": "(1) Even assuming that you have the state information during training is a strong assumption. You might not always have that. A lot of works do not make that assumption and perform pretty well.\n\n(2) There should have been comparisons to commonly used POMDP works like Dreamer [] and others. But the comparisons were made only with (in some words) different variants of the algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive comments and suggestions. We have revised our paper accordingly. Below, we present detailed responses to each point.\n\n> Weakness 1 and Question 1: \n>\n> Even assuming that you have the state information during training is a strong assumption. You might not always have that. A lot of works do not make that assumption and perform pretty well. \n>\n> Do you think in most real world settings you described in the introduction, you will have access to the state information for training the oracle critic?\n\nWe fully understand your concerns. To address your concerns, we included a more detailed explanation of the assumptions regarding state information accessibility and provided a representative **real-world scenario (autonomous driving)** of the POMDP problem.\n\nIncorporating state information during the training process is a common assumption adopted by many previous works. In POMDP problems such as MiniGrid and MiniWorld, researchers often use state information to assist agents during training [UAAC2022AAMAS, Believer2023ICML]. In various partially observable games, including **card games (e.g., Mahjong, DouDiZhu) and MOBA games (e.g., Honor of Kings)**, we can readily obtain the state information (opponents' hands or attributes) during training. Consequently, researchers often leverage this information to enhance agent training, resulting in significant performance improvements [Suphx2020Arxiv, PerfectDou2022NIPS, Honor-of-Kings2020NIPS].\n\nIn autonomous driving applications, due to the high costs of specific sensors and the challenges of acquiring high-definition maps in some areas, existing solutions mainly rely on end-to-end visual models [Tesla, PhiGent]. During the training phase, autonomous driving companies typically equip a small fleet of training vehicles with expensive sensors and **use the collected high-definition maps and other information as state information to facilitate the training of these end-to-end visual models**. This approach aids in constructing 3D environments, thereby enhancing the performance of autonomous driving systems [BEV2023TPAMI].\n\n> Weakness 2:\n>\n> There should have been comparisons to commonly used POMDP works like Dreamer [] and others. But the comparisons were made only with (in some words) different variants of the algorithm.\n\nThank you for this valuable suggestion. We have followed your advice and considered the commonly used POMDP baseline DreamerV3 [DreamerV32023Arxiv].  We compared the performance of DCRL, DreamerV3, and additional baselines across eight tasks in the MiniGrid environment. Figure 3 of the rebuttal PDF illustrates the average scores from five training seeds over $1e5$ steps for each method. **The results indicate that DCRL consistently outperforms DreamerV3 across all eight tasks.** Importantly, DCRL requires fewer computational resources and less training time than DreamerV3.\n\nThank you for your thought-provoking and discussion-worthy suggestions, which have significantly improved the quality of our paper. If you find that these concerns have been resolved, we would appreciate it if you would consider reflecting this in your rating of our paper.\n\n**References**\n\n- [UAAC2022AAMAS] Andrea Baisero, et al. Unbiased asymmetric reinforcement learning under partial observability. In AAMAS, pages 44–52, 2022.\n- [Believer2023ICML] Andrew Wang, et al. Learning belief representations for partially observable deep RL. In ICML, pages 35970–35988, 2023.\n- [Suphx2020Arxiv] Junjie Li, et al. Suphx: Mastering mahjong with deep reinforcement learning. Arxiv, 2020.\n- [PerfectDou2022NIPS] Guan Yang, et al. PerfectDou: Dominating Doudizhu with perfect information distillation. In NIPS, pages 34954–34965, 2022.\n- [Honor-of-Kings2020NIPS] Deheng Ye, et al. Towards playing full MOBA games with deep reinforcement learning. In NIPS, pages 621-632, 2020.\n- [Tesla] Tesla AI Day. [Online]. Available: https://www.youtube.com/watch?v=j0z4FweCy4M\n- [PhiGent] PhiGent: Technical Roadmap. [Online]. Available: https://43.132.128.84/coreTechnology\n- [BEV2023TPAMI] Hongyang Li, et al. Delving into the devils of bird's-eye-view perception: A review, evaluation and recipe. IEEE TPAMI, 2023.\n- [DreamerV32023Arxiv] Hafner Danijar, et al. Mastering diverse domains through world models. Arxiv, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wo0OfRfiTZ", "reviewer": "Reviewer_3R7k", "summary": "The paper presents Dual Critic Reinforcement Learning (DCRL), a framework designed to handle partial observability in RL. Traditional RL methods often struggle with high variance and instability when relying on full-state information. DCRL addresses this by integrating two critics: an oracle critic with access to complete state information and a standard critic operating within a partially observable context. This dual approach aims to improve efficiency and reduce variance during training, leading to optimized performance in online environments.", "questions": "- There is a typo in the definition of $R(h,a)$ in line 122.\n- Can you explain in more detail how the weighting mechanism between the oracle critic and the standard critic is designed to reduce variance? Are there specific conditions or thresholds for adjusting the weights?\n- Is it necessary to ensure the alignment of $V(s)$ and $V(h,s)$ in DCRL? if yes, how to ensure it?", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- This paper is well structured and easy to read. The authors provide thorough analysis and experiment results.\n- The paper provides theoretical proof that DCRL reduces learning variance while maintaining unbiasedness. The dual value function is defined as a weighted combination of the oracle and standard value functions, and it is proven to be an unbiased estimate of the true value function with lower variance compared to using the oracle critic alone.\n- The effectiveness of DCRL is validated through extensive experiments in the Box2D and Box3D environments. Results show that DCRL significantly outperforms baseline methods, including Recurrent Actor-Critic and Asymmetric Actor-Critic frameworks, across various tasks in the MiniGrid and MiniWorld environments. DCRL achieves faster convergence and higher returns, particularly in complex and high-uncertainty scenarios.", "weaknesses": "Refer to Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging words and constructive feedback. We appreciate your time reviewing our paper and provide point-by-point responses to your comments below.\n\n> Question 1:\n>\n> There is a typo in the definition of $R(h,a)$ in line 122.\n\nWe greatly appreciate your pointing out the issues. We have corrected the typos in the revised version.\n\n> Question 2:\n>\n> Can you explain in more detail how the weighting mechanism between the oracle critic and the standard critic is designed to reduce variance? Are there specific conditions or thresholds for adjusting the weights?\n\nTo address the high variance issue arising from an over-reliance on state information, we propose a weighting mechanism that leverages the benefits of two critics. As demonstrated in Theorem 2, the no-variance standard critic can effectively mitigate the variance introduced by the oracle critic, thereby reducing overall variance.\n\n**The weight adjustment is governed by the advantage function of the standard critic.** When the advantage function yields a non-positive value, $\\beta$ is clipped to $0$. Conversely, when the advantage is positive, $\\beta$ activates the standard critic to accelerate training and reduce variance. Theorem 5 establishes that this weighting mechanism is a form of lower-bound-soft-Q-learning. This approach significantly enhances the performance of DCRL while maintaining high stability concerning hyperparameters.\n\n> Question 3:\n>\n> Is it necessary to ensure the alignment of $V(s)$ and $V(h,s)$ in DCRL? if yes, how to ensure it?\n\nThank you for this question. The answer is no. It is unnecessary to ensure the alignment of $V(s)$ and $V(h,s)$. In the POMDP setting, the policy operates based on partial observations, resulting in a standard policy gradient derived from $V(h)$. Maintaining the policy gradient's unbiasedness is crucial when incorporating state information to enhance training. **This stipulates that the value function must be unbiased concerning $V(h)$ without considering its relationship with $V(s)$.** Our DCRL framework emphasizes two critics: $V(h)$ and $V(h,s)$. It has been established that $V(h,s)$ is unbiased about $V(h)$. Furthermore, Theorem 1 demonstrates that $V_{dual}(h,s)$ in the DCRL framework is also unbiased.\n\nThank you very much for recognizing our work! Your time and effort are greatly appreciated!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TxuZEJ52Rj", "reviewer": "Reviewer_VBT3", "summary": "The authors propose a methood that uses a weighted dual critic structure for tackling POMDPs. The dual critic structure consists of a critic that receives global state information while the other critic receives only the partially observations of the state. The authors provides some simple yet concrete analytical results that show the method induces unbiased critic estimates and can reduce variance compared to that of the Unbiased Asymmetric Actor-Critic.", "questions": "Q1. The authors state that $V^\\pi(h)$ has nonzero variance. Using the definition of the dual value function, setting $\\beta\\equiv 1$ implies that $V_{\\rm dual}^\\pi(h,s)=V^\\pi(h)$. However, setting $\\beta=1$ in equation 8 implies that $Var_{s|h}[V_{\\rm dual}^\\pi(h,s)]=0$. Can the authors explain this.\n\nQ2. How does the method perform over a range of values of $\\beta$?\n\nQ3. What environments should we expect the method to do well in (and not so well)?", "limitations": "L1.The experiment section does not contain an ablation on the value of $\\beta$. Without this it is difficult to understand the size of the range of values of $\\beta$ for which the method performs well.\n\n\nL2. The experiment environments are quite simple - it would be useful to see how the method performs on complex benchmark environments.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well written and easy to follow. The motivation for their approach is well laid out and although the focus of the paper is to provide a new methodology which is supported by experiments, the paper includes some lite-touch analytics that support the overall ideas well.\n\nThe idea is quite simple and intutive, the presentation is nevertheless very clean with the authors having laid out both the rationale and benefits for the method in a very clear way.", "weaknesses": "1. While I understand the reason for doing so, such a strategy introduces a harsh jump-discontinuity in the dual advantage and critic - this may lead to numerical instabilities. The paper would benefit from a discussion on this issue.\n\n2. The method also seems related to centralised training and decentralised execution in multi-agent reinforcement learning. There, the critic is trained with global information while the actor is trained using only local inputs. It would be a nice connection to make if the authors could include some discussion on degenerate case of ct-de with N=1.\n\n3. Notwithstanding the discussions on why the method performs well in the chosen environments (lines 297-314) I would have liked to have seen some discussion and analysis of when one could expect this method to be most useful. Indeed, it's conceivable that including more information may slow training even if one can expect improved asymptotic performance.  There may also be situations where we do not have access to the full global state even during training.  I think the paper would benefit from such discussions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions. Below, we provide detailed responses to each point.\n\n> Weakness 1:\n>\n> ... harsh jump-discontinuity ... numerical instabilities.\n\nWe fully understand your concerns. **The numerical instability arises primarily from the variance of $V(h, s)$ due to the expectation of $V(h, s)$ is equal to $V(h)$.** Figure 8 in the appendix illustrates that our DCRL demonstrates a more stable value distribution and experiences fewer numerical instabilities than UAAC, attributable to its reduced variance.\n\n> Weakness 2:\n>\n> ... degenerate case of ct-de with N=1.\n\nWe appreciate the valuable insights you have provided. The degenerate case of CTDE when $N=1$ corresponds to a standard POMDP. In this context, the well-known multi-agent actor-critic algorithm COMA, utilizes both state information ($s$) and history ($h$) as inputs for the critic during centralized training. Conversely, during execution, the actor relies solely on history as its input. **This situation can be interpreted as a specific instance of the UAAC framework.** We have incorporated relevant discussions on this topic in the revised manuscript.\n\n> Weakness 3.1 and Question 3:\n>\n> ... when one could expect this method to be most useful. \n>\n> What environments should we expect the method to do well in (and not so well)?\n\n**Incorporating state information can offer significant advantages for training, as it often encapsulates high-level features that improve the learning of $V(h,s)$ compared to $V(h)$.** For example, in an extreme scenario where the state information $s$ represents the true value of $V(h)$, the agent can effortlessly infer this true value through $V(h,s)$. Specifically, in the context of the Empty Navigation task, when the state information indicates the agent's relative distance to the goal, the value function learning process is significantly streamlined. Furthermore, the reward function is defined based on the state rather than the observation. Consequently, the standard critic's learning necessitates implicit consideration of the distribution $b(s|h)$. In contrast, **the oracle critic bypasses this requirement by directly learning the reward**, as the distribution is explicitly provided through the sampling operation during the policy gradient calculation.\n\n**However, state information does not always enhance training.** When the state information contains significant noise, **introducing excessive noisy features to the input can hinder learning the value function**. In unfamiliar environments, whether state information genuinely supports the learning process remains to be determined.\n\nThe motivation behind this work is that state information may be accessible during training in many scenarios. However, introducing a state could potentially lead to bias or variance. **In most cases, state information offers higher-level features that assist agents in decision-making. In these situations, DCRL enhances the process by further reducing variance. Conversely, when state information and partial observations correspond one-to-one, the oracle critic demonstrates no variance; DCRL may become ineffective under these circumstances.**\n\n> Weakness 3.2\n>\n> ... more information may slow training ... do not have access to the full global state ...\n\nThe scope of our study encompasses situations where state information is accessible during training. This assumption is prevalent in previous works and reflects real-life common scenarios.\n\nWe provide the runtime in seconds for each method below regarding the potential issue of increased training duration due to additional information. DCRL requires more time to execute compared to other baselines, primarily because it trains an additional critic network. Nevertheless, **DCRL demonstrates significantly higher sample efficiency and performs better with fewer environment steps, thus compensating for the slower wall time.**\n\nTable 1: Comparison of DCRL and other baselines in MiniGrid. The first number represents the mean ± std of performance, while the second number indicates the runtime in seconds.\n\n|                                  |   A2C ($1e5$ frames)   |   A2C ($1e6$ frames)    |   PPO ($1e5$ frames)   |   PPO ($1e6$ frames)    |\n| :------------------------------: | :--------------------: | :---------------------: | :--------------------: | :---------------------: |\n|      Recurrent Actor-Critic      |   0.32±0.13, 108.41s   |   0.28±0.09, 1026.31s   |   0.26±0.07, 111.90s   |   0.52±0.03, 1130.45s   |\n|     Asymmetric Actor-Critic      |   0.26±0.10, 106.49s   |   0.32±0.14, 1000.69s   |   0.27±0.08, 110.21s   |   0.60±0.06, 1107.77s   |\n| Unbiased Asymmetric Actor-Critic |   0.31±0.08, 108.89s   |   0.39±0.09, 1066.49s   |   0.28±0.10, 114.35s   |   0.59±0.08, 1160.74s   |\n|         **DCRL (Ours)**          | **0.63±0.07, 130.12s** | **0.89±0.01, 1224.29s** | **0.47±0.08, 132.70s** | **0.84±0.03, 1331.69s** |\n\n> Question 1:\n>\n> ... $V^{\\pi}(h)$ has nonzero variance.\n\nWe apologize for the misunderstanding caused by a typo in our paper. In line 155, we clarify that $V(h,s)$ possesses nonzero variance, while $V(h)$ has zero variance. \n\n> Question 2:\n>\n> How does the method perform over a range of values of $\\beta$?\n\nTo address your concerns, we conducted ablation experiments on the values of the $\\beta$ parameter ($\\beta \\in \\\\{1/5, 1/3, 1/2, 2/3, 4/5\\\\}$). Figure 1 of the rebuttal PDF illustrates that **the choice of $\\beta$ exhibits notable stability**. This stability primarily arises from the characteristics of lower-bound-soft-Q-learning. Consequently, we selected $\\beta = 1/2$ as the parameter for reporting our results in the paper, as it consistently exhibits superior performance.\n\nThank you for your thoughtful review and insights into our work, which have significantly improved the quality of our paper. If you find that these concerns have been resolved, we would appreciate it if you would reconsider your ratings of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l5muGpAH2z", "reviewer": "Reviewer_RNGm", "summary": "This paper proposes a dual-critic architecture for the asymmetric RL regime where a policy deployable in partially observable settings is trained under full observability. The proposal is meant to be an improved version of Unbiased Asymmetric Actor Critic that, through its two critics, improves learning efficiency while reducing variance. This is demonstrated through theory and experiments in MiniGrid and MiniWorld.", "questions": "1. Do you know how the dual critic architecture affects the variance of the policy gradient estimator? \n2. How was the clipping method for $\\beta$ chosen?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Originality**: The authors provide a new algorithm in the context of asymmetric RL. The contribution over related works is accurate and clearly stated. \n\n**Quality**: The theory supporting the work appears sound. The experiments seem solid and are conducted across a number of partially observable domains, including both discrete gridworlds and continuous control environments. The results show that the method outperforms existing methods in terms of sample efficiency, sometimes by a substantial margin. \n\n**Clarity**: The writing is mainly clear, particularly the motivation and the exposition of related works. A few technical details can be clarified however. \n\n**Significance**: Deep RL for POMDP problems remains a significant challenge and new algorithms can have a substantial impact. Asymmetric RL has emerged as a promising paradigm for such problems and so I believe the work targets an important research topic.", "weaknesses": "**Originality**: Ultimately, the proposed method is somewhat derivative of two other well-known methods. The main contribution is the dual critic architecture, which is a linear combination of a typical recurrent critic $V(h)$ and the unbiased asymmetric critic $V(h,s)$. On its own, this would not be a major issue since the method is shown to perform well, but I believe the insights behind this approach are also discussed in the original Unbiased Asymmetric Actor Critic paper. \n\n**Quality**: While the theorems appears correct, I wonder how relevant they are to the topic at hand. To my knowledge, what we *really* care about is the variance of the policy gradient Monte-Carlo estimator --- not the value estimator. The former is important since it plays a big role in the sample efficiency of policy learning. While the variance of the value estimator may indicate the amount of data required to train a reasonable value function, this is again only important insofar as reducing policy gradient variance through its role as a baseline.\n\nThere also needs to be a discussion on *why* we don't always want to reduce variance all the way to zero by setting $\\beta = 1$ (which is equivalent to the simple recurrent critic). What value does increasing the variance (in order to incorporate state information) have? I believe this is not totally well understood in the literature either, but the topic should at least be brought up to clarify the point above. An investigation that can properly answer this question would be a strong contribution. \n\nRegarding the implementation, it seems the choice of when to clip to $\\beta = 0$ is very important but I don't understand why the particular heuristic in the paper was chosen. I'm not sure why the motivation of preventing large updates when the advantage is non-positive is relevant. \n\nThe results of the experiments show a solid advantage but it's not totally clear where the benefits are coming from (likely because I didn't understand the motivation behind the clipping of $\\beta$). I think the experiments would also benefit from stronger baselines such as the Believer algorithm cited in the related work, and partially observable tasks that involve more than simple navigation. \n\n**Clarity**: As previously mentioned, the reason why the state-based critic $V(h,s)$ is beneficial could be better explained, along with the heuristic for when to clip $\\beta = 0$. Perhaps a toy example could be useful for this. I'm also unsure what the significance of the interchange method is. \n\n**Significance**: I believe there isn't enough insight into the proposed approach to justify its use compared to Recurrent PPO or Unbiased Asymmetric Actor-Critic, which are somewhat better understood.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We provide the following clarifications in response to your comments. \n\n> Weakness 1: About the insight.\n>\n> ... the proposed method is somewhat derivative of two other well-known methods. ... there isn't enough insight ...\n\n**In this study, we address the issue of high variance stemming from an over-reliance on state information. We demonstrate that state information is not always advantageous for training purposes.** In many environments, relying solely on partial observations can yield superior results compared to using state information. \n\n**Incorporating state information can offer significant advantages for training, as it often encapsulates high-level features that improve the learning of $V(h,s)$ compared to $V(h)$.** For example, in an extreme scenario where the state information $s$ represents the true value of $V(h)$, the agent can effortlessly infer this true value through $V(h,s)$. Specifically, in the Empty Navigation task, when state information indicates the agent's relative distance to the goal, the value function learning process is significantly streamlined. In addition, the reward function is defined based on the state rather than the observation. Consequently, learning for the standard critic necessitates implicit consideration of the distribution $b(s|h)$. In contrast, **the oracle critic bypasses this requirement by directly learning the reward**, as the distribution is explicitly provided through the sampling operation during the policy gradient calculation.\n\n**However, state information is not always advantageous for training.** When the state information contains significant noise, **adding excessive noisy features to the input can hinder learning the value function.** In unfamiliar environments, whether state information genuinely supports the learning process remains to be determined.\n\nTo address this issue, we introduce a DCRL framework that leverages the strengths of two critics. **We provide theoretical evidence indicating that DCRL mitigates the learning variance while maintaining unbiasedness. Moreover, our approach incorporates a dynamic weighting mechanism, which can be interpreted as a form of lower-bound-soft-Q-learning, distinguishing it from conventional linear weighting methods.** Compared to Recurrent AC and UAAC methods, our proposed DCRL demonstrates substantial improvements.\n\n> Weakness 2 and Question 1: About the variance reduction in policy gradient.\n>\n> ... the variance of the policy gradient Monte-Carlo estimator ... how the dual critic architecture affects the variance of the policy gradient estimator?\n\nWe fully understand your concerns regarding the variance of the policy gradient Monte Carlo estimator. Traditional Actor-Critic methods employ the reward-to-go term to compute the policy gradient. Nonetheless, **the variance of the reward-to-go tends to be high**, which has led researchers to substitute this term with the Q-value to mitigate variance in the standard policy gradient method, i.e., $Q(h_t, a_t) \\nabla_{\\theta}\\log\\pi_{\\theta}(a_{t}|h_{t})$. Similarly, in our study, we focus on the policy $\\pi(a|h)$, which is linked to **a unique $Q(h, a)$ but can also correspond to multiple $Q(h, s, a)$ values** due to the non-one-to-one relationship between state and history. **Consequently, the variance of the policy gradient derived from $Q(h, a)$ is lower than that obtained from $Q(h, s, a)$.** As stated in Theorem 1 of our paper, $Q_{\\mathrm{dual}}(h, s, a)$ in DCRL achieves a reduction in this variance compared to $Q(h, s, a)$ by integrating the standard critic, thereby decreasing the variance of the policy gradient.\n\n> Weakness 3 and Question 2: About the $\\beta$.\n>\n> ... reduce variance all the way to zero by setting $\\beta=1$ ... when to clip $\\beta=0$ ... the motivation behind the clipping of $\\beta$ ... How was the clipping method for $\\beta$ chosen? \n\nIn response to weakness 1, we elaborated on the advantages of employing the oracle critic $V(h, s)$, which incorporates state information for policy training. \n\nThe motivation behind the clipping mechanism is to leverage the standard critic when opportunities for policy enhancement remain following updates from the oracle critic. **This approach can be interpreted as a form of lower-bound soft Q-learning, wherein only samples exhibiting a positive advantage contribute meaningful insights into the optimal soft Q-value.** To validate this concept, we implemented a simplified weighting method that employs fixed ratios to weight the two critics (referred to as DCRL No Clip Version). As illustrated in Figure 1 of the rebuttal PDF, this method shows performance improvements in only certain environments, confirming that the dynamic weighting mechanism of DCRL is crucial. \n\nTo evaluate the impact of the parameter $\\beta$, we conducted ablation experiments testing various values of $\\beta \\in \\\\{1/5, 1/3, 1/2, 2/3, 4/5\\\\}$. **The choice of $\\beta$ demonstrates notable stability.** **This stability is primarily attributable to the characteristics inherent in lower-bound soft Q-learning.**\n\n> Weakness 4:\n>\n> ... the experiments would also benefit from stronger baselines...\n\nThank you for this valuable suggestion. We have followed your advice and considered the commonly used POMDP baseline DreamerV3.  We compared the performance of DCRL, DreamerV3, and additional baselines across eight tasks in the MiniGrid environment. Figure 3 of the rebuttal PDF illustrates the average scores from five training seeds over $1e5$ steps for each method. **The results indicate that DCRL consistently outperforms DreamerV3 across all eight tasks**. Importantly, DCRL requires fewer computational resources and less training time than DreamerV3.\n\nThank you for your insightful and thoughtful reviews, which have significantly improved the quality of our paper. If you find that these concerns have been resolved, we would appreciate it if you would reconsider your ratings of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the reviewers for their insightful and constructive feedback. In response to these helpful comments, we conducted supplementary experiments in the MiniGrid environment (see the attached PDF):

1. **Figure 1**: Ablation studies on different values of $\beta \in \\{1/5, 1/3, 1/2, 2/3, 4/5\\}$ and a simple weighting method (referred to as DCRL No Clip Version) with fixed ratios to weight the two critics.
2. **Figure 2**: Comparison of two different dual architectures: 1) $V(h)$ and $V(h,s)$ (our DCRL), and 2) $V(s)$ and $V(h,s)$ (referred to as DCRL State Version).
3. **Figure 3**: Comparison with the commonly used POMDP baseline, DreamerV3.

These experiments further demonstrate the following:

1. The dynamic weighting mechanism of DCRL is crucial, and the choice of $\beta$ exhibits notable stability.
2. Our DCRL framework outperforms the DCRL State Version due to its unbiased characteristic, which contrasts with the biased nature of the DCRL State Version.
3. DCRL consistently surpasses DreamerV3 across all tasks, confirming its superior performance.

We hope these additional experimental results provide further evidence of the effectiveness of our DCRL framework.

Moreover, we made several revisions and improvements to our paper:

1. We included a more detailed explanation of the assumptions regarding state information accessibility in the introduction section.
2. We addressed the degenerate case of CTDE when $N=1$ in the related work section.
3. We added additional toy examples in the appendix to illustrate when DCRL is effective and when it may fail.

Furthermore, we addressed each reviewer's questions or concerns with detailed point-by-point responses. As a result, the quality of our paper has markedly improved.

We are profoundly grateful for the reviewers' contributions to our paper and sincerely hope that our revisions have adequately addressed your concerns. We remain open to further feedback and are committed to implementing additional improvements if necessary.

Meta-review: The paper proposes a novel partially observable RL method for settings when the state is available during training. The approach combines a history-state critic (which has been shown to perform well) with a history critic using a dynamic weighting scheme to reduce variance and improve performance. The paper theoretically shows that variance can be reduced and the method remains unbiased. The approach works well in a number of domains using A2C and PPO as base methods. 

Asymmetric critics that incorporate privileged information in training are becoming common in partially observable RL (RL for POMDPs). While the current state-of-the-art methods typically use history-state, the variance can be high. The proposed approach is relatively simple but well-motivated, simple to implement, and performs well. 

The author response was helpful for better clarifying the method and showing the benefits of the method, including the use of dynamic weighting and against other baselines. These results and discussions should be included in the final paper.
Decision: Accept (poster)