Dual Critic Reinforcement Learning under Partial Observability

Authors: jinqiu  li
enmin  zhao
tong  wei
junliang  xing
shiming  xiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:38:37

Abstract: Partial observability in environments poses significant challenges that impede the formation of effective policies in reinforcement learning. Prior research has shown that borrowing the complete state information can enhance sample efficiency. This strategy, however, frequently encounters unstable learning with high variance in practical applications due to the over-reliance on complete information. This paper introduces DCRL, a Dual Critic Reinforcement Learning framework designed to adaptively harness full-state information during training to reduce variance for optimized online performance. In particular, DCRL incorporates two distinct critics: an oracle critic with access to complete state information and a standard critic functioning within the partially observable context. It innovates a synergistic strategy to meld the strengths of the oracle critic for efficiency improvement and the standard critic for variance reduction, featuring a novel mechanism for seamless transition and weighting between them. We theoretically prove that DCRL mitigates the learning variance while maintaining unbiasedness. Extensive experimental analyses across the Box2D and Box3D environments have verified DCRL's superior performance. The source code is available in the supplementary.

Main paper: 
 introduction: 
 State ObservationReal-world issues commonly involve a degree of partial observability, where agents lack access to the full system state and must base decisions on partial observations. These problems are typically defined as partially observable Markov decision processes (POMDPs) [1] . Despite the progress in reinforcement learning (RL) for addressing complex tasks [2] [3] [4] [5] [6] , learning effective strategies in scenarios with partial observations remains challenging.Traditional RL methods in POMDP contexts often encounter impractical computations in processing the complete history, leading to suboptimal performance. Luckily, full states are accessible during training in many training scenarios. For example, in applications like autonomous driving [7] or robot control tasks [8] , where computational and time limitations exist, only low-cost RGB images are available for deployment. At the same time, high-precision sensors can record more accurate and detailed data during training. Similarly, in card games [9] , the private hands of other players are typically hidden during deployment but revealed during training. This distinction of separating available information during training and deployment stages has driven the adoption of an offline learning and online execution paradigm in POMDP settings [10] [11] [12] [13] .This study focuses on the asymmetric actor-critic framework [8, 14, 15] , where the actor and critic rely on different inputs. The critic, serving as an auxiliary tool for training, can be transformed into an oracle critic to align with the offline learning and online execution paradigm. Specifically, the oracle critic is empowered to access the complete states while the actor remains constrained to partial observations. Using an oracle critic conditioned on the state and observation history promises to enhance performance and theoretically achieve unbiasedness [10] . However, the practical performance of such approaches raises doubts, particularly in domains characterized by high uncertainty and extensive state spaces. For instance, in the illustrated first-person 3D navigation task depicted in Figure 1 , the agent is tasked with reaching a dynamic target that varies across episodes within as few steps as possible. This task involves a first-person view observation and an aerial view state.Introducing an oracle critic incorporating state information in this complex environment may increase variance, potentially undermining learning efficiency and overall performance.To tackle the challenges mentioned above, we argue that the indiscriminate utilization of state information is inadvisable. We introduce a Dual Critic Reinforcement Learning framework (DCRL). DCRL consists of two critics: the oracle critic conditioned on the <history, state> tuple, and the standard critic conditioned solely on history. It introduces a synergistic strategy to meld the strengths of the oracle critic for efficiency improvement and the standard critic for variance reduction. In summary, the main contributions of this work are as follows:• We propose a DCRL framework to address partially observable tasks. This framework features a novel mechanism for seamless transition and weighting between the dual critics.• We theoretically prove that DCRL mitigates learning variance while maintaining unbiasedness.• The DCRL framework is simple to implement. While our emphasis lies on the Advantage Actor-Critic (A2C) [16] , DCRL can also be integrated into other actor-critic architectures, such as Proximal Policy Optimization (PPO) [17] .We extensively validate the DCRL framework based on A2C and PPO. Experimental results conducted on both Box2D and Box3D environments confirm its effectiveness. The source code is available in the supplementary material.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We provide a detailed introduction to the proposed method and provide relevant theoretical proof in Section 4. In Section 5, we present experimental results to demonstrate the effectiveness of the proposed method.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Most methods can be adapted for partial observability by training memory-based policies to condition past observations. Recurrent Neural Networks (RNNs) [18] are commonly used for their simplicity and strong performance. Deep Recurrent Q-Networks (DRQN) [19] integrates Long Short-Term Memory (LSTM) [20] cells into the Deep Q-Network (DQN) [2] architecture, which demonstrated promising results. In addition to LSTM, other memory mechanisms such as Gated Recurrent Units (GRUs) [21, 22] , Transformers [23, 24] , and Language Models [25, 26] have been adopted in RL algorithms for memory learning. However, POMDPs are hard to solve when relying solely on memory-based policies because the size of the history grows linearly with the horizon length.It is crucial that in many environments, complete states are observable during the training phase. Utilizing full observable states during training has emerged as a popular paradigm in RL. We discuss prior works on the utilization of full observability in RL algorithms. Additionally, we provide further discussion about training the world model, applications in multi-agent reinforcement learning, and other double structures in Appendix B.Using complete states to train the critic. This topic is closely related to our paper and has been applied in many large-scale scenarios, such as DouDizhu [9] and Mahjong [27] . The critic in the actor-critic framework is a training tool not required during deployment, making its structure naturally suitable to exploit the state. This is achieved by critic asymmetry, where the actor and critic receive history and state, respectively. Asymmetric Actor-Critic [8] uses the critic conditioned on groundtruth states to address image-based robot control tasks. AACC [15] employs the critic conditioned on both states and observations to solve contextual Markov decision processes. Theoretically, using the state as the critic input is biased, whereas using the <history, state> tuple is unbiased in expectation [10] . However, it introduces high variance in practice.Using complete states to train the policy. These methods typically involve training a teacher policy using complete state information and then utilizing this policy to guide the learning of a student policy under partial observability. The forms of guidance vary, such as maintaining consistency of the Qfunction [28] , the encoder [11] , action-distributions [29] , and others. In addition to introducing extra loss terms to enforce consistency between the student policy and the teacher policy, the transition from the teacher policy to the student policy can also be facilitated through dropout of state information from the input [30] [31] [32] . Common issues include the imitation gap and suboptimal teacher, which can limit the potential capabilities of the student policy.Using complete states to learn the belief representation. Belief state methods [1, 33] are classical in POMDPs. These methods often necessitate knowledge of the dynamics and reward functions, thus rendering them more applicable to scenarios with well-defined system dynamics. Consequently, they are typically constrained to simpler settings characterized by small state spaces, where the complexities of the environment are more tractable. Several recent studies [34] [35] [36] focus on learning belief representations by predicting future observations and rewards. VRM [34] utilizes a latent stochastic variable to model sequential observations. However, learning latent states that can capture long-term dependencies is challenging. Incorporating the complete states can help alleviate this issue. DouZero+ [37] introduces auxiliary tasks for predicting states while concurrently training policies conditioned on observations and the predicted states. Believer [13] decouples belief state modeling from policy optimization. They first capture state features through representation learning and then train the policy conditioned on observations and the predicted states.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 This section introduces the background topics required to understand our work: partially observable Markov decision processes, the recurrent Advantage Actor-Critic algorithm, and the Unbiased Asymmetric Actor-Critic algorithm.A partially observable Markov decision process (POMDP) [1] is formally identified as a 7-tuple, namely P = S, A, T, Ω, R, O, γ , where S, A, and Ω represent the state space, action space, and observation space, respectively. The transition function T : S × A → ∆S is denoted as T (s, a, s ) = p(s |s, a), describing the probability of transitioning from state s to state s after taking action a. Advantage Actor-Critic (A2C) [16] is primarily designed for fully observable control problems. It can be easily adapted to handle partially observable control problems by replacing all occurrences of the state s with the respective history h. To clearly distinguish between different methods, we denote this method as AC. It trains an actor model π : H → ∆A parameterized by θ, and a critic model V : H → R parameterized by φ. The policy loss is defined as:EQUATIONwhere Q π (h t , a t ) denotes the history-action value of the policy π. In practice, Q π (h t , a t ) is replaced by the n-step bootstrapped advantage to reduce variance:EQUATIONCorrespondingly, the policy loss isL AC policy (θ) = -E ∞ t=0 δ φ (h t , a t ) log π θ (a t |h t ). In this work, the value loss is defined asL AC value (θ) = E ∞ t=0 δ φ (h t , a t ) 2 . The entropy loss is L entropy (θ) = -E ∞ t=0a∈A π θ (a t |h t ) log π θ (a t |h t ) . The total loss is L AC value (φ) + L AC policy (θ) + ηL entropy (θ), where η is a weight for entropy regularization.Unbiased Asymmetric Actor-Critic [10] is a variant of recurrent A2C that can exploit state information during training. It considers an oracle critic model V : H × S → R parameterized by ψ instead of the standard critic model. To clearly distinguish between different methods, we denote this method as UAAC. The n-step bootstrapped advantage for UAAC is defined as:EQUATIONCorrespondingly, the policy model is trained to minimize the following objective function:EQUATIONThe value loss is defined asL UAAC value (ψ) = E ∞ t=0 δ ψ (h t , s t , a t ) 2 .Compared with the standard actor-critic method, the state s provides additional context to determine the agent's true underlying situation, which may facilitate the learning of the oracle critic. Importantly, it has been proven that V π (h, s) provides Monte Carlo estimates of the respective V π (h) for arbitrary policies, which is theoretically unbiased, i.e.,V π (h) = E s|h [V π (h, s)],(5)EQUATIONTherefore, the equality ∇ θ L AC policy (θ) = ∇ θ L UAAC policy (θ) holds, indicating that the policy gradient remains unchanged in expectation.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: We provide experimental details in Appendix E.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
dual critic reinforcement learning: 
 Critic models that learn accurate values slowly often become a bottleneck for policy performance in many scenarios. When state information is incorporated, it may be easier to learn V π (h, s) with a neural network compared with learning V π (h), leading to faster policy learning. However, V π (h, s) has nonzero variance. Taking into account the variance arising from V π (h, s): 2 , with a practical batch size that is not too large, the variance tends to be high, especially in high-uncertainty belief states and large state spaces. In games like DouDizhu, when the initial information collection does not accurately infer the opponents' hands, training with V π (h, s) incorporating complete state information increases variance, thus limiting model generalization. Conversely, using V π (h), which relies solely on observations, enhances robustness in such cases. This concept is akin to replacing the reward-to-go term with the Q-value to reduce variance in the vanilla policy gradient method, as discussed in [38, 39] .Var s|h [V π (h, s)] = E s|h V π (h, s) -V π (h)Definition 1. Define the dual value function asV π dual (h, s) = (1 -β)V π (h, s) + βV π (h), where β is a weight that balances the contributions of the V π (h, s) and V π (h), and β ∈ [0, 1]. Correspondingly, the dual Q function is defined asQ π dual (h, s, a) = (1 -β)Q π (h, s, a) + βQ π (h, a).Theorem 1. For arbitrary control problems and policies, V π dual (h, s) is an unbiased estimate of V π (h) and Q π dual (h, s, a) is an unbiased estimate of Q π (h, a).Proof. Following from Equation ( 5), we have:EQUATIONSimilarly, as inferred from Equation ( 6), E s|h [Q π dual (h, s, a)] = Q π (h, a) by replacing the V with Q using the same approach outlined above.Therefore, the variance of the dual value is defined as 2 , and that of Unbiased Asymmetric Actor-Critic is defined asVar s|h [V π dual (h, s)] = E s|h V π dual (h, s) - V π (h)Var s|h [V π (h, s)] = E s|h V π (h, s) -V π (h) 2 .Theorem 2. The variance of the dual value is smaller than that of the Unbiased Asymmetric Actor-Critic, i.e., Vars|h [V π dual (h, s)] ≤ Var s|h [V π (h, s)].Proof. According to the definitions and β ∈ [0, 1], it holds thatEQUATIONSimilarly,Var s|h [Q π dual (h, s, a)] ≤ Var s|h [Q π (h, s, a)] by replacing the V with Q using the same approach outlined above.Considering the unbiasedness demonstrated above, we contemplate employing the dual Q function for policy updates. The policy gradient based on the dual Q function can be formulated as:EQUATIONThis formulation maintains the policy gradient unchanged in expectation. Theorem 3. The dual Q function maintains the policy gradient unchanged in expectation.Proof. Following Theorem 1, therefore,EQUATIONSimilarly, the dual Q function decreases the variance of the policy gradient according to Theorem 2.Policy Gradient Flow Building upon the abovementioned theory, we incorporate the dual Q function to balance variance and efficiency. This integration introduces the Dual Critic Reinforcement Learning framework (DCRL). As shown in Figure 2 , the DCRL consists of three main components: 1) a standard critic network V (h) parameterized by φ, 2) an oracle critic network V (h, s) parameterized by ψ, and 3) an actor model π(a|h) parameterized by θ. Inspired by the Equation ( 9) and Theorem 3, we propose the policy and value losses of DCRL as the following objective functions:EQUATIONEQUATIONwhere δ φ (h t , a t ) and δ ψ (h t , s t , a t ) follow the definitions from Equation (2) and Equation (3). The dual advantage is formulated asδ dual (h t , s t , a t ) = (1 -β)δ ψ (h t , s t , a t ) + βδ φ (h t , a t ).We propose a novel weighting mechanism to synergistically meld the two critics and thoroughly exploit the complete state information. When the received return does not exceed the agent's value estimate, the advantage is non-positive (δ φ (h t , a t ) ≤ 0), β is clipped to 0. In cases with δ φ (h t , a t ) > 0, β takes effect. β is a parameter associated with h but not with s. Hence, this mechanism maintains the theory mentioned above. This mechanism helps stabilize training and prevent large updates in cases where the advantage is non-positive. Similar to Self-Imitation Learning [40] , this clipping mechanism can be viewed as a form of lower-bound-soft-Q-learning, where only samples with a positive advantage provide valuable insights into the optimal soft Q-value. From an empirical perspective, this approach embodies the notion that if there is still room for policy enhancement after being updated by the oracle critic V (h, s), the standard critic V (h) should be engaged. With full-state information from the oracle critic, DCRL avoids falling into a vicious cycle of poor policy and bad value estimates. The standard critic in DCRL can effectively mitigate variance, whereas recurrent Actor-Critic methods preclude such opportunities due to entanglement in a vicious cycle. These two critics establish a mutually supportive relationship, facilitating variance reduction while effectively using full-state information. The proof regarding lower-bound-soft-Q-learning is provided in Appendix C.In addition to the mechanism mentioned above, there is another natural way to leverage the dual-critic framework by interchanging the roles of the oracle and standard critics. Specifically, we shift the weighting factor to the forefront of the δ ψ (h t , s t , a t ) term which denote as β , and the inside term of the policy loss is defined as(1 -β )δ φ (h t , a t )logπ θ (a t |h t ) + β δ ψ (h t , s t , a t )logπ θ (a t |h t ).β is clipped to 0 when δ ψ (h t , s t , a t ) ≤ 0 and otherwise takes a positive value. β is a parameter associated with both h and s. Unlike DCRL, this alternative technique lacks the characteristic of unbiasedness and fails to thoroughly exploit state information while deviating from the lower-boundsoft-Q-learning framework. As a result, its performance exhibits instability compared with DCRL.The experimental section provides a detailed discussion of this technique's performance. 
experiments: 
 We conduct a series of experiments to evaluate the effectiveness of DCRL in partially observable domains. Our research focuses on the model-free setting where complete states are available during training. We compare our DCRL approach with several baselines.Baselines. Recurrent Actor-Critic is a recurrent version of the standard actor-critic algorithm, namely A2C [16] or PPO [17] . This baseline represents methods that do not utilize state information. Asymmetric Actor-Critic [8] is a representation method based on critic asymmetry. The critic uses the state as input, independent of the history. Oracle Guiding [30] is a representation method based on teacher policy. This approach gradually removes state information from the policy input, transitioning the teacher policy to a student policy under partial observability. The dropout probability linearly increases with training iterations until only observation remains. We adopt a 50% cutoff for all tasks. Unbiased Asymmetric Actor-Critic [10] is a state-of-the-art method for POMDPs. The critic uses the <history, state> tuple as input to exploit state information while introducing no bias into the training process, thus achieving superior performance. To ensure a fair comparison, we keep the main network architectures similar across all methods. We present the mean and standard deviation of results obtained from five different seeds for each environment. For a comprehensive overview of other implementation details and hyperparameters, please refer to Appendix E.To see how useful DCRL is across various POMDP tasks, we compare various methods in Box2D environments. MiniGrid [41] is a procedurally generated environment with goal-oriented tasks. As shown in Figure 4 , the triangular agent with a discrete action space aims to reach the goal. These tasks involve interacting with different objects and solving diverse maze maps that vary in each episode.The observation comprises a first-person view representing a 7 × 7 × 3 image. At the same time, the state offers a fully observable perspective, with the image size varying according to the tasks. The image is compact and efficient encoding, rather than raw pixels. All experiments are conducted based on A2C, and the results are plotted in Figure 3 . Recurrent Actor-Critic struggles to learn a robust policy across most environments, primarily due to the agent's need for extensive exploration without state information. Despite leveraging states, Oracle Guiding and Asymmetric Actor-Critic performance exhibits instability, attributed to their heuristic nature and lack of theoretical guarantees. In contrast, the Unbiased Asymmetric Actor-Critic maintains stability and outperforms the mentioned methods. Nonetheless, it faces challenges in scenarios such as LavaCrossingS9N2 and RedBlueDoors-8x8. Our DCRL framework surpasses all other methods in eight environments, particularly excelling in challenging tasks. Taking LavaCrossingS9N2 and Empty-16x16 as examples, the map size of the former is smaller, but the layout is more complex. In the case of Empty-16x16, the states remain simple and relatively constant, making it easy to infer key information from history. Conversely, in LavaCrossingS9N2, objects change between episodes, leading to high uncertainty in the belief state. While leveraging states aids in precise value function learning, the variance poses challenges in policy optimization. DCRL effectively overcomes these limitations and emerges as a formidable competitor against other baseline methods. This section investigates whether DCRL benefits more complex continuous state space tasks and whether it can be applied to other Actor-Critic algorithms, such as PPO. Unlike A2C, PPO lacks a strong theoretical connection to DCRL, but we claim they can still complement each other. To empirically verify this, we implement all methods based on PPO and evaluate them on the MiniWorld domain [41] . The observations and states are represented as 80 × 60 × 3 images rendered as pixels.The results depicted in Figure 5 illustrate that DCRL enhances PPO performance across all four environments. In contrast to MiniGrid, the state format in this benchmark is not a concise encoding and tends to contain redundancy and noise. Although states assist the agent in effectively discerning information about the environment, the variance introduced by noise poses a challenge to the learning process. Furthermore, it is observed that the Unbiased Asymmetric Actor-Critic performs poorer than the Recurrent Actor-Critic across all tasks. Nevertheless, DCRL converges to superior policies by synergistically melding the two critics. Ablation analysis of components in DCRL. We conduct detailed ablation studies to analyze how DCRL improves efficiency and enhances performance. DCRL contains two key factors: the dual-critic structure and the weighting mechanism. We replace the dual-critic structure with the single-critic structure to verify the structure's effectiveness. We select four environments from the MiniGrid domain, where state information plays a crucial role. In these environments, methods that utilize states perform significantly better than those that do not. We compare our DCRL approach with four ablation baselines, and the results are presented in Figure 6 . It can be observed that DCRL converges to optimal policies faster than other methods. Intuitively, incorporating the concept of positive advantage (δ + ) can optimize the utilization of beneficial samples and expedite the training process. Consequently, single-critic structure methods integrating a δ + term outperform those lacking it, as demonstrated by the superior performance of approaches like the Standard Critic Version compared with the Recurrent Actor-Critic. However, it is crucial that the two critics fulfill distinct and orthogonal roles within the dual-critic structure. DCRL incorporates an oracle critic that utilizes state information compared with the Standard Critic Version. The inclusion of state information provides the agents with additional context to accurately determine the environment's true state, facilitating learning an accurate value function and ultimately improving performance. In comparison to the Oracle Critic Version, DCRL reduces variance. These factors contribute to DCRL's efficiency. As for the DCRL Exchange Version, although it is also a dual-critic structure, its policy gradient is biased, and its weighting mechanism Ablation analysis of β in DCRL. Our DCRL incorporates a dynamic weighting mechanism, which can be interpreted as a form of lower-bound-soft-Q-learning, distinguishing it from conventional linear weighting methods. This mechanism substantially improves DCRL's performance while maintaining high stability concerning hyperparameters. To substantiate this claim, we implement a simplified weighting method with fixed ratios for weighting the two critics, referred to as the DCRL No Clip Version. Additionally, we conduct ablation experiments on the parameter β ∈ {1/5, 1/3, 1/2, 2/3, 4/5}. The results depicted in Figure 7 indicate that the DCRL No Clip version shows performance improvements in only some environments compared to the Unbiased Asymmetric Actor-Critic. In contrast, DCRL consistently enhances performance across all environments, underscoring the importance of the dynamic weighting mechanism. Furthermore, the stability of β is notably evident. We choose β = 1/2 for reporting results in the paper, given its consistently superior performance. 
conclusion: 
 This paper introduces a novel reinforcement learning framework designed for partially observable tasks. The motivation behind this work is that complete state information may be accessible during training in many scenarios. However, introducing a state could potentially lead to bias or variance. The proposed DCRL method leverages two asymmetric critics to use the state to expedite training and mitigate variance in practice. We underscore the benefits of training with two critics and provide theoretical evidence supporting the unbiasedness and variance reduction achieved through our approach. Empirical results obtained across multiple tasks serve to validate the efficacy of DCRL.Limitations. Our framework operates under the assumption that state information is accessible during training. However, this assumption may not always hold, as only part of the state information might be available within a broader context. It is crucial to discuss whether this limitation will impact the performance of our framework, necessitating further investigation. Furthermore, our observations suggest that the availability of complete state information can sometimes hinder performance in certain environments. This observation underscores the need for a more nuanced approach to the design and utilization of state information. Addressing these issues could lead to enhancements in the performance and applicability of our framework. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cited the original paper that produced the code package. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Incorporating state information can offer significant advantages for training, as it often encapsulates high-level features that improve the learning of V (h, s) compared to V (h) [10] . For example, in an extreme scenario where the state information s represents the true value of V (h), the agent can effortlessly infer this true value through V (h, s). Specifically, in the context of the Empty Navigation task, when the state information indicates the agent's relative distance to the goal, the value function learning process is significantly streamlined. Besides, in most POMDPs, rewards are directly defined based on state information rather than on observations. Consequently, learning the value function V (h, s) with the reward as the target is typically simpler than learning V (h). This characteristic makes state information effective in most practical applications, a conclusion that numerous previous studies have empirically validated.However, state information does not always enhance training. When the state information contains significant noise, introducing excessive noisy features to the input can hinder learning the value function. In unfamiliar environments, whether state information genuinely supports the learning process remains to be determined. The motivation behind this work is that state information may be accessible during training in many scenarios. However, introducing a state could potentially lead to bias or variance. In most cases, state information offers higher-level features that assist agents in decision-making. In these situations, DCRL enhances the process by further reducing variance. Conversely, when state information and partial observations correspond one-to-one, the oracle critic demonstrates no variance; DCRL may become ineffective under these circumstances.Using complete states to train the world model. Traditional POMDP solutions rely on accurate environment models. Informed Dreamer [12] improves world modeling by incorporating state information. However, the dynamics models of POMDP environments are complex, posing challenges to modeling even with state information. Model inaccuracies will inevitably limit the performance. Our focus in this paper is on model-free methods.Using complete states in multi-agent reinforcement learning. Centralized Training Decentralized Execution (CTDE) [42] [43] [44] is a prevalent paradigm for addressing challenges in multi-agent reinforcement learning (MARL). The optimization objective in MARL is to maximize the rewards under the global states, optimizing each agent accordingly. These methods mostly presuppose that the combination of optimized actions under partial observations leads to optimality under global states. Building on CTDE, MANSA [45] introduces a Global agent to decide when to activate global information usage. However, the policy that maximizes rewards under partial observations does not necessarily align with the policy that maximizes rewards under global states, i.e., Q(h, a) = Q(s, a) [19] . Contrary to CTDE's assumptions, the learning objective of DCRL is to derive the optimal policy under partial observations. Double structure in the actor-critic framework. In our work, we employ two asymmetric critics to address partially observable tasks. Previous studies have explored various double structures. In the context of double critics, Double DQN [46] utilizes a replicated Q-network as the target network to handle overestimation issues, a method later extended to subsequent works such as TD3 [47] and SAC [48] . Considering the opposite properties of single-critic and double-critic methods, TADD [49] incorporates a novel triplet critics mechanism to reduce the estimation bias. Additionally, ADC [50] takes advantage of environmental models by training a model-based critic, which is then combined with the model-free critic. This approach enables learning from the model-free path despite model noise, enhancing the robustness to environmental models. In the context of double actors, DARC [51] introduces a double actor structure to boost the agent's exploration ability by providing two paths for policy optimization, thus reducing the risk of the agent getting stuck locally. Moreover, DAC [52] utilizes a parallel actor-critic architecture to learn better options, indicating that the two critics can be effectively consolidated into one.Theorem 4. In the DCRL Interchange Version,E s|h (1 -β )V π (h) + β V π (h, s) = V π (h).Proof. Since β is a parameter associated with both h and s, it cannot shift to the forefront of the expected symbol. Therefore, the dual value of the DCRL Interchange Version is biased. Also, the policy gradient in the DCRL Interchange Version is biased.Theorem 5. The learning objective δ φ (h, a)) + logπ(a|h) in DCRL can be viewed as a form of lower-bound-soft-Q-learning.Proof. According to Self-Imitation Learning [40] , the Lower bound soft Q-learning updates Q(h, a) as follows:EQUATION)where µ is a behavior policy, (h t , a t , R t ) is the trajectory triples from µ,R t = r t + ∞ k=t+1 γ k-t (r k + αH µ k )is the entropy-regularized return, α represents the weight of entropy bonus, andH µ k = -logπ(a k |h k )is the entropy of the policy. According to the form of optimal soft value function and optimal policy in the entropy-regularized RL, it is natural to consider the following forms of a value function V and a policy π:EQUATIONEQUATION)For convenience, we can define the following:EQUATIONEQUATIONWe can derive the gradient estimator of lower-bound-soft-Q-learning for the actor-critic architecture as follows:∇ θ E h,a,R∼µ 1 2 (R -Q(h, a)) + 2 = E [-∇ θ Q(h, a)∆ + ] = E [-∇ θ (α log π(a | h) + V (h)) ∆ + ] = E [-α∇ θ log π(a | h)∆ + -∇ θ V (h)∆ + ] = E α∇ θ L lb policy -∇ θ V (h)∆ + (19) = E α∇ θ L lb policy -∇ θ V (h) (R -Q(h, a)) + = E α∇ θ L lb policy -∇ θ V (h) R -V (h) + = E α∇ θ L lb policy + ∇ θ 1 2 R -V (h) + 2 = E α∇ θ L lb policy + ∇ θ L lb value ,whereL lb policy = -log π(a|h) R -V (h) + and L lb value = 1 2 ( R -V (h)) + 2 .As n → ∞, the learning objective in DCRL is∞ d=0 γ d r t -V (h) + logπ(a|h).It can be viewed as a form of lower-bound-soft-Q-learning when α → 0.In the DCRL Interchange Version, the equation E s|h [ R -V (h, s)] = R -V (h) holds according to Equation (5) . However, the equation E s|h [( R -V (h, s)) + ] = ( R -V (h)) + does not hold due to the presence of the + operation. The inconsistency in the equation arises from the fact that the + operation does not commute with the expectation operator, leading to a difference in the results. Consequently, the learning objective in the DCRL Interchange Version cannot be considered a form of lower-bound soft Q-learning.To illustrate the effectiveness of DCRL in variance reduction, we visualize the values of different critics in the Empty-Random-6x6 environment with a first-person view size of 3 × 3, as depicted in Figure 8 . In this configuration, the same history is linked to different states. We collect 8 history samples, each associated with over 300 state samples, meaning that each V π (h) is connected to multiple V π (h, s) and multiple V π dual (h, s). The converged critic models are used to predict V π (h), V π (h, s), and V π dual (h, s), respectively. The 8 violin plots in Figure 9 depict the details of each history sample. These plots show density estimation of the values, with the center line indicating the mean value. It is evident that DCRL (shown in orange) demonstrates lower variance compared with the oracle critic (shown in blue). V(h) V(h, s) Vdual(h, s)Although our research focuses on the model-free setting, we also compare DCRL with DreamerV3 [53] , a widely used model-based method in POMDPs. DreamerV3 constructs a model of the environment and enhances its behavior through the imagination of future scenarios. We describe each task environment used in our experiments. More detailed introductions can be found on the official project homepage of each benchmark. The source code is available in the supplementary.MiniGrid [41] comprises Box2D environments featuring various room layouts, interactive objects, and goals. A red triangular agent operates within a discrete action space, with actions such as "Move forward", "Turn left", "Turn right", "Pick up an object", "Toggle/activate an object", and others. Interactive objects include keys, doors, balls, and more. The agent must learn specific action sequences to achieve the final goal within its limited forward field of view, with rewards given only upon goal attainment. Each episode samples a different grid layout, and the agent has a first-person partially observable view of the environment.We selected different experimental settings. For instance, as depicted in Figure 4 , the agent in the LavaCrossingS9N2 environment aims to reach the green goal square in the room's opposite corner while avoiding deadly lava rivers that terminate the episode in failure. Each lava stream runs either horizontally or vertically across the room, with a single safe crossing point. Fortunately, a path to the goal is guaranteed to exist.The agent receives a 7 × 7 × 3 dimensional image of the environment as observation. The three channels represent object ID, color ID, and state. The agent is positioned at the observation's bottom center, and the state is encoded similarly to the observation, consistently displaying all grids without adjusting to the agent's orientation. The final channel may include 1, 2, 3, or 4 values at the agent's current coordinates, indicating the agent's facing direction (east, south, west, and north).Similar to MiniGrid, MiniWorld [41] is a minimalistic Box3D interior environment simulator consisting of connected rooms with objects inside. In MiniWorld, the agent has a 3D partial view of the environment and must navigate to the red box. For simplicity, actions are discrete, including "Turn left", "Turn right", "Move forward", "Move back" and others. Turn and move actions rotate or move the agent by a small fixed interval, simulating the behavior of a differential drive robot. Rewards are based on the number of time steps required to successfully complete the task, with a small penalty incurred. If the task is not completed within the maximum steps, the current episode is terminated, and a reward of 0 is assigned.We selected four experimental settings. For instance, as depicted in Figure 1 , the WallGap environment features two rooms connected by a gap in a wall. The agent must navigate toward the red box, randomly placed in one of the rooms.Observations consist of RGB images sized 80 × 60 × 3 captured from the agents perspective, while states are RGB images of size 80 × 60 × 3, representing an aerial view of the environment. In our experiments, RGB images are converted into grayscale images.We run all experiments on a single server with 64 Intel(R) Xeon(R) Gold 5218 CPU processors @ 2.30GHz and 1 Tesla V100 GPU. The hyperparameters for training each method are summarized in Table 2 . In this section, we describe the structures of the policy network, the standard critic network, and the oracle critic network. The general architecture for all environments is the same, except for the representation of input images. We utilize the CNN model for representation, and the structures are contingent upon the dimensions (n, m) of the input images. Detailed information for each environment is provided below. Representation Network, FC(in=input_dim, out=64), Relu, LSTM(in=64, out=64). FC(in=128, out=128), Relu, FC(in=128, out=number of actions).The structure is the same as that of the policy network, except for the dimension of the final MLP output head, which is set to 1.The structure is the same as that of the policy network, except for the dimension of the final MLP output head, which is set to 1.Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper proposes a novel partially observable RL method for settings when the state is available during training. The approach combines a history-state critic (which has been shown to perform well) with a history critic using a dynamic weighting scheme to reduce variance and improve performance. The paper theoretically shows that variance can be reduced and the method remains unbiased. The approach works well in a number of domains using A2C and PPO as base methods. 

Asymmetric critics that incorporate privileged information in training are becoming common in partially observable RL (RL for POMDPs). While the current state-of-the-art methods typically use history-state, the variance can be high. The proposed approach is relatively simple but well-motivated, simple to implement, and performs well. 

The author response was helpful for better clarifying the method and showing the benefits of the method, including the use of dynamic weighting and against other baselines. These results and discussions should be included in the final paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a dual critic architecture for learning POMDPs. One critic is the standard critic that uses the history information while the other critic is the unbiased asymmetric critic that uses the history and state information. The authors prove that using both critics reduces variance and propose a weighting mechanism between the two critics based on the advantage. Empirical results show that DCRL provides a substantial benefit over the asymmetric critic or unbiased asymmetric critic.
strength_weakness: strength_weakness: The primary weakness of this paper is novelty. While I can appreciate that the dual critic architecture reduces variance, the proofs seems fairly obvious. In my opinion, the main novel contribution of this paper then relies on the weighting mechanism and the empirical results showing that DCRL can beat AC, UAAC, and also recurrent A2C. The novelty therefore seems marginal.

Additionally, asymmetric actor critic (AC) was proposed to utilize additional state information to learn a policy more efficiently over the standard recurrent A2C where the critic only uses history information. UAAC then corrects the bias in AC by incorporating both the state and the history. However this paper proposes going back to the standard recurrent A2C method for one of the critics to achieve unbiasedness. It would have been interesting to compare two dual critic architectures: 1) V(h) and V(h,s) and 2) V(s) and V(h,s). Although the second approach would be biased, I would assume that using V(s) would lead to more efficiency gains. This seems to be an obvious comparison that was not considered.
questions: questions: See weaknesses
limitations: limitations: The authors clearly discuss the limitations of their work.
paper_summary: paper_summary: In this work, the authors aim to learn policies in the POMDP setting. They make use of two critics - one that has the privileged state information and one that doesn't and uses only history. Creating a dual value function which is a convex combination of these two value functions. They use this dual value function for policy optimization. They perform experiments in box2d and box3d environments.
strength_weakness: strength_weakness: (1) Even assuming that you have the state information during training is a strong assumption. You might not always have that. A lot of works do not make that assumption and perform pretty well.

(2) There should have been comparisons to commonly used POMDP works like Dreamer [] and others. But the comparisons were made only with (in some words) different variants of the algorithm.
questions: questions: (1) Do you think in most real world settings you described in the introduction, you will have access to the state information for training the oracle critic?
limitations: limitations: The authors have discussed their limitation.
paper_summary: paper_summary: The paper presents Dual Critic Reinforcement Learning (DCRL), a framework designed to handle partial observability in RL. Traditional RL methods often struggle with high variance and instability when relying on full-state information. DCRL addresses this by integrating two critics: an oracle critic with access to complete state information and a standard critic operating within a partially observable context. This dual approach aims to improve efficiency and reduce variance during training, leading to optimized performance in online environments.
strength_weakness: strength_weakness: Refer to Questions.
questions: questions: - There is a typo in the definition of $R(h,a)$ in line 122.
- Can you explain in more detail how the weighting mechanism between the oracle critic and the standard critic is designed to reduce variance? Are there specific conditions or thresholds for adjusting the weights?
- Is it necessary to ensure the alignment of $V(s)$ and $V(h,s)$ in DCRL? if yes, how to ensure it?
limitations: limitations: N/A
paper_summary: paper_summary: The authors propose a methood that uses a weighted dual critic structure for tackling POMDPs. The dual critic structure consists of a critic that receives global state information while the other critic receives only the partially observations of the state. The authors provides some simple yet concrete analytical results that show the method induces unbiased critic estimates and can reduce variance compared to that of the Unbiased Asymmetric Actor-Critic.
strength_weakness: strength_weakness: 1. While I understand the reason for doing so, such a strategy introduces a harsh jump-discontinuity in the dual advantage and critic - this may lead to numerical instabilities. The paper would benefit from a discussion on this issue.

2. The method also seems related to centralised training and decentralised execution in multi-agent reinforcement learning. There, the critic is trained with global information while the actor is trained using only local inputs. It would be a nice connection to make if the authors could include some discussion on degenerate case of ct-de with N=1.

3. Notwithstanding the discussions on why the method performs well in the chosen environments (lines 297-314) I would have liked to have seen some discussion and analysis of when one could expect this method to be most useful. Indeed, it's conceivable that including more information may slow training even if one can expect improved asymptotic performance.  There may also be situations where we do not have access to the full global state even during training.  I think the paper would benefit from such discussions.
questions: questions: Q1. The authors state that $V^\pi(h)$ has nonzero variance. Using the definition of the dual value function, setting $\beta\equiv 1$ implies that $V_{\rm dual}^\pi(h,s)=V^\pi(h)$. However, setting $\beta=1$ in equation 8 implies that $Var_{s|h}[V_{\rm dual}^\pi(h,s)]=0$. Can the authors explain this.

Q2. How does the method perform over a range of values of $\beta$?

Q3. What environments should we expect the method to do well in (and not so well)?
limitations: limitations: L1.The experiment section does not contain an ablation on the value of $\beta$. Without this it is difficult to understand the size of the range of values of $\beta$ for which the method performs well.


L2. The experiment environments are quite simple - it would be useful to see how the method performs on complex benchmark environments.
paper_summary: paper_summary: This paper proposes a dual-critic architecture for the asymmetric RL regime where a policy deployable in partially observable settings is trained under full observability. The proposal is meant to be an improved version of Unbiased Asymmetric Actor Critic that, through its two critics, improves learning efficiency while reducing variance. This is demonstrated through theory and experiments in MiniGrid and MiniWorld.
strength_weakness: strength_weakness: **Originality**: Ultimately, the proposed method is somewhat derivative of two other well-known methods. The main contribution is the dual critic architecture, which is a linear combination of a typical recurrent critic $V(h)$ and the unbiased asymmetric critic $V(h,s)$. On its own, this would not be a major issue since the method is shown to perform well, but I believe the insights behind this approach are also discussed in the original Unbiased Asymmetric Actor Critic paper. 

**Quality**: While the theorems appears correct, I wonder how relevant they are to the topic at hand. To my knowledge, what we *really* care about is the variance of the policy gradient Monte-Carlo estimator --- not the value estimator. The former is important since it plays a big role in the sample efficiency of policy learning. While the variance of the value estimator may indicate the amount of data required to train a reasonable value function, this is again only important insofar as reducing policy gradient variance through its role as a baseline.

There also needs to be a discussion on *why* we don't always want to reduce variance all the way to zero by setting $\beta = 1$ (which is equivalent to the simple recurrent critic). What value does increasing the variance (in order to incorporate state information) have? I believe this is not totally well understood in the literature either, but the topic should at least be brought up to clarify the point above. An investigation that can properly answer this question would be a strong contribution. 

Regarding the implementation, it seems the choice of when to clip to $\beta = 0$ is very important but I don't understand why the particular heuristic in the paper was chosen. I'm not sure why the motivation of preventing large updates when the advantage is non-positive is relevant. 

The results of the experiments show a solid advantage but it's not totally clear where the benefits are coming from (likely because I didn't understand the motivation behind the clipping of $\beta$). I think the experiments would also benefit from stronger baselines such as the Believer algorithm cited in the related work, and partially observable tasks that involve more than simple navigation. 

**Clarity**: As previously mentioned, the reason why the state-based critic $V(h,s)$ is beneficial could be better explained, along with the heuristic for when to clip $\beta = 0$. Perhaps a toy example could be useful for this. I'm also unsure what the significance of the interchange method is. 

**Significance**: I believe there isn't enough insight into the proposed approach to justify its use compared to Recurrent PPO or Unbiased Asymmetric Actor-Critic, which are somewhat better understood.
questions: questions: 1. Do you know how the dual critic architecture affects the variance of the policy gradient estimator? 
2. How was the clipping method for $\beta$ chosen?
limitations: limitations: Yes.

Comments: title: Re:
Have we addressed your concerns?
Response
Reviewer Response
comment: Thank you very much for your response. We are happy to hear that we have addressed some of your concerns. We will incorporate your suggestions regarding the concepts of variance and provide a more precise description in the revised manuscript. For your remaining questions, we provide point-to-point responses as follows:

> Q1: About the dynamic weighting mechanism.

We apologize for the misunderstandings that have arisen. To clarify, our claim is that DCRL reduces variance compared to UAAC (which relies solely on $V(h,s)$) and is unbiased with respect to $V(h)$ in expectation, leading to an unbiased policy gradient. **However, this does not imply that we always want to choose $\beta=1$.**  When $\beta=1$, state information can not be utilized. In fact, $\beta=1$ is not used in either the derivations or experiments of DCRL. As shown in Figure 1 of the rebuttal PDF, we tested values of $\beta$ in $\\{1/5, 1/3, 1/2, 2/3, 4/5\\}$ and selected $\beta=1/2$ for reporting our results in the paper.

We agree that the key question is when each critic is more effective. Our motivation is to enhance the balance between the two critics. DCRL achieves this by dynamically adjusting the weight rather than always setting $\beta$ to $1$ or $0$. Specifically, the adjustment mechanism is based on the advantage $\delta(h, a)$. This approach aims to reduce variance while preserving the acceleration provided by state information. From the perspective of lower-bound-soft-Q-learning, **the policy gradient induced by $V(h)$ supports the lower bound of the optimal Q-values only when the advantage is positive.** This allows the policy to update towards higher returns without altering the update direction from $V(h, s)$. Conversely, when the advantage is non-positive, $V(h)$ does not provide useful information about the optimal Q-values and may interfere with updates from $V(h, s)$, thus diminishing the benefit of state information. Therefore, $V(h)$ is excluded from training in DCRL under these conditions. As shown in Figure 1 of the rebuttal PDF, the results of the DCRL No Clip Version experiment support this conclusion.

We also agree that $V(h)$ ought to be weighted higher near the end of training.  However, this presents a challenge: accurately identifying the later stages of training in an unfamiliar environment can be difficult. Mismanagement of this weighting may lead to suboptimal solutions. Therefore, we adopt a more intuitive and straightforward approach: **applying $V(h)$ whenever it reduces variance without disrupting updates of state information.** The idea that $V(h)$ ought to be weighted higher near the end of training is the direction of our future efforts.

> Q2: About additional baselines that leverage state information.

Thank you for your valuable feedback. Our paper has included three relevant latest baselines that utilize state information: UAAC, AAC, and Oracle Guiding. Following your insightful suggestions, we are conducting experiments about Believer. As Believer focuses on representation models instead of learning frameworks like DCRL, it involves collecting random samples to pre-train the representation model for $5000$ epochs, which considerably extends training time. We will consider incorporating additional baselines in the revised version of our manuscript as per your recommendation.

We sincerely appreciate the time and effort you have invested to the discussions! Please feel free to let us know if you have any further concerns.
Thank you very much for your feedback! We are glad to have addressed some of your concerns.

**Indeed, there are situations where the local and joint policy coincide, this holds ture when the state encompasses history (rather than just the observation).** The centralized critic is equivalent to using $V(h, s)$, which theoretically ensures that the policy is unbiased and converges to the optimal solution. Since MARL primarily deals with cooperation and competition among multiple agents, partial observability often stems from other agents. When the problem is reduced to $N=1$, most tasks fall into this situation.

**In POMDPs, it is more common for the state not to fully include history**, which is equivalent to using $V(s)$. The state alone does not typically reveal whether the agent has previously collected the necessary information and thus cannot adequately indicate if the current state is favorable or adverse. **The policy is biased and may not converge to the optimal solution.** The UAAC paper has proved it in Theorem 4.2 and includes a toy example to illustrate the concept in Appendix B.2. 

We appreciate your constructive feedback and apologize for the lack of clarity in our description. We will provide a more detailed explanation in the revised version of our manuscript.
Thank you to the authors for clarifying my questions. 

I appreciate your clarification on the connection between the policy gradient variance and the value function variance. The paper should better distinguish these two concepts -- "variance" on its own is typically associated with the policy gradient estimator and it's important that these concepts not be conflated. Your explanation of how they are mutually related helps allay my concerns.

I also appreciate the insight behind why we would sometimes opt for the higher variance $V(h, s)$ critic. **However, I'm not sure I'm convinced by the explanation of the dynamic weighting mechanism.** As I understand it, there are some situations where we want the $V(h, s)$ critic to have higher weight and others when we want $V(h)$ to have higher weight -- presumably $V(h)$ ought to be weighted higher near the end of training when we've converged on a good $V(h)$ critic and $V(h, s)$ unnecessarily adds (irreducible) sampling variance. The argument that the DCRL critic has lower variance and is unbiased compared to $V(h)$ is misleading since that would suggest we always want to choose $\beta = 1$. In reality, the question of *when* each critic is more effective is the key question here. 

That leads me to my next point. The authors implicitly claim that the dynamic weighting is an effective way to decide when to apply the DCRL critic vs the $V(h)$ critic. To me, the explanation in the paper (and in the rebuttal) is a bit brief and lacking -- it still doesn't quite answer why this works so well. Why is the sign of the advantage $\delta_\phi(h_t, a_t)$ a good determinant of how we should weight the two critics?

Lastly I appreciate the inclusion of Dreamer-v3 as a baseline that does not utilize state information. I still think including additional baselines that do leverage state information would be an improvement.
Thanks again to the authors for their detailed response. 

My comment about the relationship with MARL is that the centralised training with decentralised execution (CT-DE) paradigm closely resembles the central idea being presented, albeit for a distributed setting with a different need to aggregate local information. In particular, the methods that stem from CT-DE degenerate into something similar when considering the case of $N=1$. While it is true that in MARL, one has to consider how to best promote the so-called individual-global-max condition, in the degenerate $N=1$ case this may not be a concern, I believe as the local and joint policy coincide (even if the critic is centalised). Note also that the solution to a POMDP is in general, different to the solution when the agent is endowed with the missing data (which recovers the MDP setup) - this is not specific to the multi-agent setting.

I would like to thank the authors for their response to point 2 - this is a helpful clarification.
Thanks again for your time and effort in reviewing our paper! As the discussion period is coming to a close, we would like to know if we have resolved your concerns expressed in the original reviews. We remain open to further feedback and are committed to implementing additional improvements if necessary. If you find that these concerns have been resolved, we would appreciate it if you would consider reflecting this in your rating of our paper!
We sincerely appreciate the time and effort you have invested to the discussions! The above inspiring discussion has greatly enhanced the quality of our paper. The scope of our study encompasses situations where state information is accessible during training, which is an essential topic in POMDPs with many applications like autonomous driving. For problem settings without access to state information, we can leverage a pre-trained teacher network to provide labels, thereby facilitating training without compromising deployment. We will incorporate this suggestion and provide a more comprehensive discussion in the revised manuscript.

--Best wishes from all the authors!
Thank you for your response and additional experiments. I am increasing my score based on the clarifications. I agree that there might be environments where obtaining state-information is possible and you can use that state observation during training but that would severely limit your work in terms of applicability to any POMDP setting.
Thank you very much for your support! We appreciate the time and effort you've invested in reviewing our manuscript!
Thank you to the authors for answering my questions! I will keep my score.
We sincerely appreciate the time and effort you have dedicated to the discussions! The above inspiring discussion has greatly improved the quality of our paper. Thank you for recognizing our efforts. We want to assure you that we are committed to incorporating these discussions into the final version.

Regarding the lower-bound soft-Q-learning, we employ an approximation in which the entropy parameter approaches zero to meet the condition. We will incorporate this suggestion and provide a more comprehensive discussion in the revised manuscript.

--Best wishes from all the authors!
I thank the authors for their detailed and focused responses. I am honestly impressed that they were able to perform all of the new experiments and the new results strengthen the paper greatly. I've raised my score.

I'm now more convinced that DCRL is a good approach and outperforms other possible reasonable methods. In particular, the fixed weighting $\beta$ experiments show that dynamic reweighting is crucial, increasing this paper's contribution. 

One minor point in the rebuttal is that I'm not convinced of the argument that the mechanism resembles lower-bound-soft-Q-learning since soft Q learning fundamentally optimizes a different objective (max entropy RL or soft Bellman equation), while here the critic is still optimized for the standard Bellman equation, but again this is minor.
Thank you for your prompt response. We also appreciate your recognition of our supplementary experiment and are glad to have addressed some of your concerns. We will incorporate your suggestions regarding notation and jump discontinuities in our revision of the manuscript. For your remaining questions, we provide point-to-point responses:

> Q1: Regarding the relevance to MARL problems.

We apologize for the misunderstandings that have arisen. The setup presented in our paper differs from the MARL problem formulation.

In MARL, the optimization objective is to maximize the reward under the global states, optimizing each agent accordingly. These methods mostly presuppose that the combination of optimized actions under partial observations leads to optimality under global states. However, this assumption does not always hold, even when $N=1$. The joint-action learner can not be decomposed into independent actors. The policy that maximizes rewards under partial observations does not necessarily align with the policy that maximizes rewards under global states, i.e., $Q(h, a) \neq Q(s, a)$ [DRQN2015Arxiv]. In contrast to MARL, the optimization objective in the POMDP problem discussed in this paper does not rely on this assumption, focusing instead on maximizing rewards under partial observations. Therefore, compared to our DCRL, the applicability of MARL methods in POMDP contexts is limited.

> Q2: Regarding the effectiveness of state information.

We fully understand your concerns. In POMDPs, rewards are directly defined based on state information rather than on observations. Consequently, learning the value function $V(h,s)$ with the reward as the target is typically simpler than learning $V(h)$. This characteristic makes state information effective in most practical applications, a conclusion that numerous previous studies have empirically validated [UAAC2022AAMAS, Suphx2020Arxiv, PerfectDou2022NIPS, Honor-of-Kings2020NIPS]. Additionally, our DCRL leverages the advantages of state information while reducing variance, significantly enhancing training efficiency. Extensive experiments demonstrate that our DCRL consistently outperforms methods that do not incorporate state information across all tested tasks.

Thank you very much for your contribution in clarifying our method. Please feel free to let us know if you have any further concerns.

**Reference**

- [DRQN2015Arxiv] Hausknecht Matthew, et al. Deep recurrent q-learning for partially observable MDPs. Arxiv, 2015.
- [UAAC2022AAMAS] Andrea Baisero, et al. Unbiased asymmetric reinforcement learning under partial observability. In AAMAS, pages 44–52, 2022.
- [Suphx2020Arxiv] Junjie Li, et al. Suphx: Mastering mahjong with deep reinforcement learning. Arxiv, 2020.
- [PerfectDou2022NIPS] Guan Yang, et al. PerfectDou: Dominating Doudizhu with perfect information distillation. In NIPS, pages 34954–34965, 2022.
- [Honor-of-Kings2020NIPS] Deheng Ye, et al. Towards playing full MOBA games with deep reinforcement learning. In NIPS, pages 621-632, 2020.
I would like to thank the authors for their detailed responses to my comments. I would also like to congratulate the authors for having produced useful additional results, particularly the ablation on $\beta$ in the short space of time.

My opinion is that overall the paper has merit and the additional analyses have allayed some of my concerns. In light of the fact that parts of the main insights exist albeit within the multi-agent reinforcement literature and that the corresponding methodology may be applied in the degenerate, single-agent case means that the contribution is somewhat constrained. In MARL, there are also techniques to learn how best to interpolate or switch between using a value function with global state inputs and that with local inputs which are worth mentioning e.g. [1]. This is an important consideration as it is likely not a priori known which environments would benefit from global information during training should this information be available.  For this reason, I will keep my score as is.

**Other points**

* I recommend the authors use a different notation for the value functions $V^\pi(h,s)$ and $V^\pi(h)$ and similarly for the action-value functions.

* I think my earlier point about jump-discontinuities can be addressed by showing that $V^\pi_{\rm dual}$ is a smooth function w.r.t. the update parameters of $V^\pi(h,s)$ and $V^\pi(h)$  whenever the latter two functions are differentiable and, showing the corresponding statement for $Q^\pi_{\rm dual}$ .

[1] Mguni, David Henry, et al. "Mansa: Learning fast and slow in multi-agent systems." International Conference on Machine Learning. PMLR, 2023.