Latent Trajectory: A New Framework for Deep Actor-Critic Reinforcement Learning with Uncertainty Quantification
Authors: Frank Shih
Faming Liang
Venue: Accepted by TMLR ()

Abstract: Uncertainty quantification in deep learning is challenging due to the complexity of deep neural networks. This challenge is particularly pronounced in deep reinforcement learning (RL), where agents interact with stochastic environments. In deep actor-critic RL, this challenge is further exacerbated due to the interdependence between the actor and critic updates. Existing uncertainty quantification methods for RL are predominantly developed within the Bayesian framework. While these methods estimate the uncertainty of the value function, their confidence intervals are often misleading, with the coverage rate frequently falling well below the nominal level. To address this issue, we introduce a novel deep RL framework that treats transition trajectories as latent variables. Leveraging this framework, we propose an adaptive Stochastic Gradient Markov Chain Monte Carlo algorithm to train deep actor-critic models, which naturally accounts for the interdependence between the actor and critic updates.  We provide theoretical guarantees for the convergence of the proposed method and offer empirical evidence for its effectiveness in uncertainty quantification of the value function. The proposed latent trajectory framework  is highly flexible,  allowing for the integration of advanced RL strategies to further enhance deep actor-critic learning.

Published in Transactions on Machine Learning Research (09/2025)
Latent Trajectory: A New Framework for Deep Actor-Critic Reinforcement Learning with Uncertainty Quantification
Frank Shih
shihf@msskc.org
Deportment of Epidemiology and Biostatistics Memorial Sloon Kettering Cancer Center New York, NY 10017
Faming Liang
fmllian @purduc..ccccummmm edu
Department of Statistics
Purdue University
West Lafayelle, IN 47907
Reviewed on OpenReview: http::://ppereviiiee net/ formm id= 8B74
Abstract
Uncertainty quantification deep learning is challenging due the complexity of deep neural networks. This challenge is particularly pronounced deep reinforcement learning (RL), where agents interact with stochastic enviromments. In deep actor-critic RL, this challenge is further exacerbated due to the interdependence between the actor and critic updates. Existing uncertainty quantification methods for RL are predominantly developed within the Bayesian framework. While these methods estimate the uncertainty of the value function, their confidence intervals are often misleading, with the coverage rate frequently falling well below the nominal level. To address this issue, we introduce a novel deep RL framework that treats transition trajectories latent variables. Leveraging this framework, we propose an adaptive Stochastic Gradient Markov Chain Monte Carlo algorithm to train deep actor-critic models, which naturally accounts for the interdependence between the actor and critic updates. We provide theoretical guarantees for the convergence of the proposed method and offer empirical evidence for its effectiveness in uncertainty quantification of the value function. The proposed latent trajectory framework is highly flexible, allowing for the integration of advanced RL strategies to further enhance deep actor-critic learning.
1 Introduction
Reinforccment learning (RL) tackles sequential decision-making problems by designing agent that interacts with its environment to learn optimal policy, aiming maximize value function. Accurately quantifying the uncertainty of the value function is crucial for ensuring reliable and robust RL applications. However, this task is particularly challenging in the context of deep RL due to two main factors: the complex architectures of deep neural networks (DNNs) and the adaptive nature of the RL process. Even in supervised learning, accurately quantifying the uncertainty for DNNs has proven to be difficult (see, e.g., Blundell al. 2015 Lakshminarayanan et al., 2017 and [Snn et al. 2021), and the dynamic, evolving process in RL exacerbates al. 2017). A significant step toward addressing this challenge has been made in Shih & Liang (2024), where deep Q-networks simulated from their posterior distribution under the Kalman Temporal Difference (KTD) framework (Geist & Pictquin 2010 Shashua & Mannor 2020), enabling accurate quantification of the uncertainty of the Q-value function throughout the RL process. However, this approach is difficult to extend to deep actor-critic RL, where the presence of an additional actor network adds complexities to uncertainty quantification.
this challenge (Osband et al. 2016 Bellemare
To address this challenge, we introduce a novel deep RL framework which treats transition trajectories latent variables. Leveraging this framework, propose adaptive Stochastic Gradient. Markov Chain

---

Published in Transactions on Machine Learning Research (09/2025)
Monte Carlo (SGMCMC) algorithm to train deep actor-critic networks, which simultaneously updates the actor network through a stochastic gradient descent (SGD) step and samples from the conditional distribution of the critic network - conditioned on the current actor network - via an SGMCMC step. Under mild conditions, we establish the convergence of this adaptive SGMCMC algorithm. Specifically, we show that the parameters of the actor network converge in probability to a fixed point, while the parameters of the critic network converge weakly to a target distribution, thereby enabling accurate uncertainty quantification for the associated value function.
In summary, this study has made three primary contributions:
We introduce novel latent trajectory framework for training deep actor-critic models that natu- rally accounts for the interdependence between the actor and critic updates, simplifying theoretical analysis.
We provide theoretical guarantees for the convergence of the proposed deep actor-critic RL method the latent trajectory framework coupled with an adaptive SGMCMC algorithım, which ensures effective training for the actor network while enabling proper uncertainly quantification for the critic network and, consequently, the value function.
The proposed latent trajectory framework is highly flexible, allowing for the integration of advanced RL strategies to enhance deep actor-critic training and uncertainty quantification.
Related Works Convergence analysis for deep actor-critic RL is inherently challenging due to the inter- dependence between the actor and critic updates. To the best of our knowledge, only limited progress has been made in this area; see, e.g., Wang et al. (2020), Cayci et al. (2022), and Tian et al. (2023). Both Wang et al. (020)) and Cayci et al. (2022) considered single-layer neural networks and employed double-loop strategy. In the inner loop, the critic network performs sufficiently many updates to accurately estimate the value function, ensuring that the actor network operates with reliable approximation of the true value function. This setup enables the actor-critic RT to be analyzed as gradient method with approximation error. In contrast, Tian et al. (2023) applied small-gain approach that accommodates multi-layer networks but is limited to finite state spaces. In this work, we establish convergence guarantees for deep actor-critic RL with multi-layer networks and general state spaces, which can be either discrete or continuous.
The proposed method provides accurate uncertainty quantification for the critic network. While Bayesian methods have been widely adopted for uncertainly quantification in machine learning, applying them rigor- ously to deep actor-critic RL presents significant challenges. For example, Osband et al. (2018) introduced uncertainty in deep RL through randomized priors within maximum posteriori (MAP) framework. How- ever, the posterior consistency for such priors remains unestablished, and inappropriate priors can lead to severely biased inference. The deep actor-critic setting further complicates the uncertainty quantification issue due to the interdependence between the actor and critic updates. Other methods, such as those based on bootstrapping (Osband et al.
2016 Tasdighi et al. 2024 Poer et al.
2021
Kalweit & Boodecker 2017)
deep ensembles (Mai et al. 2022), Bayesian dropout (Moerland et al.
2017
Gaussian processes (Geist
Pictquin 2010 Engel et al. 2003), quantile regression (Dabney et al.
2017
Kalman temporal difference
(Shashua & Mannor 2020), and distributional RL (Bellemare al.
2017
Clements et al. 2019), have
been proposed for uncertainty quantification in RL. However, their theoretical guarantees have not been established in the deep actor-critic setting.
In contrast, our method explicitly accounts for the evolving nature of online deep RL and ensures that uncertainty in the critic network is accurately quantified, despite challenges posed by the interdependence between the actor and critic updates.
2 Preliminaries on Actor-Critic Models
We consider discounted, finite horizon policy optimization problems. Let 0 and y denote the parameters of the actor and critic networks, respectively. Let ($1, 01, 82, o2....) be the transition trajectory generated by stochastic policy na; where each action a, is sampled from the distribution no(as(s)) and indexes the
2

---

Published in Transactions on Machine Learning Research (09/2025)
state transitions. Let (s denote the Q-value corresponding the state-action pair (st, at) under the policy wo. At each time step 1, the agent receives immediate reward = r(t,, 0z). Let Re = "===?"==r,. denote the truncated return
to horizon n, which is an unbiased estimator of Qxo (st, at). Let V₂ be the critic network approximation to the value function V..- For convenience, we denote a single transition of the state and action as æ = (8, a), the relurn estimate as In this paper, we focus on the advantage actor-critic algorithm (uutonn al. 2000 Schulman al. 2018) with the advantage function expressed as:
Av (tt 0z) R. - (o((s),
(1)
where Ag indicates the dependence of the advantage function on the critic network th.
The policy gradient [Sutton Barto 2018) for the advantage actor-critic algorithm takes the form
9
E
Σ As Vo
(2)
where the expectation is taken with respect to on-policy trajectories generated by wg. Note that different parameterization strategies can be used for the advantage function. For example, one may parameterize the Q-function, using temporal difference (TD) or Monte Carlo methods to estimate it (Schulman et al.) 2018). The parameters and s are then iteratively updated using SGD algorithms till convergence, at which a solution to the following equation is reached:
( 0,
(3)
which characterizes (local) optimum of the policy objective (not necessarily the global optimumm). However; the convergence theory for such an iterative optimization algorithm is hard to establish except for special cases under restrictive assumptions, such as linear function approximation (Chen et al.) 2023 Wu et al. 2022), greedy policies (Holzleitner et al.) 2020), shallow neural network approximation (Wang et al. 2020 Cayci et al. 2022), or finite state space (Tian al. 2023). In practice, various RT. strategies have been proposed, such as Advantage Actor-Critic (A2C), Asynchronous Advantage Actor-Critic (A3C) (Mnih al. 2016), Proximal Policy Optimization (PPO) (Schulman al. 2017). Soft Actor-Critt (SAC) (Haarnoja al. 2019), which employ different tuning techniques for policy gradients to enhance the convergence and stability of the training process.
al. 2018), and Deep Deterministic Policy Gradient
R
e'
(A)
0--2
0
02
0
Figure 1: Latent Markov Sampling process, where tha is conditionally independent of 0x-1 given (((), i.e., th 10 and the dashed line indicates that including the latent variables (x) R(k) breaks the original dependence between the and 0--1-
Consider an actor-critic training process as depicted by Figure I Let & index the updates of 0 in the training process, and let (R) = (2( p denote batch of state-action transition trajectories drawn independently from the stationary distribution [x where x; == (* aa
(k)) and n is the batch size. Additionally,
let (k) (k) ( be the estimated returns corresponding to (k), At each iteration k transition trajectory of size is generated from the policy _e the critic network parameter thk is then updated based on the transition trajectory.
Given actor-network parameters 44-1, the induced policy Tep_, is fixed. Together with the environment transition dy- namics, this policy induces stationary distribution over state acate-action trajectories (50, 40, 81, 01s...), which we denate (( (2-1) under standard ergodicity conditions.
3

---

Published in Transactions on Machine Learning Research (09/2025)
The ultimate goal of RL is to learn optimal policy, with the transition trajectory serving evolving path of the learning process and the critic network serving as an auxiliary guiding device for the path. Therefore, the transition trajectory (x(*), R(()) and the critic parameter th can be naturally treated as latent variables that facilitate policy optimization. The whole RL process is Markovian as depicted by Figure [ Leveraging the latent Markov sampling process, thk can also be viewed as a sample drawn from the conditional distribution x1)1/00-1) by noting that
xVW1(0x11)= = (x) R( On (0-2)) (
(4)
To enhance the mathematical rigor of equation 4 in practical implementations, we assume that 2x(h) is sampled from eseudo-populatinn (denoted by )..-,) of size .N, while the pseudo-ppopulations can vary for different values of 0 Intuitively, when the pseudo-population is small, л x==(0==1) can be well ap- proximated with a limited number of samples, thereby resulting in a good approximation to x(vu/0,-1) and strengthening the Markovian property the process. The concept of poouloopopulation allows for the flexibility of using different mini-batch size at different. iterations, while ensuring the stability of the target conditional distribution л (1)(6)) In what follows, we use Tx(l)) to denote the conditional distribution of t for given value of 0.
Furthermore, this latent Markov process view of RL allows frame the training of the actor network in terms of solving the following equation:
g(0) = (0) = 0,
(5)
which accounts for the variation of sp in 3 and makes its solution (denoted by 0*) essentially invariant to the trajectories of db and (æ, Rt). Notably, 0* is not necessarily unique. However, as with other applications of neural networks, this non-uniqueness of the optimal solution does not impact the network's ' performance,
Equation (5) can be efficiently solved using an adaptive SGMCMC algorithm (Liang et al. 2022 Deng et al. 2019), leading to an innovative Latent Trajectory Framework (LTF) for training actor-critic models.
By examining equation (5) through the lens of Fisher's identity (see, e.g., [Song et al.) 2020). it becomes evident that g()) corresponds to the marginal gradient of the actor network when the critic network is treated as latent variable. Consequently, training deep actor-critic networks falls into the class of problems amenable to stochastic approximation MCMC (see e.g., Benveniste et al., 1990). This formulation allows the interdependence between the actor and critic updates to be naturally handled by the proposed adaptive SGMCMC algorithm (sce Section 3). leading to a simplified convergence analysis compared to the double-
loop (Wang el al. 2020: Cayci et al. 2022 and small-gain (Tian al. 2023 approaches.
3 A Latent Trajectory Framework for Actor-Critic Models
3.1 An Overview of the SGMCMC Algorithm
To solve equation 5 an adaptive SGMCMC algorithm can be applied. Each iteration of the algorithm consists of two steps:
1. (v--smpling) Simulate the (16668)))1 by SGMCMC algorithm.
(0-updating) Update 0g = 00 + Ukgac (0.-2) where vi denotes the step size used the stochastic approximation procedure (Robbins & Monro. 1951), and goe (0<-1) represents an unbiased estimator of 9(0=-1).
2Given fixed policy ny and stationary, ergodic sequence of transition tuples, the number N of observed tuples tends to infinity, any consistent estimator e' of the parameter & in the value function Vis collapses to point (zero variance). To avoid the resulting degeneracy the conditional distribution x(>) 8), we work with pseudo-populatinn population so that n(p 8) remains well defined and does not depend on N.
4

---

Published in Transactions on Machine Learning Research (09/2025)
Under mild conditions, establish the convergence of the proposed algorithm. Specifically, we show that |00-01 + 0 in probability as k - xo, where 0* denotes a solution to (5) defined previously. Additionally, Ve converges weakly (in 2- Wasserstein distance) to the conditional distribution TT(++)0)). Consequently, the algorithm enables proper uncertainty quantification for gi-related quantitics, such as the V- and Q-value functions, which are central to RL. Notably, uncertainty quantification for the value functions is generally beyond the capabilities of iterative optimization algorithms conventionally used to train deep actor-critic models (see, e.g., Tian et al. (2023)). Leveraging this latent trajectory formulation, we establish actor- critic training framework that is essentially independent of the sample trajectory.
3.2 Adaptive Stochastic Gradient MCMC for Deep Actor Critic Learning
To perform d-sampling using SGMCMC, we need to evaluate the gradient V (W This be done using the following identity established in Song al. 2020
V log x (v)0) / V log (v) (z) 0)dz,
where denotes latent variable. By treating the trajectory (zx(A, (2(h)) the latent variable, we can derive the following formula (refer to Appendix A for the derivation):
V log (x((h/00-11) / V log (1
x (R (xx()) shh) (x)( R(() 0k 11-1)d (R()) R())
(6)
(R) xx)
provided that the mini-batch size has been chosen to be sufficiently large, ensuring that (x(k) serves as good representative of the underlying ceuudo-oopulation
For convenience, though not requirement, we assume that the reward distribution x T(R,a v)) is Gaussian, as given by
Re vb ~ (VV
It worth noting that the Gaussian assumption for the reward has also been employed under the Kalman Temporal Difference framework, see c.g., Geist & Pietquin 2010). Tripp & Shachter (2013), and Shashua & Mannor 2020
Remark 3.1 How to evaluate V log (v2)) R())) Based
have
V lvg (R())
x (R) (b) + V (thx) V log T. (R())
N
(8)
v log T (R((a) |x
(k) (b) + V log ((hc),
where (Tbx) denotes the prior distribution of vh.
- .vt) Since the numerator can be Remark 3.2 How to evaluate the importance weight wy * [ (|ক)|
evaluated based on 7. we consider the evaluation of the denominator in this remark. One way is to evaluate the denominator based on the relationship:
TT (200) f (2) whe ( (e'x (b) (d)
(9)
i.e., estimating the denominator by averaging the density Tt (p wha) over a set of samples of the drawn from (k)x) The auziliary samples of th can be simeulated using a SGMCMC algorithm based on the following gradient estimation:
V log (0)o(( = Vg
(
() G)d
(10)
which be estimated based auriliary somples of R drawm from (k) (0)(())))))))) H), defined in

---

Published in Transactions on Machine Learning Research (09/2025)
Alternatively, can estimate x ((R(()(())) using the Nadaraya- Watson (NW) conditional density kernel estimalor:
A ( Ei=1 Kig (r - 21 ( Kng (R R2(A))
(11)
E Kng (xc - (10)
where both (-) and (+) are Gaussian kernels, and h1 and h2 are their respective bandwidths. The NW estimator is known to be consistent provided h1 -> 0, h2 - 0, and nh1h2
n (Hyndman et al. 1996). Exlensions of the NW eslimalor based on local polynomial smoolhing are available, see e.g., Pan et al. (996]) and Gooijer & Zerom 2003). See Izbicki & Lee (016)) for an estimator in high-dimensional
regression setting.
As a summaar, we have Algorithm 1 which provides an efficient implementation for the proposed LTF. Although the algorithm is described perform a single update of at each sampling step, multiple updates are also allowed. This does not interfere with the convergence theory of the algorithm.
Algorithm Latent Trajectory for A2C (LT-A2C)
1: Step 0 (Initialization). Tnitialize actor network #eo with learning rate sequence {1r,, and initialize critic network Vig with learming rate sequence {ex}
2: for k = I,...,K do
3: Generate trajectories (kk) = (ar (k) (m)
Ji=1 and returns R(() = " with policy TBk-1
4: Step (Auxiliary sampling). Draw auxiliary d-samples.
5: for j =;=1.... I,..., m do
6: Presetting: If j = 1, set di = W
Jي أي + 6y v log (6)2))))) +
(12)
where 6j the learning rate, V log (W)o(())) El V log (1) Ri,) is calculated based on (10 using L auxiliary
samples of R drawn from x Rly,, ,()),, and & 8,)b))
7: end for
8: Step (Policy Evaluation). Sampling the critic network d/k Computing importance weight: calculate
Ti T(R()) (o))) 1h--1
(13)
ΣT 7 (2((k) v3))
10: Sampling: Draw en N, Beulp) and caleulate
" "VIL(On-I. Oks ve x
2
where the gradient term is given by
V VeL(Ox-1, = 10
V log ( lat (K)
x)- N V log log
(14)
[=
Step (Policy Control). Updating the actor network 0x-11 Compute advantage functions [ and update
0
02 0k-1 Ui Σ Ave (x) , (1) (1) log TH (0 (K)).
12: end for
Furthermore, the proposed LTF is highly ficxible and can be scamlessly integrated with various actor-critic learning strategies. Specifically, we can modily the critic network distribution ( Re, and/or the policy gradient to align with different actor-critic learning strategies. For instance, we replaced the A2C policy gradient with the one used in PPO, resulting the LT-PPO algorithm, as demonstrated in our numerical experiments. Potential alternatives for the critic network distribution will be discussed in the final section of the paper.

---

Published in Transactions on Machine Learning Research (09/2025)
3.3 Convergence Theory
In the proposed LTF, we simulate thn T(bx(0x-1) with SGMCMC algorichms while Os changes from iteration to iteration. We establish the Le-convergence of {0s h = .. and the Wg-convergence of fel : h = 1, ... in Theorem 3.1 and Theorem 3.2 respectively. This implics that the actor network achieves an optimal policy, while the critic network converges weakly to the stationary distribution (())0").
Theorem 3.1 (Convergence of 0&) Suppose Assumplions A1 A5 (in Appendir B hold, and the sample size of auriliary s-samples is sufficiently large. Set the learning rate sequence {ek] and the step size sequence {2x le in the form:
C.
C.
c
c + ka
Cy + kD
(15)
for some constants C. > 0, c. 0, C. a root 0* E (0 g()) = 0} such that
0, Cu
0, a, E (0, I], and B s min Then there exists
0 - 1-11² S EvR,
k k
(16)
for some constants ko 0 and {= λu + 21222(11111111 ) Go. where Ao denotes constant, S2 is defined in Assumplion A3 and G and Go are given in Lemsma B.2
The constants Gy and Gg depend, respectively, on the critic and actor networks, including their dimensions and structures. It worth noting that condition (15) requires the learning rate of the critic network decay no slower than that of the actor network, ensuring that valid samples of critic networks are used for the update of the actor network in the later period of the simulation. This aligus with conditions commonly seen in iterative optimization methods, which typically require accurate estimate of the value function, see c.g., the double-loop methods (Wang et al. 2020 Cayci et al. 2022
Let n" (v+)0",, let T = Ei=1 G, and let μT,, denote the probability law of t/. Theorem 3.2 establishes convergence of μr,, in 2- W-ssserstein distance.
Theorem 3.2 (Wz-convergence of vkk) Suppose Assumptions
A1 A6 (in Appendia B hold, the sample
size of auailiary 1 (o-ammllss is sufficiently large, and the sequences Then, for any ls E N,
ck and {ux)a=1 are set as in Theorem
3.1
Wo(uT, x') S (C66l/4
4 + CLu TT + Cze-Tn/ess
(17)
for some posilive constants Co Ći, and O2, where Wa(-,)) denotes the 2- W-sassrsleei distance, ets denotes the logarithmic Sobolev constant of x", and ôú is a coefficient defined in Assumption A3 and reflects the variation of the stochastic gradient V (()
The hand side of can be made sufficiently small by selecting a large enough mini-batch size, a sufficiently small value of UI, and large number of iterations.
We prove Theorems 3.1 and 3.2 by following the proofs of adaptive SGLD provided in Dong et al. 2023
and Liang et al. (2025) respectively (see Appendix B). The proof for the expression of can be found in Theorem A.1 of Dong et al. (2023). It is worth noting that the proposed LTF can also be implemented using an adaptive SGHMC algorithm (Liang et al. 2022). In this case, Theorems 3.1 and 3.2 can still be established similar to the convergence theory presented in Liang et al. (2022)
4 Experiments
In this section, evaluate the performance and effectiveness of the proposed LTF in enhancing deep actor- critic RL. We conduct experiments in two environments: the Indoor Escape Environment, where we demon- strate the performance of the L.Fneeh deep actor-critic methods in uncertainly quantification; and the PyBullet Enviromment (Ellenberger, 2018 2019), where we compare the performance of LTF-enhanced deep actor-critic methods to their vanilla counterparts on continuous control benchmarks, demonstrating the flexibility of the LTF integrating different RL strategics enhance deep actor-critic training. These experiments highlight the improvements in deep actor-critic RL achieved through the adoption of the I.F..
7

---

Published in Transactions on Machine Learning Research (09/2025)
4.1 Indoor Escape Environment
Figure 2 depicts a simple indoor escape environment (Shih & Liang 2024), where the state space consists of 100 grids and the agent's objective is to navigate to the goal positioned at the top right corner. The agent starts its task from the bottom left grid at time = O. For every time step t, the agent identifies its current position, represented by the coordinate & E {(i,j): = ,,..., 10]. Given a policy wo, the agent chooses an action {N, S, W} with respect to the probability ne (a(ss)) The action taken by the agent determines the adjacent grid to which it moves. Following cach action, the agent is awarded an immodiate reward, Tt: drawn independently from the Gaussian distribution (V(-100.01.
We evaluate the performance of the proposed method from three aspects: (i) Policy Diversity: The policy, coded by the actor network and denoted as Te(a))), should converge to a distribution that assigns equal probabilities optimal actions and zero probability critic network is expected to accurately approximate the state value function (s)s) across the entire state space. (iii) Value Uncertainty: Algorithms should be capable of quantifying the uncertainty associated with the value function.
others. (ii) Value Accuracy: The
To quantify policy diversity, we define the optimal policy distribution x" (+1s) to a probability distribution over all actions at state s, which is uniform over optimal actions and zero on sub-optimal actions. For a given policy no(+ss), the Kullback-Leibler (KL) divergence between n² and ma, denoted by can be used to measure the diversity of the policy distribution. It's worth noting that for most states, actions N and E are identically optimal. Hence, the policy me Talss) should assign equal probabilities on these two actions. Figure visualizes the policy distribution To at each state 8. The left plot shows that the LTF achieves a nearly optimal policy distribution at each state 8, which has a small value of (xl(==([mm)) The right plot shows the policy distributions achieved by A2C, which is severely biased toward single policy and has large value of (x= Refer Appendix C for the experimental settings.
Str
Figure 2: Indoor escape envi- ronment
DKu(T
Suppose that the actor network converges to fixed policy man, and the state value function (s) coded by the critic network should be dis- tributed around the optimal value function (s).. To evaluate such estimation, we collect the last 1000 critic parameter updates to form pool of di-samples, denoted by vb = ( which naturally in- duces a sample pool of values V, (Vo()) e v.)). From the value sample pool, we can obtain a point estimate of the state value at state by calculating the sample average ((s) - Σ Vi, (s, a). For inter- val estimation, we can achieve one-step value tracking by constructing 95% prediction interval with the state value samples. We replicate each experiment 100 times and calculate the following three metrics: (i) the KL-divergence between x² and Tg, denoted by De (xr" (T=),, (ii) the mean squared error (MSE) between (s) and (s),), defined by MSE(V) E.Es (s) - (s))², where B(-) denotes the empirical average over the state space S; and (iii) the coverage rate (CR) of the 95% prediction intervals.
Action Probabi
Action Protabl by
1.0
08
Figure 3: (Left) Policy distribution each state 8, achieved by LTF in single run; (right) Policy distribution at each state s, achieved by A2C in single run.
Table demonstrates that, when enhanced with LTF, both A2C and PPO show substantial improvements across all three performance metrics. Compared to the vanilla A2C and PPO methods, which exhibit notable bias in value function estimation, the L.Te-en methods yield more consistent estimates and offer
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Comparison of A2C, PPO, AT-A and LT-PPO for Indoor Escape. The number in the parentheses represents the standard deviation of the corresponding metric, which was calculatod by averaging over 100 independent runs for each method.
Algorithm
N
DRL( [|
MSE(V)
Coverage Rate
CI-Width
A2C
4.647 (0.0729)
0.53527 (0.03974)
0.489 (0.0061)
0.413 (0.0023)
LT-A2C
10000
0.010 (0.0010)
0.00038 (0.00001)
0.947 (0.0004)
0.457 (0.0009)
LT-A2C
20000
0.014 (0.0014)
0.00039 (0.00001)
0.947 (0.0004)
0.452 (0.0010)
LT-A2C
40000
0.014 (0.0013)
0.00033 (0.00001)
0.947 (0.0004)
0.449 (0.0009)
PPO
4.773 (0.0893)
0.56112 (0.04272)
0.487 (0.0066)
0.416 (0.0024)
LT-PPO
10000
0.011 (0.0010)
0.00041 (0.00001)
0.947 (0.0004)
0.458 (0.0009)
LT-PPO
20000
0.009 (0.0009)
0.00038 (0.00001)
0.947 (0.0005)
0.452 (0.0009)
LT-PPO 40000
0.011 (0.0011)
0.00032 (0.00001)
0.947 (0.0004)
0.449 (0.0008)
reliable uncortainty quantification. Across all three performance metries, LT-A2C and LT-PPO consistently outperform the original A2C and PPO methods, yielding lower MSE values, reduced KL-divergence, and improved coverage rates. The lower KL-divergence values suggest that the policy distribution more effectively converges to uniform distribution over optimal actions, thereby enhancing exploration efficiency across the state space. These findings can be further visualized in Figure AT and Figure A2 The former summarizes the results on KT-divergence and CI-Width, and the latter summarizes the results on MSE(V) and coverage rates. For MSE(V), the LTF-enhanced methods exhibit significantly smaller values and tighter box plots, reflecting greater training stability. In terms of uncertainty quantification, only the LTF-enhancod methods achieve coverage rales close to the nominal 95%. Furthermore, as the pseudo-populattin size increases, uncertainty quantification for the value function becomes more accurate, as indicated by reduced MSE values and narrower confidence interval widths.
Many existing methods claim the ability to quantify the uncertainty of the critic network. These methods often rely on approaches such as Bootstrap DQN, Quantile Regression DQN (distributional RL), and Ran- dom Prior Networks (RPN). While these methods provide estimates of uncertainby, they fail to construct statistically honest confidence intervals for the Q-function. As demonstrated in Table 2 we evaluated these methods in the Indoor Escape environment using a single neural network to approximate the Q-value func- tion. None of them was able to accurately construct the 95% confidence interval for the Q-value, highlighting their limitations in uncertainty quantification: when neural networks are involved, the existing methods can be challenging to calibrate, often leading to inaccurate coverage rates of Q-values. This raises significant concerns about their ability in quantifying uncertainty for more complex actor-critic models.
Table 2: Comparison of different methods for Indoor Escape: (i) BootDQN: Bootstrapped DQN (Osband et al.] 2016), (ii) Quantile regression (QR) DQN: Distributional RL (Bellemare et al. 2017), (ii) RPN; Randomized Prior Networks (Osband et al. 2018). The number in the parentheses represents the standard deviation of the corresponding metric, which was calculated by averaging over 100 independent runs.
Algorithm
MSE(2)
Coverage Rate
CI-Width
BootDQN QR-DQN
0.09979 (0.01609)
0.388 (0.0186)
0.188 (0.0032)
0.00459 (0.00028)
0.821 (0.0089)
0.278 (0.0033)
RPN: prior scale=0.1
0.03339 (0. (0.00290)
0.802 (0.0147)
0.679 (0.0243)
RPN: prior scale=1.0
0.03724
(0.00412)
0.816 (0.0157)
0.693 (0.0243)
RPN: prior scale=5.0
0.03658 (0.00297)
0.793 (0.0190)
0.782 (0.0341)
In the Indoor Escape environment, each episode is capped at 200 steps in our implementations. As result, reaching the goal without relying on additional exploration techniques like c-greedy - depends hcavily on the ability of the method in performing parameter space exploration. Figure 4(a) plots the best-so-far reward versus time steps for each method: LT-A2C reaches the global optimum (-18) in every run, whereas A2C fails to do so a substantial fraction of runs. Figure 4b) reports the proportion of experiments which each method successfully attained the optimal policy: LT-A2C reached the optimal policy in 100% of the runs, whereas A2C succeeded in only 78% of the runs. The comparison highlights the capability of the
9

---

Published in Transactions on Machine Learning Research (09/2025)
LTF in parameter space exploration and suggests that it substantially outperforms A2C in identifying the optimal policy, while accurately quantifying the uncertainty of the resulting values.
Regarding computational complexity, we note that although LTF requires additional SGMCMC sampling for the critic network at each iteration, the per-iteration complexity of SGLD and SGHMC is comparable to that of SGD. As a result, the overall time complexity remains on par with standard iterative optimization methods, supporting the scalability of the proposed method. The LTF methods can be readily applied to large-scale neural networks.
Max Rewards Timsteps
Cumulative Success Rote (100 experiments)
1.0
-5-
0.6-
-75
0.6-
-100
A2C
1125-
Cumuatus
044
-150
-175
0.78 11 1.0
-200
0.2
(a) Maximum reward achieved along time steps
(b) Successs rate of achieving optimal policy
Figure 4: Optimal policy exploration for the Indoor Escape environment.
4.2 PyBullet Environment
To demonstrate the applicability of the I.TF-enhanced actor-critic RT, methods in complex environments, we evaluate their performance on continuous control benchmarks using the RL Baselines3 Zoo (Raffin 2020) and the PyBullet environment (Ellenberger 2018-2019). We compare the LT-A2C and LT-PPO methods with their vanilla counterparts (A2C and PPO, respectively) across four continuous control tasks: HalfCheetah, Hopper, Reacher, and Walker. The hyperparameters for A2C and PPO follow the default configurations
from RL Baselines3 Zoo [Raflin 2020). Additional experimental details are provided in Appendix g
Figure 5 presents the best reward up to the current iteration obtained by these methods for the control tasks. Each curve in the plots represents the mcan of the best-so-far reward obtained by the corresponding method in 100 independent runs. The shaded region indicates the 95% confidence interval of the expected return, computed as the mean +1.96 times the standard error. The comparison shows that both A.T-22C and LT-PPO outperform their vanilla counterparts in Walker-2D) while performing comparably in other control tasks. This underscores the flexibility of the proposed LTF, which allows for the integration of various advanced RL strategies to enhance actor-critic training while ensuring proper uncertainly quantification for the critic network and the resulting value function. Specifically, for PPO, its gradient clipping mechanism prevents the actor's policy from changing too drastically, helping ensure stable updates and improving the training of the actor-critic networks, even under the proposed LTF.
In terms of computation, the LT framework introduces an inner loop that samples the critic network (Algo- rithm [), which increases training cost. However, the number of sampling steps need not be large; 10 steps typically suffice. To assess how this hyperparameter affects wall-clock time and performance, we conducted an ablation over the number of sampling steps, with results presented in Figure [ The comparison shows that reward improvements slow after about 10 steps. Accordingly, we recommend using 10 sampling steps to reliably achieve good performance while maintaining efficient training.
A close inspection of Figure G shows that LT-A2C's performance continues to improve as the number of critic-network sampling steps increases. This, in turn, suggests that a high-quality SGMCMC algorithm could further enhance the method's performance, though it is not strictly essential.
10

---

Published in Transactions on Machine Learning Research (09/2025)
HallChestahBuuleeEEn-vv
HopperBulletEnn-vv
2500
2500
2250
2000
2000
1750
1750
1500
1500
1250
APC
A2C
1000
LTAZC
1000
ITC
PPO
PPO
ITAPO
750
TENO
1.0
0.4 training progress
training progress
ReacherBulletEnv-vo
Walker20Bullettnnw-o
25
1750
15
reward
1250
10
1000
A2C
A2C
750
PPO
PPO
-5
0.2
0.6
0.8
1.0
0.2
0.4
1.0
training progress
training progress
Figure 5: Comparison of A2C, PPO, LT-A2C, and LT-PPO for the PyBullet environments: training, evalu- ation/testing /esting, and best/highesi rewards (up the current iteration) achieved during training. The results were summarized from 100 independent runs for each method.
5 Conclusion
In this paper, have introduced novel Latent Trajectory Framework, coupled with adaptive SGMCMC algorithm, for training deep aclor-critic worksr The proposed method naturally captures the interdepen- dence between the actor and critic updates. We provide theoretical guarantees that, under mild conditions, the proposed method ensures consistency in parameter ostimation for the actor network and weak conver- gence in parameter sampling for the critic network. Compared to existing ilerative optimization methods, the proposed method ensures accurate uncertainty quantification for the critic network and, consequently, the value function, while offering greater flexibility and robustness in training the actor network. Exist- ing Bayesian methods generally lack the mathematical rigor required to achieve this level of uncertainty quantification for deep actor-critic networks. Notably, the proposed method accomplishes these improve- ments without increasing the order of computational complexity compared to existing iterative optimization methods, making it scalable to large-scale DNNs.
In searching for high-reward policies, LTF is expected outperform the vanilla baselines when the landscape of the actor-network objective rugged. The added randomness from sampling critic networks compared with iterative optimization facilitates escape from local optima and speeds convergence to higher-reward policies. On smoother landscapes, LTF and iterative iiaimiiztion-baseed methods perform similarly, with no loss from sampling.
The proposed LTF is also highly flexible. In addition to replacing the SGLD steps with SGHMC (Chen) et al. 2014) or substituting the SGD step with Adam (Kingma Ba 2014), one can modify the policy control step and/or the critic network distribution to incorporate advanced RL strategics, further accelerating convergence toward high-reward polices. In this paper, we have demonstrated that substituting the policy control step with PPO update significantly enhances the training of actor-critic models in complex
11

---

Published in Transactions on Machine Learning Research (09/2025)
HallChesttaBuulleeEnvvv
HopperBulletEnn-vv
2500
1600
2250
1400
1750
1500
1250
1000
APC
A2C
1000
800
IT-A2C (samp5)
LTA2C (semp10)
UTA2C (semp10)
LT-22C
(samp15)
LFA2C Isemp15)
1000
2500
3000
3500
1000
1500
2000
2500
3000
well time
wall
ReacherBulletEnn-vo0
Waller22DBulletEnv-v00
20
1000
15
950
5-
A2C
A2C
LT-A2C (camp5)
(samp5)
LTA2C (samp13)
LT-A2C (samp10)
(samp15)
(semp15)
-5
800
500
1000
1500
2000
500
1000
1500
2000
2500
3000
3500
wall time (s)
wall time Is]
Figure 6: Comparison of A2C and LT-A2C (with varying numbers of sampling steps) on PyBullet environ- ments. The plots show the best-so-far rewards (up to the current. wall-clock time) during training. Results are averaged over 100 independent runs for each method.
environments. Similarly, the critic network distribution can be replaced with one derived from the critic update of an advanced RL stralegy. For example, by leveraging the randomized ensembled double Q- learning (REDQ) strategy (Chen et al. 2021),, we can replace the critic network distribution a R,, xx) used in g with the following ensembled version:
N
T ( xx) exp
Σ S.. (s.a.r,a")BB
Σ
(s, - -y) T
B
where T f1/1 the, ..., vN denotes an ensemble of critic networks, B represents a mini-batch with size [B, and y is defined as in Algorithm 1 of Chen al.) (022)) The detailed definition of is omitted here for simplicity. Based (OAC) (Ciosek et al. 2019), the critic network distribution
other strategies, such as SUNRISE (ee et al.) 2021 and Optimistic Actor-Critic also be defined accordingly. In summary, the proposed LTF is able to accelerate the search for optimal policies by leveraging the strengths of modern RI. strategies as well as the inherent randomness from sampling critic networks, while providing calibrated uncertainty quantification for the critic network and value functions. It is readily applicable to diverse actor-critic training tasks.
Acknowledgments
Liang's research supported in part by the NSF grant DMS-2210819 and the NIH grant 201-GM152717. Shih's research is partially supported by MSK Cancer Center Support Grant/Core Grant (P30 CA008748).
12

---

Published in Transactions on Machine Learning Research (09/2025)
References
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning, 2017.
Albert Benveniste, Michel Métivier, and Pierre Priourct. Adaptive Algorithms and Stochastic Approzima- lions. Springer-Vellag, New York, 1990.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15, pp. 1613-1622. JMr.R.org, 2015.
Semih Cayci, Niao He, and Rayadurgam Srikant. Finite-time analysis of entropy-reeullaizeed neural nat- ural actor-critic algorithm. ArXiv, (bs/22206.00833 2022. URL https://api htw.aap....aaatiacccocoarr..o.
CorpusID:249282240
Tianqi Chen, Emily B. Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In ICML, 2014.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learn- ing fast without a model. ICLR, 2021. URL https://appl https::/app..eemaart sea.aacccoaaamaittccccccooaaa org/CorpusID :t3pppuss..:33217737337273 231627733
Xuyang Chen, Jingliang Duan, Yingbin Liang, and Lin Zhao. Global convergence of two-timescale actor-critic for solving linear quadratic regulator, 2023.
Kamil Ciosek, Quan Ho Vuong, Robert Tyler Loftin, and Katja Hofmann. Better exploration with optimistic actor-critic. In Neural Information Processing Systems, 2019. URL htcp .rg/ CorpusID:202766222
William R. Clements, Benost-Marie Robaglia, Bastien van Delft, Reda Bahi Slaoui, and S'ebastien Toth. Estimating risk and uncertainty in deep reinforcement. learning. ArXin, abs/ abs/1995.09663 2019. URL
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learn- ing with quantile regression. ArXiu, abs/ 10001100.0044, 2017. URL https://apl.
Corpus
Wei Deng, Xiao Zhang, Faming Liang, and Guang Lin. An Adaptive Empirical Bayesian Method for Sparse Deep Learning. In NeurIPS, 2019.
Tianning Dong, Peiyi Zhang, and Faming Liang. A stochastic approxinaation-angeviiee ensemble kalman filter algorithm for state space models with unknown parameters. Journal of Computational and Graphical Statistics, 32(2))448 469, 2023. doi: 111110//00116000222221777533 URL 10618600. 2022.22222 2107531
Benjamin Ellenberger. Pybullet gymperium. atttp:://gttuub..mmm//oao//////p
gym
2018-2011
Yaakov Engel, Shic Mannor, and Ron Mcir. Bayes meets bellman: the gaussian process approach to temporal difference learning. In Proceedings of the Twenlieth International Conference on International Conference on Machine Learning, ICML '03, pp. 154 161. AAAI Press, 2003. ISBN 1577351894.
Jiaqqin Fan, Qiwei Yao, and Howell Tong. Estimation of conditional densities and sensitivity measures in nonlincar dynamical systems. LSE Research Online Documents on Economics, 1996. URL https: . seappaiimmnt www.aaarrrrrCoooooroaeeeaa ID: 17065829
Matthieu Geist and Olivier Pietquin. Kalman temporal differences. J. Artif. Int. Res., 39(1):483 532, sep 2010. ISSN 1076-9757.
Jan G. De Gooijer and Dawit Zerom. On conditional density estimation. Statistica Neerlandica, 57, 2003.
URL https://app. https://api.smmattiico www.nticcccaalarr....... wtwpccooiaaarrrrg//oouusulll.ai. 122069058
13

---

Published in Transactions on Machine Learning Research (09/2025)
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum
entropy deep reinforcement learning with stochastic actor. ArXiv, abs/ ftss/1801.01200 2018. URL https:
Markus Holzleitner, Lukas Gruber, José Arjona- Media,, Johannes Brandstetter, and Sepp Hochreiter. Con- vergence proof for actor-critic methods applied to ppo and rudder, 2020.
Rob J Hyndman, David M. Bashtannyk, and Gary K. Grunwald. Estimating and visualizing condi- tional densities. Journal of Compulational and Graphical Slatistics, 5:315-336 336, 1996. URL. https:
//api.semant seappaicmmnttc
Prr//orppsID:121000 121045552
Rafael Izbicki and Ann B. Lee. Nonparametric conditional density estimation in a high-dimensional regression sctting. Journal of Computational and Graphical Statistics, 25:1297 1316, 2016. URL https://api.
semanticscholar.org org/ Sttp:/oorpulDD:2887770 bbg
Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continuous deep reinforcement org//orpusID otg.eaaatttccccoo...orrooooooopa
learning. In Conference on Robot Learning, 2017. URL
4083410
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.600 69800 2014.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 6405 6416, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Kimin Lee, Michael Laskin, A. Srinivas, and Abbeel. Sunrise: simple unified framework for ensem- ble learning in deep reinforcement learning. ICML, 2021. URL https://api. http://pp...mmatn.ccccccccllla..
ID:
F. Liang, Sehwan Kim, and Yan Sun. Extended fiducial inference: Toward an automated process of statistical inference. Journal of the Royal Statistical Society, Series B, (7(1):98-113) 2025.
Siqi Liang, Yan Sun, and Faming Liang. Nonlinear sufficient dimension reduction with a stochastic neural network. NewrIPS, 2022.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2019.
Vincent Mai, Kaustubh Mani, and Liam Paull. Sample efficient deep reinforcement learning via uncer- tainty estimation. ArXie, abs/2201.01666 2022. URL https://api. wwwantcschoollnn...m.o.o Corp/orpp
245704549
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1928 1937, New York, New York, USA, 20-22 Jun 2016. PMLR. URL httts:://pocceeeeeee wwriiinnnssm.r....mmm www.m pttet//////////mmmmee.m mtt.8//mniaaalt html
Thomas M. Mocrland, Joost Brockens, and Catholijn M. Jonker. Efficient exploration with double uncertain value networks. ArXiv, abs/1711.10789, 2017. URL https::/aapi.smmanticcccooo ottpa//atcccccccoooooooooooooopo
21565462
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (ods.), Advances in Neural Informa- tion Processing Systems, volume 29. Curran Associates, Inc., 2016. wttp.//////rpppeppoooaaago..o.e cc/paper__ htt.a//www..a.....aaaaaaaaaaaaaa
14

---

Published in Transactions on Machine Learning Research (09/2025)
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Proceedings of the 32nd International Conference on Newral Information. Processing Systems, NTPS'18, pp. 8626 8638, Red Hook, NY, USA, 2018, Curran Associates Inc.
Oren Poer, Chen Tessler, Nadav Merlis, and Ron Meir. Ensemble bootstrapping for Q-learning. In Inter- national Conference on Machine Learning, 2021. URL httpp:://ap..senaattccctttt http://oolllaooooooopoosssseee/e org/CorpusID:
232076148
Antonin Raffin. RI baselines3 zoo. http:///gihhu...oo///---/-111/i1
2020
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-basclines:: Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1 8, 2021. URL http::///jal..orr/aapper////////
Herbert Robbins and Sutton Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22:400-407, 1951.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms, 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018.
Shirli Di-Castro Shashua and Shie Mannor. Kalman meets Bellman: Improving policy evaluation through value tracking. ArXiv, abs/ 20002002.0717,, 2020.
Frank Shih and Faming Liang. Fast value tracking for deep reinforcement learning. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL. https:://openreviex net/forum?id= do
LZIOBA2oDU
Q. Song, Y. Sun, M. Ye, and F. Liang. Extendod stochastic gradient meme algorithms for large-scale bayesian variable selection. Biometrika, 107:997-1004, 2020.
Yan Sun, Qifan Song, and Faming Liang. Consistent sparse deep learning: Theory and computation. Journal of the American Statistical Association, 117:1981-1995, 2022.
Yan Lindsay Sun, Wenjun Xiong, and Faming Liang. Sparse deep learning: new framework immune to local traps and miscalibration. NewrIPS, 2021. URL https: wow.ae/teeeeoooo.....oooorrrpppp org/CorpusID: 238259983
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, sccond edition, 2018. URL http:///nccoppetoideess http://hoonpp.tttaaa.a.eeeeorbbr 2nd. html
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for re- inforcement learning with function approximation. In Advances in Neural Information Processing Systems (NIPS), pp. 1057-1063, 2000.
Baharch Tasdighi, Manuel Haussmann, Nicklas Werge, Yi-Shan Wu, and Melih Kandemir. Deep exploration with pac-bayes, 2024. URL
W.Y. Teh, Alexandre Thiery, and Sebastian Vollmer. Consistency and fluctuations for stochastic gradient langevin dynamics. Journal of Machine Learning Research, 17:1-33, 2016.
Haoxing Tian, Toannis Ch. Paschalidis, and Alexander Olshevsky. Convergence of actor-critic methods with multi-layer neural networks. NewrIPS, 2023. URL .ttp://cp...mmmtticcccccc....... org/CorpusID: 274397963
Charles Edison Tripp and Ross D. Shachter. Approximate kalman filter q-learning for continuous state-space mdps. ArXio, abs/ 1/////1399.0888 2013. httpt:tt////t
15

---

Published in Transactions on Machine Learning Research (09/2025)
Lingxiao Wang. Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global op- timality and rates of convergence. ICLR, 2020. URL ccctps/// org/CorpusID: ottpr//cccoooaoooooooorroroooo 202121859
Yue Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite time analysis of two time-scale actor critic methods, 2022.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Appendix
A Derivation of Equation (6)
V log (ognN(d100011
/ Vp log (v (k) On (ax Ib On Oa-1 1ddx d
/ v. log (og R(()
T (xx(1) ((()) wek OR 0k-1)
(2x)k)), R(()) (0=000011 x (b) Rk( 10:- 1) dR((k)
(A1)
/ Vp log (V R()))
(IR( ()) (k), 1 On TT (ax
/un, 0k-1)
n R ax (R), On n (k--)) n (xx
R 10=-1 (8) (.R())))
/ V log (1 R)
π R lx: (A) wha) (x (K) R 1 1x-11) dR(()
TT (k))
x (8))
provided that the mini-batch size n has been chosen to be sufficiently large, ensuring that x(*) serves as a good representative of the underlying pseudo-population. This guarantees that t/e conditionally independent of 0Ş-1 given (k)), see Figure | for the graphical illustration of the latent Markov sampling process. Consequently, x (1bel0x-1). The former leads to the second equality in (A1 while the latter leads to the last equality in
have v log (vv (xx(k) R((), 02-1) V log ( (zx()) RR(()) and T(tx/ (x())))(())))))))))))))))))
A1 by the following equality:
π ( (k) Ruk. Ox-1)
T(V2lx(1) 0k-1)
(1tx (z0 1.
(A2)
(k) (k=1))
oT (1/x (x-11 o (1/a (0x-1)
B Proof of Theorem 3.1 and Theorem 3.2
For convenience, we denote the Irajectory observation (ax(k) R(8)) as zk, and assume ZŞ 2 be a compact set. The Latent Trajectory Framework can be written in general form
1 -4- + ex V L(Ox 11- + 22RER
(A3)
0/ 0 (xğ(Ok- thk, 2k),
where EN denotes the learning rate, ex is a standard Gaussian noise, VpL(On-1 1,-1, zx) denotes an unbi- ased estimate of ((0--1 1)) v log (6- and g(0x-1, 2/k 2x) is an unbiased estimator of g (0x-1). Convergence of adaptive stochastic gradient MCMC algorithms has been studied in Deng et al.
(2019). Dong et al. (2023) and Liang et al. (2025). The convergence theory of LTF can be established by slightly modilying some of the assumptions used therein.
Notation: We use Eg [u(O, b)] to denote the expectation of u(O, vb) with respect to the conditional distri- bution xW)()), and use E[u(-)] to denote the expectation with respect to the joint distribution of all the variables involved in the integrand u(-).
Assumption A1 The step size sequence JeEN is a positive decreasing sequence of real numbers such that
20 Uk 3c.
lim Uk 0, k-000
(A4)
There exist > O and stationary point 6* such that for any f E θ
00 0" (()))
- 0-
17

---

Published in Transactions on Machine Learning Research (09/2025)
where g(0) Ex and, in addition,
lim inf 26
Uk
+
Dk+1
Uk
0,
(A5)
R-00c
Uk+1
v
where || denotes the L -norm.
Assumption A2 L(0, 1) M-smooth on and with M > 0, and (m,b), b)-dissipative on &b for some constants > and b > 0. In other words, for any up a s/" T and 0,0' E the following inequalities
are salisfied:
V E M - V + M - 11).
(A6)
(VoL(0) 2) 1) I - m||b
(A7)
where 6* stationary point as defined in Assumption
Assumption is critical and standard assumption in the convergence of SGMCMC algorithms. In the context of deep neural networks, the dissipativity condition can be easily achieved by imposing Gaussian prior on the critic network parameter, which further guarantees convergence.
Lemma B.1 [(., s 3M + 3M31
0-||2 + 3B2 for some conslant B.
PROOF: Follow the proof of Lemma A.1 in Dong al. 2023
Assumption A3 Let Sa= = V v L(k,, thk, zk) V L(Ok, v'x). white noises, and they satisfy the conditions
Assume that SA 's are mutually independent
M - 1- + B²), B((a|F&) = O, EI|c < 6,
(A8)
where 0j and B are positive constants, and F& - o 10,, 11, O2 t2 O2 vh denotes a-filtration.
Assumption A4 There exist positive constants M and such that for all E 2,
10 4b = - M - 1- + 12 B2
where g(0,b, vb x) is as defined in A3).
By the formulation defined in section 3.2 let g(0) [ v, 2) and n -0(0, 2) g(0)) Since
( (lo,))llo(0 v/, =) 2l ((xa)) [|0], this implies Bl]g(0)112 I Elg(0, v. 2) and |/ s
E El v, =)1
Lemma B.2 (Uniform L bounds) Suppose Assumptions A1 A4 fied:
hold. If the following conditions are satis-
C.
C
(A9)
C. + ka
C, + KB'
for some constants C. > c. > 0, C. 0, Co > 0, a, (0, 1), and 8 S a S min Then there exist
constants G& and Go such that Bl[yvb| 2 Gz and E 1 Go for all 0,1,
PROOF: Follow the proof of Lemma A.2 in Dong et al. (0023). We slightly modify Assumption 4 in Dong ct al. (2023) by Assumption A4 where the stochastic gradient is replaced with (0). Then the proof is straight torward.
Assumption A5 (Solution of Poisson equation) For any 0 E O, + V, and function V( 1 || there exists function pe on V that solves the Poisson equation mo(D) - ToToo()) gg (0)), where To denotes probability transition kernel with (op Jw ( To( eo d')dt' such that
got. (0k) = -((0)) + ua. (1b+1) Te, Ma, (vh++1), k == 1,
(A10)
Moreover, for all 0, e and v E V we have (1)) per (1) + (T) To n (1)) S - (1) and [|,0(b)]] + [T)] I (16) for some conslants > 0 and <2 > 0.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Proof of Theorem 3.1
PROOF: For Algorithm assuine that the sample size of auxiliary vi-samples is sulficiently large, ensuring the denominator estimator in Eq. (13) converges almost surely to its mean value (Teh et al. 2016). Therefore,
the resulting stochastic gradient [14 is almost surely unbiased.
Dong el al. (2223) proved the result 16 for more general adaptive Langevinized ensemble Kalman filter (LEnKF)) algorithm, which is equivalent to an adaptive pre-conditioned SGLD algorithm. Extending their
proof to Algorithm straight forward.
Assumption A6 The probability law os of the initial hypothesis dg has a bounded and strictly positive density Po with respect to the Lebesgue measure on Bad, and
No = log
ello1 po( < 00.
R2'0
Proof of Theorem 3.2
PRoOF: This theorem is proved in Liang et al. (2025) with the same Assumptions A1 A6 For Algorithm we only need to assume that the sample size m of auxiliary ti-samples is sufficiently large, as explained in the proof of Theorem 3.1
C Experiment Settings
C.1 Escape environment
In this experiment, both Te and Ve are approximated by deep neural networks with two hidden layers of sizes (128, 128). The agent updates the network paramcters every 50 interactions, for a total of 10 action steps. Each experiment is replicated for 100 times. For initial exploration, an entropy penalty coefficient of 0.01 is added, and gradually decay to O. To achieve sparse deep neural network, we follow the suggestion in Sun et al. 2022 to impose mixture Gaussian prior onto both network parameters:
0,0 (1 - X)A ON
(A11)
where À E 10, is the mixture proportion and oi is usually set to small number compare to of. We set 0.01, = 0.001 and À = 0.5 in all LTF-enhanced algorithms. For indoor escape environment, the reward given by AV 0.01); that is, we set 6² 0.01. To make the estimated return y/2 = R₂ stationary, the reward at the goal state set to (M11, 0.01 ), where the discount factor ? = 0.9. guarantee the convergence of LTF, set the decay policy learning rate e 0 (gds) and constant critic learning rate ед = 2 x 10-4, The sample size L in 12 is set to 50, and the auxiliary sample size in [3 is set to 5.
In practical implementation, drawing samples from the conditional distribution (th (#)) can be performed with a short sub-loop of SGMCMC updates, which we set the length to be 10. That is, for each itcration k we repeat the v-sampling update times. The sub-loop sampling scheme is given by
s/a. thee--
kk 2
th
V log TI (R rhe.e1))
v V log (ogg(Weee))) + ex,e
(A12)
where the sub-loop is indexed by é. And the importance weight can be calculated by
π R (1) (k), v4.6-1)
(k)) (a(1)) 33)) + m (R(())r vhx.e-1) 1=
m++
where m denote the number of auxiliary samples and the importance weight is bounded by m +1 The boundedness of the importance weights wx,e's further ensures the stability of SGMCMC sampling step. We note that including the ther---term in the denominator is reasonable. As implied by the definition of
19

---

Published in Transactions on Machine Learning Research (09/2025)
=< (*) vt) *( ( }
the numerator term should be part the denominator and, therefore, we need to include LI as an auxiliary sample b. Furthermore, we refer to Theorem of Song et al. (2020) for the sample equally weighted formula calculating the denominator.
the importance weight wik
ITA0E: w-1 X 101
LLAZC: 2 107
Algorithi
CLAZE: Nwe4x I01
LTMMO: Mw
Kiddivereence
Alcorthin
PT 1-
LTMPO 2x
LT Aw 307
Erterva Famgee
Figure A1: KT-divergence (top plot) and interval widths (bottom plot) achieved by A2C, PPO, LT-A2C, and LT-PPO for the Indoor Escape example. The results were summarized from 100 independent runs for each method.
UTA2G x-11
Algoritim
LTA 2x
ITADE:
LT 10
LT 2x 10
LT LPO, www
10
10-7
10
MSE(V)
Agonthm
LT 1- 10"
LT A-
LT wwx.
Caverage rate
Figure A2: Top plot: MSE of V: Bollom plot: Coverage rate of the 95% prediction interval for the value (s)... The results were computed from 100 independent runs for each method.
C.2 PyBullet environment
In this experiment, we conduct experiments on PyBullet environments, including Ant, HalfChectah, Hopper, Reacher, and Walker2D. The training framework and hyperparameters of A2C and PPO are based on RL baselines3 zoo, and our LT-A2C and LT-PPO is implemented on top of Stable- Baselines3 Raffin et al. (0021). The hyperparameters are listed in Tables AT tables is set the default value used its corresponding vanilla counterpart. The actor and critic networks in AT-A2C and LT-PPO follow the default configurations of A2C and PPO, respectively, implemented in
A2 A3, and A4 Any hyperparameter not specified these
20

---

Published in Transactions on Machine Learning Research (09/2025)
the Stable- Baselinss Zoo. There are 2 types of learning rate, constant and linear decay. To balance between exploration and exploitation in L.T-enhanced algorithms, adopt annealing technique, where the pseudo population size increases as training steps increase, starting from 500. This method allows the algorithm to gradually shift from exploration to exploitation, improving overall performance and stability. A2C optimizes both the actor and critic networks using RMSprop, whereas PPO employs the Adam optimizer for both networks. In contrast, LT-A2C and LT-PPO update the actor network using RMSprop while performing SGHMC sampling for the critic parameters. For the prior distribution, both LT-A2C and LT-PPO adopt the same Gaussian mixture prior used in the Escape environment.
In theory, the auxiliary sampling step requires large sample size to guarantee good approximation. To improve the sampling efficiency of the auxiliary sampling step, we modify the approximation procedure of the importance weight. We replace the auxiliary samples bi 's with the SGMCMC samples v/x. derived in A12. The importance weight can then be approximated by
T R (1)
V4.8-1)
i A
D T (x((), vk,e)
With this modification, we can eliminate the auxiliary sampling step and further lower the computation complexity and memory complexity. The rationale behind this replacement is that the drift term V log (8)), R2()), used in AA12) provides an unbiased estimator of the drift V (( xx()) that governs the dynamics in 12 for simulations of auxiliary samples.
Table Al: Hyperparameters for A2C and LT-A2C
Environment
HalfCheetah
Hopper
Hyperparamctors
LT-A2C
A2C
LT-A2C
A2C
learning rate
lin 0.00067
0.00096
0.00042
lin 0.00096
a (obscrvation)
0.1
0.1
.V
50000
10000
"[discount factor)
0.95
0.99
0.99
0.99
gae-X
0.9
0.9
1.0
0.9
train batch
32
32
32
32
training steps
2c6
2c6
2c6
2c6
Table A2: Hyperparameters for A2C and LT-A2C (cont.)
Environment
Reacher
Walker2D
Hyperparameters
LT-A2C
A2C
LT-A2C
A2C
learning rate
lin 0.00096
lin 0.0008
lin 0.00037
lin 0.00096
(obscrvation)
0.1
0.1
.V
1000
500
(discount factor)
0.99
0.99
gae-X
1.0
train batch
training steps
2c6
2c6
21

---

Published in Transactions on Machine Learning Research (09/202))
Table A3: Hyperparamotors for PPO and LT-PPO
Environment
HalfCheetah
Hopper
Hyperparameters
LT-PPO
PPO
LT-PPO
PPO
learning rate
3e-5
3e-5
3e-5
3e-5
(observation)
0.1
0.1
.V
50000
50000
rddiscount factor)
0.99
0.99
0.99
0.99
gae-A
0.9
0.9
0.9
0.9
train batch
128
128
128
128
training steps
2e6
2e6
2e6
2e6
Table A4: Hyperparameters for PPO and LT-PPO (cont.)
Euvironment
Reacher
Walker2D
Hyperparameters
LT-PPO
PPO
LT-PPO
PPO
learning rate
3e-5
3e-5
3e-5
3e-5
(observation)
0.1
.V
50000
50000
(discount factor)
0.99
0.99
0.99
0.90
gao-À
0.9
0.9
0.9
0.9
train batch
64
64
128
128
training steps
2e6
2e6
266
2e6
22


Reviews:
[{"review_id": "Om1l3VFudF", "reviewer": "Reviewer_eHaw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- Deep actor-critic algorithms are important for the RL community.\n- Both theoretical and empirical validation of the proposed method.\n\n# Weaknesses\n\nThe paper suffers from three main limitations.\n\n## 1) The empirical validation is not significant.\n- Regarding the escape environment, is not clear why we should care about, e.g., the notion of policy diversity (which is not a meaningful objective in general, and the authors do not provide enough arguments in support of using it), and not about the expected return induced by the learned policy. This is seen also in Table 1, where it is not shown the difference in performance (i.e., expected return) between the policies learned through PPO/A2C and the proposed method.\n- Regarding the experiments in the PyBullet environment, that compare the expected returns of the policies learned by the various algorithms, it is clear from FIg. 7 that the std is quite large, so we cannot confidently conclude that the proposed method actually improves over the existing ones.\n## 2) The presentation is quite confusing.\nEspecially in Sections 2 and 3, where the technical contribution of the paper is presented, more clarity and intuition would be appreciated. To make some examples, in Section 2:\n- Why is $R_t$ \"denoted by $Q^{\\pi_\\theta}$\"? What does it mean?\n- What is the stationary distribution $\\pi(x|\\theta_{k-1})$? I don't get it.\n- What is an optimal policy? It is not defined.\n- Why do you include also $R^{(k)}$ in the transition trajectory?\n- What is a pseudo-population?\n- Why you say that the training is completed when Eqs. (3)-(5) are obtained? I mean, the goal is to maximize the expected return, not finding a point where the gradient is 0.\n\n## 3) Other parts of the submission lack quality.\n- In the discussion of related works, some are missing. E.g., \"Kalman meets Bellman - Improving Policy Evaluation through Value Tracking\".\n- Why do you claim that distributional RL is about uncertainty quantification? It is not. In distributional RL, you care about the distribution of the return as a random variable. Instead, in previous works on uncertainty quantification (e.g., \"Kalman Temporal Differences\"), the focus is on the distance between your estimate of the expected return, and the true expected return. These are two quite different objects.\n\n# Questions\n\nIn section 4.1, what is the coverage rate?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on how we can successfully quantify the uncertainty in our Q and value-function estimates in deep RL actor-critic algorithms. Specifically, prior works in uncertainty quantification in RL and deep RL (without actor-critic) have shown its importance. The idea of this paper is to introduce a novel framework that treats the transition trajectories as latent variables, and then use it to improve any existing actor-critic deep RL algorithm. The authors provide theoretical convergence results for the proposed method, and then conduct numerical simulations for validating it.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- The presentation of sections 2 and 3 should be changed almost completely.\n- Additional experiments that clarify precisely that the proposed method improves over the sota should be conducted.\n- Fix the other issues I identified in the block above.", "broader_impact_concerns": "/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SGbUtccyqE", "reviewer": "Reviewer_Dfgk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The core idea is nicely built upon previous work and advances in an interesting and underexplored area of uncertainty based RL methods\n- Decoupling the actor/critic by treating the trajectories as the latent variables is a neat idea and this work could lead to interesting follow ups that bridge it with Bayesian approaches\n- The flexibility to integrate the framework into a variety of actor critic methods increases its useability\n- The toy model evaluation helps showcase how and why the framework provides an advantage\n\nCons:\n- Figure 7 could be improved. Since each variant is connected/looked at with its LT pair, it’s worth visually connecting them in some way (e.g. make them the same color and vary the line style or something). The dpi should be higher (it looks a bit blurry if you zoom in). All three columns are not necessary, pick the column to use, then make the 4 environments in a row and move the rest to the appendix.\n- Too much time is spent on the toy model results. They are interesting and quite helpful to an extent, but 5 figures and 2 tables is overkill. Consolidate them to focus on the key points of explaining how and why the framework is advantageous, and move the rest to the appendix. Table 2 is important and very good to have and should center more in the explanations. The figures should highlight the answer to the question “why is it that existing algorithms fail to capture this in the same way?”\n- “Regarding computational complexity, we note that although LTF requires additional SGMCMC sampling for the critic network at each iteration, the per-iteration complexity of SGLD and SGHMC is comparable to that of SGD. As a result, the overall time complexity remains on par with standard iterative optimization methods, supporting the scalability of the proposed method. The LTF methods can be readily applied to large-scale neural networks.” This is a top question I had while reading the paper. While the per iteration complexity for many SGMCMC methods is comparable to SGD, the real question is how many iterations are necessary. To this end, the work (possibly in the appendix) would benefit from a wall clock/GPU hours plot comparison.\n- Related to the previous point, one key thing is how big of `m` is necessary to get good results. For the scaled up pybullet work, the appendix showed “With this modification, we can eliminate the auxiliary sampling step and further lower the computation complexity and memory complexity.” This seems like a pretty important aspect that is relegated to the appendix?\n- “In this experiment, we omit the reports for the uncertainty of the value functions, as their true values are unknown.” An interesting plot (although not one useful for quality comparison, but one that could be an interesting insight into the functioning of the algorithms), is to plot the uncertainty estimate for a variety of algorithms over training and compare what they look like (there isn’t a “right” or “wrong”, but might elucidate where some things happen).\n- There could be some additional comparison/literature included, some works such as https://arxiv.org/abs/1711.10789, https://arxiv.org/abs/1905.09638, https://arxiv.org/abs/2103.00445, https://arxiv.org/abs/2201.01666, https://proceedings.mlr.press/v78/kalweit17a/kalweit17a.pdf, etc. \n- The quality/importance of the SGMCMC algorithm is not discussed or ablated. It would be interesting to see how crucial the success of that algorithm is to the success of the outer loop.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a method for integrating uncertainty estimation of the value function into actor critic algorithms. By sampling the value function weights and weighting the update of the value function from these samples, this framework is able to incorporate actor informed uncertainty estimates into the training of the critic. This algorithm is shown to have certain theoretical convergences and is verified on a toy environment and a selection of pybullet environments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The key changes (outside of the minor citation/figure stuff) are:\n- Consolidation of grid results\n- Ensuring the necessary algorithmic adjustments are in the main text that are required for scaling\n- Further consideration of time/compute\n\nWhich are outlined in greater detail above.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lNefBi11nQ", "reviewer": "Reviewer_udfF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nOverall a well-organized, mathematically clear paper that is easy to follow.\n\n**Weaknesses**\n\n* The motivation of incorporating uncertainty quantification (UQ) into RL needs further elaboration, especially how it impacts current methods. As shown in Figure 7, there is very little difference between the performance of the original PPO algorithm and the LTF-enhanced PPO (LT-PPO) approach, suggesting that it might not be a serious issue in deep RL.\n\n* In Table A1 and A2, they show that LT-A2C and A2C use different hyperparameters (e.g., GAE factor $\\lambda$ and discount factor $\\gamma$). An ablation study is needed to examine their effects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a framework that treats transition trajectories as latent variables, which then motivates the adaptive Stochastic Gradient Markov Chain Monte Carlo algorithm to train deep actor-critic models, which naturally accounts for the interdependence between the actor and critic updates.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest the authors to motivate the problem studied in this paper in a better and more convincing way. For the empirical results, the authors should either use the same hyperparameter for both A2C and LTF-A2C, or conduct an ablation study to show that the performance improvement is due to the LTF-enhanced approach, not hyperparameter tuning.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)