SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first establish the necessary and sufficient conditions to guarantee monotonic policy improvement in the presence of critic error (Sec. 4.1). In Sec. 4.2, we prove that Algorithm 1 is guaranteed to converge to the neighbourhood (that depends on the critic error) of a stationary point.
4.1 Conditions for monotonic policy improvement
According to inequality (I), to guarantee monotonic policy improvement at iteration  $t$ , one must find a  $(\theta, c)$  pair to guarantee that the RHS of (I) is positive. In Prop. 2 (proved in App. D), we derive the conditions on the critic error to ensure that it possible to find such an  $(\theta, c)$  pair.
Proposition 2. For any policy representation and any policy or critic parameterization, there exists a  $(\theta ,c)$  pair that makes the RHS of inequality (I) strictly positive, and hence guarantees monotonic policy improvement  $(J(\pi_{t + 1}) > J(\pi_t))$  , if and only if
$$
\langle b _ {t}, \tilde {H} _ {t} ^ {\dagger} b _ {t} \rangle > \langle [ \nabla J (\pi_ {t}) - \hat {g} _ {t} ], [ \nabla_ {\pi} ^ {2} \Phi (\pi_ {t}) ] ^ {- 1} [ \nabla J (\pi_ {t}) - \hat {g} _ {t} ] \rangle ,
$$
where  $b_{t}\in \mathbb{R}^{n}\coloneqq \sum_{s\in \mathcal{S}}\sum_{a\in \mathcal{A}}[\hat{g}_{t}]_{s,a}\nabla_{\theta}[\pi (\theta_{t})]_{s,a},\tilde{H}_{t}\in \mathbb{R}^{n\times n}\coloneqq \nabla_{\theta}\pi (\theta_{t})^{\top}\nabla_{\pi}^{2}\Phi (\pi_{t})\nabla_{\theta}\pi (\theta_{t})$  and  $\tilde{H}_t^\dagger$  denotes the pseudo-inverse of  $\tilde{H}_t$ . For the special case of the tabular policy parameterization, the above condition becomes equal to,
$$
\left\langle \hat {g} _ {t}, \left[ \nabla_ {\pi} ^ {2} \Phi (\pi_ {t}) \right] ^ {- 1} \hat {g} _ {t} \right\rangle > \left\langle \left[ \nabla J (\pi_ {t}) - \hat {g} _ {t} \right], \left[ \nabla_ {\pi} ^ {2} \Phi (\pi_ {t}) \right] ^ {- 1} \left[ \nabla J (\pi_ {t}) - \hat {g} _ {t} \right] \right\rangle .
$$
For the Euclidean mirror map with the tabular policy parameterization, this condition becomes equal to  $\| \hat{g}_t\| _2^2 >\| \nabla J(\pi_t) - \hat{g}_t\| _2^2$  meaning that the relative error in estimating  $\nabla J(\pi_t)$  needs to be less than 1. For a general mirror map, the relative error is measured in a different norm induced by the mirror map. The above proposition also quantifies the scenario when the critic error is too large to guarantee policy improvement. In this case, the algorithm should either improve the critic by better optimization or by using a more expressive model, or resort to using sufficiently many (high-variance) Monte-Carlo samples as in REINFORCE [59]. Finally, we see that the impact of a smaller function class for the actor is a potentially lower value for  $\langle b_t,\tilde{H}_t^\dagger b_t\rangle$ , making it more difficult to satisfy the condition. The improvement guarantee in Prop. 2 holds regardless of the policy representation and parameterization of the policy or critic. This is in contrast to existing theoretical results [41, 28, 18] that focus on either the tabular or linear function approximation setting for the policy and/or critic, or rely on using expressive models to minimize the critic error and achieve good performance for the actor. Moreover, this result only depends on the magnitude of the critic loss (after updates), irrespective of the optimizer, step-size or other factors influencing the critic optimization. The actor and critic are coupled via the threshold (on the critic loss) required to guarantee policy improvement.
4.2 Convergence of Algorithm 1
Prop. 2 holds when the critic error is small. We now analyze the convergence of Algorithm 1 for an arbitrary critic error. Define  $\bar{\theta}_{t + 1} \coloneqq \arg \max_{\theta} \ell_t(\theta)$ ,  $\bar{\pi}_{t + 1} = \pi (\bar{\theta}_{t + 1}) = \arg \max_{\pi \in \Pi} \left\{ \langle \hat{g}_t, \pi - \pi_t \rangle - \left( \frac{1}{\eta} + \frac{1}{c} \right) D_\Phi (\pi , \pi_t) \right\}$ . Note that  $\bar{\pi}_{t + 1}$  is the iterate obtained by using the inexact mirror ascent (MA) update (because it does not use the true gradient  $\nabla_{\pi} J(\pi_t)$ ) starting from  $\pi_t$ , and that the inner-loop (Lines 10-13) of Algorithm 1 approximates this update. This connection allows us to prove the following guarantee (see App. D.1 for details) for Algorithm 1.
Proposition 3. For any policy representation and mirror map  $\Phi$  such that (i)  $J + \frac{1}{\eta}\Phi$  is convex in  $\pi$ , any policy parameterization such that (ii)  $\ell_t(\theta)$  is smooth w.r.t  $\theta$  and satisfies the Polyak-Lojasiewicz (PL) condition, for  $c > 0$ , after  $T$  iterations of Algorithm 1 we have that,
$$
\mathbb {E} \left[ \frac {D _ {\Phi} (\bar {\pi} _ {\mathcal {R} + 1} , \pi_ {\mathcal {R}})}{\zeta^ {2}} \right] \leq \frac {1}{\zeta T} \left[ \underbrace {J (\pi^ {*}) - J (\pi_ {0})} _ {\text {T e r m (i)}} + \sum_ {t = 0} ^ {T - 1} \left(\underbrace {\frac {1}{c} \mathbb {E} D _ {\Phi^ {*}} \left(\nabla \Phi (\pi_ {t}) - c \delta_ {t} , \nabla \Phi (\pi_ {t})\right)} _ {\text {T e r m (i i)}} + \underbrace {\mathbb {E} [ e _ {t} ]} _ {\text {T e r m (i i i)}}\right) \right]
$$
where  $\delta_t \coloneqq \nabla J(\pi_t) - \hat{g}_t$ ,  $\frac{1}{\zeta} \coloneqq \frac{1}{\eta} + \frac{1}{c}$ ,  $\mathcal{R}$  is a random variable chosen uniformly from  $\{0, 1, 2, \ldots, T-1\}$  and  $e_t \in \mathcal{O}(\exp(-m_a))$  is the projection error (onto  $\Pi$ ) at iteration  $t$ .
Prop. 3 shows that Algorithm 1 converges to the neighbourhood of a stationary point of  $J$  for an arbitrary critic error. The LHS of the above expression is a measure of sub-optimality similar to the one used in the analysis of stochastic mirror descent [65]. For the Euclidean mirror map, the LHS becomes equal to  $\| \nabla_{\pi}J(\pi_R)\| _2^2$ , the standard characterization of a stationary point. Term (i) on the RHS is the initial sub-optimality, whereas Term (ii) is equal to the critic error and can be further decomposed into variance and bias terms. The variance decreases as the number of samples used to train the critic (Line 4 in Algorithm 1) increases. The bias can be decomposed into an optimization error (that decreases as  $m_c$  increases) and a function approximation error (that decreases as we use more expressive models for the critic). Finally, Term (iii) is the projection (onto II) error, is equal to zero for the tabular policy parameterization, and decreases as  $m_a$  increases. Hence, the performance of Algorithm 1 improves as we increase both  $m_a$  and  $m_c$ .
In Sec. 5, we specify the step-size  $\eta$  such that Assumption (i) is satisfied for both the direct and softmax representations. Assumption (ii) is satisfied when using a linear and, in some cases, a neural network policy parameterization [33]. For the above proposition to hold, we require that step-sizes  $\alpha_{c}$  and  $\alpha_{a}$  in Algorithm 1 be set according to the smoothness of critic  $(\mathcal{L}_t(\omega))$  and actor  $(\ell_t(\theta))$  objectives respectively. These choice of step-sizes guarantee ascent for the actor objective, and descent for the critic objective (refer to the proof of Prop. 3 in App. D for details). In practice, we set both step-sizes using an Armijo line-search, and refer the reader to App. F for details. Since Algorithm 1 does not update the actor and critic in a two time-scale setting, unlike [29, 60], the relative scales of the step-sizes and the number of inner iterations  $(m_a, m_c)$  do not affect the algorithm's performance.
In contrast to Prop. 3, Dong et al. [11] prove that their proposed algorithm results in an  $O(1 / T)$  convergence to the stationary point (not the neighbourhood). However, they make a strong unjustified
assumption that the minimization problem w.r.t the parameters modeling the policy and distribution over states is jointly PL. Compared to [2, 61, 36] that focus on proving convergence to the neighbourhood of the optimal value function, but bound the critic error in the  $\ell_2$  or  $\ell_{\infty}$  norm, we focus on proving convergence to the (neighbourhood) of a stationary point, but define the critic loss in a decision-aware manner that depends on  $D_{\Phi^*}$ . Since Algorithm 1 is not a two time-scale algorithm, unlike Konda and Tsitsiklis [29], Wu et al. [60], the proof of Prop. 3 does not require analyzing coupled recursions between the actor and critic. Furthermore, the guarantees in Prop. 3 are independent of how the critic loss is minimized. We could use any policy evaluation method in order to estimate the value function. Hence, unlike the standard two time-scale analyses, we do not make assumptions about the mixing time of the underlying Markov chain.
Compared to the existing theoretical work on general (not decision-aware) AC methods [62, 60, 8, 28, 22, 30, 18, 41, 9] that prove convergence for the tabular or linear function approximation settings, (a) our theoretical results require fewer assumptions on the function approximation. For instance, Prop. 2 holds for any actor or critic parameterization (including complex neural networks), while the guarantees in Prop. 3 hold for any critic parameterization, but require that  $\ell_t$ , the surrogate function for the actor satisfy smoothness and gradient domination properties. (b) On the other hand, since our analysis does not explicitly model how the critic error is minimized, we can only converge to the neighbourhood of a stationary point. This is in contrast to the existing two time-scale analyses that jointly analyze the actor and critic, and show convergence to a stationary point [60]. (c) Finally, we note that the proposed algorithm supports off-policy updates i.e. the actor can re-use the value estimates from the critic to update the policy multiple times (corresponding to Lines 10-13 in Algorithm 1). This is in contrast to existing theoretically principled actor-critic methods that require interacting with the environment and gathering new data after each policy update. Hence, compared to the existing literature on AC methods, Algorithm 1 is more practical, has weaker theoretical guarantees but requires fewer assumptions on the function approximation.
5 Instantiating the generic actor-critic framework
We now instantiate Algorithm 1 for the direct (Sec. 5.1) and softmax (Sec. 5.2) representation.
5.1 Direct representation
Recall that for the direct functional representation, policy  $\pi$  is represented by the set of distributions  $p^{\pi}(\cdot |s)$  over actions for each state  $s\in S$ . Using the policy gradient theorem [53],  $[\nabla_{\pi}J(\pi)]_{s,a} = d^{\pi}(s)Q^{\pi}(s,a)$ . Similar to [57, 61], we use a weighted (across states) negative entropy mirror map implying that  $D_{\Phi}(p^{\pi},p^{\pi '}) = \sum_{s\in \mathcal{S}}d^{\pi t}(s)D_{\phi}(p^{\pi}(\cdot |s),p^{\pi '}(\cdot |s))$  where  $\phi (p^{\pi}(\cdot |s)) = -\sum_{a}p^{\pi}(a|s)\log (p^{\pi}(a|s))$  and hence,  $D_{\phi}(p^{\pi}(\cdot |s),p^{\pi '}(\cdot |s)) = \mathrm{KL}(p^{\pi}(\cdot |s)||p^{\pi '}(\cdot |s))$ . We now instantiate inequality (I) in Sec. 3.2 in the proposition below (see App. E for the derivation).
Proposition 4. For the direct representation and negative entropy mirror map,  $c > 0$ ,  $\eta \leq \frac{(1 - \gamma)^3}{2\gamma|A|}$
$$
\begin{array}{l} J (\pi) - J (\pi_ {t}) \geq C + \mathbb {E} _ {s \sim d ^ {\pi_ {t}}} \left[ \mathbb {E} _ {a \sim p ^ {\pi_ {t}} (· | s)} \left[ \frac {p ^ {\pi} (a | s)}{p ^ {\pi_ {t}} (a | s)} \left(\hat {Q} ^ {\pi_ {t}} (s, a) - \left(\frac {1}{\eta} + \frac {1}{c}\right) \log \left(\frac {p ^ {\pi} (a | s)}{p ^ {\pi_ {t}} (a | s)}\right)\right) \right] \right] \\ \left. \right. - \mathbb {E} _ {s \sim d ^ {\pi_ {t}}} \left[ \mathbb {E} _ {a \sim p ^ {\pi_ {t}} (\cdot | s)} \left[ Q ^ {\pi_ {t}} (s, a) - \hat {Q} ^ {\pi_ {t}} (s, a) \right] + \frac {1}{c} \log \left(\mathbb {E} _ {a \sim p ^ {\pi_ {t}} (\cdot | s)} \left[ \exp \left(- c \left[ Q ^ {\pi_ {t}} (s, a) - \hat {Q} ^ {\pi_ {t}} (s, a) \right]\right)\right]\right)\right] \\ \end{array}
$$
where  $C$  is a constant and  $\hat{Q}^{\pi_t}$  is the estimate of the action-value function for policy  $\pi_t$ .
For incorporating policy (with parameters  $\theta$ ) and critic (with parameters  $\omega$ ) parameterization, we note that  $p^{\pi}(\cdot |s) = p^{\pi}(\cdot |s,\theta)$  and  $\hat{Q}^{\pi}(s,a) = Q^{\pi}(s,a|\omega)$  where the model is implicit in the notation. Using the reasoning in Sec. 3.2 with Prop. 4 immediately gives us the actor and critic objectives  $(\ell_t(\theta)$  and  $L_{t}(\omega)$  respectively) at iteration  $t$  and completely instantiates Algorithm 1. Observe that the critic error is asymmetric and penalizes the under/over-estimation of the  $Q^{\pi}$  function differently. This is different from the standard squared critic loss:  $E_{s\sim d^{\pi_t}}\mathbb{E}_{a\sim p^{\pi_t}(\cdot |s)}[Q^{\pi_t}(s,a) - Q^{\pi_t}(s,a|\omega)]^2$  that does not take into account the sign of the misestimation.
To demonstrate the effectiveness of the proposed critic loss, we consider a two-armed bandit example in Prop. 5 (see App. E for details) with deterministic rewards (there is no variance due to sampling), use the direct representation and tabular parameterization for the policy, linear function approximation
for the critic and compare minimizing the standard squared loss vs the decision-aware loss in Prop. 4.
Proposition 5. Consider a two-armed bandit example with deterministic rewards where arm 1 is optimal and has a reward  $r_1 = Q_1 = 2$  whereas arm 2 has reward  $r_2 = Q_2 = 1$ . Consider using linear function approximation to estimate the  $Q$  function i.e.  $\hat{Q} = x\omega$  where  $\omega$  is the parameter to be learned and  $x$  is the feature of the corresponding arm. Let  $x_1 = -2$  and  $x_2 = 1$  implying that  $\hat{Q}_1(\omega) = -2\omega$  and  $\hat{Q}_2(\omega) = \omega$ . Let  $p_t$  be the probability of pulling the optimal arm at iteration  $t$  and consider minimizing two alternative objectives to estimate  $\omega$ :
(1) Squared loss:  $\omega_{t}^{(1)} := \arg \min \left\{\frac{p_{t}}{2}\left[\hat{Q}_{1}(\omega) - Q_{1}\right]^{2} + \frac{1 - p_{t}}{2}\left[\hat{Q}_{2}(\omega) - Q_{2}\right]^{2}\right\}$ .
(2) Decision-aware critic loss:  $\omega_{t}^{(2)} = \arg \min \mathcal{L}_{t}(\omega)\coloneqq p_{t}\left[Q_{1} - \hat{Q}_{1}(\omega)\right] + (1 - p_{t})\left[Q_{2} - \hat{Q}_{2}(\omega)\right] +$ $\frac{1}{c}\log \Big(p_t\exp \Bigl (-c\bigl [Q_1 - \hat{Q}_1(\omega)\bigr ] + (1 - p_t)\exp \Bigl (-c\bigl [Q_2 - \hat{Q}_2(\omega)\bigr ]\Bigr)\Bigr)\Bigg{]}$
For  $p_0 < \frac{2}{5}$ , minimizing the squared loss results in convergence to the sub-optimal action, while minimizing the decision-aware loss (for  $c, p_0 > 0$ ) results in convergence to the optimal action.
Hence, minimizing the decision-aware critic loss results in a better, more well-informed estimate of  $\omega$  which when coupled with the actor update results in convergence to the optimal arm. For this simple example, at every iteration  $t$ ,  $\mathcal{L}_t(\omega_t^{(2)}) = 0$ , while the standard squared loss is non-zero at  $\omega_t^{(1)}$ , though we use the same linear function approximation model in both cases. In Prop. 16, we prove that for a 2-arm bandit with deterministic rewards and linear critic parameterization, minimizing the decision-aware critic loss will always result in convergence to the optimal arm.
5.2 Softmax representation
Recall that for the softmax functional representation, policy  $\pi$  is represented by the logits  $z^{\pi}(s,a)$  for each  $s\in S$  and  $a\in \mathcal{A}$  such that  $p^\pi (a|s) = \frac{\exp(z^\pi(s,a))}{\sum_{a'}\exp(z^\pi(s,a'))}$ . Using the policy gradient theorem,  $[\nabla_{\pi}J(\pi)]_{s,a} = d^{\pi}(s)A^{\pi}(s,a)p^{\pi}(a|s)$  where  $A^{\pi}$  is the advantage function. Similar to Vaswani et al. [57], we use a weighted (across states) log-sum-exp mirror map implying that  $D_{\Phi}(z,z^{\prime}) = \sum_{s\in S}d^{\pi t}(s)D_{\phi}(z(s,\cdot),z^{\prime}(s,\cdot))$  where  $\phi (z(s,\cdot)) = \log (\sum_{a}\exp (z(s,a)))$  and hence,  $D_{\phi}(z(s,\cdot),z^{\prime}(s,\cdot)) = \mathrm{KL}(p^{\pi^{\prime}}(\cdot |s),p^{\pi}(\cdot |s))$  (see Lemma 11 for a derivation). We now instantiate inequality (I) in Sec. 3.2 in the proposition below (see App. E for the derivation).
Proposition 6. For the softmax representation and log-sum-exp mirror map,  $c > 0$ ,  $\eta \leq 1 - \gamma$
$$
\begin{array}{l} J (\pi) - J \left(\pi_ {t}\right) \geq \mathbb {E} _ {s \sim d ^ {\pi_ {t}}} \mathbb {E} _ {a \sim p ^ {\pi_ {t}} (\cdot | s)} \left[ \left(\hat {A} ^ {\pi_ {t}} (s, a) + \frac {1}{\eta} + \frac {1}{c}\right) \log \left(\frac {p ^ {\pi} (a | s)}{p ^ {\pi_ {t}} (a | s)}\right) \right] \\ - \frac {1}{c} \mathbb {E} _ {s \sim d ^ {\pi_ {t}}} \mathbb {E} _ {a \sim p ^ {\pi_ {t}} (\cdot | s)} \left[ \left(1 - c \left[ A ^ {\pi_ {t}} (s, a) - \hat {A} ^ {\pi_ {t}} (s, a) \right]\right) \log \left(1 - c \left[ A ^ {\pi_ {t}} (s, a) - \hat {A} ^ {\pi_ {t}} (s, a) \right]\right) \right] \\ \end{array}
$$
where  $\hat{A}^{\pi_t}$  is the estimate of the advantage function for policy  $\pi_t$ .
For incorporating policy (with parameters  $\theta$ ) and critic (with parameters  $\omega$ ) parameterization, we note that  $p^{\pi}(a|s) = \frac{\exp(z^{\pi}(s,a|\theta))}{\sum_{a'}\exp(z^{\pi}(s,a'|\theta))}$  and  $\hat{A}^{\pi}(s,a) = A^{\pi}(s,a|\omega)$  where the model is implicit in the notation. Using the reasoning in Sec. 3.2 with Prop. 6 immediately gives us the actor and critic objectives  $(\ell_t(\theta)$  and  $L_{t}(\omega)$  respectively) at iteration  $t$  and completely instantiates Algorithm 1. Similar to the direct representation, observe that  $\mathcal{L}_t$  is asymmetric and penalizes the under/overestimation of the advantage function differently.
To demonstrate the effectiveness of the proposed critic loss, we construct a two-armed bandit example in Prop. 7 below (see App. E for details), use the softmax representation and tabular parameterization for the policy and consider a discrete hypothesis class (with two hypotheses) as the model for the critic. We compare minimizing the squared loss on the advantage:  $E_{s\sim d^{\pi_t}}\mathbb{E}_{a\sim p^{\pi_t}(\cdot |s)}[A^{\pi_t}(s,a) - A^{\pi_t}(s,a|\omega)]^2$  with minimizing the decision-aware loss. We see that minimizing the decision-aware critic loss can distinguish between the two hypotheses and choose the correct hypothesis resulting in convergence to the optimal action.
Proposition 7. Consider a two-armed bandit example and define  $p \in [0,1]$  as the probability of pulling arm 1. Given  $p$ , let the advantage of arm 1 be equal to  $A_1 := \frac{1}{2} > 0$ , while that of arm 2 is  $A_2 := -\frac{p}{2(1 - p)} < 0$  implying that arm 1 is optimal. For  $\varepsilon \in \left(\frac{1}{2}, 1\right)$ , consider approximating the advantage of the two arms using a function approximation model with two hypotheses that
depend on  $p$ :  $\mathcal{H}_0: \hat{A}_1 = \frac{1}{2} + \varepsilon, \hat{A}_2 = -\frac{p}{1 - p}\left(\frac{1}{2} + \varepsilon\right)$  and  $\mathcal{H}_1: \hat{A}_1 = \frac{1}{2} - \varepsilon \operatorname{sgn}\left(\frac{1}{2} - p\right), \hat{A}_2 = -\frac{p}{1 - p}\left(\frac{1}{2} - \varepsilon \operatorname{sgn}\left(\frac{1}{2} - p\right)\right)$  where  $\operatorname{sgn}$  is the signum function. If  $p_t$  is the probability of pulling arm 1 at iteration  $t$ , consider minimizing two alternative loss functions to choose the hypothesis  $\mathcal{H}_t$ :
(1) Squared loss:  $\mathcal{H}_t = \arg \min_{\{\mathcal{H}_0,\mathcal{H}_1\}}\left\{\frac{p_t}{2} [A_1 - \hat{A}_1]^2 +\frac{1 - p_t}{2} [A_2 - \hat{A}_2]^2\right\}$ .
(2) Decision-aware critic loss with  $c = 1$ :  $\mathcal{H}_t = \arg \min_{\{\mathcal{H}_0,\mathcal{H}_1\}}$ $\left\{p_t\left(1 - [A_1 - \hat{A}_1]\right)\log (1 - [A_1 - \hat{A}_1]) + (1 - p_t)\left(1 - [A_2 - \hat{A}_2]\right)\log (1 - [A_2 - \hat{A}_2])\right\}$ .
For  $p_0 \leq \frac{1}{2}$ , the squared loss cannot distinguish between  $\mathcal{H}_0$  and  $\mathcal{H}_1$ , and depending on how ties are broken, minimizing it can result in convergence to the sub-optimal action. On the other hand, minimizing the divergence loss (for any  $p_0 > 0$ ) results in convergence to the optimal arm.
In Prop. 13 in App. E, we study the softmax representation with the Euclidean mirror map and instantiate inequality (I) for this case. Finally, in App. B, we instantiate our actor-critic framework to handle stochastic value gradients used for learning continuous control policies [21]. In the next section, we consider simple RL environments to empirically benchmark Algorithm 1.
Figure 1: Comparison of decision-aware, Adv-MSE and MSE loss functions using a linear actor and linear (with three different dimensions) critic in the Cliff World environment for direct and softmax policy representations. For  $d = 80$  (corresponding to an expressive critic), all algorithms have the same performance. For  $d = 40$  and  $d = 60$ , MSE does not have monotonic improvement and converges to a sub-optimal policy. Adv-MSE almost always reaches the optimal policy. Compared to the Adv-MSE and MSE, minimizing the decision-aware loss always results in convergence to the optimal policy at a faster rate, especially when using a less expressive critic ( $d = 40$ ).
We demonstrate the benefit of the decision-aware framework over the standard AC algorithm where the critic is trained by minimizing the squared error. We instantiate Algorithm 1 for the direct and softmax representations, and evaluate the performance on two grid-world environments, namely Cliff World [53] and Frozen Lake [6] (see App. F for details). We compare the performance of three AC algorithms that have the same actor, but differ in the objective function used to train the critic.
Critic Optimization: For the direct and softmax representations, the critic's objective is to estimate the action-value  $(Q)$  and advantage  $(A)$  functions respectively. We use a linear parameterization for the  $Q$  function implying that for each policy  $\pi$ ,  $Q^{\pi}(s,a|\omega) = \langle \omega ,\mathbf{X}(s,a)\rangle$ , where  $\mathbf{X}(s,a)\in \mathbb{R}^{d}$  are features obtained via tile-coding [53, Ch. 9]. We vary the dimension  $d\in \{80,60,40\}$  of the tile-coding features to vary the expressivity of the critic. Given the knowledge of  $p^\pi$  and the estimate  $Q^{\pi}(s,a|\omega)$ , the estimated advantage can be obtained as:  $A^{\pi}(s,a|\omega) = Q^{\pi}(s,a|\omega) - \sum_{a}p^{\pi}(a|s)Q^{\pi}(s,a|\omega)$ . We consider two ways to estimate the  $Q$  function for training the critic: (a) using the known MDP to exactly compute the  $Q$  values and (b) estimating the  $Q$  function using Monte-Carlo (MC) rollouts. There are three sources of error for an insufficient critic – the bias due to limited model capacity, the optimization error due to an insufficient number of inner iterations  $(m_c)$  and the variance due to Monte-Carlo sampling. We use a large value of  $m_{c}$  and sufficiently large
Monte-Carlo rollouts to control the optimization error and variance respectively (see App. F). This ensures that the bias dominates, and enables us to isolate the effect of the form of the critic loss.