SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanlin Zhu  
EECS, UC Berkeley; Meta AI  
hanlinzhu@berkeley.edu
Amy Zhang  
ECE, UT Austin; Meta AI  
amy.zhang@austin.utexas.edu
Goal-conditioned reinforcement learning (GCRL) refers to learning general-purpose skills that aim to reach diverse goals. In particular, offline GCRL only requires purely pre-collected datasets to perform training tasks without additional interactions with the environment. Although offline GCRL has become increasingly prevalent and many previous works have demonstrated its empirical success, the theoretical understanding of efficient offline GCRL algorithms is not well established, especially when the state space is huge and the offline dataset only covers the policy we aim to learn. In this paper, we provide a rigorous theoretical analysis of an existing empirically successful offline GCRL algorithm. We prove that under slight modification, this algorithm enjoys an  $\tilde{O}(\mathrm{poly}(1/\epsilon))$  sample complexity (where  $\epsilon$  is the desired suboptimality of the learned policy) with general function approximation thanks to the property of (semi-)strong convexity of the objective functions. We only require nearly minimal assumptions on the dataset (single-policy concentrability) and the function class (realizability). Moreover, this algorithm consists of two uninterleaved optimization steps, which we refer to as  $V$ -learning and policy learning, and is computationally stable since it does not involve minimax optimization. We also empirically validate our theory by showing that the modified algorithm outperforms the previous algorithm in various real-world environments. To the best of our knowledge, this is the first algorithm that is both provably efficient with general function approximation and single-policy concentrability, and empirically successful without requiring solving minimax optimization problems.
Goal-conditioned reinforcement learning (GCRL) aims to design agents that are able to learn general-purpose skills to reach diverse goals [Kaelbling, 1993, Schaul et al., 2015, Plappert et al., 2018]. In particular, offline GCRL learns goal-reaching policies by purely pre-collected data without any further interactions with the environment [Chebotar et al., 2021, Yang et al., 2022]. Since such interaction can be expensive or even unsafe in practice, offline GCRL is increasingly popular as a way to learn generalist agents in real-world environments [Lange et al., 2012, Levine et al., 2020].
Although offline GCRL is promising and achieves great success in various practical scenarios [Lynch et al., 2020, Chebotar et al., 2021, Yang et al., 2022, Ma et al., 2022b,c], designing practical algorithms that are provably efficient still remains an open question. On the practical side, an ideal algorithm should be scalable to huge (or infinite) state spaces and only require minimal dataset coverage assumptions. Moreover, the algorithm should be computationally efficient and stable (e.g., only
using regression-based methods to train policies to avoid unstable minimax optimization). On the theoretical side, we aim to provide finite-sample guarantees of the learned policy.
Unfortunately, most existing algorithms are not both theoretically and practically efficient. On the one hand, many empirically efficient algorithms do not enjoy finite-sample guarantees [Lynch et al., 2020, Chebotar et al., 2021, Yang et al., 2022, Ma et al., 2022c] or even suffer constant suboptimality in favorable settings given infinite data (e.g., Ma et al. [2022c]). On the other hand, although many previous offline RL algorithms with theoretical finite-sample guarantees can be naturally extended to offline GCRL settings, they either cannot handle general value function approximation in the presence of huge (or infinite) state spaces [Jin et al., 2021, Rashidinejad et al., 2021, Yin et al., 2021, Shi et al., 2022, Li et al., 2022], or require impractically strong dataset coverage assumptions, such as all policy concentrability [Antos et al., 2008, Munos and Szepesvári, 2008, Xie and Jiang, 2021b].
Recently, several provably efficient algorithms have been proposed under general function approximation and single-policy concentrability [Zhan et al., 2022, Cheng et al., 2022, Rashidinejad et al., 2022]. In particular, the algorithm of [Zhan et al., 2022], based on the duality form of regularized linear programming formulation of RL, only requires the realizability assumption of function class. However, they all require solving minimax optimization problems which can be difficult or computationally unstable [Daskalakis et al., 2021]. On the contrary, some practically efficient algorithms (e.g., Ma et al. [2022b,c]) do not involve minimax optimization and thus are computationally more stable. This naturally raises an important question:
Can we design an efficient offline GCRL algorithm that enjoys favorable theoretical guarantees under mild assumptions and performs well empirically in real-world scenarios without a minimax formulation?
In this paper, we answer the above question affirmatively by providing rigorous theoretical guarantees for an empirically successful offline GCRL algorithm named GoFAR proposed by Ma et al. [2022c]. We made some slight yet critical modifications to GoFAR. For deterministic MDPs, we need to carefully select the value of one hyperparameter that is set to 1 in the original GoFAR (which can be tuned in practice). For stochastic MDPs, we need to first learn the true transition model via maximum likelihood (MLE) and then plug in the learned model in the algorithm. To distinguish the difference between the original algorithm and the modified ones, we name the modified versions VP-learning (Algorithm 1).
We show that the VP-learning algorithm has both good empirical performance in real-world scenarios (already shown by Ma et al. [2022c], and we compare VP-learning and GoFAR empirically and show that our modification further improves the performance of the previous algorithm GoFAR) and favorable theoretical guarantees under mild assumptions. Specifically, it achieves  $\tilde{O} (\mathrm{poly}(1 / \epsilon))$  sample complexity (where  $\epsilon$  is the desired suboptimality level of the learned policy) under general function approximation with realizability-only assumption and partial data coverage with single-policy concentrability assumption. Moreover, the VP-learning algorithm can be decomposed into two interleaved learning (optimization) procedures (i.e.,  $V$ -learning and policy learning), which only require solving regression problems without minimax optimization.
Note that the VP-learning algorithm can be naturally applied to single-task RL settings, and all the analysis in this paper does not rely on whether the setting is goal-conditioned. Since the original algorithm is proposed and empirically validated in goal-conditioned settings, we analyze the algorithm in goal-conditioned settings as well.
Since our algorithm can be naturally applied to single-task offline RL settings, we discuss the related work in a broader scope, which also includes single-task offline RL.
Offline RL in tabular and linear function approximation settings. In tabular and linear settings, a line of work proposed efficient (both statistically and computationally) algorithms under single-policy concentrability [Jin et al., 2021, Rashidinejad et al., 2021, Yin et al., 2021, Shi et al., 2022, Li et al., 2022]. These algorithms construct uncertainty quantifiers to ensure pessimism such that policies not well covered by the dataset (which, by single-policy concentrability assumption, are thus suboptimal) suffer a large penalty. Yin and Wang [2021] also consider offline RL with single-policy
concentrability and achieve instance-dependent characterization. However, the above algorithms cannot be directly applied to many practical scenarios when non-linear function approximators are required, since it is hard to obtain uncertainty quantifiers without oracle access when function approximators are non-linear [Jiang and Huang, 2020, Jin et al., 2021, Uehara and Sun, 2021, Xie et al., 2021]. In our algorithm (V-learning step), we use a regularizer in the form of  $f$ -divergence instead of uncertainty quantifiers to ensure pessimism, which makes our algorithm efficient in the presence of non-linear function approximators without additional oracle access.
Offline RL with all-policy concentrability. Besides huge state spaces, another central challenge for offline RL is the lack of dataset coverability. Concentrability, defined as the ratio of occupancy frequency induced by a policy to the dataset distribution, is one of the most widely used definitions to characterize the dataset coverability [Munos, 2007, Scherrer, 2014]. Many previous works require all-policy concentrability to make the algorithm efficient [Szepesvári and Munos, 2005, Munos, 2007, Antos et al., 2007, 2008, Farahmand et al., 2010, Scherrer, 2014, Liu et al., 2019, Chen and Jiang, 2019, Jiang, 2019, Wang et al., 2019, Feng et al., 2019, Liao et al., 2020, Zhang et al., 2020, Uehara et al., 2020, Xie and Jiang, 2021a]. However, in practice, it is unreasonable to require that the offline dataset can cover all candidate policies, and our algorithm only requires single-policy concentrability.
Offline RL with general function approximation and single-policy concentrability. A recent line of work, which is based on marginalized importance sampling (MIS) formulation of RL, has shown success either empirically [Nachum et al., 2019a,b, Lee et al., 2021, Kim et al., 2021] or theoretically [Zhan et al., 2022, Rashidinejad et al., 2022]. In particular, Zhan et al. [2022], Rashidinejad et al. [2022] provide finite-sample guarantees for their algorithms under general function approximation and only single-policy concentrability. Another line of work [Xie et al., 2021, Cheng et al., 2022] also proposes provably efficient algorithms based on an actor-critic formulation under similar assumptions. However, all the above algorithms require solving minimax optimization, which could be difficult or computationally unstable [Daskalakis et al., 2021]. Instead, our algorithm only involves uninterleaved regression-based optimization without minimax optimization.
Offline GCRL. In the context of offline GCRL, the sparsity of the reward is another core challenge [Kaelbling, 1993, Schaul et al., 2015]. Several previous works aim to solve the issue and show empirical success without finite-sample guarantee [Ghosh et al., 2019, Chebotar et al., 2021, Yang et al., 2022, Ma et al., 2022c]. Although there exist theoretical studies of relevant problems such as the offline stochastic shortest path problem [Yin et al., 2022], theoretical understanding of the offline GCRL problem is still lacking. The most relevant work to this paper is Ma et al. [2022c], which shows great performance in several real-world settings without minimax optimization but lacks theoretical guarantee. In this paper, we proved that a slightly modified version of their algorithm (i.e., our VP-learning algorithm shown in Algorithm 1) is provably efficient. Also, we note that Ma et al. [2022a] can be viewed as the single-task version of Ma et al. [2022c], and our algorithm and analysis can be naturally extended to single-task offline RL settings.
Basic Notations. Throughout this paper, we use  $|\mathcal{X}|$  and  $\Delta (\mathcal{X})$  to denote the cardinality and probability simplex of a given set  $\mathcal{X}$ . We use  $x\lesssim y$  to denote that there exists a constant  $c > 0$  such that  $x\leq cy$ , use  $x\gtrsim y$  if  $y\lesssim x$  and use  $x\succ y$  if  $x\lesssim y$  and  $y\lesssim x$ . Also, we use the standard  $O(\cdot)$  notation where  $f(n) = O(g(n))$  if there exists  $n_0,C > 0$  such that  $|f(n)|\leq Cg(n)$  for all  $n\geq n_0$ , and denote  $f(n) = \Omega (g(n))$  if  $g(n) = O(f(n))$ . For any  $x\in \mathbb{R}$ , define  $x_{+}\triangleq \max \{x,0\}$ ; for any general function  $f:\mathcal{X}\to \mathbb{R}$ , define  $f_{+}(x) = \max \{f(x),0\},\forall x\in \mathcal{X}$ . Also, for any function  $f:\mathbb{R}\rightarrow \mathbb{R}$ , define  $\bar{f} (x) = f(x) - \min_{u\in \mathbb{R}}f(u)$  for any  $x\in \mathbb{R}$  if  $\min_{u\in \mathbb{R}}f(u)$  exists and further overwrite the notation  $f_{+}(x) = \mathbb{1}\{f^{\prime}(x)\geq 0\} \cdot \bar{f} (x)$  where  $\mathbb{1}\{\cdot \}$  is the indicator function.
Markov decision process. We consider an infinite-horizon discounted Markov decision process (MDP), which is described by a tuple  $M = (\mathcal{S}, \mathcal{A}, P, R, \rho, \gamma)$ , where  $\mathcal{S}$  and  $\mathcal{A}$  denote the state and action spaces respectively,  $P: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$  is the transition kernel,  $R: \mathcal{S} \times \mathcal{A} \to \Delta([0,1])$  encodes a family of reward distributions given state-action pairs with  $r: \mathcal{S} \times \mathcal{A} \to [0,1]$  as the expected reward function,  $\rho: \mathcal{S} \to [0,1]$  is the initial state distribution, and  $\gamma \in [0,1)$  is the discount factor. We assume  $\mathcal{A}$  is finite while  $\mathcal{S}$  could be arbitrarily complex (even continuous) as in many
real-world scenarios. A stationary (stochastic) policy  $\pi : \mathcal{S} \to \Delta(\mathcal{A})$  outputs a distribution over action space for each state.
Goal-conditioned reinforcement learning. In goal-conditioned RL, we additionally assume a goal set  $\mathcal{G}$ . Similar to Ma et al. [2022c], in goal-conditioned settings, the reward function  $R(s;g)$  (as well as the expected reward function  $r(s;g)$ ) and policy  $\pi(a|s,g)$  also depend on the commanded goal  $g \in \mathcal{G}$ , and the reward no longer depends on the action  $a$  and is deterministic.
Each (goal-conditioned) policy  $\pi$  induces a (discounted) occupancy density over state-action pairs for any commanded goal  $d^{\pi} : \mathcal{S} \times \mathcal{A} \times \mathcal{G} \to [0,1]$  defined as  $d^{\pi}(s,a;g) \coloneqq (1 - \gamma)\sum_{t=0}^{\infty}\gamma^{t}\operatorname*{Pr}(s_{t} = s, a_{t} = a;\pi)$ , where  $\operatorname*{Pr}(s_{t} = s, a_{t} = a;\pi)$  denotes the visitation probability of state-action pair  $(s,a)$  at step  $t$ , starting at  $s_0 \sim \rho(\cdot)$  and following  $\pi$  given commanded goal  $g$ . We also write  $d^{\pi}(s;g) = \sum_{a \in \mathcal{A}} d^{\pi}(s,a;g)$  to denote the marginalized state occupancy. Let  $p(g)$  be a distribution over desired goals, then we denote  $d^{\pi}(s,a,g) = d^{\pi}(s,a;g)p(g)$  and  $d^{\pi}(s,g) = d^{\pi}(s;g)p(g)$ . An important property of occupancy density  $d^{\pi}$  is that it satisfies the following Bellman flow constraint:
$$
\sum_ {a} d (s, a; g) = (1 - \gamma) \rho (s) + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P \left(s \mid s ^ {\prime}, a ^ {\prime}\right) d \left(s ^ {\prime}, a ^ {\prime}; g\right) \tag {1}
$$
for all  $s \in S$  and  $g \in G$  when letting  $d = d^{\pi}$  for any policy  $\pi$ . Moreover, any  $d$  satisfying (1) is the occupancy density of a policy  $\pi_d$  where
$$
\pi_ {d} (a | s, g) = \left\{ \begin{array}{l l} d (s, a; g) / d (s; g), & d (s; g) > 0 \\ 1 / | \mathcal {A} |, & d (s; g) = 0 \end{array} \right. \quad \text {a n d} \quad d (s; g) = \sum_ {a \in \mathcal {A}} d (s, a; g). \tag {2}
$$
An important quantity associated with a policy  $\pi$  is the value function, which is the expected discounted cumulative reward defined as  $V^{\pi}(s;g)\coloneqq \mathbb{E}\left[\sum_{t = 0}^{\infty}\gamma^{t}r_{t}\mid s_{0} = s,a_{t}\sim \pi (\cdot |s_{t},g)\forall t\geq 0\right]$  starting at state  $s\in S$  with a commanded goal  $g\in \mathcal{G}$  where  $r_t = R(s_t;g) = r(s_t;g)$ . We use the notation  $J(\pi)\coloneqq (1 - \gamma)\mathbb{E}_{s\sim \rho ,g\sim p}[V^{\pi}(s;g)] = \mathbb{E}_{(s,a,g)\sim d^{\pi}}[r(s;g)]$  to represent a scalar summary of the performance of a policy  $\pi$ . We denote by  $\pi^{*}$  the optimal policy that maximizes the above objective and use  $V^{*}\coloneqq V^{\pi^{*}}$  to denote the optimal value function.
Offline GCRL. In this paper, we focus on offline GCRL, where the agent is only provided with a previously-collected offline dataset  $\mathcal{D} = \{(s_i,a_i,r_i,s_i',g_i)\}_{i=1}^N$ . Here,  $r_i \sim R(s_i;g_i)$ ,  $s_i' \sim P(\cdot \mid s_i, a_i)$ , and we assume that  $g_i$  are i.i.d. sampled from  $p(\cdot)$  and it is common that data are collected by a behavior policy  $\mu$  of which the discounted occupancy density is  $d^\mu$ . Therefore, we assume that  $(s_i,a_i,g_i)$  are sampled i.i.d. from a distribution  $\mu$  where  $\mu(s,a,g) = p(g)d^\mu(s,a;g) = d^\mu(s,a,g)$ . Note that we use  $\mu$  to denote both the behavior policy and the dataset distribution. We also assume an additional dataset  $\mathcal{D}_0 = \{(s_{0,i},g_{0,i})\}_{i=1}^{N_0}$  where  $s_{0,i}$  are i.i.d. sampled from  $\rho(\cdot)$  and  $g_{0,i}$  are i.i.d. sampled from  $p(\cdot)$ . The goal of offline RL is to learn a policy  $\hat{\pi}$  using the offline dataset so as to minimize the sub-optimality compared to the optimal policy  $\pi^*$ , i.e.,  $J(\pi^*) - J(\hat{\pi})$ , with high probability.
Function approximation. To deal with huge state spaces, (general) function approximation is necessary for practical scenarios. In this paper, we assume access to two function classes: a value function class  $\mathcal{V} \subseteq \{V : \mathcal{S} \times \mathcal{G} \to [0, V_{\max}]\}$  that models the value function of the (regularized) optimal policies, and a policy class  $\Pi \subseteq \{\pi : \mathcal{S} \times \mathcal{G} \to \Delta(\mathcal{A})\}$  consisting of candidate policies. For stochastic MDP (Section 3.1.2), we also need a transition kernel class  $\mathcal{P} \subseteq \{P : \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})\}$  which contains the ground-truth transition kernel. Additionally, for any function  $f : \mathcal{S} \times \mathcal{G} \to \mathbb{R}$ , we denote the operator  $\mathcal{T} : \mathbb{R}^{\mathcal{S} \times \mathcal{G}} \to \mathbb{R}^{\mathcal{S} \times \mathcal{A} \times \mathcal{G}}$  as  $(\mathcal{T}f)(s, a; g) = \mathbb{E}_{s' \sim P(\cdot | s, a)}[f(s'; g)]$ . Also, for any function  $V : \mathcal{S} \times \mathcal{G} \to [0, V_{\max}]$ , we define  $A_V(s, a; g) = r(s; g) + \gamma T V(s, a; g) - V(s; g)$ .
Offline data coverage assumption. Our algorithm works within the single-policy concentrability framework [Rashidinejad et al., 2021], which is defined as below.
Definition 1 (Single-policy concentrability for GCRL). Given a policy  $\pi$ , define  $C^{\pi}$  to be the smallest constant that satisfies  $\frac{d^{\pi}(s,a,g)}{\mu(s,a,g)} \leq C^{\pi}$  for all  $s \in S$ ,  $a \in \mathcal{A}$  and  $g \in \mathcal{G}$ .
The single-policy concentrability parameter  $C^\pi$  captures the coverage of policy  $\pi$  in the offline data. Our algorithm only requires this parameter to be small for  $\pi_{\alpha}^{*}$  which is a regularized optimal policy
(see Section 3 for formal definition) and is close to some optimal policy. This assumption is similar to Zhan et al. [2022] and and is much weaker than the widely used all-policy concentrability that assumes bounded  $C^\pi$  for all  $\pi$  (e.g., Scherrer [2014]).
Offline GCRL can be formulated as the following program:
$$
\max  _ {\pi} \quad \mathbb {E} _ {(s, g) \sim d ^ {\pi} (s, g)} [ r (s; g) ]. \tag {3}
$$
(3) requires solving an optimization problem over the policy space. One can also optimize over occupancy density  $d(s, a; g)$  s.t.  $d = d^{\pi}$  for some policy  $\pi$  which is equivalent to that  $d$  satisfies Bellman flow constraint (1). Therefore, the program (3) can be represented equivalently as follows:
$$
\max  _ {d (s, a; g) \geq 0} \quad \mathbb {E} _ {(s, g) \sim d (s, g)} [ r (s; g) ]
$$
$$
\text {s . t .} \quad \sum_ {a} d (s, a; g) = (1 - \gamma) \rho (s) + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P \left(s \mid s ^ {\prime}, a ^ {\prime}\right) d \left(s ^ {\prime}, a ^ {\prime}; g\right), \quad \forall (s, g) \in \mathcal {S} \times \mathcal {G}. \tag {4}
$$
Let  $d^{*}$  denote the optimal solution of (4), then (one of) the optimal policy can be induced by  $\pi^{*} = \pi_{d^{*}}$  as in (2). Under partial data coverage assumptions, (4) might fail in empirical settings by choosing a highly suboptimal policy that is not well covered by the dataset with constant probability. Similar to Zhan et al. [2022], Ma et al. [2022c], a regularizer is needed to ensure that the learned policy is well covered by the dataset. Therefore, one should instead solve a regularized version of (4), which is stated as follows:
$$
\max  _ {d (s, a; g) \geq 0} \mathbb {E} _ {(s, g) \sim d (s, g)} [ r (s; g) ] - \alpha D _ {f} (d \| \mu)
$$
$$
\text {s . t .} \quad \sum_ {a} d (s, a; g) = (1 - \gamma) \rho (s) + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P \left(s \mid s ^ {\prime}, a ^ {\prime}\right) d \left(s ^ {\prime}, a ^ {\prime}; g\right), \quad \forall (s, g) \in \mathcal {S} \times \mathcal {G}, \tag {5}
$$
where the  $f$ -divergence is defined as  $D_{f}(d\| \mu)\triangleq \mathbb{E}_{(s,a,g)\sim \mu}[f(d(s,a,g) / \mu (s,a,g))]$  for a convex function  $f$ . Throughout this paper, we choose  $f(x) = \frac{1}{2} (x - 1)^{2}$  as in Ma et al. [2022c], where the  $f$ -divergence is known as  $\chi^2$ -divergence under this specific choice of  $f$  and it is shown to be more stable than other divergences such as KL divergence [Ma et al., 2022c]. Let  $d_{\alpha}^{*}$  denote the optimal solution of (5), then the regularized optimal policy can be induced by  $\pi_{\alpha}^{*} = \pi_{d_{\alpha}^{*}}$  as in (2). The following single-policy concentrability assumption assumes that  $\pi_{\alpha}^{*}$  is well covered by the offline dataset.
Assumption 1 (Single-policy concentrability for  $\pi_{\alpha}^{*}$ ). Let  $d_{\alpha}^{*}$  be the optimal solution of (5), and let  $\pi_{\alpha}^{*} = \pi_{d_{\alpha}^{*}}$  as defined in (2). We assume  $C^{\pi_{\alpha}^{*}} \leq C_{\alpha}^{*}$  where  $C^{\pi_{\alpha}^{*}}$  is defined in Definition 1 and  $C_{\alpha}^{*} > 0$  is a constant.
Under Assumption 1, it can be observed that the performance difference between the regularized optimal policy  $\pi_{\alpha}^{*}$  and the optimal policy  $\pi^{*}$  is bounded by  $O(\alpha)$ . The following proposition formally presents this observation.
Proposition 3.1. Let  $d_{\alpha}^{*}$  be the optimal solution of (5), and let  $\pi_{\alpha}^{*} = \pi_{d_{\alpha}^{*}}$  as defined in (2). Then under Assumption 1, it holds that  $J(\pi^{*}) - J(\pi_{\alpha}^{*}) \leq O\left(\alpha (C_{\alpha}^{*})^{2}\right)$ .
The proof of Proposition 3.1 is deferred to Appendix A.1. Proposition 3.1 shows that by solving the regularized program (5), we can obtain a near-optimal policy as long as  $\alpha$  is small. The algorithm of Ma et al. [2022c] also aims to solve (5) and they simply choose  $\alpha = 1$ . We show empirically in Section 5 that  $\alpha < 1$  achieves better performance than  $\alpha = 1$ . In theory, we must carefully choose the value of  $\alpha$  s.t. the suboptimality of our learned policy vanishes to 0 with a reasonable rate. Finally, as in Ma et al. [2022c], we convert (5) to the dual form, which is an unconstrained problem and amenable to solve:
Proposition 3.2 (Dual form of (5)). The duality form of (5) is
$$
\min  _ {V (s; g) \geq 0} (1 - \gamma) \mathbb {E} _ {(s, g) \sim (\rho , p (g))} [ V (s; g) ] + \mathbb {E} _ {(s, a, g) \sim \mu} [ \mathbb {1} \{g _ {*} ^ {\prime} (A _ {V} (s, a; g)) \geq 0 \} \bar {g} _ {*} (A _ {V} (s, a; g)) ] \tag {6}
$$
where  $g_{*}$  is the convex conjugate of  $g = \alpha \cdot f$ . Moreover, let  $V_{\alpha}^{*}$  denote the optimal solution of (6), then it holds
$$
d _ {\alpha} ^ {*} (s, a; g) = \mu (s, a; g) g _ {*} ^ {\prime} (r (s; g) + \gamma \mathcal {T} V _ {\alpha} ^ {*} (s, a; g) - V _ {\alpha} ^ {*} (s; g)) _ {+} \tag {7}
$$
for all  $(s,a,g)\in \mathcal{S}\times \mathcal{A}\times \mathcal{G}$