SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate the performance of the decision-aware loss defined for the direct (Prop. 4) and softmax representations (Prop. 6). For both representations, we minimize the corresponding objective at each iteration  $t$  (Lines 6-8 in Algorithm 1) using gradient descent with the step-size  $\alpha_{c}$  determined by the Armijo line-search [4]. We use a grid-search to tune the trade-off parameter  $c$ , and propose an alternative albeit conservative method to estimate  $c$  in App. F. We compare against two baselines (see App. F for implementation details) - (i) the standard squared loss on the  $Q$  functions (referred to as MSE in the plots) defined in Prop. 5 and (ii) squared loss on  $A$  function (referred to as Adv-MSE in the plots) defined in Prop. 7. We note that the Adv-MSE loss corresponds to a second-order Taylor series expansion of the decision-aware loss (see Prop. 14 for details), and is similar to the loss in Pan et al. [42]. Recall that the critic error consists of the variance when using MC samples (equal to zero when we exactly compute the  $Q$  function) and the bias because of the critic optimization error (controlled since the critic objective is convex) and error due to the limited expressivity of the linear function approximation (decreases as  $d$  increases). Since our objective is to study the effect of the critic loss and its interaction with function approximation, we do not use bootstrapping to estimate the  $Q^{\pi}$  since it would result in a confounding bias term.
Actor Optimization: For all algorithms, we use the same actor objective defined for the direct (Prop. 4) and softmax representations (Prop. 6). We consider both the tabular and linear policy parameterization for the actor. For the linear function approximation, we use the same tile-coded features and set  $n = 60$  for both environments. We update the policy parameters at each iteration  $t$  in the off-policy inner-loop (Lines 11-13 in Algorithm 1) using Armijo line-search to set  $\alpha_{a}$ . For details about the derivatives and closed-form solutions for the actor objective, refer to [57] and App. F. We use a grid-search to tune  $\eta$ , and compare different values. Our experimental setup enables us to isolate the effect of the critic loss, without non-convexity or optimization issues acting as confounders.
Results: For each environment, we conduct four experiments that depend on (a) whether we use MC samples or the true dynamics to estimate the  $Q$  function, and (b) on the policy parameterization. We only show the plot corresponding to using the true dynamics for estimating the  $Q$  function and linear policy parameterization, and defer the remaining plots to App. G. For all experiments, we report the mean and  $95\%$  confidence interval of  $J(\pi)$  averaged across 5 runs. In the main paper, we only include 2 values of  $\eta \in \{0.01, 0.1\}$  and vary  $d \in \{40, 60, 80\}$ , and defer the complete figure with a broader range of  $\eta$  and  $d$  to App. G. For this experiment,  $c$  is tuned to 0.01 and we include a sensitivity (of  $J(\pi)$  to  $c$ ) plot in App. G. From Fig. 1, we see that (i) with a sufficiently expressive critic ( $d = 80$ ), all algorithms reach the optimal policy at nearly the same rate. (ii) as we decrease the critic capacity, minimizing the MSE loss does not result in monotonic improvement and converges to a sub-optimal policy, (iii) minimizing the Adv-MSE usually results in convergence to the optimal policy, whereas (iv) minimizing the decision-aware loss results in convergence to better policies at a faster rate, and is more beneficial when using a less-expressive critic (corresponding to  $d = 40$ ). We obtain similar results for the tabular policy parameterization or when using sampling to estimate the  $Q$  function (see App. G for additional results).
We designed a generic decision-aware actor-critic framework where the actor and critic are trained cooperatively to optimize a joint objective. Our framework can be used with any policy representation and easily handle general policy and critic parameterization, while preserving theoretical guarantees. Instantiating the framework resulted in an actor that supports off-policy updates, and a corresponding critic loss that can be minimized using first-order optimization. We demonstrated the benefit of our framework both theoretically and empirically. We note that Algorithm 1 can be directly used with any complex actor/critic parameterization in order to generalize across states/actions. The theoretical guarantees of Prop. 2 would still hold. From a practical perspective, w.r.t tuning hyperparameters,  $\eta$  does not depend on the actor/critic parameterization. On the other hand,  $\alpha_{a}$  and  $\alpha_{c}$  are set adaptively using an Armijo line-search that only requires the smoothness of actor/critic objectives, and does not depend on their convexity. However, Algorithm 1 does require tuning hyper-parameter  $c$ , and we will aim to investigate automatic adaptive ways to set it. In the future, we aim to benchmark Algorithm 1 for complex deep RL environments. Finally, we aim to broaden the scope our framework to applications such as variational inference.
$^{1}$ Code to reproduce the experiments is available at https://github.com/amirrezakazemi/ACPG
We would like to thank Michael Lu for feedback on the paper. This research was partially supported by the Canada CIFAR AI Chair program, the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant RGPIN-2022-04816.
[1] Romina Abachi. Policy-aware model learning for policy gradient methods. PhD thesis, University of Toronto (Canada), 2020.
[2] Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in Markov decision processes. In Conference on Learning Theory (COLT), pages 64-66, 2020.
[3] Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2): 251-276, February 1998.
[4] Larry Armijo. Minimization of functions having lipschitz continuous first partial derivatives. Pacific Journal of mathematics, 16(1):1-3, 1966.
[5] Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica, 45(11):2471-2482, 2009.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[7] Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning, 8(3-4):231–357, 2015.
[8] Semih Cayci, Niao He, and R Srikant. Finite-time analysis of entropy-regularized neural natural actor-critic algorithm. arXiv preprint arXiv:2206.00833, 2022.
[9] Tianyi Chen, Yuejiao Sun, and Wotao Yin. Tighter analysis of alternating stochastic gradient method for stochastic nested problems. arXiv preprint arXiv:2106.13781, 2021.
[10] Bo Dai, Albert Shaw, Niao He, Lihong Li, and Le Song. Boosting the actor with dual critic. arXiv preprint arXiv:1712.10282, 2017.
[11] Jing Dong, Li Shen, Yinggan Xu, and Baoxiang Wang. Provably efficient convergence of primal-dual actor-critic with nonlinear function approximation. arXiv preprint arXiv:2202.13863, 2022.
[12] Ryan D'Orazio, Nicolas Loizou, Issam Laradji, and Ioannis Mitlagkas. Stochastic mirror descent: Convergence analysis and adaptive variants via the mirror stochastic polyak stepsize. arXiv preprint arXiv:2110.15412, 2021.
[13] Pierluca D'Oro and Wojciech Jaskowski. How to learn a useful critic? model-based action-gradient-estimator policy optimization. Advances in Neural Information Processing Systems, 33:313–324, 2020.
[14] Pierluca D'Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, and Marcello Restelli. Gradient-aware model-based policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3801-3808, 2020.
[15] Radu Alexandru Dragomir, Mathieu Even, and Hadrien Hendrikx. Fast stochastic bregman gradient methods: Sharp analysis and variance reduction. In International Conference on Machine Learning, pages 2815-2825. PMLR, 2021.
[16] Amir-massoud Farahmand. Iterative value-aware model learning. Advances in Neural Information Processing Systems, 31, 2018.
[17] Amir-massoud Farahmand, Andre Barreto, and Daniel Nikovski. Value-aware loss function for model-based reinforcement learning. In Artificial Intelligence and Statistics, pages 1486-1494. PMLR, 2017.
[54] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS), pages 1057-1063, 2000.
[55] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804.10332, 2018.
[56] Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprint arXiv:2005.09814, 2020.
[57] Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert Müller, Shivam Garg, Matthieu Geist, Marlos C Machado, Pablo Samuel Castro, and Nicolas Le Roux. A general class of surrogate functions for stable and efficient reinforcement learning. arXiv preprint arXiv:2108.05828, 2021.
[58] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
[59] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
[60] Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems, 33: 17617-17628, 2020.
[61] Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1-36, 2022.
[62] Tengyu Xu, Zhe Wang, and Yingbin Liang. Non-asymptotic convergence analysis of two time-scale (natural) actor-critic algorithms. arXiv preprint arXiv:2005.03557, 2020.
[63] Rui Yuan, Robert M Gower, and Alessandro Lazaric. A general sample complexity analysis of vanilla policy gradient. In International Conference on Artificial Intelligence and Statistics, pages 3332-3380. PMLR, 2022.
[64] Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36 (4):1307-1319, 2020.
[65] Siqi Zhang and Niao He. On the convergence rate of stochastic mirror descent for nonsmooth nonconvex optimization. arXiv preprint arXiv:1806.04781, 2018.
Organization of the Appendix
A Definitions
B Extension to stochastic value gradients
C Proofs for Sec. 3
D Proofs for Sec. 4
E Proofs for Sec. 5
F Implementation Details
G Additional Experiments
- [Solution set]. We define the solution set  $\mathcal{X}^*$  for a function  $f$  as  $\mathcal{X}^* := \{x^* | x^* \in \arg \min_{x \in \operatorname{dom}(f)} f(x)\}$ .
- [Convexity]. A differentiable function  $f$  is convex iff for all  $v$  and  $w$  in  $\operatorname{dom}(f)$
$$
f (v) \geq f (w) + \langle \nabla f (w), v - w \rangle . \tag {Convexity}
$$
- [Lipschitz continuity]. A differentiable function  $f$  is  $G$ -Lipschitz continuous, meaning that for all  $v$  and  $w$  and constant  $G >$ ,
$$
| f (v) - f (w) | \leq G \| v - w \| \Rightarrow \| \nabla f (v) \| \leq G. \tag {LipschitzContinuity}
$$
- [Smoothness]. A differentiable function  $f$  is  $L$ -smooth, meaning that for all  $v$  and  $w$  and some constant  $L > 0$
$$
f (v) \leq f (w) + \langle \nabla f (w), v - w \rangle + \frac {L}{2} \| v - w \| _ {2} ^ {2}. \quad \text {(S m o o t h n e s s)}
$$
- [Polyak-Lojasiewicz inequality]. A differentiable function  $f$  satisfies the Polyak-Lojasiewicz (PL) inequality if there exists a constant  $\mu_p > 0$  s.t. for all  $v$ ,
$$
\mu_ {p} (f (v) - f ^ {*}) \leq \frac {1}{2} \| \nabla f (v) \| _ {2} ^ {2}, \tag {PL}
$$
where  $f^{*}$  is the optimal function value i.e.  $f^{*} \coloneqq f(x^{*})$  for  $x^{*} \in \mathcal{X}^{*}$ .
- [Restricted Secant Inequality]. A differentiable function  $f$  satisfies the Restricted Secant Inequality (RSI) inequality if there exists a constant  $\mu_r > 0$  that for all  $v$
$$
\langle \nabla f (v), v - v _ {p} \rangle \geq \mu_ {r} \| v - v _ {p} \| _ {2} ^ {2}, \tag {RSI}
$$
where  $v_{p}$  is the projection of  $v$  onto  $\mathcal{X}^*$ .
- [Bregman divergence]. For a strictly-convex, differentiable function  $\Phi$ , we define the Bregman divergence induced by  $\Phi$  (known as the mirror map) as:
$$
D _ {\Phi} (w, v) := \Phi (w) - \Phi (v) - \langle \nabla \Phi (v), w - v \rangle . \quad (\text {B r e g m a n d i v e r g e n c e})
$$
- [Relative smoothness]. A function  $f$  is  $\rho$ -relatively smooth w.r.t.  $D_{\Phi}$  iff  $f + \rho \Phi$  is convex. Furthermore, if  $f$  is  $\rho$ -relatively smooth w.r.t.  $\Phi$ , then,  $|f(w) - f(v) - \langle \nabla f(v), w - v \rangle| \leq \rho D_{\Phi}(w, v)$ .
- [Mirror Ascent]. Optimizing  $\max_{x\in \mathcal{X}}f(x)$  using mirror ascent (MA), if  $x_{t}$  is the current iterate, then the update at iteration  $t\in \{0,1,\dots ,T - 1\}$  with a step-size  $\eta_t$  and mirror map  $\Phi$  is given as:
$$
x _ {t + 1} := \underset {x \in \mathcal {X}} {\arg \max } \left\{\langle \nabla f (x _ {t}), x \rangle - \frac {1}{\eta_ {t}} D _ {\Phi} (x, x _ {t}) \right\}, \tag {MDupdate}
$$
The above update can be formulated into two steps Bubeck [7, Chapter 4] as follows:
$$
y _ {t + 1} := (\nabla \Phi) ^ {- 1} (\nabla \Phi (x _ {t}) + \eta_ {t} \nabla f (x _ {t}))
$$
(Move in dual space)
$$
x_{t + 1}:= \operatorname *{arg  min}_{x\in \mathcal{X}}\left\{D_{\Phi}(x,y_{t + 1})\right\}
$$
(Projection step)
B Extension to stochastic value gradients
In Sec. 5, we have seen alternative ways to represent a policy's conditional distributions over actions  $p^{\pi}(\cdot |s)$  for each state  $s\in S$ . On the other hand, stochastic value gradients [21] represent a policy by a set of actions. Formally, if  $\varepsilon$  are random variables drawn from a fixed distribution  $\chi$ , then policy  $\pi$  is a deterministic map from  $S\times \chi \to A$ . This corresponds to the functional representation of the policy, and is particularly helpful for continuous control, i.e. when the action-space is continuous. The action  $a$  chosen by  $\pi$  in state  $s$ , when fixing the random variable  $\varepsilon = \epsilon$ , is represented as  $\pi (s,\epsilon)$ , and the value function for policy  $\pi$  is given as:
$$
J (\pi) = \sum_ {s} d ^ {\pi} (s) \int_ {\varepsilon \sim \chi} r (s, \pi (s, \varepsilon)) d \varepsilon \tag {1}
$$
and Silver et al. [49] showed that  $\frac{\partial J(\pi)}{\partial\pi(s,\epsilon)} = d^{\pi}(s)\nabla_{a}Q^{\pi}(s,a)\big|_{a = \pi (s,\epsilon)}$ . In order to characterize the dependence on the policy parameterization, we note that  $\pi (s,\epsilon) = \pi (s,\epsilon ,\theta)$  where  $\theta$  are the model parameters. For a fixed  $\epsilon$ , we will use a Euclidean mirror map implying that  $D_{\Phi}(\pi ,\pi^{\prime}) = \sum_{s\in S}d^{\pi_{t}}(s)D_{\phi}(\pi (s,\epsilon),\pi^{\prime}(s,\epsilon)$  and choose  $\phi (\pi (s,\epsilon)) = \frac{1}{2}\| \pi '(s,\epsilon)\| _2^2$  implying that  $D_{\phi}(\pi (s,\epsilon),\pi '(s,\epsilon) = \frac{1}{2}\left[\pi (s,\epsilon) - \pi '(s,\epsilon)\right]^2$ . In order to instantiate the generic lower bound in Prop. 1 at iteration  $t$ , we prove the following proposition in App. E.
Proposition 8. For the stochastic value gradient representation and Euclidean mirror map,  $c > 0$ ,  $\eta$  such that  $J + \frac{1}{\eta}\Phi$  is convex in  $\pi$ .
$$
\begin{array}{l} J (\pi) - J (\pi_ {t}) \geq C + \mathbb {E} _ {s \sim d ^ {\pi_ {t}}} \mathbb {E} _ {\varepsilon \sim \chi} \left[ \widehat {\nabla_ {a} Q ^ {\pi_ {t}}} (s, a) \big | _ {a = \pi_ {t} (s, \varepsilon)} \pi (s, \varepsilon) - \frac {1}{2} \left(\frac {1}{\eta} + \frac {1}{c}\right) [ \pi_ {t} (s, \epsilon) - \pi (s, \epsilon) ] ^ {2} \right] \\ - \frac {c}{2} \mathbb {E} _ {s \sim d ^ {\pi_ {t}}} \mathbb {E} _ {\varepsilon \sim \chi} \left[ \nabla_ {a} Q ^ {\pi_ {t}} (s, a) \big | _ {a = \pi_ {t} (s, \varepsilon)} - \widehat {\nabla_ {a} Q ^ {\pi_ {t}}} (s, a) \big | _ {a = \pi_ {t} (s, \varepsilon)} \right] ^ {2} \\ \end{array}
$$
where  $C$  is a constant and  $\left.\widehat{\nabla_a Q^{\pi_t}}(s, a)\right|_{a = \pi_t(s, \varepsilon)}$  is the estimate of the action-value gradients for policy  $\pi$  at state  $s$  and  $a = \pi_t(s, \epsilon)$ .
For incorporating policy (with parameters  $\theta$ ) and critic (with parameters  $\omega$ ) parameterization, we note that  $\pi(s,\varepsilon) = \pi(s,\varepsilon|\theta)$  and  $\widehat{\nabla_a Q^{\pi_t}}(s,a)_{a=\pi_t(s,\varepsilon)} = \nabla_a Q^{\pi_t}(s,a|\omega)_{a=\pi_t(s,\varepsilon,\theta_t)}$  where the model is implicit in the notation. Using the reasoning in Sec. 3.2 with Prop. 8 immediately gives us the actor and critic objectives  $(\ell_t(\theta)$  and  $L_t(\omega)$  respectively) at iteration  $t$  and completely instantiates Algorithm 1. The actor objective is similar to Eq (15) of Silver et al. [49], with the easier to compute  $Q^{\pi_t}$  instead of  $Q^{\pi}$ , whereas the critic objective is similar to the one used in existing work on policy-aware model-based RL for continuous control [13].
C Proofs for Sec. 3
$$
J (\pi) \geq J (\pi^ {\prime}) + \langle \hat {g} (\pi^ {\prime}), \pi - \pi^ {\prime} \rangle - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} (\pi , \pi^ {\prime}) - \frac {1}{c} D _ {\Phi^ {*}} \Bigg (\nabla \Phi (\pi^ {\prime}) - e [ \nabla J (\pi^ {\prime}) - \hat {g} (\pi^ {\prime}) ], \nabla \Phi (\pi^ {\prime}) \Bigg)
$$
Proof. For any  $\eta$  such that  $J + \frac{1}{\eta}\Phi$  is convex, we use Lemma 2 to form the following lower-bound,
$$
\begin{array}{l} J (\pi) \geq J (\pi^ {\prime}) + \langle \nabla J (\pi^ {\prime}), (\pi - \pi^ {\prime}) \rangle - \frac {1}{\eta} D _ {\phi} (\pi , \pi^ {\prime}) \\ = J (\pi^ {\prime}) + \langle \hat {g} (\pi^ {\prime}), (\pi - \pi^ {\prime}) \rangle + \langle \nabla J (\pi^ {\prime}) - \hat {g} (\pi^ {\prime}), (\pi - \pi^ {\prime}) \rangle - \frac {1}{\eta} D _ {\phi} (\pi , \pi^ {\prime}) \\ \end{array}
$$
Defining  $\delta \coloneqq \nabla J(\pi') - \hat{g}(\pi')$ , and assuming that  $c\delta$  is small enough to satisfy the requirement for Lemma 1, we use Lemma 1 with  $x = \delta$ ,  $y = \pi$  and  $y' = \pi'$ .
$$
\begin{array}{l} = J (\pi^ {\prime}) + \langle \hat {g} (\pi^ {\prime}), (\pi - \pi^ {\prime}) \rangle - \frac {1}{\eta} D _ {\phi} (\pi , \pi^ {\prime}) - \frac {1}{c} \left[ D _ {\phi} (\pi , \pi^ {\prime}) + D _ {\phi *} \left(\nabla \phi (\pi^ {\prime}) - c \delta , \nabla \phi (\pi^ {\prime})\right) \right] \\ \Longrightarrow J (\pi) \geq J (\pi^ {\prime}) + \hat {g} (\pi^ {\prime}) ^ {\top} (\pi - \pi^ {\prime}) - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} (\pi , \pi^ {\prime}) - \frac {1}{c} D _ {\phi^ {*}} \left(\nabla \phi (\pi^ {\prime}) - c [ \nabla J (\pi^ {\prime}) - \hat {g} (\pi^ {\prime}) ], \nabla \phi (\pi^ {\prime})\right) \\ \end{array}
$$
Lemma 1 (Bregman Fenchel-Young). Let  $x \in \mathcal{Y}^*$ ,  $y \in \mathcal{V}$ ,  $y' \in \mathcal{V}$ . Then, for sufficiently small  $c > 0$  and  $x$  s.t.  $(\nabla \phi)^{-1}[\nabla \phi(y') - cx] \in \mathcal{V}$ , we have
$$
\langle y - y ^ {\prime}, x \rangle \geq - \frac {1}{c} \left[ D _ {\phi} \left(y, y ^ {\prime}\right) + D _ {\phi^ {*}} \left(\nabla \phi \left(y ^ {\prime}\right) - c x, \nabla \phi \left(y ^ {\prime}\right)\right) \right]. \tag {2}
$$
For a fixed  $y'$ , this inequality is tight for  $y = \arg \min_{v} \left\{ \langle x, v - y' \rangle + \frac{1}{c} D_{\Phi}(v, y) \right\}$ .
Proof. Define  $f(y) \coloneqq \langle x, y - y' \rangle + \frac{1}{c} D_{\Phi}(y, y')$ . If  $y^{*} = \arg \min f(y)$ , then,
$$
\begin{array}{l} \nabla f (y ^ {*}) = 0 \Longrightarrow \nabla \phi (y ^ {*}) = \nabla \phi (y ^ {\prime}) - c x \\ y ^ {*} = (\nabla \phi) ^ {- 1} [ \nabla \phi (y ^ {\prime}) - c x ] \Longrightarrow y ^ {*} = \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \\ \end{array}
$$
Note that according to our assumption,  $y^{*}\in \mathcal{V}$  . For any  $y$
$$
f (y) \geq f \left(y ^ {*}\right) = \langle x, y ^ {*} - y ^ {\prime} \rangle + \frac {1}{c} D _ {\Phi} \left(y ^ {*}, y ^ {\prime}\right) \tag {3}
$$
In order to simplify  $D_{\Phi}(y^{*},y^{\prime})$  , we will use the definition of  $\phi^{\ast}(z)$  . In particular, for any  $y$
$$
\begin{array}{l} \phi (y) = \max  _ {z} [ \langle z, y \rangle - \phi^ {*} (z) ] \quad ; \quad z ^ {*} = \arg \max  _ {z} [ \langle z, y \rangle - \phi^ {*} (z) ] \Longrightarrow y = \nabla \phi^ {*} (z ^ {*}) \Longrightarrow z ^ {*} = \nabla \phi (y) \\ \Rightarrow \phi (y) = \left\langle \nabla \phi (y), y \right\rangle - \phi^ {*} (\nabla \phi (y)) \tag {4} \\ \end{array}
$$
$$
\begin{array}{l} D _ {\Phi} \left(y ^ {*}, y ^ {\prime}\right) = \phi \left(y ^ {*}\right) - \phi \left(y ^ {\prime}\right) - \langle \nabla \phi \left(y ^ {\prime}\right), y ^ {*} - y ^ {\prime} \rangle \\ = \left[ \left\langle \nabla \phi \left(y ^ {*}\right), y ^ {*} \right\rangle - \phi^ {*} \left(\nabla \phi \left(y ^ {*}\right)\right) \right] - \phi \left(y ^ {\prime}\right) - \left\langle \nabla \phi \left(y ^ {\prime}\right), y ^ {*} - y ^ {\prime} \right\rangle \\ \end{array}
$$
(using Eq. (4) to simplify the first term)
Let us focus on the first term and simplify it,
$$
\begin{array}{l} \langle \nabla \phi (y ^ {*}), y ^ {*} \rangle - \phi^ {*} (\nabla \phi (y ^ {*})) = \langle \nabla \phi (\nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ]), \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \rangle - \phi^ {*} (\nabla \phi (\nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ])) \\ = \langle [ \nabla \phi (y ^ {\prime}) - c x ], \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \rangle - \phi^ {*} ([ \nabla \phi (y ^ {\prime}) - c x ]) (\text {F o r a n y} z, \nabla \phi (\nabla \phi^ {*} (z)) = z) \\ \end{array}
$$
Using the above relations,
$$
\begin{array}{l} D _ {\Phi} \left(y ^ {*}, y ^ {\prime}\right) = \left\langle \left[ \nabla \phi \left(y ^ {\prime}\right) - c x \right], \nabla \phi^ {*} \left[ \nabla \phi \left(y ^ {\prime}\right) - c x \right] \right\rangle - \phi^ {*} \left(\left[ \nabla \phi \left(y ^ {\prime}\right) - c x \right]\right) - \phi \left(y ^ {\prime}\right) \\ - \left\langle \nabla \phi (y ^ {\prime}), \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] - y ^ {\prime} \right\rangle \\ = \left\langle \nabla \phi (y ^ {\prime}), \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \right\rangle - c \left\langle x, \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \right\rangle \\ - \phi^ {*} ([ \nabla \phi (y ^ {\prime}) - c x ]) - \phi (y ^ {\prime}) - \langle \nabla \phi (y ^ {\prime}), \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] - y ^ {\prime} \rangle \\ \Rightarrow D _ {\Phi} \left(y ^ {*}, y ^ {\prime}\right) = - c \langle x, \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \rangle - \phi^ {*} ([ \nabla \phi (y ^ {\prime}) - c x ]) - \phi (y ^ {\prime}) + \langle \nabla \phi (y ^ {\prime}), y ^ {\prime} \rangle \\ \end{array}
$$
Using the above simplification with Eq. (3),