SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shicheng Liu & Minghui Zhu
School of Electrical Engineering and Computer Science
University Park, PA 16802, USA
{sf15539,muz16}@psu.edu
This paper considers the problem of inferring the behaviors of multiple interacting experts by estimating their reward functions and constraints where the distributed demonstrated trajectories are sequentially revealed to a group of learners. We formulate the problem as a distributed online bi-level optimization problem where the outer-level problem is to estimate the reward functions and the inner-level problem is to learn the constraints and corresponding policies. We propose a novel "multi-agent behavior inference from distributed and streaming demonstrations" (MA-BIRDS) algorithm that allows the learners to solve the outer-level and inner-level problems in a single loop through intermittent communications. We formally guarantee that the distributed learners achieve consensus on reward functions, constraints, and policies, the average local regret (over  $N$  online iterations) decreases at the rate of  $O(1 / N^{1 - \eta_1} + 1 / N^{1 - \eta_2} + 1 / N)$ , and the cumulative constraint violation increases sub-linearly at the rate of  $O(N^{\eta_2} + 1)$  where  $\eta_1,\eta_2\in (1 / 2,1)$ .
Multi-agent systems (MASs) are an effective tool to model networked systems where multiple entities interact with each other to reach certain goals. Due to the lack of a centralized authority, the data in MASs is usually distributed [1, 2]. Therefore, distributed learning is desired for MASs where machine learning models are trained over distributed data sets. Current works on distributed learning include distributed supervised learning [3, 4], distributed unsupervised learning [5, 6], distributed reinforcement learning [7, 8], etc. Recently, distributed learning is applied to learn the behaviors in MASs from distributed experts' demonstrations via inverse reinforcement learning (IRL) [9].
In IRL [10, 11, 12, 13, 14], a learner aims to learn a policy that imitates the expert behaviors in the demonstrations by first learning a reward function. Multi-agent IRL [15, 16, 17] extends IRL to MASs where the reward functions and policies of a group of experts are learned. However, current works on multi-agent IRL have the following two limitations: (i) As mentioned, the demonstration data in MASs is usually distributed while these works assume that there is a centralized learner which can obtain all the demonstrations. (ii) There are usually many underlying constraints in MASs, e.g., avoiding collision with each other and obstacles, and thus learning a reward function combined with a set of constraints is better than learning a single reward function in terms of explaining the experts' behaviors [18, 19]. Therefore, distributed inverse constrained reinforcement learning (D-ICRL) [9] is proposed where a group of learners cooperatively learn the behaviors in an MAS by estimating the experts' reward functions and constraints and each learner can only access a local demonstration set.
While D-ICRL performs over pre-collected distributed data, recent applications [20, 21] of IRL motivate the need for algorithms that can learn from sequentially revealed demonstrations and continuously improve the learned models. For example, inferring a person's intent by observing her
ongoing daily routine [20] and updating evasion strategies by continuously observing the patrollers' behaviors [21]. However, D-ICRL is not efficient to deal with streaming data because it has a double-loop learning structure where the outer loop is to update the constraints and the inner loop needs to find a corresponding reward estimate. While the computation overhead of this double-loop structure is reluctantly acceptable in offline settings, it is too time-consuming for streaming data since the computation may not be finished before the next data arrives. As D-ICRL is the most important baseline of our work, we include a section in Appendix to summarize our improvements from it.
Related works and our improvements. Current theoretical works on online IRL [20, 21] only consider linear reward functions and thus cast the problem as online convex optimization. While these approaches achieve sub-linear regret, their analysis does not hold for non-linear reward functions (e.g., neural networks) as the corresponding objective functions can be non-convex. To quantify the algorithmic performance on online non-convex optimization, "local regret" [22, 23] is proposed which, at each online iteration, quantifies the gradient norms of the average sum of some already-revealed loss functions. Current state-of-the-arts [22, 23] use follow-the-leader-based methods to minimize the local regret. While their methods can achieve the tight regret bound, the follow-the-leader-based methods require multiple gradient descent steps until reaching a near-stationary point at each online iteration. However, if the data arrival rate is fast, the computation may not be done before the next data arrives. To alleviate the computation burden at each online iteration, we use an online gradient descent (OGD) method which only updates the decision variable by one gradient descent step at each online iteration. To the best of our knowledge, no works can quantify the local regret of OGD.
Inspired by [9, 24], we formulate a distributed online bi-level optimization problem where the learners cooperatively learn the reward functions in the outer-level problem and the constraints and corresponding policies in the inner-level problem. Papers [9, 24, 25] use double-loop methods [26, 27, 28] to solve their bi-level optimization problems where they first find a (sub)optimal solution of the inner-level problem in a faster loop and then solve the outer-level problem in a slower loop. However, when it comes to streaming data, the double-loop method needs multiple steps for the inner-level problem before updating the decision variable of the outer-level problem, which can be too slow to finish the update of the outer decision variable when the data is revealed in a fast speed. Therefore, we use a single-loop method which updates both the outer and inner decision variable only once at each iteration. Notice that the state-of-the-arts on single-loop bi-level optimization [29, 30] cannot be directly applied to our problem because they are centralized and require the inner objective function to be strongly convex while our problem does not have these properties.
Contribution statement. Our contributions are threefold. First, we consider the problem where a group of learners cooperatively recover the policies by estimating the reward functions and constraints from distributed streaming demonstrations of cooperative experts. We formulate this "multi-agent behavior inference from distributed and streaming demonstrations" (MA-BIRDS) problem as a distributed online bi-level optimization problem. Second, we propose a novel distributed online gradient descent algorithm for the learners to learn the reward functions, constraints, and the corresponding policies in a single loop where the decision variables of both the outer-level and inner-level problems are updated only once at each online iteration. Third, we prove that the distributed learners achieve consensus in reward functions, constraints, and policies, respectively, at the rate of  $O(1 / N^{\eta_1} + \bar{\epsilon}^N)$ ,  $O(1 / N^{\eta_2} + \bar{\epsilon}^N)$ , and  $O(1 / N^{\eta_1} + 1 / N^{\eta_2} + \bar{\epsilon}^N)$  where  $\bar{\epsilon} \in (0,1)$  and  $\eta_1, \eta_2 \in (1/2,1)$ . The local regret averaged over  $N$  iterations decreases at the rate of  $O(1 / N^{1 - \eta_1} + 1 / N^{1 - \eta_2} + 1 / N)$  and the cumulative constraint violation grows sub-linearly at the rate of  $O(N^{\eta_2} + 1)$ . Moreover, if the reward functions are linear, we prove that the average cumulative reward difference between the experts and learners diminishes at the rate of  $O(1 / N^{1 - \eta_1} + 1 / N)$ .
This section presents the models of the experts and learners.
Experts. There are  $N_{E}$  experts whose decision making is modeled as a constrained Markov game (CMG) [31]. A CMG  $(S, \mathcal{A}, \gamma, T, P_{0}, P, r_{E}, c_{E}, b)$  consists of a state set  $S \triangleq \prod_{i=1}^{N_{E}} S^{(i)}$ , an action set  $\mathcal{A} \triangleq \prod_{i=1}^{N_{E}} \mathcal{A}^{(i)}$ , a discount factor  $\gamma$ , a time horizon  $T$ , and an initial state distribution  $P_{0}$ . The state transition function is  $P$  and  $P(s'|s, a)$  represents the probability of transitioning to state  $s'$  from  $s$  by taking action  $a \triangleq (a^{(1)}, \dots, a^{(N_{E})})$ . Expert  $i$ 's reward function is  $r_{E}^{(i)}: S \times \mathcal{A} \to \mathbb{R}$  and the
experts are cooperative, i.e.,  $r_E \triangleq \sum_{i=1}^{N_E} r_E^{(i)}$ . The cost function of expert  $i$  is  $c_E^{(i)} \triangleq (\omega_E^{(i)})^\top \phi^{(i)}$  where  $\phi^{(i)}: S \times \mathcal{A} \to [0, d_1]^{l^{(i)}}$  is an  $l^{(i)}$ -dimensional cost feature vector,  $d_1$  is a bounded constant, and  $\omega_E^{(i)} \in \mathbb{R}_+^{l^{(i)}}$  is the weight. The cost function of all the experts is  $c_E \triangleq \sum_{i=1}^{N_E} c_E^{(i)}$ . Expert  $i$ 's policy  $\pi_E^{(i)}(a^{(i)}|s)$  represents the probability of expert  $i$  taking action  $a^{(i)}$  at state  $s$  and the joint policy of all the experts is  $\pi_E(a|s) \triangleq \prod_{i=1}^{N_E} \pi_E^{(i)}(a^{(i)}|s)$ . We define  $J_{r_E}(\pi) \triangleq E_{S,A}^{\pi}[\sum_{t=0}^T \gamma^t r_E(S_t, A_t)]$  as the expected cumulative reward under policy  $\pi$  where the initial state is drawn from  $P_0$  and  $J_{c_E}(\pi) \triangleq E_{S,A}^{\pi}[\sum_{t=0}^T \gamma^t c_E(S_t, A_t)]$  as the expected cumulative cost. The experts' policy  $\pi_E$  wants to maximize  $J_{r_E}(\pi)$  subject to  $J_{c_E}(\pi) \leq b$  where  $b$  is the budget. Following [9, 18, 19], we study hard constraints (i.e.,  $b = 0$ ). These experts use  $\pi_E$  to demonstrate  $N_L$  trajectories  $\{\zeta^{[v]}(n)\}_{v=1}^{N_L}$  at each online iteration  $n$  where each trajectory  $\zeta^{[v]}(n) \triangleq s_0^{[v]}(n), a_0^{[v]}(n), \dots, s_T^{[v]}(n), a_T^{[v]}(n)$  is a state-action sequence of all the experts. This distributed online data style can also be found in [32].
Learners. There are  $N_{L}$  learners where each learner  $v$  knows  $(\gamma, T, \{\phi^{(i)}\}_{i=1}^{N_{E}}, \zeta^{[v]}(n))$  and  $\zeta^{[v]}(n)$  is a demonstration observed by learner  $v$  at online iteration  $n$ . Each learner wants to use communications to learn the cost functions by estimating  $\omega_{E} \triangleq [(\omega_{E}^{(1)})^{\top}, \dots, (\omega_{E}^{(N_{E})})^{\top}]^{\top}$  and reward functions using parameterized models  $\{r_{\theta^{(i)}}^{(i)}\}_{i=1}^{N_{E}}$  where  $\theta^{(i)} \in \mathbb{R}^{d^{(i)}}$  is  $d^{(i)}$ -dimensional. Here we relax the linear reward assumption in [9] but keep its linear cost assumption because non-linear cost functions can make the problem ill-defined (explained in Appendix).
Assumption 1. The reward function  $r_{\theta}$  satisfies the following:  $|r_{\theta}(s,a)| \leq C, ||\nabla_{\theta}r_{\theta}(s,a)|| \leq \bar{C}$ , and  $||\nabla_{\theta \theta}^{2}r_{\theta}(s,a)|| \leq \tilde{C}$  for any  $(s,a) \in S \times \mathcal{A}$  and any  $\theta$  where  $C, \bar{C},$  and  $\tilde{C}$  are positive constants.
Notice that Assumption 1 is standard in RL [33, 34, 35].
The communication network is modeled as a time-varying directed graph  $\mathcal{G}(n) \triangleq (\mathcal{V}, \mathcal{E}(n))$  where  $\mathcal{V} \triangleq \{1, \dots, N_L\}$  is the node (learner) set and  $\mathcal{E}(n) \subseteq \mathcal{V} \times \mathcal{V}$  is the set of directed edges (communication links) at time  $n$ . The edge  $(v, v') \in \mathcal{E}(n)$  means that learner  $v$  receives information from learner  $v'$  at time  $n$  and  $(v, v) \in \mathcal{E}(n)$  for all  $n \geq 0$ . The adjacency matrix of the graph at time  $n$  is  $W(n) \triangleq [W^{[vv']}(n)]_{v, v' \in \mathcal{V}} \in \mathbb{R}^{N_L \times N_L}$  where  $W^{[vv']}(n) = 0$  if and only if  $(v, v') \notin \mathcal{E}(n)$ . The set of neighbors of learner  $v$  at time  $n$  is  $\mathcal{N}^{[v]}(n) \triangleq \{v' \in \mathcal{V} | (v, v') \in \mathcal{E}(n)\}$ .
Assumption 2. There exists an integer  $B \geq 1$  such that the graph  $(\mathcal{V}, \mathcal{E}(n) \cup \dots \cup \mathcal{E}(n + B - 1))$  is strongly connected for all  $n \geq 0$ .
Assumption 3. The adjacency matrix  $W(n)$  has the following properties: (i)  $\mathbf{1}^{\top}W(n) = \mathbf{1}^{\top}$  and  $W(n)\mathbf{1} = \mathbf{1}$  where  $\mathbf{1}$  is the column vector whose entries are all ones. (ii) There is an  $\epsilon \in (0,1)$  such that  $W^{[vv]}(n) \geq \epsilon$  for all  $v \in V$  and  $W^{[vv']}(n) \geq \epsilon$  if  $(v,v') \in \mathcal{E}(n)$ .
Notice that these two assumptions are standard in distributed learning [3, 9, 36].
Figure 1 shows that the learners stand outside the MAS observing the sequential data revealed by the experts and collaboratively learn the reward functions and constraints. Each learner aims to learn the reward functions and constraints of all the experts where  $\theta^{[v]}$  and  $\omega^{[v]}$  are the reward and cost function parameter estimates of all the experts learned by
Figure 1: Relation between experts and learners
Notions and Notations. Define  $\theta \triangleq [(\theta^{(1)})^{\top},\dots ,(\theta^{(N_E)})^{\top}]^{\top}$  and  $r_{\theta} \triangleq \sum_{i=1}^{N_E} r_{\theta^{(i)}}^{(i)}$ . We use  $\theta^{[v]}$  to represent learner  $v$ 's learned  $\theta$ . Therefore, the dimension of each  $\theta^{[v]}$  is  $\sum_{i=1}^{N_E} d^{(i)}$ . Given a trajectory  $\zeta = s_0, a_0, \dots, s_T, a_T$ , the empirical cumulative reward under  $r_{\theta}$  is  $\hat{J}_{r_{\theta}}(\zeta) \triangleq \sum_{t=0}^{T} \gamma^t r_{\theta}(s_t, a_t)$ , the empirical cumulative cost feature is defined as  $\hat{\mu}(\zeta) \triangleq \sum_{t=0}^{T} \gamma^t \phi(s_t, a_t)$  where  $\phi \triangleq [(\phi^{(1)})^{\top}, \dots, (\phi^{(N_E)})^{\top}]^{\top}$ . The expectation of cumulative cost feature under a given policy  $\pi$  is  $\mu(\pi) \triangleq E_{S,A}^{\pi}[\sum_{t=0}^{T} \gamma^t \phi(S_t, A_t)]$  and the expectation of cumulative reward  $r_{\theta}$  under a
given policy  $\pi$  is  $J_{r_{\theta}}(\pi) \triangleq E_{S,A}^{\pi}[\sum_{t=0}^{T} \gamma^{t} r_{\theta}(S_{t},A_{t})]$ . The set of all stochastic policies is  $\Pi$  where every  $\pi \in \Pi$  satisfies  $\pi(a|s) \geq 0$  for any  $(s,a) \in S \times \mathcal{A}$  and  $\int_{a \in \mathcal{A}} \pi(a|s) da = 1$  for any  $s \in S$ .
In MA-BIRDS, the learners collaboratively learn the experts' policy  $\pi_E$  by estimating the reward function  $r_E$  and cost function  $c_{E}$ , and each learner  $v$  does not share its private data such as local trajectory  $\zeta^{[v]}$  and local estimates  $\hat{J}_{r_\theta}(\zeta^{[v]})$  and  $\hat{\mu} (\zeta^{[v]})$ . While it seems that learning a well-structured reward function is enough to prevent "bad" movements by assigning negative reward, we include a section (in Appendix) to further discuss the benefits of learning both reward and cost functions.
Many IRL works [9, 11, 24, 37?] have a bi-level learning structure where the outer level is to learn a reward function and the inner level is to learn a corresponding policy by solving an RL problem under the current learned reward function. Inspired by their bi-level structure, we formulate a bi-level optimization problem where the outer level is to learn a reward function  $r_{\theta}$  and the inner level is to find the cost function and policy corresponding to  $r_{\theta}$ . In what follows, we first define the inner-level optimization problem and then introduce the bi-level optimization problem.
The inner-level optimization. Given a learned reward function  $r_{\theta}$ , the corresponding policy  $\pi_{r_{\theta}}$  is the optimal solution of the following constrained RL problem where the reward function is  $r_{\theta}$ :
$$
\pi_ {r _ {\theta}} = \underset {\pi \in \Pi} {\arg \max } \left\{H (\pi) + J _ {r _ {\theta}} (\pi), \quad \text {s . t .} \mu (\pi) = \frac {1}{N _ {L}} \sum_ {v = 1} ^ {N _ {L}} \hat {\mu} \left(\zeta^ {[ v ]} (n)\right) \right\}, \tag {1}
$$
where  $H(\pi) \triangleq \sum_{t=0}^{T} E_{S,A}\left[-\gamma^{t} \ln \pi(A_{t}|S_{t})\right]$  is causal entropy [38]. The constraint in problem (1) is cost feature expectation matching similar to the spirit of "feature expectation matching" in [11, 38].
However, the policy  $\pi_{r_\theta}$  is hard to get because problem (1) is non-convex. A standard way to tackle this difficulty is dual methods [39], therefore, we introduce the dual function of problem (1):  $G(\omega; \theta, n) \triangleq \max_{\pi \in \Pi} H(\pi) + J_{r_\theta}(\pi) + \omega^\top (\mu(\pi) - \frac{1}{N_L} \sum_{v=1}^{N_L} \hat{\mu}(\zeta^{[v]}(n)))$  where the dual variable  $\omega$  is used to estimate  $\omega_E$ . Notice that the dual function  $G(\omega; \theta, n)$  is convex in  $\omega$  [39].
Lemma 1. The optimal solution of problem (1) is the constrained soft Bellman policy  $\pi_{\omega^{*}(\theta ,n);\theta}$  and its parameter  $\omega^{*}(\theta ,n)$  is the optimal solution of the dual problem  $\min_{\omega}G(\omega ;\theta ,n)$ .
The expression of constrained soft Bellman policy is in Appendix. Lemma 1 shows that  $\pi_{\omega^{*}(\theta ,n);\theta} =$ $\pi_{r_\theta}$  and  $\omega^{*}(\theta ,n) = \arg \min_{\omega}G(\omega ;\theta ,n)$ . Therefore, we can solve problem (1) by solving its dual problem. We use the dual problem to be the inner-level problem where  $\omega^{*}(\theta ,n)$  is the learned cost function and  $\pi_{\omega^{*}(\theta ,n);\theta}$  is the learned policy corresponding to the current learned reward function  $r_\theta$ .
The bi-level optimization. Given a reward function  $r_{\theta}$ , the inner-level problem  $\arg \min_{\omega} G(\omega; \theta, n)$  can find the corresponding cost function and policy. The outer level aims to learn  $r_{\theta}$  via minimizing the following loss function over  $\theta$ :
$$
L \left(\theta , \omega^ {*} (\theta , n), n\right), \quad \text {s . t .} \quad \omega^ {*} (\theta , n) = \underset {\omega} {\arg \min } G (\omega ; \theta , n), \tag {2}
$$
where  $L(\theta, \omega^{*}(\theta, n), n) \triangleq -\sum_{v=1}^{N_{L}} \sum_{t=0}^{T} \gamma^{t} \ln \pi_{\omega^{*}(\theta, n); \theta}(a_{t}^{[v]}(n) | s_{t}^{[v]}(n))$  is the negative log likelihood of the trajectories  $\{\zeta^{[v]}(n)\}_{v=1}^{N_{L}}$  received at time  $n$  under the policy  $\pi_{\omega^{*}(\theta, n); \theta}[9, 40]$  and  $\zeta^{[v]}(n) = \{(s_{t}^{[v]}(n), a_{t}^{[v]}(n))\}_{0 \leq t \leq T}$ . The likelihood function is widely used in IRL [9?, 40] to learn the reward function.
Notice that (2) requires all the demonstrations  $\{\zeta^{[v]}(n)\}_{v = 1}^{N_L}$  at time  $n$ . However, each learner  $v$  can only observe  $\zeta^{[v]}(n)$  and formulate its local negative log likelihood function  $L^{[v]}(\theta ,\omega^{*}(\theta ,n),n)\triangleq -\sum_{t = 0}^{T}\gamma^{t}\ln \pi_{\omega^{*}(\theta ,n);\theta}(a_{t}^{[v]}(n)|s_{t}^{[v]}(n))$  and local dual function  $G^{[v]}(\omega ;\theta ,n)\triangleq \max_{\pi \in \Pi}H(\pi) + J_{r_\theta}(\pi) + \omega^\top (\mu (\pi) - \hat{\mu} (\zeta^{[v]}(n)))$ . Notice that  $L = \sum_{v = 1}^{N_L}L^{[v]}$  and  $G = \frac{1}{N_L}\sum_{v = 1}^{N_L}G^{[v]}$ .
As the demonstrations are streaming, we have a sequence of loss functions  $\{L(\cdot ,\omega^{*}(\cdot ,n),n)\}_{n\geq 1}$ . We use this sequence of loss functions to formulate an online learning problem and a common problem for online learning is to minimize the regret:  $\sum_{n = 1}^{N}L(\theta (n),\omega^{*}(\theta (n),n),n) - \sum_{n = 1}^{N}L(\theta^{*},\omega^{*}(\theta^{*},n),n)$
which quantifies the difference of the accumulative losses between the learned parameter  $\theta$  and the best parameter  $\theta^{*}$  in hindsight. However, it is too challenging to minimize the regret in our case because  $L$  is non-convex [22]. Therefore, we use local regret [22, 23] which is widely used in online non-convex optimization. It quantifies the general stationarity of a sequence of loss functions. In specific, given a sequence of loss functions  $\{f(\cdot ,n)\}_{n\geq 1}$ , the local regret [22, 23] at online iteration  $n$  is defined as  $\| \frac{1}{l}\sum_{i = 0}^{l - 1}\nabla f(x(n),n - i)\| ^2$  which quantifies the gradient norms of the average of  $l$  previously received loss functions under the current learned parameter  $x(n)$ . The total local regret is defined as the sum of the local regret at every online iteration  $n$ , i.e.,  $\sum_{n = 1}^{N}\| \frac{1}{l}\sum_{i = 0}^{l - 1}\nabla f(x(n),n - i)\| ^2$ . In our case, we replace  $f$  with our loss function (2) and thereby formulate the local regret (3)-(4) which has a distributed bi-level formulation. We want to minimize the local regret (3)-(4).
$$
\sum_ {n = 1} ^ {N} \left|\right| \frac {1}{l} \sum_ {i = 0} ^ {l - 1} \sum_ {v = 1} ^ {N _ {L}} \nabla L ^ {[ v ]} (\theta (n), \omega^ {*} (\theta (n), n), n - i) \left. \right| | ^ {2}, \tag {3}
$$
$$
\text {s . t .} \omega^ {*} (\theta (n), n) = \arg \min  _ {\omega} \sum_ {v = 1} ^ {N _ {L}} G ^ {[ v ]} (\omega ; \theta (n), n). \tag {4}
$$
The time window length is  $1 \leq l \leq N$  and  $L^{[v]}(\theta, \omega, i) = 0$  if  $i \leq 0$  [22]. The outer-level problem (3) is to learn the reward parameter  $\theta$  and the inner-level problem (4) is to learn the cost parameter  $\omega$  given  $\theta$ . The learned policy is the constrained soft Bellman policy  $\pi_{\omega; \theta}$  with parameters  $(\theta, \omega)$ .
4 Algorithm and Performance Guarantee
This section consists of two subsections where the first one introduces an approximation method to solve the bi-level optimization problem (3)-(4) in a single loop and the second one introduces a consensus-based method for the multiple learners to solve the problem in a distributed way.
4.1 Approximation-based single-loop method
In this part, we develop an approximation-based single-loop method which (i) does not use the exact gradient of the outer-level problem (3) but an approximation of the gradient; (ii) solves the outer-level and inner-level problems in a single loop. In the following analysis, for simple notations, we omit the time index  $n$  and imply that the analysis holds for all  $n$ .
Lemma 2. The problem  $\min_{\omega} G(\omega; \theta)$  has a unique optimal solution  $\omega^{*}(\theta)$  for any  $\theta$ .
Since the inner-level problem (4) is unconstrained and  $\omega^{*}(\theta)$  is its optimal solution, then we have  $\nabla_{\omega}G(\omega^{*}(\theta);\theta) = 0$ . Taking derivative with respect to  $\theta$  on both sides renders:
$$
\nabla_ {\omega \theta} ^ {2} G (\omega^ {*} (\theta); \theta) + \nabla_ {\omega \omega} ^ {2} G (\omega^ {*} (\theta); \theta) \nabla \omega^ {*} (\theta) = 0 \Rightarrow \nabla \omega^ {*} (\theta) = - M (\theta , \omega^ {*} (\theta)) ^ {\top},
$$
where  $M(\theta, \omega) \triangleq \nabla_{\theta \omega}^2 G(\omega; \theta)[\nabla_{\omega \omega}^2 G(\omega; \theta)]^{-1}$ . Then, using the chain rule, we have:
$$
\nabla L \left(\theta , \omega^ {*} (\theta)\right) = \nabla_ {\theta} L \left(\theta , \omega^ {*} (\theta)\right) - M \left(\theta , \omega^ {*} (\theta)\right) \nabla_ {\omega} L \left(\theta , \omega^ {*} (\theta)\right). \tag {5}
$$