SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
l _ {\mathrm {m e m}} \left(\pi^ {*}\right) \leq \max  \left(m _ {\text {r e w a r d}} ^ {\mathcal {M}}, m _ {\text {t r a n s i t}} ^ {\mathcal {M}}, l _ {\mathrm {m e m}} \left(\pi^ {*}\right) - 1\right) \tag {21}
$$
$$
l _ {\text {m e m}} \left(\pi^ {*}\right) - 1 <   \max  \left(m _ {\text {r e w a r d}} ^ {\mathcal {M}}, m _ {\text {t r a n s i t}} ^ {\mathcal {M}}\right) \tag {22}
$$
$$
l _ {\text {v a l u e}} ^ {\mathcal {M}} \left(\pi^ {*}\right) \leq \max  \left(m _ {\text {r e w a r d}} ^ {\mathcal {M}}, m _ {\text {t r a n s i t}} ^ {\mathcal {M}}\right) \tag {23}
$$
$c(\pi^{*})$  may be not unique among all optimal policies. For example, consider an MDP described in Fig. 9, and two optimal deterministic Markov policy,  $\pi_1^*, \pi_2^*$ , only different at the state P2:  $\pi_1^*(\mathrm{P}2) = x$  while  $\pi_2^*(\mathrm{P}2) = y$ . It is easy to compute the credit assignment length in this MDP, because only state P1 contains suboptimal action. At state P1, the optimal action  $y$  starts to be better than suboptimal action  $x$  at the next 2 steps for  $\pi_2^*$ , while at the next 3 steps for  $\pi_1^*$ . Thus,  $c(\pi_1^*) = 3$  and  $c(\pi_2^*) = 2$ .
Figure 9: An MDP that shows the credit assignment length over optimal policies is not a constant. Each node is a state with its name on the top and its reward at the bottom. The initial state is P1. The action space is  $\{x, y\}$ . The finite horizon  $T \geq 4$  and  $\gamma = 1$ .
B Details on Environments Related to Memory and Credit Assignment
B.1 Abstract Problems
Example 3 (n-order MDPs). It is a POMDP where the recent  $n$  observations form the current state, i.e. the transition is  $P(o_{t + 1} \mid o_{t - n + 1:t}, a_t)$  and the reward is  $R_t(o_{t - n + 1:t}, a_t)$ .
Example 4 (Episodic reward). It is a POMDP with a finite horizon of  $T$ . The reward function is episodic in that  $R_{t} = 0$  for  $\forall t < T$ .
Example 5 (Decomposed episodic reward (Ren et al., 2022)). It has an episodic reward (Eg. 4), decomposed into the sum of Markovian rewards:  $R_{T}(h_{1:T}, a_{T}) = \sum_{t=1}^{T} r'(o_{t}, a_{t})$ , where  $r': \mathcal{O} \times \mathcal{A} \to \mathbb{R}$  is a Markovian reward function. The transition  $P(o_{t+1} \mid o_{t}, a_{t})$  is Markovian. The smallest policy memory length for an optimal policy  $l_{\mathrm{mem}}(\pi^{*})$  can be shown as 1 (see below).
Proof of decomposed episodic reward problem requiring  $l_{\mathrm{mem}}(\pi^{*}) = 1$  at most. Suppose the policy is non-stationary, composed of  $\{\pi_t\}_{t=1}^T$ . For any time step  $t \leq T$  in the POMDP  $\mathcal{M}$ ,
$$
\begin{array}{l} Q ^ {\pi} \left(h _ {1: t}, a _ {t}\right) = \mathbb {E} _ {\pi , \mathcal {M}} \left[ \sum_ {k = t} ^ {T} r _ {k} \mid h _ {1: t}, a _ {t} \right] (24) \\ = \mathbb {E} _ {\pi , \mathcal {M}} \left[ r _ {T} \mid h _ {1: t}, a _ {t} \right] = \mathbb {E} _ {\pi , \mathcal {M}} \left[ \sum_ {k = 1} ^ {T} r ^ {\prime} \left(o _ {k}, a _ {k}\right) \mid h _ {1: t}, a _ {t} \right] (25) \\ = \sum_ {k = 1} ^ {t} r ^ {\prime} \left(o _ {k}, a _ {k}\right) + \mathbb {E} _ {\pi_ {t + 1: T}, o _ {t + 1} \sim P \left(\left| o _ {t}, a _ {t}\right), \mathcal {M} \right.} \left[ \sum_ {k = t + 1} ^ {T} r ^ {\prime} \left(o _ {k}, a _ {k}\right) \mid h _ {1: t}, a _ {t} \right] (26) \\ \end{array}
$$
By taking argmax,
$$
\underset {a _ {t}} {\operatorname {a r g m a x}} Q ^ {\pi} \left(h _ {1: t}, a _ {t}\right) = \underset {a _ {t}} {\operatorname {a r g m a x}} r ^ {\prime} \left(o _ {t}, a _ {t}\right) + \mathbb {E} _ {\pi_ {t + 1: T}, o _ {t + 1} \sim P \left(\left| o _ {t}, a _ {t}\right), \mathcal {M}\right)} \left[ \sum_ {k = t + 1} ^ {T} r ^ {\prime} \left(o _ {k}, a _ {k}\right) \mid h _ {1: t}, a _ {t} \right] \tag {27}
$$
Now we show that there exists an optimal policy that is Markovian by backward induction. When  $t = T$ , (27) reduces to
$$
\underset {a _ {T}} {\operatorname {a r g m a x}} Q ^ {\pi} \left(h _ {1: T}, a _ {T}\right) = \underset {a _ {T}} {\operatorname {a r g m a x}} r ^ {\prime} \left(o _ {T}, a _ {T}\right) \tag {28}
$$
thus Markovian policy  $\pi_T(o_T) = \mathrm{argmax}_{a_T} r'(o_T, a_T)$  is optimal. Now assume  $\pi_{t+1:T}$  are Markovian and optimal, consider the step  $t$  in (27), all future observations and actions  $(o_{t+1:T}, a_{t+1:T})$  rely on  $(o_t, a_t)$  by the property of Markovian transition and induction, thus optimal action  $a_t^*$  can only rely on  $o_t$ .
Example 6 (Delayed rewards of  $n$  steps (Arjona-Medina et al., 2019)). The transition  $P(o_{t+1} \mid o_t, a_t)$  is Markovian, and the reward  $R_t(o_{t-n+1}, a_{t-n+1})$  depends on the observation and action  $n$  steps before. Decomposed episodic rewards (Eq. 5) can be viewed as an extreme case of delayed rewards.
Example 7 (Delayed execution of  $n$  steps (Derman et al., 2021)). The transition is  $P(o_{t+1} \mid o_t, a_{t-n})$ , and the reward is  $R_t(o_t, a_{t-n})$ .
Table 2: The memory and credit assignment lengths required in the related abstract problems, using our notion. Assume all tasks have a horizon of  $T$ .
Abstract Problem M	l_mem(π*)	m_Mreward	m_Mtransit	c^M
MDP	[0,1]	[0,1]	[0,1]	[1,T]
n-order MDP (Eg. 3)	[0,n]	[0,n]	[0,n]	[1,T]
POMDP	[0,T]	[0,T]	[0,T]	[1,T]
Episodic reward (Eg. 4)	T	T	[0,T]	T
Decomposed episodic reward (Eg. 5)	[0,1]	T	[0,1]	T
Delayed rewards of n steps (Eg. 6)	[0,1]	n	[0,1]	[n,T]
Delayed execution of n steps (Eg. 7)	n	n	n	[n,T]
Delayed observation of n steps (Eg. 8)	n	n	n	[0,T]
Example 8 (Delayed observation of  $n$  steps (Katsikopoulos and Engelbrecht, 2003)). The transition is  $P(o_{t+1} \mid o_t, a_{t-n})$ , and the reward is  $R_t(o_t, a_{t-n:t})$ .
We summarize our analysis in Table 2.
B.2 Concrete Benchmarks for Memory
T-Maze (Bakker, 2001). The task is almost the same as our Passive T-Maze, except that the agent is not penalized if it moves left instead of right in the corridor. The modification results in a credit assignment length closer to  $T$ . The authors test RL algorithms with the largest horizon of 70.
MiniGrid-Memory (Chevalier-Boisvert et al., 2018). This task has the same structure as our Active T-Maze. In the worst-case scenario, the optimal policy can traverse the horizontal corridor back and forth, and then follow the vertical corridor to solve the task. Thus, the memory and credit assignment lengths of optimal policy is upper bounded by three times the maze side length, which is  $3 \times 17 = 51$  for a  $17 \times 17$  maze (MiniGrid-MemoryS17Random-v0). The reward memory length can equal the horizon of 1445.
Passive Visual Match (Hung et al., 2018). The environment consists of a partially observable  $7 \times 11$  grid-world, where the agent can only observe a  $5 \times 5$  grid surrounding itself. The task is separated into three phases. In the first, the agent passively observes a randomly generated color. In the second, the agent must pick up apples with immediate reward. In the final phase, three random colored squares are displayed, and the agent must pick up the one corresponding to the color observed in the first phase, which tests the agent's ability to recall temporally distant events. Since rewards are relatively immediate with respect to the agent's actions, the credit assignment length of optimal policy is upper bounded by the length of the shortest path needed to traverse the grid-world, i.e.  $7 + 11 = 18$ .
T Maze Long and T Maze Long Noise (Beck et al., 2019). Both tasks are structurally similar to our Passive T-Maze. In T Maze Long, the agent is forced to take the forward action until reaching the junction (i.e., the action space is a singleton except at the junction). At the junction, the agent is required to choose the goal candidate matching the color it observed at the starting position. Therefore, the memory length is the horizon  $T$  and the credit assignment length is 1 since only the terminal action can be credited. T Maze Long Noise introduces uniform noise appended to the observation space. Both tasks have a horizon and corridor length of 100.
PsychLab (Fortunato et al., 2019). This environment simulates a lab environment in first person, and can be sub-categorized into four distinct tasks. In all tasks, one or many images are shown, and the overall objective of all tasks is to correctly memorize the contents of the images for future actions. Images are passively observed by the agent, thus eliminating the need for long-term credit assignment, and instead focusing on long-term memory.
Spot the Difference (Fortunato et al., 2019). In this task, two nearly identical rooms are randomly generated, and the agent must correctly identify the differences between the rooms. The agent must navigate through a corridor of configurable length to go from one room to another. While this task was designed to test an agent's memory, effective temporal credit assignment is also necessary for an optimal policy. Similar in flavor to Active Visual Match (Hung et al., 2018), an agent must first explore the initial room to be able to identify the differences in the second room. Rewards for this exploratory phase are only given near the end of the task once the agent has identified all differences. Long-term credit assignment is needed to incorporate these actions into an optimal policy.
Goal Navigation (Fortunato et al., 2019). A random maze and goal state is generated at the start of each episode. The agent is rewarded every time it reaches the goal state, and is randomly re-initialized in the maze each time it does so. Crucial information about the maze needs to be memorized throughout the episode for optimal performance, while exploratory actions may also be taken early for better quicker navigation later. Consequently, this environment fails to cleanly disentangle temporal credit assignment from memory.
Numpad (Parisotto et al., 2020; Humplik et al., 2019). This is a visual RL problem with continuous control. In this task, there is an  $N \times N$  number pad, and the goal of the agent is to activate up to  $N^2$  pads in a specific order. The order is randomized at the beginning of each episode. The agent does not know the order, and has to explore to figure it out. The agent will receive an immediate reward of 1 if it hits the next pad correctly; otherwise, the agent must restart. As a result, the credit assignment length is short, although the exact value is hard to determine. The optimal policy needs to memorize the order of  $N^2$  pads during the whole episode, thus the memory length is upper bounded by the full horizon of 500.
Memory Length (Osband et al., 2020, A.6). The task is similar to our Passive T-Maze in terms of reward memory and credit assignment lengths. However, we did not adopt this task because the agent cannot change the observation through its action, making the task solvable by supervised learning. Moreover, in this task, the observation is i.i.d. sampled at each time step, making a transition memory length of only 1. The authors set the horizon at a maximum of 100.
Reacher-pomdp (Yang and Nguyen, 2021). In this task, the goal position in Reacher is only revealed at the first step. The optimal policy has to memorize the goal until it is reached. The policy memory length is thus long, but not necessarily the full horizon. The task provides dense rewards, so the credit assignment length is short, although the exact value is unclear. The horizon is set to 50.
T-Maze (Lambrechts et al., 2021). This task has the same structure as T-Maze (Bakker, 2001). The RL algorithms are tested with the largest horizon of 200.
Ballet (Lampinen et al., 2021). ballet is a 2D gridworld task with the agent situated at the center. In its most challenging variant, there are 8 dancers around the agent, each executing a 16-step dance sequentially. A 48-step interval separates two consecutive dances. Consequently, the agent observes the environment for a minimum of  $16 * 8 + 48 * (8 - 1) = 464$  steps. Following the final dance, the agent receives a reward upon reaching the dancer performing a specific routine. Hence, the optimal policy's memory length is at least 464 steps, and the reward memory length is capped by the horizon of 1024. The agent's actions are constrained to the post-dance phase, resulting in a short credit assignment length determined by the time taken to reach the correct dancer.
HeavenHell (Esslinger et al., 2022; Thrun, 1999). This task has the same structure as our Active T-Maze. The agent has to first move south to reach the oracle and then move north to reach the heaven. The reward is given only at the terminal step. The credit assignment length is the full horizon, which is around 20. The Gym-Gridverse and Car Flag tasks in their work have the same structures as HeavenHell but feature more complex observation spaces.
Memory Cards (Esslinger et al., 2022). The task can be viewed as a discrete version of Numpad. There are  $N$  pairs of cards, with each pair sharing the same value. The agent can observe a random card's value at each step, after which the card value is hidden. The agent then takes action to choose the paired card that matches the observed card. It will get an immediate reward of 0 if it makes the correct choice, otherwise, a reward of  $-1$ . The credit assignment length is thus 1. The minimal memory length is around  $O(N)$  to observe all cards' values, but hard to determine exactly due to randomness. The finite horizon is set to 50.
Memory Maze (Pasukonis et al., 2022). In this environment, the agent is placed in a randomly generated  $N \times N$  maze with  $N = 15$  at most. There are  $K$  colored balls in the maze, and the agent must pick up the  $K$  colored balls in a specified order (randomized in every episode). As for the reward memory, the reward is given only when the agent picks up the correct ball and solely depends on current observation and action, thus  $m_{\text{reward}}^{\mathcal{M}} = 1$ . Although their longest horizon  $T$  can be 4000, the credit assignment length is upper bounded by the size of the maze. In the worst-case scenario, an optimal policy can perform a naive search on the entire maze and uncover the maze plan to obtain a reward, which costs  $O(N^{2})$  steps. Nevertheless, the optimal policy memory length can be much longer than  $O(N^{2})$  steps and close to the horizon  $T$ . This is because the optimal policy may need to memorize the initial subsequence of observations to infer the maze plan when taking current actions.
POPGym (Morad et al., 2023). This benchmark comprises 15 tasks from which we examine several representative ones:
- The diagnostic POMDP task, Repeat First, provides a reward based on whether the action repeats the first observation. Thus, the credit assignment length is 1. Although this task indeed has a reward memory length of  $T$ , the optimal policy memory length can be as low as 2 simply by recalling the previous optimal action. The horizon  $T$  of this task can be 16 decks (831 steps).
- In another diagnostic POMDP task, Autoencode, an agent first observes a sequence of cards, and then must reproduce the exact sequence in a reverse order to earn rewards. Here, the credit assignment length is 1, and the optimal policy memory length is at most twice the number of cards, which is 6 decks (311 steps).
- The control POMDP tasks, Stateless Cartpole and Pendulum, occlude velocities, making them similar to the "-P" tasks we previously evaluated. These tasks require short-term dependencies.
- The game POMDP task, Battleship, requires the agent to deduce the locations of all ships on a board without observing the board. This requires a memory length upper bounded by the area of the board, i.e.  $12^2 = 144$ , and short-term credit assignment. Similarly, the Concentration game requires the agent to maximize the number of card matches in an episode. Although the reward memory length of horizon  $T$ , the optimal policy memory length can be restricted to the number of cards, i.e.  $2 * 52 = 104$ .
Overall, this benchmark indeed emphasizes the evaluation of memory capabilities with short-term credit assignment, where the memory length is limited by the maximal horizon of around 831.
Memory Gym (Pleines et al., 2023). This benchmark has a pixel-based observation space and a discrete action space for controlling movement within a gridworld. The Mortar Mayhem (MM) task is considered as one of the hardest tasks in this benchmark. MM requires the agent to first memorize a sequence of five commands, and then execute the exact commands in order. A reward is given when the agent successfully executes one command. Thus, the credit assignment length for optimal policy is at most the area of the gridworld, which is  $5 * 5 = 25$ . The memory length is at most the full horizon, which according to their Table 3 is at most 135.
B.3 Concrete Benchmarks for Credit Assignment
Active Visual Match (Hung et al., 2018). The agent has to open the door to observe the color in the first phase, picks apples up in the second phase, and goes to the door with the corresponding color in the final phase. Thus, the policy needs to memorize the color.
Episodic MuJoCo (Liu et al., 2019; Ren et al., 2022). It is a decomposed episodic reward problem (Eg. 5). The terminal reward is the sum of the original Markov dense rewards. They tried the maximal horizon as 1000.
Umbrella Length (Osband et al., 2020, A.5). It is an MDP and the initial action taken by the agent will be carried through the episode. The terminal reward depends on the terminal state, which is determined by the initial action. Thus, the credit assignment length is the full horizon, at most 100. There are also some random intermediate rewards that are independent of actions.
Key-to-Door (Raposo et al., 2021; Mesnard et al., 2021; Chen et al., 2021). The agent has to reach the key in the first phase, picks apples up in the second phase, and goes to the single door. Thus, the policy has little memory, while the terminal reward depends on whether the agent picks the key up in the first phase. In this sense, it can be also viewed as a decomposed episodic reward problem with some noisy immediate rewards. Raposo et al. (2021) tried the maximal horizon as 90. The horizon in Mesnard et al. (2021); Chen et al. (2021) is unknown but should be similar.
Catch with delayed rewards (Raposo et al., 2021). It is a decomposed episodic reward problem (Eg. 5). The terminal reward is given by how many times the agent catches the ball in an episode. They tried the maximal horizon as 280.
Push-r-bump (Yang and Nguyen, 2021). The agent has to first explore to find out the correct bump, and then move towards it to receive a terminal reward. The credit assignment length is thus long. The horizon is 50.
C.1 Memory-Based RL Implementation
Our implementation builds upon prior work (Ni et al., 2022), providing a strong baseline on a variety of POMDP tasks, including Key-to-Door and PyBullet tasks used in this work. We denote this implementation as POMDP-baselines. All parameters of our agents in our experiments are trained end-to-end with model-free RL algorithms from scratch. The following paragraphs provide detailed descriptions.
Figure 10: Agent architectures for memory-based DDQN (left) and TD3 (right) in our implementation. While rewards are not used as inputs in our experiments, they are included in this figure for completeness. Compared to POMDP-baselines (Ni et al., 2022), we have removed shortcuts from  $o_t$  to the MLPs and shared the same sequence model between the actor and critic.
Sharing the sequence model in actor-critic with frozen critic parameters in the actor loss. Our agent architecture, depicted in Fig. 10, simplifies the one in POMDP-baselines by sharing the sequence model between the actor and critic. Previously, POMDP-baselines found sharing the RNNs between actor and critic causes instability in gradients, thus they adopt the separate sequence encoders for actor and critic. We examine their code and find that it is mainly due to the gradient bias in the actor loss. This issue is also discussed in prior (Yang and Nguyen, 2021) and concurrent works (Grigsby et al., 2023) on memory-based RL and related to sharing CNN encoders in visual RL (Yarats et al., 2021a,b). Below we formalize the issue and introduce our solution.
In the separate encoder setting, we consider an actor and a critic with distinct history encoders  $f_{\phi_{\pi}}$  and  $f_{\phi_Q}$ , mapping a history  $h$  into latent states  $z_{\pi}$  and  $z_{Q}$ , respectively. The history encoder parameters are denoted as  $\phi_{\pi}$  and  $\phi_{Q}$ . The actor is parameterized by  $\pi_{\nu}(f_{\phi_{\pi}}(h))$  and the critic is parameterized by  $Q_{\omega}(f_{\phi_Q}(h),a)$  with  $\nu$  and  $\omega$  denoting their respective MLP parameters. Considering the DDPG algorithm (Lillicrap et al., 2016) for training<sup>10</sup> and a tuple of data  $(h,a,o',r)^{11}$ , the loss of critic parameters is
$$
L _ {Q} \left(\phi_ {Q}, \omega\right) = \frac {1}{2} \left(Q _ {\omega} \left(f _ {\phi_ {Q}} (h), a\right) - Q ^ {\operatorname {t a r}} \left(h, a, o ^ {\prime}, r\right)\right) ^ {2}, \tag {29}
$$
where  $Q^{\mathrm{tar}}(h, a, o', r) \coloneqq r + \gamma Q_{\overline{\omega}}(f_{\overline{\phi_Q}}(h'), \pi_{\overline{\nu}}(f_{\overline{\phi_\pi}}(h'))))$  and  $\overline{\theta}$  denotes the stopped-gradient version of  $\theta$  and  $h' = (h, a, o')$ . The loss of actor parameters is
$$
L _ {\pi} \left(\phi_ {\pi}, \nu\right) = - Q _ {\omega} \left(f _ {\phi_ {Q}} (h), \pi_ {\nu} \left(f _ {\phi_ {\pi}} (h)\right)\right). \tag {30}
$$
The total loss of actor and critic parameters is thus  $L_{\mathrm{sep}}(\phi_{\pi}, \phi_Q, \omega, \nu) \coloneqq L_Q(\phi_Q, \omega) + L_{\pi}(\phi_{\pi}, \nu)$ . The gradients of  $L_{\mathrm{sep}}$  are
$$
\nabla_ {\phi_ {Q}} L _ {\mathrm {s e p}} = \left(Q _ {\omega} \left(f _ {\phi_ {Q}} (h), a\right) - Q ^ {\mathrm {t a r}} \left(h, a, o ^ {\prime}, r\right)\right) \nabla_ {z _ {Q}} Q _ {\omega} \left(f _ {\phi_ {Q}} (h), a\right) \nabla_ {\phi_ {Q}} f _ {\phi_ {Q}} (h) \tag {31}
$$
$$
\nabla_ {\omega} L _ {\mathrm {s e p}} = \left(Q _ {\omega} \left(f _ {\phi_ {Q}} (h), a\right) - Q ^ {\operatorname {t a r}} \left(h, a, o ^ {\prime}, r\right)\right) \nabla_ {\omega} Q _ {\omega} \left(f _ {\phi_ {Q}} (h), a\right) \tag {32}
$$
$$
\nabla_ {\phi_ {\pi}} L _ {\operatorname {s e p}} = - \nabla_ {a} Q _ {\omega} \left(f _ {\phi_ {Q}} (h), \pi_ {\nu} \left(f _ {\phi_ {\pi}} (h)\right)\right) \nabla_ {z _ {\pi}} \pi_ {\nu} \left(f _ {\phi_ {\pi}} (h)\right) \nabla_ {\phi_ {\pi}} f _ {\phi_ {\pi}} (h) \tag {33}
$$
$$
\nabla_ {\nu} L _ {\mathrm {s e p}} = - \nabla_ {a} Q _ {\omega} \left(f _ {\phi_ {Q}} (h), \pi_ {\nu} \left(f _ {\phi_ {\pi}} (h)\right)\right) \nabla_ {\nu} \pi_ {\nu} \left(f _ {\phi_ {\pi}} (h)\right) \tag {34}
$$
<sup>9</sup>https://github.com/twni2016/pomdp-baselines
10The analysis can be generalized to stochastic actors by the reparametrization trick.
<sup>11</sup>It denotes current history, current action, next observation, and current reward.
Now we consider an actor and a critic with a shared encoder denoted as  $f_{\phi}$ , mapping a history  $h$  into a latent state  $z$ . The history encoder has parameters  $\phi$ . The actor is parameterized by  $\pi_{\nu}(f_{\phi}(h))$  and the critic is parameterized by  $Q_{\omega}(f_{\phi}(h),a)$ . The total loss of actor and critic
$$
L _ {\mathrm {s h a}} (\phi , \omega , \nu) = L _ {Q} (\phi , \omega) - Q _ {\omega} \left(f _ {\phi} (h), \pi_ {\nu} \left(f _ {\phi} (h)\right)\right) \tag {35}
$$
has such gradients:
$$
\begin{array}{l} \nabla_ {\phi} L _ {\mathrm {s h a}} = \left(Q _ {\omega} \left(f _ {\phi} (h), a\right) - Q ^ {\operatorname {t a r}} \left(h, a, o ^ {\prime}, r\right)\right) \nabla_ {z} Q _ {\omega} \left(f _ {\phi} (h), a\right) \nabla_ {\phi} f _ {\phi} (h) \\ - \nabla_ {a} Q _ {\omega} \left(f _ {\phi} (h), \pi_ {\nu} \left(f _ {\phi} (h)\right)\right) \nabla_ {z} \pi_ {\nu} \left(f _ {\phi} (h)\right) \nabla_ {\phi} f _ {\phi} (h) - \underbrace {\nabla_ {z} Q _ {\omega} \left(f _ {\phi} (h) , \pi_ {\nu} \left(f _ {\phi} (h)\right)\right) \nabla_ {\phi} f _ {\phi} (h)} _ {\text {e x t r a g r a d i e n t t e r m}} \tag {36} \\ \end{array}
$$