SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Tsallis entropy regularized policy has general expression for all  $q$ . Moreover,  $q$  and  $\tau$  are interchangeable for controlling the truncation (Theorem 1).
2. The policies can be expressed by  $q$ -exponential (Theorem 1).
3. We present a computable approximate threshold  $\hat{\psi}_q$  (Theorem 2).
General expression for Tsallis entropy regularized policy. The original definition of Tsallis entropy is  $S_{q^*}(\pi(\cdot|s)) = \frac{p}{q^*-1} \left( 1 - \sum_{a} \pi^{q^*}(a|s) \right)$ ,  $q^* \in \mathbb{R}$ ,  $p \in \mathbb{R}_+$ . Note that similar to Appendix A, we can choose whichever convenient of  $q$  and  $q^*$ , since the domain of the entropic index is  $\mathbb{R}$ . To obtain the Tsallis entropy-regularized policies we follow [Chen et al., 2018]. The derivation begins with assuming an actor-critic framework where the policy network is parametrized by  $w$ . It is well-known that the parameters should be updated towards the direction specified by the policy gradient theorem:
$$
\Delta w \propto \mathbb {E} _ {\pi} \left[ Q _ {\pi} \frac {\partial \ln \pi}{\partial w} + \tau \frac {\partial \mathcal {H} (\pi)}{\partial w} \right] - \sum_ {s} \lambda (s) \frac {\partial \langle \mathbf {1} , \pi \rangle}{\partial w} =: f (w), \tag {13}
$$
Recall that  $\mathcal{H}(\pi)$  denotes the Shannon entropy and  $\tau$  is the coefficient.  $\lambda (s)$  are the Lagrange multipliers for the constraint  $\langle 1,\pi \rangle = 1$ . In the Tsallis entropy framework, we replace  $\mathcal{H}(\pi)$  with  $S_{q^{*}}(\pi)$ . We can assume  $p = \frac{1}{q^*}$  to ease derivation, which is the case for sparsemax.
We can now explicitly write the optimal condition for the policy network parameters:
$$
\begin{array}{l} f (w) = 0 = \mathbb {E} _ {\pi} \left[ Q _ {\pi} \frac {\partial \ln \pi}{\partial w} + \tau \frac {\partial S _ {q ^ {*}} (\pi)}{\partial w} \right] - \sum_ {s} \lambda (s) \frac {\partial \langle \mathbf {1} , \pi \rangle}{\partial w} \\ = \mathbb {E} _ {\pi} \left[ Q _ {\pi} \frac {\partial \ln \pi}{\partial w} - \tau \frac {1}{q ^ {*} - 1} \left\langle \mathbf {1}, \pi^ {q ^ {*}} \frac {\partial \ln \pi}{\partial w} \right\rangle - \tilde {\psi} _ {q} (s) \frac {\partial \ln \pi}{\partial w} \right] \tag {14} \\ = \mathbb {E} _ {\boldsymbol {\pi}} \left[ \left(Q _ {\boldsymbol {\pi}} - \tau \frac {1}{q ^ {*} - 1} \pi^ {q ^ {*} - 1} - \tilde {\psi} _ {q} (s)\right) \frac {\partial \ln \boldsymbol {\pi}}{\partial w} \right], \\ \end{array}
$$
where we leveraged  $\frac{\partial S_{q^*}(\pi)}{\partial w} = \frac{1}{q^* - 1}\left\langle 1, \pi^{q^*} \frac{\partial \ln \pi}{\partial w} \right\rangle$  in the second step and absorbed terms into the expectation in the last step.  $\tilde{\psi}_q(s)$  denotes the adjusted Lagrange multipliers by taking  $\lambda(s)$  inside the expectation and modifying it according to the discounted stationary distribution.
Now it suffices to verify either  $\frac{\partial\ln\pi}{\partial w} = 0$  or
$$
Q _ {\pi} (s, a) - \tau \frac {1}{q ^ {*} - 1} \pi^ {q ^ {*} - 1} (a | s) - \tilde {\psi} _ {q} (s) = 0
$$
$$
\Leftrightarrow \pi^ {*} (a | s) = \sqrt [ q ^ {*} - 1 ]{\left[ \frac {Q _ {\pi} (s , a)}{\tau} - \frac {\tilde {\psi} _ {q} (s)}{\tau} \right] _ {+} (q ^ {*} - 1)}, \tag {15}
$$
$$
\text {o r} \quad \pi^ {*} (a | s) = \sqrt [ 1 - q ]{\left[ \frac {Q _ {\pi} (s , a)}{\tau} - \frac {\tilde {\psi} _ {q} (s)}{\tau} \right] _ {+} (1 - q)},
$$
where we changed the entropic index from  $q^*$  to  $q$ . Clearly, the root does not affect truncation. Consider the pair  $(q^* = 50, \tau)$ , then the same truncation effect can be achieved by choosing  $(q^* = 2, \frac{\tau}{50 - 1})$ . The same goes for  $q$ . Therefore, we conclude that  $q$  and  $\tau$  are interchangeable for the truncation, and we should stick to the analytic choice  $q^* = 2(q = 0)$ .
Tsallis policies can be expressed by  $q$ -exponential. Given Eq. (15), by adding and subtracting 1, we have:
$$
\pi^ {*} (a | s) = \sqrt [ 1 - q ]{\left[ 1 + (1 - q) \left(\frac {Q _ {\pi} (s , a)}{\tau} - \tilde {\psi} _ {q} \left(\frac {Q _ {\pi} (s , \cdot)}{\tau}\right) - \frac {1}{1 - q}\right) \right] _ {+}} = \exp_ {q} \left(\frac {Q _ {\pi} (s , a)}{\tau} - \hat {\psi} _ {q} \left(\frac {Q _ {\pi} (s , \cdot)}{\tau}\right)\right),
$$
where we defined  $\hat{\psi}_q = \tilde{\psi}_q + \frac{1}{1 - q}$ . Note that this expression is general for all  $q$ , but whether  $\pi^*$  has closed-form expression depends on the solvability of  $\tilde{\psi}_q$ .
Let us consider the extreme case  $q = \infty$ . It is clear that  $\lim_{q\to \infty}\frac{1}{1 - q}\rightarrow 0$ . Therefore, for any  $x > 0$  we must have  $x^{\frac{1}{1 - q}}\rightarrow 1$ ; i.e., there is only one action with probability 1, with all others being 0. This conclusion agrees with the fact that  $S_{q}(\pi)\rightarrow 0$  as  $q\to \infty$ : hence the regularized policy degenerates to arg max.
A computable Normalization Function. The constraint  $\sum_{a\in K(s)}\pi^{*}(a|s) = 1$  is exploited to obtain the threshold  $\psi$  for the sparsemax [Lee et al., 2018, Chow et al., 2018]. Unfortunately, this is only possible when the root vanishes, since otherwise the constraint yields a summation of radicals. Nonetheless, we can resort to first-order Taylor's expansion for deriving an approximate policy. Following [Chen et al., 2018], let us expand Eq. (15) by the first order Taylor's expansion  $f(z) + f'(z)(x - z)$ , where we let  $z = 1$ ,  $x = \left[\frac{Q_{\pi}(s,a)}{\tau} -\tilde{\psi}_q\left(\frac{Q_{\pi}(s,\cdot)}{\tau}\right)\right]_+$ $(1 - q)$ ,  $f(x) = x^{\frac{1}{1 - q}}$ ,  $f'(x) = \frac{1}{1 - q} x^{\frac{q}{1 - q}}$ . So that the unnormalized approximate policy has
$$
\begin{array}{l} \tilde {\pi} ^ {*} (a | s) \approx f (z) + f ^ {\prime} (z) (x - z) \\ = 1 + \frac {1}{1 - q} \left(\left(\frac {Q _ {\pi} (s , a)}{\tau} - \tilde {\psi} _ {q} \left(\frac {Q _ {\pi} (s , \cdot)}{\tau}\right)\right) (1 - q) - 1\right). \tag {16} \\ \end{array}
$$
Therefore it is clear as  $q \to \infty$ ,  $\tilde{\pi}^{*}(a|s) \to 1$ . This concords well with the limit case where  $\pi^{*}(a|s)$  degenerates to  $\arg \max$ . With Eq. (16), we can solve for the approximate normalization by the constraint  $\sum_{a \in K(s)} \pi^{*}(a|s) = 1$ :
$$
\begin{array}{l} 1 = \sum_ {a \in K (s)} \left[ 1 + \frac {1}{1 - q} \left(\left(\frac {Q _ {\pi} (s , a)}{\tau} - \tilde {\psi} _ {q} \left(\frac {Q _ {\pi} (s , \cdot)}{\tau}\right)\right) (1 - q) - 1\right) \right] \\ = | K (s) | - \frac {1}{1 - q} | K (s) | + \sum_ {a \in K (s)} \left[ \frac {Q _ {\pi} (s , a)}{\tau} - \tilde {\psi} _ {q} \left(\frac {Q _ {\pi} (s , \cdot)}{\tau}\right) \right] \\ \Leftrightarrow \tilde {\psi} _ {q} \left(\frac {Q _ {\pi} (s , \cdot)}{\tau}\right) = \frac {\sum_ {a \in K (s)} \frac {Q _ {\pi} (s , \cdot)}{\tau} - 1}{| K (s) |} + 1 - \frac {1}{1 - q}. \\ \end{array}
$$
In order for an action to be in  $K(s)$ , it has to satisfy  $\frac{Q_{\pi}(s,\cdot)}{\tau} > \frac{\sum_{a \in K(s)} \frac{Q_{\pi}(s,\cdot)}{\tau} - 1}{|K(s)|} + 1 - \frac{1}{1 - q}$ . Therefore, the condition of  $K(s)$  satisfies:
$$
1 + i \frac {Q _ {\pi} (s , a _ {(i)})}{\tau} > \sum_ {j = 1} ^ {i} \frac {Q _ {\pi} (s , a _ {(j)})}{\tau} + i \left(1 - \frac {1}{1 - q}\right).
$$
Therefore, we see the approximate threshold  $\hat{\psi}_q = \tilde{\psi}_q + 1$ . When  $q = 0$  or  $q^{*} = 2$ ,  $\hat{\psi}_q$  recovers  $\psi$  and hence  $\tilde{\pi}^*$  recovers the exact sparsemax policy.
C Proof of convergence of  $\Omega (\pi) = D_{KL}^{q}(\pi \| \cdot)$  when  $q = 2$
Let us work with  $\ln_{q^*}$  from Appendix A and define  $\| \cdot \| _p$  as the  $l_{p}$ -norm. The convergence proof for  $\Omega (\pi) = D_{KL}^{q}(\pi \| \cdot)$  when  $q = 2$  comes from that  $\Omega (\pi)$  is strongly convex in  $\pi$ :
$$
\Omega (\pi) = D _ {K L} ^ {q ^ {*} = 2} (\pi | | \cdot) = \left\langle \pi , \ln_ {2} \frac {\pi}{\cdot} \right\rangle = \left\langle \pi , \frac {\left(\frac {\pi}{\cdot}\right) ^ {2 - 1} - 1}{2 - 1} \right\rangle \propto \left\| \frac {\pi}{\cdot} \right\| _ {2} ^ {2} - 1. \tag {17}
$$
Similarly, the negative Tsallis sparse entropy  $-S_{2}(\pi)$  is also strongly convex. Then the propositions of [Geist et al., 2019] can be applied, which we restate in the following:
Lemma 1 ([Geist et al., 2019]). Define regularized value functions as:
$$
Q _ {\pi , \Omega} = r + \gamma P V _ {\pi , \Omega}, \quad V _ {\pi , \Omega} = \langle \pi , Q _ {\pi , \Omega} \rangle - \Omega (\pi).
$$
If  $\Omega(\pi)$  is strongly convex, let  $\Omega^{*}(Q) = \max_{\pi} \langle \pi, Q \rangle - \Omega(\pi)$  denote the Legendre-Fenchel transform of  $\Omega(\pi)$ , then
Algorithm 1: MVI(q)
Input: number of iterations  $T$ , entropy coefficient  $\tau$ , TKL coefficient  $\alpha$   
Initialize  $Q_0, \pi_0$  arbitrarily  
Let  $\{|A|\} = \{1, 2, \dots, |A|\}$   
for  $k = 1, 2, \dots, T$  do  
    # Policy Improvement  
    for  $(s, a) \in (\mathcal{S}, \mathcal{A})$  do  
        Sort  $Q_k(s, a_{(1)}) > \dots > Q_k(s, a_{(|A|)})$   
        Find  $K(s) = \max \left\{i \in \{|A|\} \mid 1 + i \frac{Q_k(s, a_{(i)})}{\tau} > \sum_{j=1}^{i} \frac{Q_k(s, a_{(j)})}{\tau} + i\left(1 - \frac{1}{1-q}\right)\right\}$   
        Compute  $\hat{\psi}_q\left(\frac{Q_k(s, \cdot)}{\tau}\right) = \frac{\sum_{a \in K(s)} \frac{Q_k(s, a)}{\tau} - 1}{|K(s)|} + 1$   
    #Normalize when  $q \neq 2$ $\pi_{k+1}(a|s) \propto \exp_q\left(\frac{Q_k(s, a)}{\tau} - \hat{\psi}_q\left(\frac{Q_k(s, \cdot)}{\tau}\right)\right)$   
end for  
#Policy Evaluation  
for  $(s, a, s') \in (\mathcal{S}, \mathcal{A})$  do  
     $Q_{k+1}(s, a) = r(s, a) + \alpha \tau (Q_k(s, a) - M_{q, \tau} Q_k(s)) + \gamma \sum_{b \in A} \pi_{k+1}(b|s') (Q_k(s', b) - \tau \ln_q \pi_{k+1}(b|s'))$   
end for  
end for
-  $\nabla \Omega^{*}$  is Lipschitz and is the unique maximizer of  $\arg \max_{\pi} \langle \pi, Q \rangle - \Omega(\pi)$ .
-  $T_{\pi, \Omega}$  is a  $\gamma$ -contraction in the supremum norm, i.e.  $\| T_{\pi, \Omega} V_1 - T_{\pi, \Omega} V_2 \|_\infty \leq \gamma \| V_1 - V_2 \|_\infty$ . Further, it has a unique fixed point  $V_{\pi, \Omega}$ .
- The policy  $\pi_{*,\Omega} = \arg \max_{\pi}\langle \pi ,Q_{*,\Omega}\rangle -\Omega (\pi)$  is the unique optimal regularized policy.
Note that in the main paper we dropped the subscript  $\Omega$  for both the regularized optimal policy and action value function to lighten notations. It is now clear that Eq. (6) indeed converges for entropic indices that make  $D_{KL}^{q}(\pi \| \cdot)$  strongly convex. But we mostly consider the case  $q = 2$ .
D Derivation of the Tsallis KL Policy
This section contains the proof for the Tsallis KL-regularized policy (7). Section D.1 shows that a Tsallis KL policy can also be expressed by a series of multiplications of  $\exp_q(Q)$ ; while Section D.2 shows its more-than-averaging property.
D.1 Tsallis KL Policies are Similar to KL
We extend the proof and use the same notations from [Lee et al., 2020, Appendix D] to derive the Tsallis KL regularized policy. Again let us work with  $\ln_{q^*}$  from Appendix A. Define state visitation as  $\rho_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{t=0}^{\infty} \mathbb{1}(s_t = s)]$  and state-action visitation  $\rho_{\pi}(s, a) = \mathbb{E}_{\pi}[\sum_{t=0}^{\infty} \mathbb{1}(s_t = s, a_t = a)]$ . The core of the proof resides in establishing the one-to-one correspondence between the policy and the induced state-action visitation  $\rho_{\pi}$ . For example, Tsallis entropy is written as
$$
S _ {q ^ {*}} (\pi) = S _ {q ^ {*}} (\rho_ {\pi}) = - \sum_ {s, a} \rho_ {\pi} (s, a) \ln_ {q ^ {*}} \frac {\rho_ {\pi} (s , a)}{\sum_ {a} \rho_ {\pi} (s , a)}.
$$
This unique correspondence allows us to replace the optimization variable from  $\pi$  to  $\rho_{\pi}$ . Indeed, one can always restore the policy by  $\pi(a|s) := \frac{\rho_{\pi}(s,a)}{\sum_{a'} \rho_{\pi}(s,a')}$ .
Let us write Tsallis KL divergence as \(D_{KL}^{q^*}(\pi \parallel \mu) = D_{KL}^{q^*}(\rho \parallel \nu) = \sum_{s,a} \rho(s,a) \ln_{q^*} \frac{\rho(s,a) \sum_{a'} \nu(s,a')}{\nu(s,a) \sum_{a'} \rho(s,a')} \text{ by replacing the policies } \pi, \mu \text{ with their state-action visita-
tion  $\rho, \nu$ . One can then convert the Tsallis MDP problem into the following problem:
$$
\max _ {\rho} \sum_ {s, a} \rho (s, a) \sum_ {s ^ {\prime}} r (s, a) P (s ^ {\prime} | s, a) - D _ {K L} ^ {q ^ {*}} (\rho \| \nu)
$$
subject to  $\forall s,a,\quad \rho (s,a) > 0,$  (18)
$$
\sum_ {a} \rho (s, a) = d (s) + \sum_ {s ^ {\prime}, a ^ {\prime}} P (s | s ^ {\prime}, a ^ {\prime}) \rho (s ^ {\prime}, a ^ {\prime}),
$$
where  $d(s)$  is the initial state distribution. Eq. (18) is known as the Bellman Flow Constraints [Lee et al., 2020, Prop. 5] and is concave in  $\rho$  since the first term is linear and the second term is concave in  $\rho$ . Then the primal and dual solutions satisfy KKT conditions sufficiently and necessarily. Following [Lee et al., 2020, Appendix D.2], we define the Lagrangian objective as
$$
\begin{array}{l} \mathcal {L} := \sum_ {s, a} \rho (s, a) \sum_ {s ^ {\prime}} r (s, a) P \left(s ^ {\prime} \mid s, a\right) - D _ {K L} ^ {q ^ {*}} (\rho \| \nu) + \sum_ {s, a} \lambda (s, a) \rho (s, a) \\ + \sum_ {s} \zeta (s) \left(d (s) + \sum_ {s ^ {\prime}, a ^ {\prime}} P (s | s ^ {\prime}, a ^ {\prime}) \rho (s ^ {\prime}, a ^ {\prime}) - \sum_ {a} \rho (s, a)\right) \\ \end{array}
$$
where  $\lambda(s, a)$  and  $\zeta(s)$  are dual variables for nonnegativity and Bellman flow constraints. The KKT conditions are:
$$
\forall s, a, \quad \rho^ {*} (s, a) \geq 0,
$$
$$
d (s) + \sum_ {s ^ {\prime}, a ^ {\prime}} P (s | s ^ {\prime}, a ^ {\prime}) \rho^ {*} (s ^ {\prime}, a ^ {\prime}) - \sum_ {a} \rho^ {*} (s, a) = 0,
$$
$$
\lambda^ {*} (s, a) \leq 0, \quad \lambda^ {*} (s, a) \rho^ {*} (s, a) = 0,
$$
$$
\begin{array}{l} 0 = \sum_ {s ^ {\prime}} r (s, a) P (s ^ {\prime} | s, a) + \gamma \sum_ {s ^ {\prime}} \zeta^ {*} (s ^ {\prime}) P (s ^ {\prime} | s, a) - \zeta^ {*} (s) + \lambda^ {*} (s, a) - \frac {\partial D _ {K L} ^ {q ^ {*}} (\rho^ {*} \| \nu)}{\partial \rho (s , a)}, \\ \mathrm {w h e r e} - \frac {\partial D _ {K L} ^ {q ^ {*}} (\rho^ {*} \| \nu)}{\partial \rho (s , a)} = - \ln_ {q ^ {*}} \frac {\rho^ {*} (s , a) \sum_ {a ^ {\prime}} \nu (s , a ^ {\prime})}{\nu (s , a) \sum_ {a ^ {\prime}} \rho^ {*} (s , a ^ {\prime})} - \left(\frac {\rho^ {*} (s , a) \sum_ {a ^ {\prime}} \nu (s , a ^ {\prime})}{\nu (s , a) \sum_ {a ^ {\prime}} \rho^ {*} (s , a ^ {\prime})}\right) ^ {q ^ {*} - 1} \\ + \sum_ {a} \left(\frac {\rho^ {*} (s , a)}{\sum_ {a ^ {\prime}} \rho^ {*} (s , a ^ {\prime})}\right) ^ {q ^ {*}} \left(\frac {\sum_ {a ^ {\prime}} \nu (s , a)}{\nu (s , a)}\right) ^ {q ^ {*} - 1}. \\ \end{array}
$$
The dual variable  $\zeta^{*}(s)$  can be shown to equal to the optimal state value function  $V^{*}(s)$  following [Lee et al., 2020], and  $\lambda^{*}(s,a) = 0$  whenever  $\rho^{*}(s,a) > 0$ .
By noticing that  $x^{q^* - 1} = (q^* - 1) \ln_{q^*} x + 1$ , we can show that  $-\frac{\partial D_{KL}^{q^*}(\rho^* \| \nu)}{\partial \rho(s, a)} = -q^* \ln_{q^*} \frac{\rho^*(s, a) \sum_{a'} \nu(s, a')}{\nu(s, a) \sum_{a'} \rho^*(s, a')} - 1 + \sum_a \left( \frac{\rho^*(s, a)}{\sum_{a'} \rho^*(s, a')} \right)^{q^*} \left( \frac{\sum_{a'} \nu(s, a)}{\nu(s, a)} \right)^{q^* - 1}$ . Substituting  $\zeta^*(s) = V^*(s)$ ,  $\pi^*(a|s) = \frac{\rho^*(s, a)}{\sum_{a'} \rho^*(s, a)}$ ,  $\mu^*(a|s) = \frac{\nu^*(s, a)}{\sum_{a'} \nu^*(s, a)}$  into the above KKT condition and leverage the equality  $Q^*(s, a) = r(s, a) + \mathbb{E}_{s' \sim P} [\gamma \zeta^*(s')]$  we have:
$$
\begin{array}{l} Q ^ {*} (s, a) - V ^ {*} (s) - q ^ {*} \ln_ {q ^ {*}} \frac {\pi (a | s)}{\mu (a | s)} - 1 + \sum_ {a ^ {\prime}} \pi (a | s) \left(\frac {\pi (a | s)}{\mu (a | s)}\right) ^ {q ^ {*} - 1} = 0 \\ \Leftrightarrow \pi^ {*} (a | s) = \mu (a | s) \exp_ {q ^ {*}} \left(\frac {Q ^ {*} (s , a)}{q ^ {*}} - \frac {V ^ {*} (s) + 1 - \sum_ {a ^ {\prime}} \pi (a | s) \left(\frac {\pi (a | s)}{\mu (a | s)}\right) ^ {q ^ {*} - 1}}{q ^ {*}}\right). \\ \end{array}
$$
By comparing it to the maximum Tsallis entropy policy [Lee et al., 2020, Eq.(49)] we see the only difference lies in the baseline term  $\mu (a|s)^{-(q^{*} - 1)}$ , which is expected since we are exploiting Tsallis KL regularization. Let us define the normalization function as
$$
\psi \left(\frac {Q ^ {*} (s , \cdot)}{q ^ {*}}\right) = \frac {V ^ {*} (s) + 1 - \sum_ {a} \pi (a | s) \left(\frac {\pi (a | s)}{\mu (a | s)}\right) ^ {q ^ {*} - 1}}{q ^ {*}},
$$
then we can write the policy as
$$
\pi^ {*} (a | s) = \mu (a | s) \exp_ {q ^ {*}} \left(\frac {Q ^ {*} (s , a)}{q ^ {*}} - \psi \left(\frac {Q ^ {*} (s , \cdot)}{q ^ {*}}\right)\right).
$$
In a way similar to KL regularized policies, at  $k + 1$ -th update, take  $\pi^{*} = \pi_{k + 1}, \mu = \pi_{k}$  and  $Q^{*} = Q_{k}$ , we write  $\pi_{k + 1} \propto \pi_{k} \exp_{q} Q_{k}$  since the normalization function does not depend on actions. We ignored the scaling constant  $q^{*}$  and regularization coefficient. Hence one can now expand Tsallis KL policies as:
$$
\pi_ {k + 1} \propto \pi_ {k} \exp_ {q ^ {*}} (Q _ {k}) \propto \pi_ {k - 1} \exp_ {q ^ {*}} (Q _ {k - 1}) \exp_ {q ^ {*}} (Q _ {k}) \propto \dots \propto \exp_ {q ^ {*}} Q _ {1} \dots \exp_ {q ^ {*}} Q _ {k},
$$
which proved the first part of Eq. (7).
D.2 Tsallis KL Policies Do More than Average
We now show the second part of Eq. (7), which stated that the Tsallis KL policies do more than average. This follows from the following lemma:
Lemma 2 (Eq. (25) of [Yamano, 2002]).
$$
\left(\exp_ {q} x _ {1} \dots \exp_ {q} x _ {n}\right) ^ {1 - q} = \exp_ {q} \left(\sum_ {j = 1} ^ {k} x _ {j}\right) ^ {1 - q} + \sum_ {j = 2} ^ {k} (1 - q) ^ {j} \sum_ {i _ {1} = 1 <   \dots <   i _ {j}} ^ {k} x _ {i _ {1}} \dots x _ {i _ {j}}. \tag {19}
$$
However, the mismatch between the base  $q$  and the exponent  $1 - q$  is inconvenient. We exploit the  $q = 2 - q^{*}$  duality to show this property holds for  $q^{*}$  as well:
$$
\begin{array}{l} \left(\exp_ {q ^ {*}} x \cdot \exp_ {q ^ {*}} y\right) ^ {q ^ {*} - 1} = [ 1 + (q ^ {*} - 1) x ] _ {+} \cdot [ 1 + (q ^ {*} - 1) y ] _ {+} \\ = \left[ 1 + (q ^ {*} - 1) x + (q ^ {*} - 1) y + (q ^ {*} - 1) ^ {2} x y \right] _ {+} \\ = \exp_ {q} (x + y) ^ {q ^ {*} - 1} + (q ^ {*} - 1) ^ {2} x y. \\ \end{array}
$$
Now since we proved the two-point property for  $q^*$ , by the same induction steps in [Yamano, 2002, Eq. (25)] we conclude the proof. The weighted average part Eq. (8) comes immediately from [Suyari et al., 2020, Eq.(18)].
We list the hyperparameters for Gym environments in Table 1. The epsilon threshold is fixed at 0.01 from the beginning of learning. FC  $n$  refers to the fully connected layer with  $n$  activation units.
The Q-network uses 3 convolutional layers. The epsilon greedy threshold is initialized at 1.0 and gradually decays to 0.01 at the end of first  $10\%$  of learning. We run the algorithms with the swept hyperparameters for full  $5 \times 10^{7}$  steps on the selected two Atari environments to pick the best hyperparameters.
We show in Figure 6 the performance of  $\mathrm{MVI}(q)$  on Cartpole-v1 and Acrobot-v1, and the full learning curves of  $\mathrm{MVI}(q)$  on the Atari games in Figure 7. Figures 8 and 9 show the full learning curves of Tsallis-VI.
Figure 6: (Left)  $\mathrm{MVI}(q)$  and MVI with logarithm  $\ln \pi$  simply replaced to  $\ln_q\pi$  on Cartpole-v1, when  $q = 2$ . The results are averaged over 50 independent runs. The flat learning curve is due to the pseudo-additivity. (Right)  $\mathrm{MVI}(q)$  on Acrobot-v1 with different  $q$  choices. Each  $q$  is independently fine-tuned. The black bar stands for  $95\%$  confidence interval, averaged over 50 independent runs.
Figure 7: Learning curves of  $\mathrm{MVI}(q)$  and M-VI on the selected Atari games.
Figure 8: Learning curves of  $\mathrm{MVI}(q)$  and Tsallis-VI on the selected Atari games.
Figure 9: (cont'd)  $\mathrm{MVI}(q)$  and Tsallis-VI on the selected Atari games.
Table 1: Parameters used for Gym.
Network Parameter	Value	Algorithm Parameter	Value
T (total steps)	5 × 105	γ (discount rate)	0.99
C (interaction period)	4	ε (epsilon greedy threshold)	0.01
|B| (buffer size)	5 × 104	τ (Tsallis entropy coefficient)	0.03
Bt (batch size)	128	α (advantage coefficient)	0.9
I (update period)	100 (Car.) / 2500 (Acro.)
Q-network architecture	FC512 - FC512
activation units	ReLU
optimizer	Adam
optimizer learning rate	10-3
Table 2: Parameters used for Atari games.
Network Parameter	Value	Algorithmic Parameter	Value
T (total steps)	5 × 107	γ (discount rate)	0.99
C (interaction period)	4	τMVI(q) (MVI(q) entropy coefficient)	10
|B| (buffer size)	1 × 106	αMVI(q) (MVI(q) advantage coefficient)	0.9
Bt (batch size)	32	τTsallis (Tsallis-VI entropy coef.)	10
I (update period)	8000	αM-VI (M-VI advantage coefficient)	0.9
activation units	ReLU	τM-VI (M-VI entropy coefficient)	0.03
optimizer	Adam	ε (epsilon greedy threshold)	1.0 → 0.01|10%
optimizer learning rate	10-4
Q-network architecture
Conv4,8,32 - Conv2,4,64 - Conv1,3,64 - FC512 - FC