SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Maximizing entropy is a commonly used approach to encourage exploration in RL [9, 10, 37]. However, in maximum entropy KGRL, when the entropy of policy distributions are different from one another, it leads to the problem of entropy imbalance. Entropy imbalance is a phenomenon in which an agent consistently selects only a single or a small set of knowledge policies. We show this in math by first revisiting the formulation of maximum entropy RL. In maximum entropy RL, an entropy term is added to the standard RL objective as  $\pi^{*} = \operatorname{argmax}_{\pi} \sum_{t} \mathbb{E}_{(\mathbf{s}_{t}, \mathbf{a}_{t} \sim \pi)}[R(\mathbf{s}_{t}, \mathbf{a}_{t}) + \alpha H(\pi(\cdot | \mathbf{s}_{t}))]$  [9, 10], where  $\alpha \in \mathbb{R}$  is a hyperparameter, and  $H(\cdot)$  represents the entropy of a distribution. By maximizing  $\alpha H(\pi(\cdot | \mathbf{s}_{t}))$ , the policy becomes more uniform since the entropy of a probability distribution is maximized when it is a uniform distribution [19]. With this in mind, we show that in maximum entropy KGRL, some of the weights in  $\{\hat{w}_{t,in}, \hat{w}_{t,g_1}, \dots, \hat{w}_{t,g_n}\}$  might always be larger than others. We provide the proofs of all propositions in Appendix A.
Proposition 4.1 (Entropy imbalance in discrete decision-making). Assume that a  $d_{a}$ -dimensional probability simplex  $\pi \in \Delta^{d_a}$  is fused by  $\{\pi_1,\dots ,\pi_m\}$  and  $\{\hat{w}_1,\dots ,\hat{w}_m\}$  following equation (3), where  $\pi_j\in \Delta^{d_a}$ ,  $\hat{w}_j\geq 0$ $\forall j\in \{1,\ldots ,m\}$  and  $\sum_{j = 1}^{m}\hat{w}_{j} = 1$ . If the entropy of  $\pi$  is maximized and  $\| \pi_1\|_{\infty}\ll \| \pi_2\|_{\infty},\| \pi_1\|_{\infty}\ll \| \pi_3\|_{\infty},\ldots ,\| \pi_1\|_{\infty}\ll \| \pi_m\|_{\infty}$ , then  $\hat{w}_1\rightarrow 1$
We show in Proposition A.1 that if  $\pi_1$  is more uniform than  $\pi_j$ , then  $\| \pi_1 \|_{\infty} < \| \pi_j \|_{\infty}$ .
Proposition 4.2 (Entropy imbalance in continuous control). Assume a one-dimensional policy distribution  $\pi$  is fused by
$$
\pi = \hat {w} _ {1} \pi_ {1} + \hat {w} _ {2} \pi_ {2}, \text {w h e r e} \pi_ {j} = \mathcal {N} \left(\mu_ {j}, \sigma_ {j} ^ {2}\right), \hat {w} _ {j} \geq 0 \forall j \in \{1, 2 \}, \text {a n d} \hat {w} _ {1} + \hat {w} _ {2} = 1. \tag {4}
$$
If the variance of  $\pi$  is maximized, and  $\sigma_1^2\gg \sigma_2^2$  and  $\sigma_1^2\gg (\mu_1 - \mu_2)^2$ , then  $\hat{w}_1\rightarrow 1$ .
We can also infer from Proposition 4.2 that the variance of  $\pi$  defined in equation (4) depends on the distance between  $\mu_{1}$  and  $\mu_{2}$ , which leads to Proposition 4.3.
Proposition 4.3 (Distribution separation in continuous control). Assume a one-dimensional policy distribution  $\pi$  is fused by equation (4). If  $\hat{w}_1, \hat{w}_2, \sigma_1^2$ , and  $\sigma_2^2$  are fixed, then maximizing the variance of  $\pi$  will increase the distance between  $\mu_1$  and  $\mu_2$ .
Proposition 4.1, 4.2, and 4.3 indicate that in maximum entropy KGRL, (1) the agent will pay more attention to the policy with large entropy, and (2) in continuous control, an agent with a learnable internal policy will rely on this policy and separate it as far away as possible from other policies. The consistently imbalanced attention prevents the agent from exploring the environment with other policies that might provide helpful suggestions to solve the task. Furthermore, in continuous control, the distribution separation can make  $\pi$  perform even worse than learning without any external knowledge. The reason is that external policies, although possibly being sub-optimal for the task, might be more efficient in approaching the goal, and moving away from those policies means being less efficient when exploring the environment.
4.4 Modified Policy Distributions
Proposition 4.1 and 4.2 show that fusing multiple policies with equation (3) can make a KGRL agent rely on a learnable internal policy for exploration. However, the uniformity of the internal policy is often desired since it encourages exploration in the state-action space that is not covered by external policies. Therefore, we keep the internal policy unchanged and propose methods to modify external policy distributions in KIAN to resolve the entropy imbalance issue. We provide the detailed learning algorithm of KGRL with KIAN in Appendix A.6.
Discrete Policy Distribution. We modify a fusion of discrete policy distributions in equation (3) as
$$
\pi (\cdot | \mathbf {s} _ {t}) = \hat {w} _ {t, i n} \pi_ {i n} (\cdot | \mathbf {s} _ {t}) + \sum_ {j = 1} ^ {n} \hat {w} _ {t, g _ {j}} \operatorname {s o f t m a x} \left(\beta_ {t, g _ {j}} \pi_ {g _ {j}} (\cdot | \mathbf {s} _ {t})\right), \tag {5}
$$
$$
w _ {t, i n} = \frac {\Phi (\mathbf {s} _ {t}) \cdot \mathbf {k} _ {i n}}{\| \Phi (\mathbf {s} _ {t}) \| _ {2} \| \mathbf {k} _ {i n} \| _ {2}}, w _ {t, g _ {j}} = \frac {\Phi (\mathbf {s} _ {t}) \cdot \mathbf {k} _ {g _ {j}}}{\| \Phi (\mathbf {s} _ {t}) \| _ {2} \| \mathbf {k} _ {g _ {j}} \| _ {2}}, \tag {6}
$$
$$
\beta_ {t, g _ {j}} = \| \Phi (\mathbf {s} _ {t}) \| _ {2} \| \mathbf {k} _ {g _ {j}} \| _ {2} \quad \forall j \in \{1, \dots , n \}, \tag {7}
$$
where  $\beta_{t,g_j} \in \mathbb{R}$  is a state-and-knowledge dependent variable that scales  $\pi_{g_j}(\cdot|\mathbf{s}_t)$  to change its uniformity after passing through softmax. If the value of  $\beta_{t,g_j}$  decreases, the uniformity, i.e., the entropy, of softmax  $(\beta_{t,g_j}\pi_{g_j}(\cdot|\mathbf{s}_t))$  increases. By introducing  $\beta_{t,g_j}$ , the entropy of knowledge policies becomes adjustable, resulting in reduced bias towards the internal policy during exploration.
Continuous Action Probability. We modify the probability of sampling  $\mathbf{a}_t \in \mathbb{R}^{d_a}$  from a continuous  $\pi(\cdot|\mathbf{s}_t)$  in equation (3) as
$$
\pi \left(\mathbf {a} _ {t} \mid \mathbf {s} _ {t}\right) = \hat {w} _ {i n} \pi_ {i n} \left(\mathbf {a} _ {t, i n} \mid \mathbf {s} _ {t}\right) + \sum_ {j = 1} ^ {n} \hat {w} _ {g _ {j}} \pi_ {g _ {j}} \left(\boldsymbol {\mu} _ {t, g _ {j}} \mid \mathbf {s} _ {t}\right), \tag {8}
$$
where  $\mathbf{a}_{t,in} \sim \pi_{in}(\cdot|\mathbf{s}_t)$  and  $\pmb{\mu}_{t,g_j} \in \mathbb{R}^{d_a}$  is the mean of  $\pi_{g_j}(\cdot|\mathbf{s}_t)$ . We show in the next proposition that equation (8) is an approximation of
$$
\pi \left(\mathbf {a} _ {t} \mid \mathbf {s} _ {t}\right) = \hat {w} _ {i n} \pi_ {i n} \left(\mathbf {a} _ {t} \mid \mathbf {s} _ {t}\right) + \Sigma_ {j = 1} ^ {n} \hat {w} _ {g _ {j}} \pi_ {g _ {j}} \left(\mathbf {a} _ {t} \mid \mathbf {s} _ {t}\right), \tag {9}
$$
which is the exact probability of sampling  $\mathbf{a}_t\in \mathbb{R}^{d_a}$  from a continuous  $\pi (\cdot |\mathbf{s}_t)$  in equation (3).
Proposition 4.4 (Approximation of a mixture of normal distributions). If the following three inequalities hold for  $\mu_{t,in},\mu_{t,g_1},\ldots ,\mu_{t,g_n}$ , and  $a_{t,in}:\| \mu_{t,in} - \mu_{t,g_j}\| _2 < \min \{\gamma_{t,in},\gamma_{t,g_j}\}$ ,  $\| a_{t,in} - \mu_{t,in}\| _2 < \min \{\gamma_{t,in},\gamma_{t,g_j}\}$ , and  $\| a_{t,in} - \mu_{t,g_j}\| _2 < \gamma_{t,g_j},\forall j\in \{1,\dots ,n\}$ , where  $\gamma_{t,in} = 1 / (2\pi_{in}(\mu_{t,in}|\mathbf{s}_t))$  and  $\gamma_{t,g_j} = 1 / (2\pi_{g_j}(\mu_{t,g_j}|\mathbf{s}_t))$ , then equation (9) for a real-valued action  $a_t$  sampled from KIAN can be approximated by
$$
\hat {w} _ {t, i n} \mathcal {U} \left(a _ {t}; \mu_ {t, i n} - \gamma_ {t, i n}, \mu_ {t, i n} + \gamma_ {t, i n}\right) + \sum_ {j = 1} ^ {n} \hat {w} _ {t, g _ {j}} \mathcal {U} \left(a _ {t}; \mu_ {t, i n} - \gamma_ {t, g _ {j}}, \mu_ {t, i n} + \gamma_ {t, g _ {j}}\right), \tag {10}
$$
$$
w h e r e \quad \mathcal {U} (\cdot ; a, b) = 1 / (b - a). \tag {11}
$$
In addition, equation (8) is a lower bound of equation (10).
With equation (8), we can show that maximizing the variance of  $\pi (\cdot |\mathbf{s}_t)$  will not separate the policy distributions. Hence, an agent can refer to external policies for efficient exploration and learn its own refined strategy based on them.
Proposition 4.5 (Maximized variance's independence of the distance between means). Assume a one-dimensional policy  $\pi$  is fused by equation (4). If  $\pi(a|\mathbf{s})$  is approximated as equation (8), and the three inequalities in Proposition 4.4 are satisfied, then maximizing the variance of  $\pi(\cdot|\mathbf{s})$  will not affect the distance between  $\mu_1$  and  $\mu_2$ .
We evaluate KIAN on two sets of environments with discrete and continuous action spaces: MiniGrid [5] and OpenAI-Robotics [24]. Through experiments, we answer the following four questions: [Sample Efficiency] Does KIAN require fewer training samples to solve a task than other external-policy-inclusive methods? [Generalizability] Can KIAN trained on one task be directly used to solve another task? [Compositional and Incremental Learning] Can KIAN combine previously learned knowledge keys and inner policies to learn a new task? After adding more external policies to  $\mathcal{G}$ , can most of the components from a trained KIAN be reused for learning?
For comparison, we implement the following five methods as our baselines: behavior cloning (BC) [3], RL [10, 29], RL+BC [21], KoGuN [36], and A2T [27]. KoGuN and A2T are modified to be compositional and applicable in both discrete and continuous action spaces. Moreover, all methods (BC, RL+BC, KoGuN, A2T, and KIAN) are equipped with the same initial external knowledge set,  $\mathcal{G}^{init}$ , for each task. This knowledge set comprises sub-optimal if-else-based programs that cannot complete a task themselves, e.g., pickup_a_key or move_forward_to_the_goal.  $\mathcal{G}^{init}$  will be expanded with learned policies in compositional- and incremental-learning experiments. We provide the experimental details in Appendix B.
Figure 3: The learning curves of sample efficiency experiments in MiniGrid (top 2 rows) and OpenAI-Robotics (last row) environments. Given a knowledge set that cannot complete a task (as shown by BC), KIAN exhibits better sample efficiency across all tasks. These results underline the effectiveness of KIAN in leveraging external policies to mitigate the need for extensive training samples.
Train in Test in	Empty-Random-5x5	DoorKey-5x5	Push	Slide	Pick-and-Place
6x6	8x8	16x16	8x8	16x16	5x	10x	5x	10x	5x	10x
RL [10, 29]	0.88	0.71	0.45	0.29	0.08	0.87	0.52	0.45	0.17	0.34	0.27
RL+BC [21]	0.87	0.60	0.24	0.40	0.09	0.89	0.60	0.44	0.16	0.34	0.30
KoGuN [36]	0.94	0.83	0.53	0.77	0.35	0.63	0.43	0.55	0.18	0.32	0.24
A2T [27]	0.92	0.78	0.51	0.53	0.11	0.03	0.05	0.00	0.01	0.01	0.06
KIAN (ours)	0.96	0.91	0.93	0.76	0.42	0.93	0.70	0.42	0.15	0.92	0.72
Table 1: (Zero-Shot S2C Experiments) The left five columns show the generalizability results of an agent trained in a  $5 \times 5$  environment and tested in environments of varying sizes. The right six columns show the results of an agent trained with a 1x goal range and tested with different goal ranges. Transferring policies from a simple task to a more complex one is a challenging setup in generalizability experiments. The results highlight the superior performance of KIAN in such setup.
5.1 Sample Efficiency and Generalizability
We study the sample efficiency of baselines and KIAN under the intra-task setup, where an agent learns a single task with the external knowledge set  $\mathcal{G}^{init}$  fixed. Figure 3 plots the learning curves in different environments. All experiments in these figures are run with ten random seeds, and each error band is a  $95\%$  confidence interval. The results of BC show that the external knowledge policies are sub-optimal for all environments. Given sub-optimal external knowledge, only KIAN shows success in all environments. In general, improvement of KIAN over baselines is more apparent when the task is more complex, e.g., Empty  $<$  Unlock  $<$  DoorKey and Push  $<$  Pick-and-Place. Moreover, KIAN is more stable than baselines in most environments. Note that in continuous-control tasks (Push, Slide, and Pick-and-Place), A2T barely succeeds since it does not consider the entropy imbalance issue introduced in Proposition 4.2 and 4.3. These results suggest that KIAN can more efficiently explore the environment with external knowledge policies and fuse multiple policies to solve a task.
Next, we evaluate the generalizability of all methods under simple-to-complex (S2C) and complex-to-simple (C2S) setups, where the former trains a policy in a simple task and test it in a complex one,
Train in Test in	DoorKey-5x5 Empty-Random	DoorKey-8x8 Unlock	DoorKey5x5	Pick-and-Place Reach	Push Push	Push Reach	Slide Push
RL [10, 29]	0.83	0.92	0.93	0.80	0.31	0.16	0.09
RL+BC [21]	0.85	0.87	0.93	0.80	0.31	0.16	0.09
KoGuN [36]	0.90	0.91	0.93	0.45	0.05	0.20	0.07
A2T [27]	0.84	0.92	0.93	0.01	0.05	0.20	0.05
KIAN (ours)	0.91	0.94	0.95	1.00	0.30	0.24	0.13
Table 2: (Zero-Shot C2S Experiments) In general, KIAN outperforms other methods when transferring policies across different tasks. Note that although distinguishing the levels of difficulty between Push, Slide, and Pick-and-Place is not straightforward, KIAN still achieves better performance.
Figure 4: The learning curves of composition and incremental experiments in MiniGrid (left 3 columns) and OpenAI-Robotics (right column) environments. KIAN requires fewer samples to learn two tasks sequentially than separately and outperforms other approaches in incremental learning.
and the latter goes the opposite way. All generalizability experiments are run with the same policies as in Section 5.1. Table 1 and 2 show that KIAN outperforms other baselines in most experiments, and its results have a smaller variance (see Table 3 to 5 in Appendix E). These results demonstrate that KIAN's flexibility in incorporating external policies improves generalizability.
5.2 Compositional and Incremental Learning
In the final experiments, we test different methods in the compositional and incremental learning setting. We modify RL, KoGuN, and A2T to fit into this setting; details can be found in Appendix C. The experiments follow the inter-task setup: (1) We randomly select a pair of tasks  $(\mathcal{M}_k^1,\mathcal{M}_k^2)$ . (2) An agent learns a policy to solve  $\mathcal{M}_k^1$  with  $\mathcal{G}^{init}$  fixed, as done in Section 5.1. (3) The learned (internal) policy,  $\pi_{in}^{1}$ , is added to the external knowledge set,  $\mathcal{G} = \mathcal{G}^{init} \cup \{\pi_{in}^{1}\}$ . (4) The same agent learns a policy to solve  $\mathcal{M}_k^2$  with  $\mathcal{G}$ . Each experiment is run with ten random seeds.
The learning curves in Figure 4 demonstrate that given the same updated  $\mathcal{G}$ , KIAN requires fewer samples to solve  $\mathcal{M}_k^2$  than RL, KoGuN, and A2T in all experiments. Our knowledge-key and query design disentangles policy representations from the action-prediction operation, so the agent is more optimized in incremental learning. Unlike our disentangled design, prior methods use a single function approximator to directly predict an action (KoGuN) or the weight of each policy (A2T) given a state. These methods make the action-prediction operation depend on the number of knowledge policies, so changing the size of  $\mathcal{G}$  requires significant retraining of the entire function approximator.
Figure 4 also shows that KIAN solves  $\mathcal{M}_k^2$  more efficiently with  $\mathcal{G}$  than  $\mathcal{G}^{init}$  in most experiments. This improvement can be attributed to KIAN reusing the knowledge keys and query, which allows an agent to know which policies to fuse under different scenarios. Note that  $\mathcal{G}$  can be further expanded with the internal policy learned in  $\mathcal{M}_k^2$  and be used to solve another task  $\mathcal{M}_k^3$ .
Figure 5: The learning curves of KIAN with and without addressing entropy imbalance as described in Section 4.4. The results indicate the adverse impact of entropy imbalance on KIAN's performance within the context of maximum entropy KGRL. In addition, our proposed modifications to external policy distributions are shown to be highly effective in alleviating this issue.
5.3 Analysis of Entropy Imbalance in Maximum Entropy KGRL
In our ablation study, we investigate (1) the impact of entropy imbalance on the performance of maximum entropy KGRL and (2) whether the proposed modifications to external policy distributions in Section 4.4 can alleviate the issue.
Figure 5 shows the learning curves comparing KIAN's performance with and without addressing the entropy-imbalance issue. The results demonstrate that when not addressing the issue using equation (5) or (8), KIAN fails to fully capitalize on the guidance offered by external policies. We also draw two noteworthy conclusions from the figure: (1) For discrete decision-making tasks, the detrimental impact of entropy imbalance becomes more evident as task complexity increases. (2) For continuous-control tasks, entropy imbalance can degrade KIAN's performance and make it perform worse than pure RL without external policies, as shown by the results of FetchPickAndPlace and FetchPush. This phenomenon can be attributed to Proposition 4.3. In contrast, by adjusting KIAN's external policy distributions using equation (5) or (8), a KGRL agent can efficiently harness external policies to solve a given task.
This work introduces KGRL, an RL paradigm aiming to enhance efficient and flexible learning by harnessing external policies. We propose KIAN as an actor model for KGRL, which predicts an action by fusing multiple policies with an embedding-based attention operation. Furthermore, we propose modifications to KIAN's policy distributions to address entropy imbalance, which hinders efficient exploration with external policies in maximum entropy KGRL. Our experimental findings demonstrate that KIAN outperforms alternative methods incorporating external policies regarding sample efficiency, generalizability, and compositional and incremental learning.
However, it is essential to acknowledge a limitation not addressed in this work. The efficiency of KIAN, as well as other existing KGRL methods, may decrease when dealing with a large external knowledge set containing irrelevant policies. This issue is examined and discussed in Appendix F. Efficiently handling extensive sets of external policies is left for future research.
Our research represents an initial step towards the overarching goal of KGRL: learning a knowledge set with a diverse range of policies. These knowledge policies can be shared across various environments and continuously expanded, allowing artificial agents to flexibly query and learn from them. We provide detailed discussions on the broader impact of this work and outline potential directions of future research in Appendix D.
A Proofs and Learning Algorithms of KIAN
A.1 Proof of Proposition 4.1
Proposition 4.1 (Entropy imbalance in discrete decision-making). Assume that a  $d_{a}$ -dimensional probability simplex  $\pi \in \Delta^{d_a}$  is fused by  $\{\pi_1,\dots ,\pi_m\}$  and  $\{\hat{w}_1,\dots ,\hat{w}_m\}$  following equation (3), where  $\pi_j\in \Delta^{d_a}$ ,  $\hat{w}_j\geq 0\forall j\in \{1,\ldots ,m\}$  and  $\sum_{j = 1}^{m}\hat{w}_{j} = 1$ . If the entropy of  $\pi$  is maximized and  $\| \pi_1\|_{\infty}\ll \| \pi_2\|_{\infty},\| \pi_1\|_{\infty}\ll$ $\| \pi_3\|_{\infty},\ldots ,\| \pi_1\|_{\infty}\ll \| \pi_m\|_{\infty}$ , then  $\hat{w}_1\rightarrow 1$
Proof. Since  $\pi \in \Delta^{d_a}$  is defined as equation (3), and its entropy is maximized,
$$
\pi = \sum_ {j = 1} ^ {m} \hat {w} _ {j} \pi_ {j} = \left[ \frac {1}{d _ {a}}, \dots , \frac {1}{d _ {a}} \right] ^ {\top}. \tag {12}
$$
Equation (12) holds since  $\pi$  becomes a uniform distribution if its entropy is maximized.
Since each  $\pi_j$  in equation (12) is a probability simplex, it can be written as
$$
\pi_ {j} = \left[ \frac {1}{d _ {a}} + \varepsilon_ {j, 1}, \frac {1}{d _ {a}} + \varepsilon_ {j, 2}, \dots , \frac {1}{d _ {a}} + \varepsilon_ {j, d _ {a}} \right] ^ {\top}, \text {w h e r e} \tag {13}
$$
$$
\sum_ {i = 1} ^ {d _ {a}} \varepsilon_ {j, i} = 0 \quad \forall j \in \{1, \dots , m \}. \tag {14}
$$
Substituting equation (13) into equation (12) and, after some rearrangement, we get
$$
\hat {w} _ {1} \left[ \frac {1}{d _ {a}} + \varepsilon_ {1, 1}, \dots , \frac {1}{d _ {a}} + \varepsilon_ {1, d _ {a}} \right] ^ {\top} = \left[ \frac {1}{d _ {a}}, \dots , \frac {1}{d _ {a}} \right] ^ {\top} - \sum_ {j = 2} ^ {m} \hat {w} _ {j} \left[ \frac {1}{d _ {a}} + \varepsilon_ {j, 1}, \dots , \frac {1}{d _ {a}} + \varepsilon_ {j, d _ {a}} \right] ^ {\top}. \tag {15}
$$
Without loss of generality, we can assume that
$$
\max  _ {i \in \{1, \dots , d _ {a} \}} \varepsilon_ {1, i} = \varepsilon_ {1, 1} \tag {16}
$$
$$
\min  _ {i \in \{1, \dots , d _ {a} \}} \varepsilon_ {j, i} = \varepsilon_ {j, 1} \quad \text {a n d} \quad \varepsilon_ {j, 1} \leq 0 \quad \forall j \in \{2, \dots , m \}. \tag {17}
$$
Then take the infinity norm for both sides of equation (15), we get
$$
\left\| \hat {w} _ {1} \left[ \frac {1}{d _ {a}} + \varepsilon_ {1, 1}, \dots , \frac {1}{d _ {a}} + \varepsilon_ {1, d _ {a}} \right] ^ {\top} \right\| _ {\infty} = \left\| \left[ \frac {1}{d _ {a}}, \dots , \frac {1}{d _ {a}} \right] ^ {\top} - \sum_ {j = 2} ^ {m} \hat {w} _ {j} \left[ \frac {1}{d _ {a}} + \varepsilon_ {j, 1}, \dots , \frac {1}{d _ {a}} + \varepsilon_ {j, d _ {a}} \right] ^ {\top} \right\| _ {\infty} \tag {18}
$$
$$
\hat {w} _ {1} \left(\frac {1}{d _ {a}} + \varepsilon_ {1, 1}\right) = \frac {1}{d _ {a}} - \sum_ {j = 2} ^ {m} \hat {w} _ {j} \left(\frac {1}{d _ {a}} + \varepsilon_ {j, 1}\right). \tag {19}
$$
After some rearrangement, equation (19) becomes
$$
\frac {1}{d _ {a}} \left(\sum_ {j = 1} ^ {m} \hat {w} _ {j}\right) + \hat {w} _ {1} \varepsilon_ {1, 1} = \frac {1}{d _ {a}} - \sum_ {j = 2} ^ {m} \hat {w} _ {j} \varepsilon_ {j, 1} \tag {20}
$$
$$
\hat {w} _ {1} \varepsilon_ {1, 1} = - \sum_ {j = 2} ^ {m} \hat {w} _ {j} \varepsilon_ {j, 1}. \tag {21}
$$
Equation (21) holds since  $\sum_{j=1}^{m} \hat{w}_j = 1$ .
Given equation (14), (17), and the assumption that  $\| \pi_1\|_{\infty}\ll \| \pi_2\|_{\infty},\| \pi_1\|_{\infty}\ll \| \pi_3\|_{\infty},\ldots ,\| \pi_1\|_{\infty}\ll$ $\| \pi_m\|_{\infty}$ , we have the following inequalities
$$
\varepsilon_ {1, 1} \ll - \varepsilon_ {j, 1} \quad \forall j \in \{1, \dots , m \}. \tag {22}
$$
Hence, for equation (21) to hold,  $\hat{w}_1\rightarrow 1$
Proposition 4.1 states that if  $\| \pi_1\|_{\infty}$  is much smaller than  $\| \pi_j\|_{\infty}$ , then maximizing the entropy of  $\pi$  results in  $\hat{w}_1\to 1$  and  $\hat{w}_j\to 0,\forall j\in \{2,\ldots ,m\}$ . Next, we provide another proposition showing that if  $\pi_1$  is a uniform distribution and  $\pi_j$  is not, then  $\| \pi_1\|_{\infty} < \| \pi_j\|_{\infty},\forall j\in \{2,\dots ,m\}$ .
Proposition A.1 (Infinity norm of a probability simplex). Given two  $d_{a}$ -dimensional probability simplices,  $\pi_1 \in \Delta^{d_a}$  and  $\pi_2 \in \Delta^{d_a}$ , if  $\pi_1$  is a uniform distribution and  $\pi_2$  is not, then  $\| \pi_1 \|_{\infty} = \frac{1}{d_a} < \| \pi_2 \|_{\infty}$ .
Proof. Since  $\pi_1 \in \Delta^{d_a}$  is a uniform distribution,  $\pi_1 = \left[\frac{1}{d_a}, \frac{1}{d_a}, \ldots, \frac{1}{d_a}\right]^\top$ . The infinity norm of  $\pi_1$  becomes
$$
\left\| \pi_ {1} \right\| _ {\infty} = \max  _ {i \in \{1, \dots , d _ {a} \}} | \pi_ {1, i} | = \frac {1}{d _ {a}}, \tag {23}
$$
where  $\pi_{1,i}$  is the  $i$ -th element of  $\pi_1$ . On the other hand, since  $\pi_2 \in \Delta^{d_a}$  is not a uniform distribution, it can be represented as
$$
\pi_ {2} = \left[ \frac {1}{d _ {a}} + \varepsilon_ {2, 1}, \frac {1}{d _ {a}} + \varepsilon_ {2, 2}, \dots , \frac {1}{d _ {a}} + \varepsilon_ {2, d _ {a}} \right] ^ {\top}, \text {w h e r e} \tag {24}
$$
$$
\sum_ {i = 1} ^ {d _ {a}} \varepsilon_ {2, i} = 0 \text {a n d} \left\{\varepsilon_ {2, 1}, \dots , \varepsilon_ {2, d _ {a}} \mid \varepsilon_ {2, i} \neq 0 \right\} \neq \emptyset . \tag {25}
$$
Equation (25) indicates that at least one element in  $\{\varepsilon_{2,1},\dots ,\varepsilon_{2,d_a}\}$  should be larger than 0. Hence,