SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \ln \mathbb {E} _ {Q (z, B _ {\tau})} [ F ^ {\prime} (z, B _ {\tau}) ] = \ln \mathbb {E} _ {Q (z, B _ {\tau})} \left[ \frac {P (Y , B _ {\tau} | \tau (z)) R (z | \tau (z))}{Q (B _ {\tau} | \tau (z)) Q (z)} \right] (71) \\ = \ln \mathbb {E} _ {Q (z, B _ {\tau})} \left[ \frac {P (Y , B _ {\tau} | \tau (z)) R (z | \tau (z))}{Q (z , B _ {\tau})} \right] (72) \\ = \ln \mathbb {E} _ {Q (z)} \left[ \frac {P (Y , \tau (z)) R (z | \tau (z))}{Q (z)} \right] (73) \\ = \ln \sum_ {\tau^ {\prime} \in \mathcal {T}} \mathbb {E} _ {Q (z)} \left[ \frac {P (Y , \tau^ {\prime}) R (z | \tau^ {\prime})}{Q (z)} \mathbb {I} [ \tau^ {\prime} = \tau (z) ] \right] (74) \\ = \ln \sum_ {\tau^ {\prime} \in \mathcal {T}} P (Y, \tau^ {\prime}) \mathbb {E} _ {R (z | \tau^ {\prime})} [ \mathbb {I} [ \tau^ {\prime} = \tau (z) ] ] (75) \\ \leq \ln \sum_ {\tau^ {\prime} \in \mathcal {T}} P \left(Y, \tau^ {\prime}\right) = \ln P (Y), (76) \\ \end{array}
$$
where, in the transition from the first to the second row, we employed the following relation:
$$
Q \left(B _ {\tau} | \tau (z)\right) = \sum_ {\tau \in \mathcal {T}} Q \left(B _ {\tau} | \tau\right) \mathbb {I} [ \tau = \tau (z) ] = \sum_ {\tau \in \mathcal {T}} Q \left(B _ {\tau} | \tau\right) Q (\tau | z) = Q \left(B _ {\tau} | z\right), \tag {77}
$$
and we have used Proposition 2 for the last inequality.
Finally, we will show that the following convergence property assuming that  $F(z,B_{\tau})$  is bounded:
$$
\mathcal {L} _ {\mathrm {I W}} ^ {(K)} [ Q, R ] \rightarrow \ln \left(\sum_ {\tau^ {\prime} \in \mathcal {T}} P (Y, \tau^ {\prime}) \mathbb {E} _ {R (z | \tau^ {\prime})} \mathbb {I} \left[ \tau^ {\prime} = \tau (z) \right]\right) \quad (K \rightarrow \infty). \tag {78}
$$
From the strong law of large numbers, it follows that  $\frac{1}{K}\sum_{k = 1}^{K}F_k'$  converges to the following term almost surely as  $K\to \infty$  ..
$$
\mathbb {E} _ {Q \left(z _ {k}, B _ {k}\right)} \left[ F ^ {\prime} \left(z _ {k}, B _ {k}\right) \right] = \sum_ {\tau^ {\prime} \in \mathcal {T}} P \left(Y, \tau^ {\prime}\right) \mathbb {E} _ {R (z | \tau^ {\prime})} \mathbb {I} \left[ \tau^ {\prime} = \tau (z) \right], \tag {79}
$$
where we have employed the same transformations as used from equation (71) to (75). Observe that the  $r.h.s$  term of the equation (78) equals to  $\ln P(Y)$  when  $\forall \tau' \in \mathcal{T}$ ,  $\mathbb{E}_{R(z|\tau')}[\mathbb{I}[\tau' = \tau(z)]] = 1$ , which completes the proof.
Estimation of marginal log-likelihood For the estimation of  $\ln P(Y)$ , we employ  $\mathcal{L}^{(K)}[Q,R]$  with  $K = 1,000$  similar to [40]. From Theorem 1, IW-ELBO  $\mathcal{L}^{(K)}[Q,R]$  is at least a better lower bound of  $\ln P(Y)$  than ELBO  $\mathcal{L}[Q,R]$ , and converges to  $\ln P(Y)$  when  $\forall \tau ,\mathbb{E}_{R(z|\tau)}[\mathbb{I}[\tau = \tau (z)]] = 1$ . According to Proposition 2, this equality condition is satisfied when we set  $R(z|\tau) = Q(z|\tau)$ , which is approached by maximizing  $\mathcal{L}[Q,R]$  with respect to  $R$  as indicated in Proposition 1.
B.8 Gradient estimators for IW-ELBO
The gradient of IW-ELBO  $\mathcal{L}_{\mathrm{IW}}^{(K)}[Q_{\theta, \phi}, R_{\psi}]$  with respect to  $\theta$  is given by
$$
\begin{array}{l} \nabla_ {\theta} \mathcal {L} _ {\mathrm {I W}} ^ {(K)} = \mathbb {E} _ {Q _ {\theta , \phi} (z ^ {(1)}, B _ {\tau} ^ {(1)}) \dots Q _ {\theta , \phi} (z ^ {(K)}, B _ {\tau} ^ {(K)})} \left[ \sum_ {k = 1} ^ {K} w _ {k} (z ^ {(1: K)}, B ^ {(1: K)}) \nabla_ {\theta} \ln F ^ {\prime} (z ^ {(k)}, B _ {\tau} ^ {(k)}) \right] \\ + \mathbb {E} _ {Q _ {\theta , \phi} (z ^ {(1)}, B _ {\tau} ^ {(1)}) \dots Q _ {\theta , \phi} (z ^ {(K)}, B _ {\tau} ^ {(K)})} \left[ \sum_ {k = 1} ^ {K} \nabla_ {\theta} \ln Q _ {\theta} (z ^ {(k)}) \cdot \ell \left(z ^ {(1: K)}, B ^ {(1: K)}\right) \right], \tag {80} \\ \end{array}
$$
where we have defined
$$
w _ {k} \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) := \frac {F ^ {\prime} \left(z ^ {(k)} , B _ {\tau} ^ {(k)}\right)}{\sum_ {k ^ {\prime} = 1} ^ {K} F ^ {\prime} \left(z ^ {(k ^ {\prime})} , B _ {\tau} ^ {(k ^ {\prime})}\right)}, \tag {81}
$$
$$
\ell \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) := \ln \left(\frac {1}{K} \sum_ {k ^ {\prime} = 1} ^ {K} F ^ {\prime} \left(z ^ {\left(k ^ {\prime}\right)}, B _ {\tau} ^ {\left(k ^ {\prime}\right)}\right)\right). \tag {82}
$$
Similarly, as  $F'$  is differentiable with respect to  $B_{\tau}$ , and  $B_{\tau} = h_{\phi}(\epsilon_B, \tau)$  is differentiable with respect to  $\phi$ , the gradient of  $\mathcal{L}_{\mathrm{IW}}^{(K)}$  with respect to  $\phi$  can be evaluated as follows:
$$
\nabla_ {\phi} \mathcal {L} _ {\mathrm {I W}} ^ {(K)} = \mathbb {E} _ {Q _ {\theta} (z ^ {(1)}) \dots Q _ {\theta} (z ^ {(K)})} \mathbb {E} _ {p _ {B} \left(\epsilon_ {B} ^ {(1)}\right) \dots p _ {B} \left(\epsilon_ {B} ^ {(K)}\right)} \left[ \sum_ {k = 1} ^ {K} w _ {k} \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) \nabla_ {\phi} \ln F ^ {\prime} \left(z ^ {(k)}, h _ {\phi} \left(\epsilon^ {(k)}, \tau\right)\right) \right]. \tag {83}
$$
Since  $\nabla_{\theta}\ln F'(z^{(k)},B_{\tau}^{(k)}) = -\nabla_{\theta}\ln Q_{\theta}(z^{(k)})$  from equation (34), an unbiased estimator of the gradient  $\nabla_{\theta}\mathcal{L}^{(K)}$  is given as follows:
$$
\widehat {g} _ {\mathrm {I W}, \theta} ^ {(K)} := \sum_ {k = 1} ^ {K} \nabla_ {\theta} \ln Q _ {\theta} (z ^ {(k)}) \cdot \left[ - w _ {k} \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) + \ell \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) \right]. \tag {84}
$$
The remaining gradient estimators are given as follows:
$$
\widehat {g} _ {\mathrm {I W}, \phi} ^ {(K)} = \sum_ {k = 1} ^ {K} w _ {k} \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) \nabla_ {\phi} \ln F \left(z ^ {(k)}, h _ {\phi} \left(\epsilon^ {(k)}, \tau\right)\right), \tag {85}
$$
$$
\widehat {g} _ {\mathrm {I W}, \phi} ^ {(K)} = \sum_ {k = 1} ^ {K} w _ {k} \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) \nabla_ {\psi} \ln F \left(z ^ {(k)}, h _ {\phi} \left(\epsilon^ {(k)}, \tau\right)\right). \tag {86}
$$
In total, the target for computing auto-gradient for these gradients is given as follows:
$$
\begin{array}{l} \widehat {\mathcal {L}} _ {\mathrm {I W}} ^ {\prime (K)} = \sum_ {k = 1} ^ {K} \ln Q _ {\theta} (z ^ {(k)}) \cdot \det a c h \left[ - w _ {k} (z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}) + \ell (z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}) \right] \\ + \sum_ {k = 1} ^ {K} \operatorname {d e t a c h} \left[ w _ {k} \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) \right] \ln F \left(z ^ {(k)}, h _ {\phi} \left(\epsilon_ {B} ^ {(k)}, \tau \left(z ^ {(k)}\right)\right)\right). \tag {87} \\ \end{array}
$$
B.9 VIMCO estimators
The VIMCO estimator [22] aims at reducing the variance in each of the terms proportional to  $\ell(z^{(1:K)}, B^{(1:K)})$  in equation (84). This is accomplished by forming for a control variate  $\ell_k(z^{(\backslash k)}, B^{(\backslash B_\tau)})$  that consists of only variables other than  $z^{(k)}$  and  $B^{(k)}$  for every  $k$  as follows:
$$
\widehat {g} _ {\theta , \mathrm {V I M C O}} ^ {(K)} := \sum_ {k = 1} ^ {K} \nabla_ {\theta} \ln Q _ {\theta} (z ^ {(k)}) \cdot \left[ - w _ {k} \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) + \ell \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) - \bar {\ell} _ {k} \left(z ^ {(\backslash k)}, B _ {\tau} ^ {(\backslash k)}\right) \right], \tag {88}
$$
where we denote that
$$
\bar {\ell} _ {k} \left(z ^ {(\backslash k)}, B ^ {(\backslash k)}\right) := \ln \left(\frac {1}{K} \left(\overline {{F ^ {\prime}}} _ {k} \left(z ^ {(\backslash k)}, B _ {\tau} ^ {(\backslash k)}\right) + \sum_ {k ^ {\prime} = 1, k ^ {\prime} \neq k} ^ {K} F ^ {\prime} \left(z ^ {(k ^ {\prime})}, B _ {\tau} ^ {(k ^ {\prime})}\right)\right)\right), \tag {89}
$$
$$
\overline {{F ^ {\prime}}} _ {k} \left(z ^ {(\backslash k)}, B _ {\tau} ^ {(\backslash k)}\right) := \exp \left(\frac {1}{K - 1} \sum_ {k ^ {\prime} = 1, k ^ {\prime} \neq k} ^ {K} \ln F ^ {\prime} \left(z ^ {\left(k ^ {\prime}\right)}, B _ {\tau} ^ {\left(k ^ {\prime}\right)}\right)\right). \tag {90}
$$
Note that the unbiasedness of  $\widehat{g}_{\theta, \mathrm{VIMCO}}$  can be confirmed through the following observation:
$$
\begin{array}{l} \mathbb {E} _ {Q _ {\theta , \phi} (z ^ {(k)}, B _ {\tau} ^ {(k)})} \left[ \nabla_ {\theta} \ln Q _ {\theta} (z ^ {(k)}) \cdot \bar {\ell} _ {k} (z ^ {(\backslash k)}, B _ {\tau} ^ {(\backslash k)}) \right] \\ = \bar {\ell} _ {k} \left(z ^ {(\backslash k)}, B _ {\tau} ^ {(\backslash k)}\right) \cdot \mathbb {E} _ {Q _ {\theta , \phi} \left(z ^ {(k)}, B _ {\tau} ^ {(k)}\right)} \left[ \nabla_ {\theta} \ln Q _ {\theta} \left(z ^ {(k)}\right) \right] \\ = \bar {\ell} _ {k} \left(z ^ {(\backslash k)}, B _ {\tau} ^ {(\backslash k)}\right) \cdot \mathbb {E} _ {Q _ {\theta , \phi} (z ^ {(k)})} \left[ \nabla_ {\theta} \ln Q _ {\theta} \left(z ^ {(k)}\right) \right] = 0. \tag {91} \\ \end{array}
$$
To employ the VIMCO estimator for  $\theta$ , the target for auto-gradient computation as stated in equation (87) needs to be adjusted as follows:
$$
\begin{array}{l} \widehat {\mathcal {L}} _ {\mathrm {V I M C O}} ^ {\prime (K)} = \sum_ {k = 1} ^ {K} \ln Q _ {\theta} (z ^ {(k)}) \cdot \det a c h \left[ - w _ {k} (z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}) + \ell (z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}) - \bar {\ell} _ {k} (z ^ {(\backslash k)}, B _ {\tau} ^ {(\backslash k)}) \right] \\ + \sum_ {k = 1} ^ {K} \operatorname {d e t a c h} \left[ w _ {k} \left(z ^ {(1: K)}, B _ {\tau} ^ {(1: K)}\right) \right] \ln F \left(z ^ {(k)}, h _ {\phi} \left(\epsilon_ {B} ^ {(k)}\right), \tau \left(z ^ {(k)}\right)\right). \tag {92} \\ \end{array}
$$
C.1 Training process
For the training of GeoPhy, we continued the stochastic gradient descent process until a total of 1,000,000 Monte Carlo (MC) tree samples were consumed. Specifically, if  $K$  MC-samples were used
per step, we performed up to  $1,000,000 / \mathrm{K}$  steps. It is noteworthy that the number of MC samples equaled the number of likelihood evaluations (NLEs), which provided us with a basis for comparing convergence speed between different runs, as shown in Fig. 2. In all of our experiments, we used Adam optimizer with an initial learning rate of 0.0001. The learning rate was then multiplied by 0.75 after every 200,000 steps. Similar to approaches taken by Zhang and Matsen IV [42], we incorporated an annealing procedure during the initial consumption of  $100,000\mathrm{MC}$  samples. Specifically, we replaced the likelihood function in the lower bound with  $P(Y|B_{\tau},\tau)^{\beta}$  and linearly increased the inverse temperature  $\beta$  from 0.001 to 1 throughout the iterations. Note that all the estimations of marginal log-likelihood (MLL) were performed with  $\beta$  set to 1.
For each estimation of MLL values, GeoPhy and VBPI-GNN used IW-ELBO with 1000 samples, whereas all CSMC-based methods used 2048 particle weights [16].
C.2 Variational branch length distributions
For the variational branch length distribution  $Q_{\phi}(B_{\tau}|\tau)$ , we followed an architecture of [40]; namely, each branch length independently followed a lognormal distribution which was parameterized with a graph neural network (GNN). Details are described in Appendix A.2.
C.3 LAX estimators
As input features of a surrogate function  $s_{\chi}(z)$  used in the LAX estimators, we employed a flattened vector of coordinates  $z \in \mathbb{R}^{N \times d}$  when  $z$  resides in Euclidean space. In cases where the coordinates were  $z \in \mathbb{H}^d$ , we first transformed  $z$  with a logarithm map  $\log_{\mu^{\circ}} z \in T_{\mu^{\circ}}\mathbb{H}^{d}$ , then omitted their constant value 0-th elements and subsequently flattened the result. We implemented a simple multi-layer perceptron (MLP) network with a single hidden layer of width  $10Nd$  and a subsequent sigmoid linear unit (SiLU) activation function as the neural network to output  $s_{\chi}(z)$ .
C.4 Replication of MLL estimates with MrBayes SS
Given the observed discrepancies in marginal log-likelihood (MLL) estimates obtained with the MrBayes stepping-stone (SS) method between references [42] and [16], we replicated the MrBayes SS runs using MrBayes version 3.2.7a. The script we used is provided below.
BEGIN MRBAYES;   
set autoclose=yes nowarn=yes Seed=123 Swapseed=123;   
lset nst=1;   
prset statefreqpr  $\equiv$  fixed(equal);   
prset brlenspr  $\equiv$  Unconstrained:exp(10.0);   
ssngen  $=$  1000000nruns  $=$  10 nchains  $= 4$  printfreq  $=$  1000 samplefreq  $= 100$  savebrrels  $\equiv$  yes filename  $\equiv$  mrbayes_ss_out;   
END;
We incorporated the results in the row named Replication in Table 3, where the values aligned more closely with those found in [42]. We deduced that the prior distribution used in [16] might have been set differently as the current default values of brlenspr are Unconstrained: GammaDir(1.0, 0.100, 1.0, 1.0)  $^{3}$ , which deviates from the model assumption used for the benchmarks. We observed that the line brlenspr was not included in the code provided in Appendix F of [16]. Having been able to replicate the results found in [42], we opted to use their values as a reference in Table 1.
C.5 Visualization of tree topologies
In Fig. 3, we visualized the sum of the probability densities for tip node distribution  $\sum_{i=1}^{N} Q(z_i)$  by projecting each hyperbolic coordinate  $z_i \in \mathbb{H}^d$  onto the Poincaré coordinates  $\overline{z}_{ik} = z_{ik} / (1 + z_{i0})$  ( $k = 1, \ldots, d$ ), then applying a transformation  $\overline{z}_i \mapsto \overline{z}_i' = \tanh(a \| \overline{z}_i \|_2) \cdot \overline{z}_i / \| \overline{z}_i \|_2$
<sup>3</sup>https://github.com/NBISweden/MrBayes/blob/develop/doc/manual/Manual_MrBayes_v3.2.pdf
with  $a = 2.1$  to emphasize the central region. To display the density  $Q$  in the new coordinates, the Jacobian term was also considered to evaluate the density  $Q(\overline{z}_i')$ .
For the comparison of consensus tree topologies, we plotted the edges of the tree by connecting each of their end node coordinate pairs with a geodesic line. The coordinate in  $\mathbb{H}^d$  of the  $i$ -th tip node was determined as the location parameter  $\mu_i \in \mathbb{H}^d$  of the wrapped normal distribution  $Q(z_i) = \mathcal{W}\mathcal{N}(z_i; \mu_i, \Sigma_i)$ . Let  $\xi_u \in \mathbb{H}^d$  denotes the coordinate of an interior node  $u$ , we defined  $\xi_u$  by using the Lorentzian centroid operation  $\mathcal{C}$  [18] as follows:
$$
\xi_ {u} := \mathcal {C} \left(\left\{c _ {s} \right\} _ {s \in \mathcal {S} _ {\tau} (u)}, \left\{\nu_ {s} \right\} _ {s \in \mathcal {S} _ {\tau} (u)}\right) = \frac {\widetilde {\xi_ {u}}}{\sqrt {- \left\langle \widetilde {\xi_ {u}} , \widetilde {\xi_ {u}} \right\rangle_ {L}}}, \tag {93}
$$
where  $\widetilde{\xi}_u\coloneqq \sum_{s\in S_\tau (u)}\nu_sc_s$  denote an unnormalized sum of weighted coordinates,  $s\in S_{\tau}(u)$  denote a subset of tip node indices partitioned by the interior node  $u$  in the tree topology  $\tau$ ,  $c_{s}\coloneqq \mathcal{C}(\{\mu_{i}\}_{i\in s},\{1\}_{i\in s})$  denote the Lorentzian centroid of the tip nodes contained in the subset  $s$ , and  $\nu_{s} = N - |s|$  denote the number of the tip nodes in the complement set of  $s$  where  $|s|$  represents the number of tip nodes in the subset  $s$ . As an unrooted tree topology  $\tau$  can be identified by the set of tip node partitions introduced by the interior nodes of  $\tau$ , the same unrooted tree topologies give the same set of interior coordinates  $\{\xi_u\}_{u\in V}$  according to equation (93).
D.1 Marginal log-likelihood estimates throughout iterations for DS1 dataset
In Fig. 2, we highlight several key findings of training processes. Specifically, (1) control variates were crucial for optimizations; (2) The LAX estimator for individual Monte Carlo (MC) samples  $(K = 1)$  and leave-one-out (LOO) CVs for multiple MC samples were effective; (3) IW-ELBO and VIMCO were not particularly effective when a full diagonal matrix is used for  $Q(z)$ . In Fig. 5 and 6, we show that these findings hold consistently across various control variates and configurations of  $Q(z)$ , including, different types of distributions (normal  $\mathcal{N}$  and wrapped normal  $\mathcal{W}\mathcal{N}$ ), embedding dimensions  $(d = 2, 3, \text{and} 4)$ , and varying numbers of MC samples  $(K = 1, 2, \text{and} 3)$ .
D.2 Marginal log-likelihood estimates for eight datasets
We present more comprehensive results in Table 3, extending upon the data from Table 1. This table showcases the marginal log-likelihood (MLL) estimates obtained with various GeoPhy configurations and other conventional methods for the datasets DS1-DS8. Once again, GeoPhy demonstrates its superior performance, consistently outperforming CSMC-based approaches that do not require preselection of tree topologies across the majority of configurations and datasets. This reaffirms the stability and excellence of our approach. Additionally, we found that a  $Q(z)$  configuration using a 4-dimensional wrapped normal distribution with a full covariance matrix was the most effective among the tested configurations.
D.3 Analysis of tree topology distributions
To assess the accuracy and expressivity of GeoPhy's tree topology distribution, we contrasted  $Q(\tau)$  with MCMC tree samples. For our analysis, we considered experimental runs using LAX estimators with either  $K = 1$  or 3. For a configuration of  $Q(z)$ , We employed a diagonal or multivariate wrapped normal distribution with 2, 4, 5, and 6 dimensions. We sampled 1000 MC trees from  $Q(\tau)$  for each experiment and then deduced the majority consensus tree. As depicted in Fig. 8 across the eight datasets, the Robinson-Foulds (RF) distances between the majority consensus tree of  $Q(\tau)$  and the one obtained with MrBayes were well aligned with the MLL values.
To highlight the expressivity limitation of  $Q(\tau)$  obtained from a simple independent distribution of  $Q(z)$ , we compiled three statistics related to the tree topology samples for DS1, DS4, and DS7 datasets (Table 4), wherein we employed the model with the least RF distance for each dataset. For DS1 and DS4, GeoPhy's tree topologies were less diverse than those of MrBayes and VBPI-GNN and had more concentration on the most frequent topology. For more diffused tree DS7, GeoPhy also showed diverse tree samples with a close diversity index to the other two.
Table 2: Comparative results of the mean MLL estimates for DS1 dataset obtained with GeoPhy for different combinations of the distribution  $Q(z)$ , per-step Monte Carlo samples ( $K$ ), and control variates (CVs). Each of the mean MLL values is obtained from five independent runs with 1000 MC samples at the last steps. Standard deviations are shown in parentheses. The bold and underlined numbers represent the best (highest) values in row-wise and column-wise comparisons, respectively.
CV	N(diag)	N(full)
d=2	d=3	d=4	d=2	d=3	d=4
K=1 (LAX)	-7126.79(10.06)	-7130.11(10.63)	-7123.95(12.03)	-7129.70(6.14)	-7119.51(10.90)	-7120.03(11.92)
K=2 (LOO)	-7122.84(11.96)	-7125.72(13.08)	-7121.79(12.52)	-7121.23(14.62)	-7130.43(10.82)	-7114.92(8.32)
K=2 (LOO+LAX)	-7129.26(8.35)	-7128.18(9.83)	-7116.96(10.39)	-7121.74(10.73)	-7129.45(10.23)	-7115.14(8.26)
K=3 (LOO)	-7130.60(10.66)	-7124.61(12.21)	-7120.88(13.15)	-7132.35(6.89)	-7128.26(9.80)	-7124.62(12.33)
K=3 (LOO+LAX)	-7128.36(9.77)	-7125.66(10.95)	-7119.81(11.71)	-7122.76(10.81)	-7123.67(11.23)	-7123.37(11.28)
CV	WN(diag)	WN(full)
d=2	d=3	d=4	d=2	d=3	d=4
K=1 (LAX)	-7126.89(10.06)	-7133.78(13.57)	-7122.10(12.29)	-7124.63(8.09)	-7119.59(10.84)	-7111.55(0.07)
K=2 (LOO)	-7121.13(13.06)	-7125.60(13.04)	-7121.80(12.52)	-7114.99(4.66)	-7130.40(10.80)	-7116.14(10.66)
K=2 (LOO+LAX)	-7120.90(10.10)	-7120.96(13.08)	-7126.49(12.01)	-7119.81(9.78)	-7129.58(10.24)	-7116.21(10.75)
K=3 (LOO)	-7130.67(10.67)	-7124.54(12.20)	-7120.94(13.11)	-7125.94(13.07)	-7122.85(11.96)	-7119.77(11.80)
K=3 (LOO+LAX)	-7128.40(9.78)	-7125.69(13.02)	-7125.78(13.10)	-7115.19(8.16)	-7120.96(13.12)	-7116.09(10.67)
We also summarize the frequency of bipartitions observed for tree topology samples in Fig. 8. The intermediate values between 0 and 1 in this frequency plot reveal topology diversities, where VBPI-GNN aligns more closely with MrBayes compared to GeoPhy. For DS1 and DS4, GeoPhy tends to take values near zero and one in the slope region. For DS7, while GeoPhy traced the curve more accurately, fluctuations around this curve highlight potential room for improvement.
D.4 Runtime comparison
We measured CPU runtimes using a single thread on a 20-core Xeon processor. In Fig. 10, we compared single-core runtime of MrBayes, VPBI-GNN, and GeoPhy (ours). As MrBayes is written in C and highly optimized its per-iteration computation time is much faster than current variational methods. In general, the per-step efficiency of GeoPhy is comparable to that of VBPI-GNN. Although the convergence of MLL estimates tends to be slow in terms of the number of consumed samples for larger  $K$  (Fig. 9 Left), the total CPU-time slightly decreases as  $K$  increases (Fig. 10 Left). We also included the estimated runtime across the eight datasets and the different model configurations, where the number of species ranges from 27 to 64 (Fig. 10 Right).
Figure 5: Comparison of marginal log-likelihood (MLL) estimates for DS1 dataset using different control variates and configurations of  $Q(z)$ . We configured  $Q(z)$  to be an independent normal distribution of dimensions  $d = 2, 3$ , or 4 with either a diagonal (diag) or full covariance matrix. The number of MC samples was set to  $K = 1, 2$ , or 3. For a detailed explanation of the legend, refer to the caption of Fig. 2.
Figure 6: Comparison of marginal log-likelihood (MLL) estimates for DS1 dataset using different control variates and configurations of  $Q(z)$ . We employed wrapped normal distributions for  $Q(z)$  instead of normal distributions shown in Fig. 5. The dimensions  $d = 2, 3$ , or 4 and the number of MC samples  $K = 1, 2$ , or 3 are the same as in the referenced figure.
Table 3: Extended results of Table 1 comparing the marginal log-likelihood (MLL) estimates with different approaches in eight benchmark datasets. The MLL values for MrBayes SS and VBPI-GNN were obtained from [40], while CSMC, VCSMC, and  $\phi$ -CSMSC are referenced from [16]. We also included replicated results for MrBayes SS. The MLL values for our approach (GeoPhy) are presented for a variety of  $Q(z)$  configurations, encompassing distribution types (normal  $\mathcal{N}$  or wrapped normal  $\mathcal{W}\mathcal{N}$ ), embedding dimensions (2 or 4), and the covariance matrix (full or diagonal). Each result features various CVs: LAX with  $K = 1$ , LOO with  $K = 3$  denoted as  $\mathrm{LOO}(3)$ , and a combination of LOO and LAX, denoted as  $\mathrm{LOO}(3)+$ . The figures highlighted in bold represent the highest values obtained with GeoPhy and the three CSMC-based methods, all of which perform an approximate Bayesian inference without the preselection of topologies. We have underlined MLL estimates where GeoPhy outperformed the other CSMC-based methods.
Figure 7: Marginal log-likelihood (MLL) estimates and Robinson-Foulds (RF) distance between the consensus trees obtained from topology distribution  $Q(\tau)$  and MrBayes (MCMC) for datasets DS1-DS8. Each dot denotes an individual experimental run. The term 'dim' refers to the dimension of wrapped normal distributions used for  $Q(z)$ . To avoid dot overlap in the plot, RF metrics are randomly jittered within  $\pm 0.2$ . The dashed line illustrates the MLL value estimated by MrBayes.
Figure 8: Comparison of bipartition frequencies from the posterior tree topology distributions of MrBayes, VBPI-GNN, and GeoPhy (ours), for DS1, DS4, and DS7 datasets. The bipartitions were ordered by descending frequency as seen in MrBayes.