PhyloVAE: Unsupervised Learning of Phylogenetic Trees via Variational Autoencoders
Authors: Tianyu Xie
Harry Richman
Jiansi Gao
Frederick A Matsen
Cheng Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Variational Autoencoders; Unsupervised Learning; Phylogenetic Trees; Representation Learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Learning informative representations of phylogenetic tree structures is essential for analyzing evolutionary relationships. Classical distance-based methods have been widely used to project phylogenetic trees into Euclidean space, but they are often sensitive to the choice of distance metric and may lack sufficient resolution. In this paper, we introduce *phylogenetic variational autoencoders* (PhyloVAEs), an unsupervised learning framework designed for representation learning and generative modeling of tree topologies. Leveraging an efficient encoding mechanism inspired by autoregressive tree topology generation, we develop a deep latent-variable generative model that facilitates fast, parallelized topology generation. PhyloVAE combines this generative model with a collaborative inference model based on learnable topological features, allowing for high-resolution representations of phylogenetic tree samples. Extensive experiments demonstrate PhyloVAE's robust representation learning capabilities and fast generation of phylogenetic tree topologies.

Published as a conference paper at ICLR 2025
PHYLOVAE: UNSUPERVISED LEARNING OF PHYLOGE- NETIC TREES VIA VARIATIONAL AUTOENCODERS
Tianyu Xie', Harry Richman', Jiansi Gao", Frederick A. Matsen IV3,4, Cheng Zhang!
1 School of Mathematical Sciences, Peking University
2 Center for Statistical Science, Peking University
3 Computational Biology Program, Fred Hutchinson Cancer Research Center
4 Howard Hughes Medical Institute
i anyuxie@pku. edu, en, (hrichmane chman, isigao, mat ) fredhutch. org, chengzhang@math h. pku. edu. cn
ABSTRACT
Learning informative representations of phylogenetic tree structures is essential for analyzing evolutionary relationships. Classical distance-based methods have been widely used project phylogenetic trees into Euclidean space, but they often sensitive to the choice of distance metric and may lack sufficient resolution. In this paper, introduce phylogenetic variational autoencoders (PhyloVAEs), an unsu- pervised learning framework designed for representation learning and generative modeling of tree topologies. Leveraging efficient encoding mechanism inspired by autoregressive tree topology generation, we develop a deep latent-variable generative model that facilitates fast, parallelized topology generation. Phylo- VAE combines this generative model with collaborative inference model based on learnable topological features, allowing for high-resolution representations of phylogenetic tree samples. Extensive experiments demonstrate PhyloVAE's ro- bust representation learning capabilities and fast generation of phylogenetic tree topologies.
INTRODUCTION
Phylogenetic trees are the foundational structure for describing the evolutionary processes among individuals or groups of biological entities. Reconstructing these trees based collected biological sequences (e.g., DNA, RNA, protein) from observed species, also known as phylogenetic inference (Felsenstein) 2004), is essential discipline of computational biology (Fitch 1971 Felsenstein 1981 Yang & Rannala 1997, Ronqqist et al. (2012).
Large collections of trees obtained from these approaches (e.g., posterior samples from MCMC runs (Ronquist et al. 2012)), however, are often difficult to summarize or visualize due to the discrete and Euclidean nature of the tree topology spacd] Yet, the importance of being able to do in practice has spurred substantial research in this direction. The classical approach to visualize and analyze distributions of phylogenetic trees is to calculate pairwise distances between the trees and project them into a plane using multidimensional scaling (MDS) (Amenta & Klingner, [002]] Hilli al. | Pombart al., (017). However, these approaches have the shortcoming that one can not map an arbitrary point in the visualization to tree, and therefore do not form actual visualization of the relevant tree space. Furthermore, their effectiveness heavily depends on the choice of distance metric and can sometimes exhibit counterintuitive behaviors (Kuhner & Yamato 2015). and the visualizations can suffer from poor resolution, where distinct sets of trees overlap within the same regions (Hillis al. 2005 Whidden & Matsen IV 2015) (see Figure in Section 5.2 for example). Recently, several vector representation methods have been developed for tree topologies (Liu) 2021] Penn al.] 2 However, these representations only provide raw features whose dimension scales to the tree topology size and hence may fail to deliver concise and useful information of
*Correpponding author. In phylogenetic terminology, free topology is just the discrete graph-theoretic component of the tree without additional information such as edge lengths.

---

Published as a conference paper at ICLR 2025
tree topologies. In general, finding good representations of tree topologies that preserve essential information for downstream tasks (e.g., comparison, visualization, clustering) remains challenging.
On the other end of the spectrum lie much more recent methods to perform density estimation sets of phylogenetic trees (Larget) 2013 Zhang & Matsen IV] 2018: Xie Zhang 2023). These methods very high resolution, as be seen by the excellent fit that they offer to empirical distributions of phylogenetic trees. However, these methods do not lend themselves to representation learning and visualization, and it is difficult understand what they are telling about the structure of the phylogenetic tree shape distribution.
In this paper, introduce phylogenetic variational autoencoders (PhyloVAEs) which is unsuper- vised learning framework that for the first time allows both representation learning and generative modeling of phylogenetic tree topologies in a satisfying and useful way. Inspired by the tree topology generating process outlined in ARTree (Xie & Zhang, 2023), we first encode tree topology into an integer vector representing the corresponding edge decisions, through linear-time algorithm. Based on this encoding mechanism, we develop deep variable generative model for tree topologies, together with an inference model for the posterior distribution of the latent variable given the tree topology using learnable topological features (Zhang. 2023). In this way, Phylo VAE provides a latent space representation that can be easily visualized like the previous MDS method, but at the same time, it is probabilistic model that gives high-resoluticn representation of the tree topology distribution. Although the main purpose of VAY representation learning, we want to emphasize that is the generative modeling objective that forces PhyloVAE learn high-resolution representations to retain more distributional information. We summarize our main contributions as follows:
We propose the first representation learning framework for phylogenetic tree topologies with deep models, which has more capacity to distinguish different shapes of tree topologies compared to traditional distance-batt based methods (see Section 5.2 for an example). Moreover, the generative nature of Phylo VAE also allows us to map an arbitrary point in the latent space to a tree topology (Figure 3). which is impossible for current methods.
In addition to providing high-resolution representation of the tree topologies, Phylo as non-autoregressive model, enjoys much faster training/generation than a previous autoregressive model ARTree (Xie & Zhang 2023), while maintaining the approximation performance.
Extensive and practical experiments demonstrate the robust representation ability and generative modeling efficiency of PhyloVAE for phylogenetic tree topologies.
BACKGROUND
Phylogenetic trees The common structure for describ- ing evolutionary history is a phylogenetic tree, which con- sists of a tree topology and the associated non-negative edge lengths on т. The tree topology T is a bifurcating tree graph (V, E), where V and E represent the sets of nodes and edges, respectively. Tree topologies can be either rooted unrooted. While our focus in this paper is on unrooted tree topologies, our proposed method can casily accommodate rooted ones as well. We will refer to unrooted tree topologies as "tree topologies" unless spec- ified otherwise. In tree topology т, the edges in E are undirected, and the nodes in V can have a degree of either 3 or 1. Degree nodes are internal nodes representing unobserved ancestor species, while degree nodes are leaf nodes representing observed species labelled with corresponding species names. An edge connecting a leaf node to an internal node is called a pendant edge. The goal of phylogenetic inference is to reconstruct the underlying phylogenetic trees from the sequence data (c.g., DNA, RNA, protein) of observed species (Felsenstein) 2004). following various criteria such as maximum likelihood (Felsenstein) 1981), maximum parsimony (Fitch 1971). and Bayesian approaches (Yang & Rannala 1997; Mau et al. 1999 Larget & Simon 1999; Ronquist et al. 2012). With leaf nodes, the number of possible tree topologies explodes combinatorially ((2N 5)!),, posing significant challenges to phylogenetic inference and related representation learning tasks. Further details on phylogenetic models can be found in Appendix A
edge
internal
leat node
Figure I: An example of a tree topology with six leaf nodes. The labels of leaf nodes are (A..BC.D.E.F).
2

---

Published as a conference paper at ICLR 2025
ARTree for tree topology modeling In ARTree (Xie & Zhang 2023), autoregressive prob- abilistic model for tree topologies. a tree topology is decomposed into a sequence of leaf node addition actions, and the associated conditional probabilities are modeled using learnable topolog- ical features via graph neural networks (GNNs) (Zhang 2023). The corresponding tree topology generating process be described as follows. Let & (f,.-..::0 :.N. denote the set of leaf nodes with pre-defined order. The generating process starts from the simplest and unique tree topology 73 (V3, Ea)) that contains the first three leaf nodes , 22; 23, and keeps adding new leaf node as follows. Assume an intermediate tree topology Tn (V,, Ex) with the first n < N leaf nodes in X, termed an ordinal tree topology of rank n as defined in Xie & Zhang (2023). Then, an edge en E En is selected according to conditional distribution Qee computed by GNNs (een is the set of previously selected edges), and Tn then extended Tn+11 1 by attaching the next leaf node xn+ to en. This process is repeated until all leaf nodes in X' are attached. This way, a tree topology
TN is uniquely transformed into an edge decision sequence D (E3..... eN-1). .. and the corresponding LRTree-based probability is Q(T) Q(D) I ((en/ <<n) where e<3 = 0. While effective, the repetitive computation of topological node embeddings and multi-round message passing steps in GNNs also add to the computational cost of ARTree. More details are deferred to Appendix B
Variational autoencoder The variational autoencoder (VAE) (Kingma & Welling, 2014 assumes generative model pe (y, =) = pe (3/ z p((), where = E R is a latent variable with prior distribution p(z), and an inference model qø z (y) an approximation for the intractable posterior (=(y))).
Given an observed data set {y1..... MM ). the generative model and inference model can be jointly learned by maximizing the following evidence lower bound (ELBO)
Polyy 2) g (z(g))
L(y: O, ②) = (z log
= log (oy) - DKL
S log (y (I)
for all data points (y, 1 ...., M). Here, (o Sex Po po z z)dz  is the marginal likelihood of y and DKL is the Kullback-Leiiblr (KI.) divergence. To remedy the approximation of the latent variable posterior and achieve more flexible generative model, the importance weighted autoencoder (IWAE) (Burda et al., 2016) instead uses the multi-sample lower bound
1 K
(o z²)
LK (y:: 0, x) Bgo(=" [p) (a(x" log
S log (z))
2 (=
The equalities in equation and 2 hold if and only if go(2))) () = po (zly) While VAEs have been effectively used in graph representation learning (Kipf & Welling, 2016 Simonovsky & Komodakis 2018; Winter et al., 202T, Zahirnia et al. 2022), they typically require transforming graphs into numerical encodings, such as adjacency matrices, and then working in the encoding space. However, the specific bifurcating structure of phylogenetic tree topologies imposes unique constraints on these adjacency matrices, posing challenges for applying these methods directly.
3 PROPOSED METHOD
In this section, we introduce phylogenetic variational autoencoders (PhyloVARs), an unsupervised learning framework designed specifically for phylogenetic tree topologies. We begin with concise overview of the problem setting and the fundamental components of PhyloVAE in Section 3.1 We then develop an encoding mechanism that bijectively maps tree topologies encoding vectors in Section 3.2 Finally, Section 3.3 elucidates how to utilize this encoding mechanism and learnable topological features to establish the generative and inference models within the Phylo VAE framework. We emphasize that the input to our algorithms is collection of phylogenetic tree topologies F Our goal is to build a probabilistic model and to learn useful representations of this collection, not infer those trees directly from sequence data, which is separate and intensively studied problem.
"These collections often come from phylogenetic analysis softwares such as MrBayes (Ronquist et al. 2012 and BEAST (Drummond & Rambaut 2007). hut they can also come from observations, open-source data sets, and other biological softwares, as long as they of scientific interest.
3

---

Published as a conference paper at ICLR 2025
Decomposition loop
edge
e M leaf node
Reconstruction loop
record remove /
locate attach
8 encoding
Figure 2: The decomposition loop and reconstruction loop for encoding the tree topology with leaf nodes X' (A.B.C.D.E.F) in Figure I Starting from the tree topology in the upper left, remove the pendant edges fa. fss ʃ4 (associated with the leaf nodes F, E, and D) sequentially, and record edge decision <5; 64, 63. Then starting from the three-leaf tree topology in the lower right, we add back fa. fs. fo and index these nodes (except for the root) sequentially. The resulting encoding vector is (3, 7, 7), which are the indexes associated with e3, E4, eg.
3.1 PHYLOVAE: AN OVERVIEW
Let = ((T', ) be a collection of tree topologie where wi the weight for the tree topology r' and Ei w² = 1. All tree topologies in T have the same leaf nodes X' {I Z2, ................................ IN with a pre-selected order. For example, T can a sample of tree topologies produced by some phylogenetic inference software, such as from a posterior sample (Ronquist et al., 2012) [Suchard al.) 2018) or the bootstrap (Felsenstein) [985] Minh et al., (013)), where the weight can equal to the frequency of a tree topology among these samples. Given the observed data set T, the data distribution is defined as Pdata(T)
Σ i=1 w (T: where 6 is a Kronecker delta function that satisfies (T) = 1 if 7 = T¹ and o- (T) = 0 elsewhere.
Similar to standard VAEs, Phy consists of a generative model and an inference model. Let Rd be latent variable with prior distribution p(z) and po(T)z) be a probabilistic model that defines the probability of generating 7 conditioned on the latent variable z. The marginal probability of is given by po(r)= JRa Po ( )z2)dd... The prior distribution p(z) is required to be analytic and easy to sample from. In this paper, we will use a standard Gaussian prior distribution, i.e., p(z) = (22; 0g, Ia). With an inference model (2(r) that approximates the posterior pe pe(2) r). the multi-sample lower bound on T takes the form
K po (2(1TT
1
Φ) = E Egg (xt))))))) (o 7) log
Σ i=1
S log (T)).
(3)
K
which reduces to the standard ELBO for log po (T) when K = The overall multi-sample lower E (r) 0, o), which serves as the objective function
bound on T is defined as LK ( for training PAY Unlike standard VAEs, the discrete nature t makes challenging to construct the generative model and inference model. In what follows, we describe how this is done using an encoding mechanism and learnable topological features respectively.
3.2
AN ENCODING MECHANISM FOR TREE TOPOLOGIES
Recall that in the generating process of ARTree, there exists map between tree topology and the edge decision sequence D = (ez, ... en-1) (see Section 2 and Appendix B for more details). We can further encode this sequence into an integer-valued vector via the following procedure.
Decomposition loop tree topology of rank N, starting from the last added node xy to the first added node x,, Suppose now we have Jn (Vn, E,) in hand, pendant edge fr. (I)) is firstly removed, where t, is
In this loop, we repetitively remove the leaf nodes from TN r. an ordinal
"The weights formed by merging replicate tree topologies the data set for compact form of T.
4

---

the internal node connected to Tn. resulting in an ordinal tree topology of rank n 1, Tn-1- Let the two neighbors of i, in Ty be Mn-1 and Un-1 (in addition to T,)). The edge cn-1 = (un-1: Un-1), 1), therefore, is the corresponding edge decision for Fn Tn-1 which we for the reconstruction Published as a conference paper at ICLR 2025
Algorithm 1: A linear-time algorithm for tree topology encoding
Input: tree topology т with N leaf nodes.
Output: An encoding vector s(T) = (83, 8N-1) E NN-3 corresponding to
TN 6 T;
for = -N,..... ...,, 4 do
Remove the pendant edge fn = (fn, Fn) from Tn and obtain Tn-1i
Record the edge decision Gn- (Un-1, Un-1) Tn-15
end
Index(#11 +- 1; [22) + Index((3) < 3; f- the unique internal node of r3! .N- ......N
for
do
Attach the pendant edge fu+1 to en (assume En, = (un, 2n)) is directed away from r) and
obtain Tn+1;
Index(tn+1) f- N + - 2; Index(xn+1) f- n2 1; 8n f- Indox(U,)):
end
loop. This procedure continues until a tree topology T3 with the first three leaf nodes is reached. See the top row in Figure 2 for an illustration.
Reconstruction loop In this loop, we start from т3 and reconstruct TN by adding back the leaf nodes from 2g to TN. At the beginning, we index the leaf nodes on T3 by setting Index(x1) 1, Index((12) 2, Index((3))
3, and denote the unique internal node in 73 as r. Suppose now we have an ordinal tree topology Tn (V,, F,,) of rank n. We first locate the edge decision en (U,, Un) on Tn (without loss of generality, can be identified via U, on Tv)) and then attach the pendant edge fo+1 = (I++1,n+1) 1) to en. Finally, and Index(2xxt)
is assumed to be directed away from In+1)
that it
we set Index(1241)
N + Tx+1 = (Vn+1, Ex+1) is constructed and all the nodes except r in Vn are indexed. This process ends upon the full reconstruction of TX. See the bottom row in Figure foor illustration.
+ 1. In this way, the next tree topology
After the above two loops, the encoding vector S()) for tree topology T takes the form
sT) [$3, 84, 8n 13' [Index(uz), Indexxu)). IIndex(r..
Nm-3
This encoding mechanism summarized in Algorithm | Theorem | (proff in Appendix C.I shows that this mechanism has linear time complexity, which is crucial to the efficient training of PhyloVAE
Theorem Given a tree topology T with N leaf nodes, the time complexity of computing its encoding vector S(T) is O(N).
Conversely, when given an encoding vector s, one can simply follow the reconstruction loop to obtain the corresponding tree topology T in linear time (deduced from Theorem Tp. This enables the fast
generation of samples from PhyloVAE (details are deferred to Appendix C.2) We note that a similar
encoding strategy has been proposed in Phylo2Vec (Penn et al., (2024). However, their approach has quadratic time complexity for vector encoding of tree topologies with unlabelled internal nodes. Our method achieves faster processing by employing smart indexing strategy, eliminating the need for repetitive relabelling of edges during the reconstruction loop.
3.3 GENERATIVE MODEL AND INFERENCE MODEL IN PHYLOVAR
Generative model The encoding mechanism in Section 3.2 allows us to define pT(T)z) through p=(s(T)(z). where S(T) (sa...., 5N-1) is the encoding vector for Similar to the diagonal Gaussian distribution used in standard VAEs, we assume the elements in s(T) are conditionally independent given z, i.e.,
N
pT(T|z)= (o(()T) = II po (s,
(5)
3
5

---

Published as a conference paper at ICLR 2025
The factorized form of equation experiments show that this non non-autoregressive structure substantially reduces the computational cost compared to autoregressive models such ARTree.
5 enables parallel computation of the probabilities po Our
Let S, == fi N: : S n n + +1515 N + n - 3] be the set of allowable indices for the edges (Index(U,) Cn (uh,; Un) E E.). For all < N, po( takes the form
(e Sh z) Discrete (softmax(m,, 3 (z))),,,
(6a)
(z) (d)(2), ... ( MLP enc ( RN
(6b)
where MLP ene is multi-layer perceptron, @ is elementwise multiplication, and m, is defined as
i Sn: (mn 3 a) = oc, elsewhere.
This mask vector is introduced to ensure that the generated encoding vector is always valid for representing tree topology.
Inference model The inference model (0 is built on top of learnable topological features as follows. Firstly, we compute the topological node embeddings {f u V} for = (V, E) by minimizing the following Dirichlet energy E(r) [(, [l foll using the efficient
two-pass algorithm described in Zhang (2033) These topological node embeddings then fed into
a gated message-passing block et al. 2015) implemented as
f GRU
f vEN(1))
W mag
f UEV,
(8)
where (uu) is the neighborhood of u, W mag is a learnable message matrix that aggregate the information from Nu)). and GRU is a gated recurrent unit (Cho et al. 2014). After L rounds of message passing, the graph embedding vector f- is computed by sum-pooling layer, i.e.,
fr = Evev ft. Finally, we use a diagonal normal distribution for the conditional distribution of the latent variable whose mean and standard deviation are defined based on f, follows
2 (z(2))) N (z. diag(o?)) μ,, MLPP, log = MLP,(f+),
(9)
where MLP,, I and MLP, are two multi-layer perceptrons, and & are the learnable parameters. The mean of the inference model, f4, E _dd is a deterministic oow-dimeniiona representation of T, and will show its representation power our experiments. Although this inference model built on top of Zhang 2223). Zhang 2023) only provides deep model architecture for extracting graph features and does not use for representation learning. This representation also induces a pairwise distance between tree topologies. For two tree topologies TI, 72, we define the [ distance between them as D1. (T, T2) = || Mrg |ral where || lpp is the p-norm in the Fuclidean space. Note that the generative model also allows us to map an arbitrary point in the latent space to tree topology, which is impossible for previous representation methods.
Optimization Thanks the Gaussian inference model, the gradient Vo (T; can be derived using the reparameterization trick (Kingma & Welling 2014) as follows
K PO(T, Hr to to, OE')
Ve,0L((T;0,0) = Eg ....... (-0,1) Ve. log
(10)
K
go (Mr + or OeIlT)
==1
During training, parameters the generative model and inference model are updated along the gradi- ent direction Ve.oL Φ) = Exun(r) Vo (T; 0 Φ), using Monte Carlo gradient estimators.
4 RELATED WORKS
For harnessing the latent-variable structure to accelerate autoregressive models, Gu al. (2018) proposed non-aue autoregressvvve machine translation by defining a factorizable distribution for the output sequence conditioned on the input sequence and latent fertility variable. Kaiser et al. (2018) extended

---

Published as a conference paper at ICLR 2025
Five-leaf tree topologics
Fight lead free topologies
Eight-leaf tree topologios
1.00
0.57
0.55
2G
B2 de--iaeee
0.50
0.87
peak pe4l
PDD 70--0
boc
post 10
-2-
otter
0.00
0:25
0.50
0.25
1.00
55 dianne
Figure 3: Performance of Phy for structural representation on simulated data sets. Left: A visualization of the 2D latent manifold for the data set of five-leaf tree topologies. o(-) refers the cumulative density function of the -me-dimenioonal standard Gaussian distribution. Different colors represent the first edge decision and different transparencies of each color represent the second edge decision. Middle: Representation vectors of all the eight-leaf tree topologies. The scatter size is proportional to the probability of the corresponding tree topology. Right: Pairwise scatter plot, linear regression, and Pearson correlation coefficients between different distance metrics of tree topologies. L Euclidean distance in PhyloVAE latent space, RF= Mobinson-Fouldee PD = Path difference.
this to discrete latent variables that summarize the input information. This approach was also integrated with normalizing flows by Ma et al. (2019).
Previous VAE frameworks for graph representation learning often encode a graph to its adjacency matrix and then define the generative models for matrices (Kipf & Welling 2016: Simonovsky & Komodakis 2018 8 Winter et al. 2021 Zahirnia et al.) 2022). However, the bifurcating structure and the unlabelled internal nodes tree topologies put special constraints on adjacency matrices, which may hinder the application of previous works to representation learning of phylogenetic trees.
The most popular means of learning an embedding of collection of phylogenetic trees is to calculate pairwise distances in some way and project to Euclidean space using multidimensional scaling (Amenta Klingner 2002 Hillis et al. 20055) Jombart al., 2017). More recently, Penn al. (2024) proposed an encoding strategy that relies tree topology branching patterns, with representation dimensions scaling according to tree size.
Some previous works integrated trees with VAEs (Shin et al. 2017 Vikram et al. 2018 Manduchi ctt al. 2023). However, they all consider a tree-shaped prior distribution or hierarchical latent variable structure for enhanced interpretability and generative quality. These papers do not consider modeling any graph or tree objects and thus are clearly distinct from our PhyloVAE.
5 EXPERIMENTS
In this section, we evaluate the performance of VAY for structural representation on simulated data sets (Section 5. and real phylogenies (Section 5.2) and generative modeling on benchmark data sets (Section 5.3) We set the latent dimension d 2 for better visualization of the latent representations in Section [.T [5.T and 5.. For all experiments, the number of particles set to = 32 and the inference model employs L 2 rounds of message passing. Leaf nodes are ordered lexicographically based on the corresponding species names. We designed our experiments with the goals of (i) verifying the effectiveness of PhyloVAE for representation learning of tree topologies and (ii) examining the generative modeling performance of Phylo with an emphasis on the generation speed. Results are gathered after 200,000 iterations with a batch size of 10. Further details can be found in Appendix D Our code is released at https::///ggtt thubb wtwp:hhww.oooooooooooooooccc.ooh tttuieyuxieeee w/w.ii/ehhhylllto
5.1 REPRESENTATION LEARNING ON SIMULATED DATA SETS
Five-leaf tree topologies In this experiment, the training set consists of all the 15 tree topologies with five leaves and a randomly generated weight vector Dirichlet(31) E R15 for these tree
7

---

Published as a conference paper at ICLR 2025
Maramal gene trees (sogien - 1000)
Mananal gene trecs (seçlen 2000)
BEAST trees of rabies phylogenios
mine ADOMAA
uaur AMP ASDMAA
settkc Spoeed mna
KiBINN 223 procad
PhyloVAR
0
XO0 un inthrola
MDS plot
ADCHAS
BIBI
ADORAS
2F3 procad touth
ZERSA
govan 1883
Figure 4: Performances of Phy and MDS plot for representing real phylogenies. Left/Middle: Latent representations of the posterior mammal gene trees for five genes with different lengths. The scatter size is proportional to the probability of the tree topology. Right: Latent representations of samples of tree topologies from five independent BEAST runs on the rabies data set.
topologies with B = 0.75. Figure 3 (left) depicts the tree topologies as partition representation (see Figure in Appendix D for an illustration), generated by a trained PhyloVAE conditioned on the Gaussian quantiles. To ease presentation, we choose the argmax index of the multinomial probability in equation (6a) instead of randomly sampling. We see that the generated tree topology exhibits nice continuity the latent variable varies.
Eight-leaf tree topologies We begin by constructing the training set, which consists of a mixture of three peaked distributions comprising all 10,395 tree topologies with eight leaves. Each peaked distribution is derived as follows: we select ground truth tree topology and simulate DNA sequences of length 500 for the eight leaf nodes using the Jukes-Cantor (JC) substitution model (Jukes et al., 1969) with the edge lengths fixed at 1. We then compute the posterior distribution of tree topologies given the simulated DNA sequences with uniform prior over tree topologies and the edge lengths fixed at 1. This posterior distribution is used the peaked distribution as it concentrates around the selected ground truth tree topology. The same procedure is repeated three times (see Figure [[a) 8(b) B for the selected ground truth tree topologies), leading to three peaked distributions which then evenly mixed to form the training data set.
The latent representations of eight-leaf tree topologies visualized in Figure where tree topologies on different peaks are clearly separated, demonstrating the effectiveness of PhyloVAL for representation learning. Following Kendall & Colijn (2016), we compare L distance Robinson- Foulds (RF) distance (Robinson & Foulds] 1981) and path difference (PD) distance (Steel & Penny 1993) in Figure 3 (right), where L distance shows a positive correlation with RF/PD distance.
5.2
REPRESENTATION LEARNING ON REAL PHYLOGENIES
Gene trees and sequence lengths In this experiment, Phy is employed to analyze phy- logenetic inference results obtained with different genes (different genes evolve under different evolutionary models) and sequence lengths. Following Hillis et al. (2005), we select five genes and the ground truth phylogenetic tree (Figure 9 44 leaves) from the carly placental mammal evolution analysis in Murphy et al. (2001). For each gene, simulate the DNA sequences with fixed
8

---

Published as a conference paper at ICLR 2025
Table 1: KL divergences the ground truth of different methods across eight benchmark data sets. VAE (d)" means Phylo VAE with latent dimension d. The "# Training set" and "# Ground truth" columns show the number of unique tree topologies in the training sets and ground truth respectively. The results are averaged over 10 replicate training sets. The tree topology probability of Phy is estimated using importance sampling with 1000 particles. The results of SBN-EM, SBN-EM-u are from Zhang & Matsen IV (2018), and the results of ARTree are from Xie & Zhang (2023). For each data set, the best result is marked in black bold font and the second best result is marked in brown bold font.
KL divergence ground truth
Sequence set
# Leaves
# Training set
Ground truth
SBN-EM
SBN-EM-o
ARTrec
PhyloVAB (2)
PhyloVAE (10)
DSI
27
1228
2784
0.0136
0.0130
0.0045
0.0273
0.0189
DS2
7
42
0.0199
0.0128
0.0097
0.0100
0,0098
DS3
36
43
351
0.1243
0.0882
0,05488
0.0529
0.0477
828
11,005
0.0763
0.0637
0,0299
0.0619
0,0469
DS5
50
33752
1516877
0.8599
0.8218
0.6266
0.7985
0.5744
DS6
35407
809765
0.3016
0.2786
0.2360
0.3241
0.2207
DS7
59
1125
11525
0.0483
0.0399
0.0191
0.0591
0,0370
DS8
64
3067
82162
0.1415
0.1236
0.0741
0.1372
0.1061
length along the ground truth tree using the corresponding evolutionary model, run MrBayes chain (Ronquist et al.] 2012) for one million iterations, and sample per 100 iterations in the last 100,000 iterations, to gather the posterior samples, done in Hillis et al. (2005). These one million iterations are enough for the MrBayes run to converge. These 5,000 tree topologies with uniform weights constitute the training set of PhyloVAE.
The upper left and upper middle plots of Figure 4 depict the learned latent representations of the posterior tree topologies for different genes obtained by PhyloVAE. We see that different groups are clearly separated. With a sequence length of 1000, the inferred posteriors from ADORA3 and APP are close the ground truth tree topology, while those from IRBP, mtRNA, and ZFX show deviations. When the sequence length is increased to 2000, the inferred posterior from ADORA3 becomes more concentrated around the ground truth tree topology, while the inferred posteriors from the other genes remain diffuse. Figure 10 in Appendix E. shows that all five genes discover the ground truth tree topology with a sequence length of 5000. For baseline visualization methods, the lower left and lower show show the multidimensional scaling (MDS) plots (Hamer & Young 2013) of
middle plots of Figure 4 mammal gene trees, where we find that different groups tend to concentrate towards the origins and overlap with each other, while our Phylo provides more clear representations of different groups.
Multiple phylogenetic analyses comparison In this experiment, we use PhyloVAE to compare multiple phylogenetic analyses and assess convergence. The sequence alignment under consideration comprises 290 rabies genomes (Viana et al. (023). We conduct 5 independent BEAST (Suchard jet al. 2018) analyses for 400 million iterations, and sample every 100,000 iterations in the last 200 million iterations. Afterward, the rooted posterior tree topologies sampled by BEAST are unrooted. The resulting 10,000 tree topologies with uniform weights form the training set of PhyLoVAE.
In Figure 4 (right), tree topologies from five independent BEAST runs form five non-overlapping groups with 2-3 sub-groups within each group, indicating the divergence of these BEAST runs. No- tably, the maximum clade credibility (MCC) tree from each run resides within the correct high-density region in the latent space. An example showcasing the convergence of independent phylogenetic analyses is provided in Appendix E.2
5.3 GENERATIVE MODELING ON BANCHMARK DATA SETS
Finally, we assess the generative modeling performance of PhyloVAE on eight benchmark sequence
sets, DS1-8, which contain biological sequences from 27 to 64 eukaryote species and are commonly considered for benchmarking tree topology density estimation and Bayesian phylogenetic inference
tasks in previous works (Zhang & Matsen IV 2018 2019 2024 Zhang 2020 Mimori & Hamadal
2023 Zhou et al. 2023 Xie & Zhang 2023 Xie et al. 2024a b Molén et al.) 2024 Hotti et al.
2024 These eight data sets cover comprehensive posterior patterns (Whidden & Matsen IV (015)
and are considered good representative cases. The training sets and ground truths for PhyloVAE are obtained the same way as in Zhang & Matsen IV (2018) (see more details in Appendix D.5). We consider SBN-EM, SBN-EM-0, and ARTree as baselines, and use the same experimental settings in

---

Published as a conference paper at ICLR 2025
the original papers (Zhang & Matsen IV 2018 Xie & Zhang 2023 For a fair comparison, uses the same batch size and number of iterations for Phylo VAE as in ARTree.
Table reports the approximation accuracies measured by KL divergence obtained by different methods. We see that PhyloVAE with latent dimension of 10 achieves compa- rable (if not better) results to ARTree. Although Phylo with latent dimension of 2 may have reduced capacity, we
25
ARfTiee (txaining)
Rontian (seconds 2 2.AGHH CPU)
AftIree (geueration)
30
PAyloVAE (training)
PhyloVAR (generation)
find it works fairly well in DS1-8, validating the reliability of
the awo-dimensional representations Sections 5.1 and 5.2 Figure [ shows the runtime comparison between AR
and Phy (d = 10) with K 32 particles. We see that both the training time and generation time of PhyloVAF. are
significantly reduced compared to ARTree, even though mul- tiple particles are used in the multi-sample lower bound (3) This is due to the non-autoregressive nature of PhyIoVAES. inherited from VAEs. Note that Phylo VAE achieves these while providing useful high-resolution representations of phylogenetic tree topologies (see Figure 14] in Appendix which is impossible for the other baseline methods that are merely designed for tree density estimation.
D151
Figure 5: Runtime comparison be- tween ARTree and PhyloVAE (d 10) with K 32 partieles. Training means 10 training iterations. Genera- tion means generating 100 topolo- gies.
CONCLUSION
This paper proposes Phy an unsupervised learning framework designed for representation learning and generative modeling of phylogenetic tree topologies. By incorporating an efficient encoding mechanism of tree topologies and leveraging latent-variable architecture, PhyloVAE facilitates fast training and generation using non-autoregrressvee generative models. The learned inference model also provides informative, high-resolution representations for tree topologies, as demonstrated in our experiments. Phylo VAE thus offers a latent space representation that can be casily visualized like the previous MDS method, while also functioning as probabilistic model that provides high-resolution view of tree topology distributions. Further applications of PhyloVAB to practical tasks phylogenetics (e.g., phylogenetic placement, Bayesian phylogenetic inference, etc.) and extending PhyloVAE for tree topologies with different leaves would be interesting future directions.
ACKNOWLEDGEMENTS
This work was supported by National Natural Science Foundation of China (grant 12201014, grant no. 12292980 and grant no. 12292983), as well as National Institutes of Health grant AI16261 1. The research of Cheng Zhang was support in part by National Engineering Laboratory for Big Data Analysis and Applications. the Key Laboratory of Mathematics and Its Applications (LMAM) and the Key Laboratory of Mathematical Economics and Quantitative Finance (LMEQF) of Peking University. Frederick Matsen is an investigator of the Howard Hughes Medical Institute. The authors appreciate Dave Rich for carefully validating the codebase. The authors are grateful for the computational resources provided by the High-performance Computing Platform of Peking University. The authors appreciate the anonymous ICLR reviewers for their constructive feedback.
REFERENCES
N. Amenta and J. Klingner. Case study: visualizing sets of evolutionary trees. IEEE Symposium on Information Visualization, 2002. INFOVIS 2002., pp. 71-74, 2002. doi: 10.1 10.1109/NFV 1173150.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In International Conference Learning Representations, 2016.
10

---

Published as a conference paper at ICLR 2025
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint vrXiv:1406 1008.078, 2014.
Alexei Drummond and Andrew Rambaut, BEAST: Bayesian evolutionary analysis by sampling trees. BMC Evolutionary Biology, 7:1-8, 2007.
Joseph Felsenstein. Evolutionary trees from DNA sequences: A maximum likelihood approach. Journal of Molecular Evolution, 17:268-276. 1981.
Joseph Felsenstein. Confidence limits on phylogenies: an approach using the bootstrap. Evolution, 39(4):783-791, 1985.
Joseph Felsenstein. Inferring Phylogenies. Sinauer associates, edition, 2004.
Walter M Fitch. Toward defining the course of evolution: minimum change for specific tree topology. Systematic Biology, 20(4))406-44) 4166 1971.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. In International Conference Learning Representations, 2018.
Robert Hamer and Forrest Young. Multidimensional scaling: History, theory, and applications. Psychology Press, 2013.
David M Hillis, Tracy A Heath, and Katherine St. John. Analysis and visualization of tree space. Systematic Biology, 54(3):471 (74244447-882 2005.
Alexandra Hotti, Oskar Kviman, Ricky Molén, Victor Elvira, and Jens Lagergren. Efficient mixture learning in black-box variational inference. In Forty-first International Conference on Machine Learning, 2024.
Thibaut Jombart, Michelle Kendall, Jacob Almagro-Garcia, and Caroline Colijn. TREESPACE: Statistical exploration landscapes of phylogenetic trees. Mol. Ecol. Resour, 17(6):1385- 1392, November 2017. ISSN 1755-098X, -0988. doi: 000111755-0998.22600 URL http://dx doi. org.lo.rr11...0
/10 1wt.e11111111111 0998. 12676
Thomas H Jukes, Charles R Cantor, et al. Evolution of protein molecules. Manmalian protein metabolism, 3:21-132, 1969.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Conference on Machine Learning, pp. 2390-2399, PMLR, 2018.
Michelle Kendall and Caroline Colijn. Mapping phylogenetic trees reveal distinct patterns of evolution. Molecular Biology and Evolution, 33(10):2735- 274--233,, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. International Conference on Learning Representations, 2014.
Thomas Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arX1v.1611.07308. 2016.
Mary K Kuhner and Jon Yamato. Practical performance of comparison metrics. Systematic Biology, 6412):205-214, 2015.
Bret Larget. The estimation of tree posterior probabilities using conditional clade probability distributions. Systematic Biology, 62(4):501-511, July 2013. ISSN 1063-5157. doi: 10.1093) sysbio/syt011. URL http::tp://eee doi org.10 http:////....1////bb////////////
Bret R. Larget and D. L. Simon. Markov chain Monte Carlo algorithms for the Bayesian analysis of phylogenetic trees. Molecular Biology and Evolution, 16:750-750, 1999.

---

Published as a conference paper at ICLR 2025
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks, arXiv preprint 1rXi::1/1100493, 2015.
Pengyu Liu. A tree distinguishing polynomial. Discrete Applied Mathematics, 288:1-8, 8, 2021.
Xuezhe Ma. Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. Flowseq: Non-autoregressive conditional sequence generation with generative flow. arXiv preprint arXiv:1909.024800 2019.
Laura Manduchi, Moritz Vandenhirtz, Alain Ryser, and Julia Vogt. Tree variational autoencoders. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Bob Mau, Michael A Newton, and Bret Larget. Bayesian phylogenetic inference via Markov chain Monte Carlo methods. Biometrics, 55(1):1-12, 1999.
Takahiro Mimori and Michiaki Hamada. Geophy: Differentiable phylogenetic inference via geometric gradients of tree topologies. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Bui Quang Minh, Minh Anh Nguyen, and Arndt von Haeseler. Ultrafast approximation for phyloge- netic bootstrap. Molecular Biology and Evolution, 30:1 18:1 1195, 2013.
Ricky Molén, Oskar Kviman, and Jens Lagergren. Improved variational bayesian phylogenetic inference using mixtures. Transactions on Machine Learning Research, 2024. ISSN 2835-8856
William J. Murphy, Eduardo Eizirik, Stephen J. O'Brien, Ole Madsen, Mark Scally, Christophe J. Douady, Emma Teeling, Oliver A. Ryder, Michael J. Stanhope, Wilfried W. de Jong, and Mark Springer. Resolution of the early placental mammal radiation using Bayesian phylogenetics. Science, 294(5550):2348- 2351, 2001. doi: 10. http//cienne.1011177.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, nigh-performang deep learning library. In Neural Information Processing Systems, 2019.
Matthew Penn, Neil Scheidwasser, Mark P Khurana, David A Duchêne, Christl A Donnelly, and Samir Bhatt. Phylo2 Vec: A vector representation for binary trees. Systematic Biology, pp. syae030, 06 2024. ISSN 1063-5157.
doi: 10. 10.1009//ssioo/yae0000. URL htttps://doi.org///
David F Robinson and Leslic R Foulds. Comparison of phylogenetic trees. Mathematical Biosciences, 13(1-2):131-147 1981.
Fredrik Ronquist, Maxim Teslenko, Paul Van Der Mark, Daniel L Ayres, Aaron Darling, Sebastian Höhna, Bret Larget, Liang Liu, Marc Suchard, and John Huelsenbeck. MrBayes 3.2: Efficient Bayesian phylogenetic inference and model choice across a large model space. Systematic Biology, 61(3):539-544 2012.
Richard Shin, Alexander A. Alemi, Geoffrey Irving, and Oriol Vinyals. Tree-structured variational autoencoder, 2017. /tLL/tttss///ppereeeee http://wwwweeeeooor//oo////////o
Martin Simonovsky and Nikos Komodakis. GraphVAE: Towards generation of small graphs using variational autoencoders. In International Conference Artificial Neural Networks, pp. 412-422, 2018.
Mike A Steel and David Penny. Distributions of tree comparison metrics-Some new results. Systematic Biology, 42(2): 1993.
Marc Suchard, Philippe Lemey, Guy Baele, Daniel L Ayres, Alexei J Drummond, and Andrew Rambaut. Bayesian phylogenetic and phylodynamic data integration using BEAST 1.10. Virus Evol, 4(1):vey01,, January 2018. ISSN 2057-1577. doi: 10.1093/ve/vey016. URL http:
7/dx.do: www.oooorg/1... 1093 wttp://0..00////evveee//l
12

---

Published as a conference paper at ICLR 2025
Mafalda Viana, Julio A Benavides, Alice Broos, Darey Ibañez Loayza, Ruby Niño, Jordan Bone, Ana da Silva Filipe, Richard Orton, William Valderrama Bazan, Jason Matthiopoulos, et al. Effects of culling vampire bats the spatial spread and spillover of rabies virus. Science Advances, 9(10): eadd7437, 2023.
Sharad Vikram, Matthew Hoffman, and Matthew J. Johnson. The loracs prior for vaes: Letting the trees speak for the data. In International Conference on Artificial Intelligence and Statistics, 2018.
Chris Whidden and Frederick A Matsen IV. Quantifying MCMC exploration of phylogenetic tree space. Systematic Biology, 499), May 2015. ISSN 1063-5157, 1076-836X. doi: 10.10933kysbiolsyy0006 URL http: doi 1ttp://1.110..00 htsp://///sssss////s/// syy01
Robin Winter, Frank Noé, and Djork-Amé Clevert. cermutation-invarian variational autoencoder for graph-level representation learning. Advances in Neural Information Processing Systems, 34: 9559-9573, 2021.
David F Wright. Bayesian estimation of fossil phylogenies and the evolution of early to middle Paleozoic crinoids (Echinodermata). Journal of Paleontology, 91(4):799-8144 2017.
Tianyu Xie and Cheng Zhang. ARTree: deep autoregressive model for phylogenetic inference. In seirty- Conference on Neural Information Processing Systems, 2023.
Tianyu Xie, Frederick A Matsen IV, Marc A Suchard, and Cheng Zhang. Variational bayesian phylo- genetic inference with semi-implicit branch length distributions. arXiv preprint arXiv:2408.05058 2024a.
Tianyu Xie, Musu Yuan, Minghua Deng, and Cheng Zhang. Improving tree probability estimation with stochastic optimization and variance reduction. Statistics and Computing, 34(6):186, 2024b.
Ziheng Yang and Bruce Rannala. Bayesian phylogenetic inference using DNA sequences: a Markov chain Monte Carlo method. Molecular Biology and Evolution, 14(7):717-724, 1997.
Kiarash Zahirnia, Oliver Schulte, Parmis Naddaf, and Ke Li. Micro and macro level graph modeling for graph variational auto-encoders. Advances in Neural Information Processing Systems, 35: 30347-30361, 2022.
Cheng Zhang. Improved variational Bayesian phylogenetic inference with normalizing flows. In Neural Information Processing Systems, 2020.
Cheng Zhang. Learnable topological features for phylogenetic inference via graph neural networks. In International Conference on Learning Representations, 2023.
Cheng Zhang and Frederick A Matsen IV. Generalizing tree probability estimation via Bayesian networks. Advances in Neural Information Processing Systems, 31, 2018.
Cheng Zhang and Frederick Matsen IV. Variational Bayesian phylogenetic inference. In Interna- tional Conference on Learning Representations, 2019.
Cheng Zhang and Frederick A Matsen IV. A variational approach to Bayesian phylogenetic infer- ence. Journal of Machine Learning Research, 25(145):1-56, 2024. URL http: http:tt////m/ooor papers/v25/22-0444
Mingyang Zhou, Zichao Yan, Elliot Layne, Nikolay Malkin, Dinghuai Zhang, Moksh Jain, Mathieu Blanchette, and Yoshua Bengio. PhyloGFN: Phylogenetic inference with generative flow networks. ArXiv, abs/2310.08774, 2023.
13

---

Published as a conference paper at ICLR 2025
A PHYLOGENETIC TREES AND PHYLOGENETIC INFERENCE
The common structure for describing evolutionary history is phylogenetic tree, which consists of bifurcating tree topology and the associated non-negative edge lengths q (see Figure 9 for real example). The tree topology I is a bifurcating tree graph (V, E), where and E are the set of nodes and edges respectively. We will be consistent with the main text and use "tree topology" for unrooted tree topology. For a tree topology т, the edges in E are undirected and the nodes in V can have either degrees or degree. Those with degrees are called internal nodes, representing unobserved ancestor species without labels, and those with 1 degree are called leaf nodes, representing the observed existing species with labels corresponding to the species names. An edge is called a pendant edge if it connects one Icaf node corresponding non-negative edge length qe, which quantifies the evolutionary intensity between two neighboring species. The set of edge lengths
internal node. For each edge E E, there exists
is given by q {ge E).
Each leaf node in V corresponds to a species with an observed biological sequence (e.g., DNA, RNA, protein). Let Y (Yi...., Ym2 E ΩN denote the observed sequences (with characters in Q) of M sites across N species. The transition probabilities of the characters along the edges of a phylogenetic 2004). With the common assumption that different sites evolve independently and identically, given phylogenetic tree (r, q). the likelihood of observing is
commonly modeled using continuous-time Markov chain (Felsenstein)
M
p(Y T, 9) ---III 1
(aa))
Pat,at (quv): (u,))EE
II
(11)
where a' ranges over all extensions of Y; to the internal nodes with a' being the character assignment of node (r represents an arbitrary internal node as the virtual root node). E the set of edges of que is the branch length of the edge (u, v) E E, (q) is the transition probability from character
through an edge of length q, and rj is the stationary distribution of the Markov chain. Parameters in equation (11) are called an evolutionary model. Assuming prior distribution P(T, q) on phy logenetic trees, Bayesian phylogenetic inference aims at properly estimating the posterior distribution
p(T, M| P(Y|T, q)p(T.9) p(Y|T, ()((T.)) PYY
(12)
There are several common phylogenetic analysis software for Bayesian phylogenetic inference such as MrBayes (Ronquist et al.) 2012), BEAST (Drummond Rambaut] 2007), etc., which implement Markov chain Monte Carlo (MCMC) runs to samples from the phylogenetic posterior.
B ARTREE
ARTrec (Xie & Zhang 2023 generates tree topology in autoregressive way. The sequential generating process in AReree facilitates probabilistic model over tree topologies which archives leading results in phylogenetic inference. We introduce the tree topology generating process of ARTree as below and most of the statements are adapted from Xie & Zhang (2023).
Let Tn (V.,, En)) be tree topology with n leaf nodes, where Vn, and E,, represent the sets of nodes and edges, respectively. A predetermined order, also known as the taxa order, is assumed for the leaf nodes X' (I1,--. ... [NN] We begin by providing the definition of ordinal tree topologies.
Definition 1 (Ordinal Tree Topology). Let X (x1,..., be set of N(N 2 2) leaf nodes. Let T, = (Vn, En) be a tree topology with n (n < N) leaf nodes in X. We say Tn is an ordinal tree topology of rank n, if its leaf nodes the first n. elements of X, i.e. V, n (s1,...,
The tree topology generation process begins with Ty (the unique ordinal tree topology of rank 3). At the n-th step, assume we have an ordinal tree topology Tn = =Vnn, steps are conducted to add the leaf node In+1 into i) Select an edge e,, = (u, E En, which is then removed from E,- ii) Add a new node w and two additional edges, (u, w) and (w, v), to the tree topology T. ii Insert the new leaf node 2n+1 and add additional edge (w, #n41) to the tree topology T,. Moreover, Xie & Zhang (2023) provides theoretical guarantees on the bijectiveness between the topology and the decision sequence D (ea, (ea...., -1- Thanks
rank n. The following
14

---

Published as a conference paper at ICLR 2025
Ordinal tree topologles of higher rank
21F3
ONN
Sampling
GNN
Sampling
GNN
Sampling
Leaf
Edge
Frobobillty vestor
vector
Addiigg node
Figure 6: An illustration of building tree topology autoregressive using ARTree. This figure is from Xie & Zhang (023).
Algorithm 2: ARTree for tree topology modeling (Xie & Zhang 2023)
Input: A set X' {ai1...., .NN of leaf nodes.
Output: An ordinal tree topology T of rank N; the ARTree probability Q(T) of T. (V3. E3) t- the unique ordinal tree topology of rank 3;
for n = . V
do
Let (u) C. (T.) + du where Tu is the parent of u;
Calculate the probability vector 9n E R Exll using the current GNN model; Sample an edge decision en from Discrete (4n) and assume en = (u, v);
Create
node wt;
En+1 + (En (e {uu, w), (w, v), In+1))
Vn+1 f Vn U {w, In41hi
Ta+1 + (Vn+1, Ent1);
end
TN: Q(7) + (a(es))((e)) ... AN qv-1 (eN1(eN-1).
for this bijectiveness, the distribution Q(T) tree topologies is modelled by Q(D) over decision sequences, i.e.
--1
Q(T) = Q(D) = II Q(enleen)
(13)
where Cen (es...., 1 and 6<3 generating process Figure and Algorithm
0.
We provide the illustration and algorithm for the
C DETAILS OF THE ENCODING MECHANISM
C.1 PROOF OF THEOREM 0
Theorem 1 Given a tree topology with N leaf nodes, the time complexity of computing its encoding vector ((T) is O(N).
Proof of Theorem Assume the tree topology (V, E) is stored in the binary tree data structure, where each node other than the root node also has parent node pointer (the root node is arbitrarily selected from the internal nodes for storage aim). Before computing the encoding т, we first perform postorder traversal over and construct index dictionary D of the mappings {(n, zn W U {(k, N
whose time complexity is O
In the n-indexed step of the decomposition loop, want remove the leaf node In., which can be located by indexing the dictionary D in O(1) time. Following the I, 's parent node pointer, we reach I, 's parent f,. We then detach 2,, t, from the tree and connect t,, 's remaining two neighbors un Un whose indexes are kj k2. Since all these local modifications of tree topologies can done in O(1) time, the time complexity of the decomposition loop is O(N).
15

---

Published as a conference paper at ICLR 2025
Algorithm 3: Converting encoding vectors to tree topologies
Input: A sequence = (83,-.. .... 8N-1) E NN-3
Output: tree topology with N leaf nodes.
T3 < the unique tree topology with the first three leaf nodes 22,
Index(x1) f 1; (x2) + 2; Index(x3) < 3; r +- the unique internal node of 73 do
for n .. N
On < the node in Tn that satisfies Index( = Sni
Un the neighbor node of un (towards the direction of r) in Tr
Create an node 1n
Attach the pendant edge (In+1 (n-11,2a+1) (un; Un): Index(tn+1) + N + +n 1; Index(Int1) t n + 1;
end
In the n-indexed step of the reconstruction loop, we have to locate the edge (um, un) to add the node Tn- This can be done in O(1) time by indexing ki A in D. Afterwards, de-connect Un Un and add t,s In to the graph. Since all these local modifications O11) time, the time complexity of the reconstruction loop O(N). Please note that the indexes in dictionary D are different from the Index(-) function.
tree topologies can be done in
Therefore, the time complexity of computing the encoding vector s(r) is O(N).
C.2 FROM ENCODINGS TO TREE TOPOLOGIES
The encoding mechanism also provides mapping from encodings to topologies, which is summarized in Algorithm 3 Algorithm 3 is similar to the reconstruction loop in Algorithm [ and has time complexity O(N).
D EXPERIMENTAL DETAILS
For all experiments, PhyloVAE is implemented in PyTorch (Paszke et al. 2019). The optimizer is Adam (Kingma & Ba 2015) with parameters (81, (2) (0.9, 0.999) and weight decay 0.0. The results are collected after 200000 iterations with batch size B 10.
D.1 FIVE-LEAF TREE TOPOLOGIES
The dimension of the latent space is set to d = 2. The generative model is a three-layer MLP with 512 hidden units and ResNet architecture. For the inference model, the number of message pass- ing rounds is = 2, and both MLP,, a and MLP, are composed of two-layer MLP with 100 hidden units. The number of particles in the multi-sample lower bound (3) is K = 32. The learning rate set to 0.0003 at the beginning and anneals according cosine schedule. The experiments are run on single 2.4 GHz CPU.
A
A _ D
E
Figure 7: An illustration of the partition presentation (left) of tree topology (right) contain- ing five leaves (A.B,C.DEE..
D.2 EIGHT-LEAF TREE TOPOLOGIES
The dimension of the latent space is set to d= 2. The generative model is three-layer MLP with 512 hidden units and ResNet architecture. For the inference model, the number of message passing rounds is L 2, and both MLP,, and MLP, are composed of two-layer MLP with 100 hidden units. The number particles the multi-sample lower bound (3) is 32. The learning rate is set to 0.0003 at the beginning and anneals according cosine schedule. The experiments on a single 2.4 GHz CPU.
The three pre-selected tree topologies for constructing the three peaked distributions are plotted in Figure 8 Here are definitions of the Robinson-Foulle (RF) distance and the path difference (PD) distance between two tree topologies (the edge lengths considered).
16

---

Published as a conference paper at ICLR 2025
A
D
F
H
G
B
D
0.714286
0.714286
0.714286
(a) The pre-selected tree topology (b) The pre pre-selected tree topology (c) The pre-selected tree topology first peak.
for second peak.
for the third peak.
Figure 8: Pre-selected tree topologies for constructing the three-peak distribution on the 10395 tree topologies with 8 leave nodes.
Table 2: The evolutionary model for each gene in the early mammal evolution analysis (Murphy et al. 2001 Hillis et al. 2005).
Base frequencies
Relative substitution rutes
Gene
Preferred model
Pinv
Alpha
C
G
AC
AG
AT
CG
CT
CG
ADORA3 APP
K2P
0.25
0.25
0.25
0.25
3
GTR [+T
0.25
0.24
0.18
0.33
1.6
3.66
0.47
0.72
2.65
1
0.78
ZFX
HKY + +I+
0.35
0.23
0.18
0.23
7.94
7.94
1
0.49
1.24
IRBP
GTR + I
0.21
0.3
0.3
0.18
1.5
4.91
1.34
0.83
5.8
1
0.18
1.04
miRNA
GTR +
0.34
0.2
0.21
0.25
5.86
14
3.85
0.58
29.3
1
0.41
0.53
Definition 2 (Robinson-Foudl Foulds distance; Robinson & Foulds (1981). For tree topology r (V, E) with leaf nodes X, an edge closeness between a leaf node and the righ//left terminal node of e. We define the unordered partition as Pa = (X,, and all the partitions on T as Pr {Pa distance between two tree topologies T1 and T2 is defined as
divides the leaf nodes into two parts (X1, X2), according to the
E}. The the Robinson-Foulee
D DRobinson- Foulds (T, T2) IPrAPEL [P +
Definition (Path difference distance; Steel & Penny 1993)). Let T be a tree topology with leaf nodes X. The path length 4 v, (T) between two leaf nodes Tr, Tj is defined as the minimum number of edges on for connecting 2; and xz.. The the path length difference distance between and T2 is
D ppoth-difference (T1,T2) T2) = Σ
Ven., (T2))
(T2)
ISiSJEN
D.3 GENE TREES AND SEQUENCE LENGTHS
The dimension of the latent space is set to d 2. The generative model is three-layer MLP with 512 hidden units and ResNer architecture. For the inference model, the number of message passing rounds 2, and both ML.P, and MT.P, are composed of two-layer MIP with 100 hidden units. The number of particles in the multi-sample lower bound 3 is K 32. The learning rate is set to 0.0003 at the beginning and anneals according to a cosine schedule. The experiments are run single 2.3 GHz CPU.
The evolutionary models the five genes of the early mammal evolutionary analysis are Table 2 For each gene, simulate the DNA sequences along the ground truth phylogenetic tree in Figure 9 We also give the Newick representation of this ground truth tree (from Hillis et al. (2005) for reproducing the results:
17

---

Published as a conference paper at ICLR 2025
Seth
Rednoder
Hedgehop
Mele
Flynng fox
Rounutta Fruibeet
Whaae
Usma
Horse
Cenform
Heuns
Flying Lemer
Terredd
Galeen Made
Stretlnn
Hyynx
Elsphast
Figure 9: The ground truth phylogenetic tree for early mammal evolution (Murphy et al. 2001
((0possum: 0. 0.072454, Diprotodontian: 0,061694):0,
((((Sloth: 0.056950, Anteater: (.061637):0.000966)
Armadillo:
((((Hedgehog: 0.137379,
Shrew: (11124147:0001111111
Mole: 0.086828): 0.011954,
(((hhyllostomid: 0.093178, Free tailed bat: 0.046665)
(False vampire bat: 0.062583, (Flying Fox: 0.018553,
Rousette Fruitbat:
(((() (Whale: 0.013788, Dolphin: 0.021978) Hippo:
(.039994):00004600 Ruminant: 0.073210): 0.008450 Pig:
0.067448):0.00588 Ilama: 0.061851):0.02773333 ((Horse:
0.043682, (Rhino: 0.028867,
Tapir: 00000030000000011000000000000000
((Cat: 0.046372, Caniform: 0.055840 Pangolli
0.0.595)) 0.001668:000001155::00022221:110
(((Sciurid: 0.083962, (Mouse: 0.042059, Rat:
0.045451):0.122018, (Hystricid: 0.074622, Caviomorph:
(Rabbit: 0.057873, Pika:
((Flying Lemur: 0.061380, Tree
Shrew: (stepeichee 0.076186, Human:
0.065099)
18

---

Published as a conference paper at ICLR 2025
((( (Tenrecid: 0.142758, Golden Mole: 0.067180): :0.009411 (Short
Eared Elephant Shrew: 0.039055, Long Eared Elephant Shrew:
0.036033) :0. :0:00888)::: 0.002240, Aardvark: (0005188)0000888000
(Sirenian:: 0. 0.038154, Hyrax: 0.089482) 002916, Elephant: 0. 0.050883) :0 014800111111111111 : 025967) :0.284326)
D.4 MULTIPLE PHYLOGENETIC ANALYSES COMPARISON
The dimension of the latent space is set to d = 2. The generative model is five-layer MLP with 512 hidden units and ResNet architecture. For the inference model, the number of message passing rounds is L = 2, and both MLP,, and MLP,, are composed of two-layer MLP with 512 hidden units. The number particles the multi-sample lower bound 33 is = 32. The learning rate is set to 0.00001 the beginning and anneals according to a cosine schedule. The experiments are run on a single 2.3 GHz CPU.
D.5 GENERATIVE MODELING CAPACITY ON BENCIMARK DATA SETS
Following Zhang & Matsen IV 2018); Xie & Zhang 2023) we construct the training data set and the ground truth follows: (i) for each sequence set, there are 10 replicate training sets of tree topologies which are gathered from 10 independent MrBayes runs until the runs have ASDSF (the standard convergence criteria used in MrBayes) less than 0.01 or a maximum of 100 million iterations (tree topologies are sampled every 100 iterations with the first 25% iterations discarded); (ii) for each sequence set, the ground truth of tree topologies is gathered from 10 single-chain MrBayes for one billion iterations (tree topologies are sampled every 1000 iterations with the first 25% iterations discarded). The numbers of unique tree topologies in the training set and the ground truth for each sequence set are reported in Table
We use the same model architecture and training strategy on all DS1-8. The generative model is five-layer MI.P with 512 hidden units and ResNet architecture. For the inference model, the number of message passing rounds with 100 hidden units. The number of particles in the multi-sample lower bound is K 32 in Table [ and Figure 5) The learning rate is set to 0.0003 at the beginning and anneals according to a cosine schedule. The experiments
= 2, and both MLP, and MLP, o are composed a two-layer MLP
single NVIDIA RTX 2080Ti GPU.
E ADDITIONAL EXPERIMENTAL RESULTS
E. GENE TREES AND SEQUENCE LENGTII
Figure depictss the latent representations of the mammal gene trees inferred from DNA sequences with length of 5000. We see that compared to shorter DNA sequences (Figure phylogenetic inference with longer DNA sequences better discovers the ground truth tree. Again, the ADORA3 gene performs best for phylogenetic discovery, the gene trees collapse to the ground truth tree. The MDS plots in the same setting are shown in Figure I]] where we see again that the tree topologies within a group tend collapse towards the origins and not sufficiently distinguished.
E.2 MULTIPLE PIYLOGENETIC ANALYSES COMPARISON
The right plot of Figure provides example that multiple phylogenetic analyses do not converge. Here we provide an example that multiple phylogenetic analyses quickly converge.
We consider the fossil data sets (Wright 2017) which contain 42 sequences (i.c., 42 leaf nodes) with length 87, which describes the biological characters of 42 types of Palcozoic crinoids. We assume the JC evolutionary model, implement two independent MrBayes (Ronquist al. 2012) runs for 10 million iterations, and gather the tree samples per 1000 iterations from the last 3 million iterations. These 6000 tree topologies with uniform weights constitute the training set of PhyloVAE.
We find that 6000 tree topologies in the training set are distinct, which reflects the diffuse posterior of fossil phylogenies. Despite this, the latent representations of tree topologies from the two independent
19

---

Published as a conference paper at ICLR 2025
Maramal gene trees (seglen 5000)
Mananal gene trecs (seçlen 5030)
Marnmal gene trees (soqion 5000)
mnce grorad tnea
ADORAAI + speead inda
APP goaad inca
PlylbV
12
4
Marmal gene trees (seqlen = 5000)
Mananal gaue trees (seçlen 5000)
x teak
uBBNA gooard inch
PhyloVA
211
Figure 10: Latent representations obtained by PhyloVAE for the mammal gene trees inferred from DNA sequences with a length of 5000.
KL divergence
K
K divergence
K
K 16
K 32
64
0.1275
0.0308
0.0273
0,0264
0.0202
0.0097
0.0100
0.0097
0.0951
0.0182
0.0177
0.0166
5
0.0202
0.0103
0.0099
0.0107
d 10
0.0997
0,0230
0.0189
0.0175
10
0.0202
0.0107
0.0098
0,0103
(a) DS1
(b) DS2
KL. divergence
K
K ==3]
K
KL divergence
K=
K 16
K 32
64
0.0674
0.0482
0.0529
0.0559
d
0.1038
0.0646
0.0619
0.0607
5
0.1397
0.0461
0.0502
0.0532
d
0.0995
0.0470
0.0467
0.0471
d 10
0.0980
0.0453
0.0477
0,0515
d= 10
0,1082
0.0470
0.0469
0,0460
(c) DS3
(d) DS4
Table 3: KL divergence of VAY to the ground truth on DS1-4 with varing d and K.
MrBayes runs seem to have the same distribution. This implies the underlying convergence of the two MrBayes runs.
E.3 GENERATIVE MODELING ON BENCHMARK DATA SETS
Table B reports the KL divergence obtained by different choices of d and K on DS1-4. We see that increasing K can generally improve the approximation accuracy, while sometimes a large d may increase the training difficulty and lead to overfitting.
Figure 1 compares the probability estimates and the ground truth. We see that Phylo provides reliable probability estimation.
Figure 14 shows latent representations of the tree topologies in the training set (repository 1) of DS1-8 produced by PAy
20

---

Published as a conference paper at ICLR 2025
Maramal gene trees (soglen - 5000)
Mananal gene trecs (seçlen - 5030)
Maznaal gene trees (soqion - 5000)
mnce goouad tnca
ABORAI+ + spoedd inda
APP goaad inca
MDS plot.
Ph
Mammal gene trees (seqlen- = 5000)
Mananal gaue trees (seglen = 5030)
x prssdi toak
uBBSA gooadd inch
MDS plot.
211
Figure 11: MDS plots for the mammal gene trees inferred from DNA sequences with a length of 5000.
MrBayes trees of fossil pla logenies
MrBayes trees of fossil phylogenics
-2
Figure 12:
Latent representation of two groups of tree topologies gathered from independent MrBayes runs on the fossil data sets.
To more comprehensively evaluate the efficiency of Phylo we report the memory usage of it and the baseline method ARTree in Table 4
Table 4: Memory usage (MB) of running ARTree and PhyloVAE (d = 10, K = 32) on DS1-8.
Data set
DSI
DS2
DS3
DS4
DS5
DS6
DS7
DS8
ARTree
400.37
428.79
518.84
600.50
784.32
778.05
932.56
1067,93 73
PhyloVAR (d = 10, K = 32)
635.67
637.72
655.69
785.75
689.15
701.30
784.41
818.72
21

---

Published as a conference paper at ICLR 2025
2, K
- -2
d 10, K 32
18
100
post
pokk
peok
oetimaaed pellabiins
ciitated pestetilly
Figure 13: Tree probability estimates obtained by PhyloVAR v.s. ground truth on DS1.
DS0
DB5
D87
D6S
Figure 14: Latent representation of the topologies in the training set (repository 1) of DS1-8. The scatter size proportional to the probability the tree topology.
22


Reviews:
[{"review_id": "4z2k3qPfhw", "reviewer": "Reviewer_SRrn", "summary": "PhyloVAE adopts the architecture of variational autoencoders and is specially designed to handle phylogenetic tree topologies. It maps any point in the latent space to a specific tree topology, and this architecture makes it excel at distinguishing between different tree shapes, outperforming traditional distance-based methods. Moreover, PhyloVAE's generative property, which enables the mapping of any point in the latent space to a specific tree topology, is an important advantage over other models that lack this functionality. This property enhances the visualization and interpretability of the learned representation.", "questions": "* **(Q1)** One of the motivations mentioned in the text is: *The classical approach to visualize and analyze distributions of phylogenetic trees is to calculate pairwise distances between the trees and project them into a plane using multidimensional scaling (MDS) .* *However, these approaches have the shortcoming that one can not map an arbitrary point in the visualization to a tree, and therefore do not form an actual visualization of the relevant tree space.* However, as the main motivation of this paper, it does not explain in detail the limitations of this mapping and its specific impact on visualization and analysis. Why is this operation a disadvantage? And what are the shortcomings of the MDS method in representing tree space?\n\n* **(Q2)** The encoding mechanism in Section 3.2 is clearly written, but there is a potential problem:\n\n  - **(a)** In the Decomposition stage, when selecting edge e_n, the conditional probability $Q(e_n | e_{<n})$ is based on the selected edge set $e_{<n}$. As the depth of the tree and the number of edges increases, the calculation of the conditional probability may become complicated.\n  - **(b)** In the Reconstruction stage, if an error occurs in the previous Decomposition process, the reconstructed tree may not be consistent with the original tree. How should this be corrected?\n\n* **(Q3)** The paper claims that PhyloVAE is ‘the first representation learning framework that targets the topology of phylogenetic trees’, but existing methods such as VBPI-GNN also have representation learning capabilities. This statement lacks discussion and comparison with relevant existing work, making its claim of innovation seem less rigorous and may make readers question the contribution of the article. Similarly, why not compare with baselines?\n\n* **(Q4)** Where is the hyper-parameter experiment, e.g., how to decide the parameters $d$, $K$, $L$, Iterations, and batch size?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* **(S1)** The proposed PhyloVAE framework is highly innovative. To my knowledge, a variational autoencoder is applied to the representation learning of phylogenetic trees for the first time, demonstrating the potential of deep learning methods in this field.\n\n* **(S2)** This paper is overall easy to follow to some extent (but also has some drawbacks, as mentioned weaknesses). The model architecture and training process are described in detail, especially the mapping function in potential space and visualization of tree topology.\n\n* **(S3)** The effectiveness of PhyloVAE is fully verified through experiments, especially its performance in dealing with complex tree structures, which shows the advantages over traditional distance-based and density-based estimation methods. The experimental design is reasonable, and the used datasets are appropriately selected.", "weaknesses": "* **(W1)** Some important assertions are unclear or unsupported. The author mentioned that the *model has high-resolution representations of phylogenetic tree samples*. How was this proven through experiments and theatrical analysis? Meanwhile, the authors mention that *their effectiveness heavily depends on the choice of distance metric and can sometimes exhibit counterintuitive behaviors*. What specifically does this counterintuitive behavior refer to?\n\n* **(W2)** The specific application scenarios described in the manuscript are vague. Although the importance of tree visualization is emphasized in the intro and abs sections, the main goal of the model is not clearly distinguished between representation learning and generative modeling. I suggest the authors further clarify the core goal of the proposed methods (or tackled problem) in the introduction section and clearly distinguish the relationship between representation learning and generative modeling. Meanwhile, the author might provide a specific discussion on the potential use of the model in different application scenarios to enhance the logic and practical significance of the manuscript.\n\n* **(W3)** This paper mentions the limitations of distance-based and density-based estimation methods. Still, it lacks an in-depth analysis of the PhyloVAE model settings to clarify its advantages in representation learning and visualization. What model settings or methods can perform well in representation learning and visualization?\n\n* **(W4)** More minor issues could be clarified:\n  - **(a)** What is the impact of different phylogenetic inference software choices?\n  - **(b)** Why is the formula for the data distribution in line 157? Are there any a priori constraints?\n  - **(c)** The simulated dataset used in the experiments in Section 5.1 only contains tree topologies with five and eight-leaf nodes, and experiments on large-scale tree structures are lacking.\n  - **(d)** The paper does not clearly explain the specific generation process of the tree topologies in the experiments in Section 5.1.\n  - **(e)** The paper's formula 3 has a variable K, and so does line 346. Do they represent the same variable?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JvsqKzVSWs", "reviewer": "Reviewer_vhRJ", "summary": "The paper introduces PhyloVAE, an unsupervised learning framework for phylogenetic tree topologies that utilizes variational autoencoders (VAE). PhyloVAE is designed to both learn informative representations of phylogenetic tree structures and generate new tree topologies efficiently. The model combines a novel encoding mechanism inspired by autoregressive topology generation, creating a deep latent-variable model that maps arbitrary points in the latent space to tree topologies, enabling high-resolution and parallelized topology generation. The framework demonstrates its effectiveness in representation learning and generative modeling, outperforming traditional distance-based methods by providing both visualization and probabilistic modeling for tree topology distributions.", "questions": "1. How does PhyloVAE perform when trained on one set of datasets (e.g., DS1-DS4) and tested on a distinct dataset like DS5? Could the authors provide insights or additional experiments to demonstrate the model’s robustness across varied evolutionary patterns? Cross-dataset evaluations would enhance the assessment of PhyloVAE’s broader applicability. To make this clearer, could the authors also analyze specific aspects of the datasets—such as the number of species, evolutionary rates, or tree shapes—that might influence generalization performance? Additionally, specific metrics or visualizations for comparing performance across datasets would help clarify how different components, like the encoding mechanism or the inference model, contribute to the model’s generalizability.\n2. Has PhyloVAE been directly compared to real, experimentally derived species trees? Such a comparison could provide essential validation for the model’s biological relevance by examining how closely the generated trees align with established evolutionary relationships. Could the authors explore potential discrepancies between PhyloVAE-generated trees and actual species trees and analyze their implications for the model’s accuracy? Including specific datasets or types of experimentally derived trees, as well as visualization techniques or metrics, would help quantify and illustrate these similarities and differences.\n3. While PhyloVAE offers efficiency gains, it appears not to outperform traditional models like ARTree in terms of accuracy metrics, as noted in Table 1’s KL divergence comparisons. Could the authors clarify the trade-offs made between efficiency and accuracy in the model design, and whether further refinements might enhance both speed and accuracy? A detailed analysis of how specific model components or hyperparameters affect this trade-off would be helpful. For example, it would be insightful to explore how increasing the latent space dimension or adjusting the number of particles in importance sampling impacts both computational efficiency and accuracy (e.g., KL divergence). These investigations could offer valuable directions for potential model refinements.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Highly Relevant and Practical Topic: The paper addresses the generation and representation of phylogenetic trees, a critical problem in evolutionary and computational biology. By introducing an unsupervised generative model, the paper offers a novel approach for analyzing complex evolutionary relationships. This work is practically significant as it enables researchers to gain deeper insights into biological evolutionary paths and diversity, advancing the field substantially.\n2. Significant Efficiency Gains: PhyloVAE achieves efficient phylogenetic tree generation through its non-autoregressive design, enabling faster and parallelized tree structure generation. Compared to traditional stepwise generation methods, this improvement is especially meaningful for handling large-scale biological data, accelerating the computational process and enhancing the model’s feasibility and applicability in real-world scenarios.", "weaknesses": "1. Need for Generalizability Testing Across Datasets: The paper primarily focuses on evaluating PhyloVAE across different benchmark datasets (DS1-DS8) without testing its generalizability across distinct datasets. For example, testing a model trained on DS1-DS4 and then assessing its performance on DS5 would offer insight into the model’s robustness across varied evolutionary patterns. Such cross-dataset evaluations are crucial for demonstrating the model’s applicability beyond specific training sets.\n2. Lack of Comparison with True Phylogenetic Trees: While PhyloVAE performs well in capturing tree structures, there is no direct comparison between the generated phylogenetic trees and real, experimentally derived species trees. This comparison would help validate the biological relevance of the model’s output by assessing how closely PhyloVAE’s generated trees align with known evolutionary relationships. In-depth analysis of these discrepancies would offer valuable insights into the model’s accuracy.\n3. Marginal Improvements in Performance Metrics: The paper demonstrates efficiency gains in tree generation time but does not show substantial improvements in accuracy metrics compared to state-of-the-art methods. As shown in Table 1, while PhyloVAE’s performance is comparable, it does not significantly outperform traditional models like ARTree in KL divergence to ground truth trees. This raises questions about the trade-offs made between efficiency and accuracy in the model’s design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BwitbNoTPG", "reviewer": "Reviewer_JV6s", "summary": "The authors introduce a cheap approximation of the ARTree generative model for phylogenetic trees (by assuming independence of the actions executed sequentially to construct a tree). This allows faster training compared to ARTree because generation (or reconstruction variational bound) can be done in parallel (over all tree construction actions) rather than sequentially, which yields a runtime speedup on modern computers. They evaluate the new model (PhyloVAE) for the purpose of 2D visualization and quantify the loss in modeling power (in terms of KL divergences to the ground truth).\n\nI have increased my ratings after reading the rebuttal.", "questions": "Please try to address the weaknesses in the above section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Phylogenetic trees are interesting objects to model and visualize and it could be the case that training on vary large collection of very large trees could be computationally expensive with previous methods such as ARTree.", "weaknesses": "(1) to be convincing, the paper should show a case where ARTree is really too slow for being practical\n\n(2) The experiments reported take on the order of seconds or less than a minute to train, so one wonder if the gain in computing power is so useful, if that would be at the expense of worse modeling of the tree distributions.\n\n(3) Although two different capacities were tested for PhyloVAE (dimension 2 and 10), that was not the case for ARTree, making it unclear if ARTree could have obtained a better KL divergence with a different capacity.\n\n(4) In terms of modeling power, assuming independence of the actions seems very strong. It is not clear that it will work well for other tree distributions not tested in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E6W8a8mix4", "reviewer": "Reviewer_7Ksu", "summary": "The authors propose a VAE for learning latent representations of phylogenetic tree topologies and generative processes thereof. Arguably the main contribution lies in the clever encoding structure that runs in the order of leaf nodes. A crux to making the VAE work for trees is to use pre-computed tree topologies as training data (inputs to the encoder). They experiment on standard and less standard real phylogenetic data.", "questions": "* Relating to the references provided regarding mixtures of variational distributions, I am wondering if your proposed method could benefit from the mixture techniques provided in [2,3,4]? Potentially they could help PhyloVAE to outperform ARTree in Table 1? In [4] the estimators make the learning of the mixture components efficient in terms of run time.\n\n* Is Algorithm 1 parallellizable on a GPU? Can you use batches of tree topologies during training?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a very nice submission that is well-written and which proposes a clever approach to learn representation and generative processes of phylogenetic trees. The experimental results are convincing and significant, especially taking into account the run times compared to ARTree. It utilizes previous results and ideas from the VI-based phylogenetic tree inference literature to, in a non-trivial way, devise a VAE algorithm.", "weaknesses": "* It is appreciated that it is clearly stated that the inputs to the encoder are collections of topologies. In [1; Fig. 5] the posterior coverage of the pre-computed topologies was evaluated and compared with those obtained from MrBayes. Can you provide a similar quantification? It might even suffice to merely reference their work to guard aganst the common suspicion that pre-computed tree topoplogies are not good representations. \n\n* On the same topic, in Sec. 5.2 the topologies are gathered from a 1 milion MCMC run, but this is not this quite short, in order to ensure that the samples are coming from the posterior?  To be clear, I do not think that it is an issue to learn posteriors over a pre-computed set of topologies, if the set of topologies is a good representation of the true posterior's support.\n\n* The collection of trees are tuples of the topologies and their corresponding weights. The weights are then in a footnote described to be the frequency of the topology (right?). Then in all experiments, the weights are explicitly mentioned to be uniform. Why is this? Is a topology never sampled more than once in the pre-processing stage? What would happen if the trees were weighted by scores obtained, for instance, from MrBayes? Or simply the Felsenstein likelihood of the topology, normalized over all samples.\n\n* Recently there have been works for applying mixtures of variational distributions in VAEs [2], phylogenetic posterior inference [3] as well as in VAEs *and* phylogenetics [4] (note that [4] does not propose a VAE for phylogenetics, but applies mixtures in both settings). I think at least [3] and [4] should be mentioned in the list of works referenced at line 463 to emphasize the great, current interest in Bayesian phylogenetics at ML conferences. \n\n## Minor\n* There is a clash of notations for $w$ notating both internal node connections in Sec. 3.2 and the weights of the tree topologies.\n\n* I would be good to disambiguate the proposed method to other tree-based VAEs such as [5, 6] and [7] (although [7] appears to not be published). None of these references experiment on phylogenetic data, but the originality of the proposed method would be emphasized by contrasting your encoder and generative processes to these works.\n\n[1] https://jmlr.org/papers/volume25/22-0348/22-0348.pdf\n\n[2] https://proceedings.mlr.press/v202/kviman23a.html\n\n[3] https://arxiv.org/abs/2310.00941\n\n[4] https://arxiv.org/abs/2406.07083\n\n[5] https://arxiv.org/pdf/2306.08984\n\n[6] https://arxiv.org/pdf/1810.06891\n\n[7] https://openreview.net/forum?id=Hy0L4t5el", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)