The tree autoencoder model, with application to hierarchical data visualization

Authors: miguel á carreira-perpiñán
kuat  gazizov
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 21:08:00

Abstract: We propose a new model for dimensionality reduction, the PCA tree, which works like a regular autoencoder, having explicit projection and reconstruction mappings. The projection is effected by a sparse oblique tree, having hard, hyperplane splits using few features and linear leaves. The reconstruction mapping is a set of local linear mappings. Thus, rather than producing a global map as in t-SNE and other methods, which often leads to distortions, it produces a hierarchical set of local PCAs. The use of a sparse oblique tree and PCA makes the overall model interpretable and very fast to project or reconstruct new points. Joint optimization of all the parameters in the tree is a nonconvex nondifferentiable problem. We propose an algorithm that is guaranteed to decrease the error monotonically and which scales to large datasets without any approximation. In experiments, we show PCA trees are able to identify a wealth of low-dimensional and cluster structure in image and document datasets.

Main paper: 
 introduction: 
 As a form of exploratory data analysis, dimensionality reduction (DR) for visualization seeks to provide a representation of a dataset in dimension at most 3 (we will focus on 2D) that, combined with the human visual ability to pick patterns, is able to find structure of interest in a high-dimensional dataset. Many DR methods have been proposed over the years. Currently, t-SNE [40] and PCA are the most popular choices and have seen widespread application in many areas.Here we advocate for a different approach, that of hierarchical local linear projections, which we call the PCA tree. Hierarchical DR is not new, but what is new is our definition of the model as a certain type of tree autoencoder and our algorithm to learn it by minimizing the reconstruction error in a self-supervised way, which brings considerable advantages in interpretability and training and inference time. From the outset, we acknowledge that DR is a hard, ill-defined problem that no single technique can solve satisfactorily. In reducing a dataset to 2D we necessarily have to lose information and produce artifacts. Indeed, the distortions of t-SNE and their impact on applications have been well documented [6, 22] . However, the PCA tree provides significant, complementary advantages over previous methods.Firstly, PCA trees are practical and intuitive to use. They optimize the reconstruction error, which has a clear meaning. They do not require a neighborhood graph (and perplexity parameter, etc.), which is tricky to estimate so it captures manifold structure, and computationally very costly. The model, a tree autoencoder (see later), directly defines nonlinear out-of-sample mappings (encoder and decoder). Its hyperparameters are natural: the depth ∆ of the tree, which directly controls the resolution (number of scatterplots and reconstruction error), and the sparsity λ, which helps to interpret the tree decision nodes. Inference on a tree (encoding or decoding) is extremely fast. Training (to find a local optimum, a hard problem because the tree is nondifferentiable) is linear on the sample size and parallelizable, hence scalable to large datasets without the need for approximations.Second, PCA trees are highly interpretable and, as demonstrated in our experiments, they extract a wealth of information from complex datasets. This comes from the fact that both trees and PCA are techniques with well understood interpretability. Specifically: 1) the use of trees to construct a nested hierarchy of maps where an input instance follows a single root-leaf path; 2) the use of sparse oblique trees, with hyperplanes using few features at the decision nodes; 3) the use of linear projections in the leaves, each a PCA scatterplot; and 4) the ability to see the data at multiple scales.Third, a PCA tree uses multiple local latent projections (an "album") hierarchically organized rather than a single global one ("atlas"). The latter is more likely to introduce distortions in trying to arrange many dimensions into a 2D map, and is hard to use with large datasets, requiring a lot of pan and zoom. This is less of a problem if one focuses on a local part of the data, particularly if there are clusters. Also, a linear projection is easy to understand and, while it does introduce distortions, it is well understood how this happens: distances can only shrink. If a PCA map shows clusters, these are real-in contrast with t-SNE's tendency to create false clusters [6, 22] , for example. Also, PC directions often correspond to some real degree of freedom, and the mean is obviously a useful summary of the data. Thus, while global PCA generally collapses different parts of a manifold, local PCAs can succeed-if properly learned. Multiple local maps are like an album of a house, where each picture shows a different room, but with an hierarchical structure.A final advantage is that the loss function is really a self-supervised regression problem. This makes it possible to use cross-validation to determine the hyperparameters (although, in an exploratory data analysis, we probably want to explore such hyperparameters manually).The PCA tree does have its limitations, as follows. It needs access to the explicit feature vectors, unlike graph-based methods, which need instead pairwise affinities. The optimization converges to a local optimum, which depends on the initialization. And, as with any DR method, PCA trees reveal only some structure, so practitioners should try multiple, complementary methods. 
related work: 
 The literature on DR is vast. We note connections to the most relevant methods.Non-hierarchical DR Of the many existing DR methods, we mention two large groups that are particularly important in this paper. One are autoencoders, which learn an encoder or projection mapping and a decoder or reconstruction mapping so as to minimize the reconstruction error of each training point. The classic examples are PCA and neural network autoencoders [34, 17] , where the encoder and decoder are linear and nonlinear, respectively. Our tree autoencoders are another example. The other are nonlinear embedding methods, which learn the 2D projection of each training point so that Euclidean distances in latent space best match distances or similarities in the high-dimensional space. These include multidimensional scaling [5] , SNE [16] , t-SNE [40] , the elastic embedding [6] and UMAP [28] , among others. Although often proposed as matching distributions of neighbors in the high-and low-dimensional spaces, these methods can be equivalently seen as optimizing a tradeoff of attraction between true neighbors and repulsion between points [6, 41, 22] .Hierarchical DR A multiscale view of DR is an attractive idea that has been explored before, often justified by the fact that a single 2D map may miss much of the structure in a complex dataset. We consider two ways of defining the hierarchical structure: soft and hard. In soft hierarchies, one defines a soft, probabilistic tree [20] so that an input instance traverses every tree path with a positive probability. Thus, the model really is a probabilistic mixture of the leaves ("experts") with mixing proportions given by the tree structure. This has the advantage that the model is differentiable, so optimization is straightforward with gradient-or EM-based approaches. Unfortunately, we lose the interpretability and fast training and inference, since an instance reaches all nodes, and the leaves may not localize in input space and overlap with each other. Examples of this are [4, 30, 38, 19] . A more detailed comparison of soft and hard oblique decision trees appears in [12] .Making the tree decisions hard makes the hierarchy nested, but the optimization is much more difficult (indeed, this is an important contribution in our paper). One can construct a hard hierarchy in a greedy sequential way by recursively partitioning the dataset, or by using some other tree con-struction (such as kd-trees [11] or cover trees [2] ); and then applying some form of DR at each node. However, this is suboptimal (since the node parameters are fixed as they are added, or independently from the leaf parameters) and produces overly large trees. Conversely, one can grow the tree bottomup by a form of hierarchical clustering, but this is also suboptimal. These approaches include work such as [1, 24, 26, 29] and others such as hierarchical self-organizing maps [32] , sequential NMF [9] and hierarchical SNE [31] .Flat, local DR This seeks to estimate a collection of local latent maps but without any hierarchy. Again, this can be done in a soft way via probabilistic mixtures (of factor analyzers [13] , PCAs [37] , etc.) or in a hard way as a form of k-PCAs [21] . A few works have also sought to align a collection of local DR models into a single global map [33, 36, 44, 23] .Other related models Some works have tried to combine graph-based methods, such as t-SNE, with hierarchical DR. For any DR method that does not define an out-of-sample mapping, such as t-SNE and other nonlinear embedding methods, it is possible to learn a tree-based projection mapping [45] . Also, some fast approximations for graph-based embedding methods are based on tree data structures, such as Barnes-Hut trees [43, 39, 42, 35] . However, this is intended to accelerate the N-body computations, not to define a hierarchical DR.In view of this, the main novelty of our work is in defining an autoencoder consisting of a hard tree encoder and a local linear decoder, and in giving an algorithm to learn it by optimizing the reconstruction error. We believe this is the first algorithm to guarantee a monotonic decrease of the error jointly over the tree parameters and local PCAs. We describe both the tree autoencoder model and the optimization algorithm next. 
definition of the pca tree as an autoencoder: 
 Like any autoencoder, the PCA tree defines an encoder F and decoder f , but in a peculiar way, as follows. Firstly, consider a fixed rooted directed tree structure with decision nodes and leaves indexed by sets D and L, respectively, and N = D ∪ L. Both the encoder and autoencoder use this tree structure. Each decision node i ∈ D has a decision function h i (x; w i , w i0 ): R D → C i , where C i = {left i , right i } ⊂ N , sending instance x to the corresponding child of i. Rather than axis-aligned trees, which are a poor model for high-dimensional data, we use oblique trees, having hyperplane decision functions "go to right if w T i x + w i0 ≥ 0", with w i ∈ R D and w i0 ∈ R. The tree's prediction for an instance x is obtained by routing x from the root to exactly one leaf and applying this leaf's predictor (defined below). We define the reduced set (RS) R i ⊂ {1, . . . , N } of a node i ∈ N as the training instances that reach i given the current tree parameters.Encoder or projection mapping F This is given by a tree mapping T e (x; Θ): R D → L × R L where the predictor for leaf j has the form of a linear mapping F j (x; U j , µ j ) = U T j (xµ j ), where U j ∈ R D×L is an orthogonal matrix and µ j ∈ R D . The encoder parameters are Θ = {w i , w i0 } i∈D ∪ {U j , µ j } j∈L . Thus, the encoder maps an input instance x ∈ R D to a leaf index j ∈ L and an L-dimensional real vector z = U T j (xµ j ), which at an optimum (see later) will be the PCA projection in that leaf. This means that the PCA tree does not have a common latent space of dimension L where all instances are projected. Instead, it has one separate L-dimensional PCA space per leaf. Thus, the latent space L × R L is mixed discrete-continuous 1 .Decoder or reconstruction mapping f This maps a leaf index j and L-dimensional vector z (in L × R L ) to a vector in R D . It consists of a set of linear mappings of the form f j (z; U j , µ j ) = U j z + µ j for j ∈ L.We can now define the autoencoder as the composition of the decoder and encoder, T = f • F. Conveniently, we can absorb the leaf index implicitly into each leaf. Thus, T(x; Θ) is simply a regression tree identical to the tree encoder T e (x; Θ) but where the predictor {3, z 3 } f 1 f 2 f 3 f 4 F 1 F 2 F 3 F 4 x 1 x 2 x 3 x y tree partitions R D into |L| leaves T e (x) f j (z j ) j = 1 j = 2 j = 3 j = 4 z 1 1 z 1 2 z 2 1 z 2 2 z 3 1 z 3 2 z 4 1 z 4 2Figure 1 : Illustration of the tree autoencoder model with a data space R 3 for a tree of depth ∆ = 2 with leaves L = {1, 2, 3, 4}, defining a latent space (album) of four 2D spaces. mapping of leaf j has the form g j (x; U j , µ j ) = U j U T j (xµ j ) + µ j . Note this is a linear mapping of rank L < D, and it is the composition f j • F of leaf j's decoder with the encoder. Fig. 1 illustrates the tree autoencoder idea. Fig. 2 shows a PCA tree on a 2D toy example using a latent dimension L = 1.∆ = 0 (PCA), E = 1.10 ∆ = 1 tree ∆ = 1 initial, E = 0.84 ∆ = 1 final, E = 0.08 w 1 , w 0 µ 1 , U 1 µ 2 , U 2 
optimization algorithm: 
 Our objective function is now the regular reconstruction error of an autoencoder T: R D → R D with an ℓ 1 regularization term of hyperparameter λ ≥ 0 on a training set {x n } N n=1 ⊂ R D :E(Θ) = N n=1x n -T(x n ; Θ)EQUATIONThus, the PCA tree is trained like a regression tree which maps real vectors to real vectors. It consists of a fixed tree structure (which we will take as complete of depth ∆); sparse oblique decision nodes, consisting of a hyperplane split using few features (as encouraged by the ℓ 1 sparsity penalty); and linear low-rank leaves. The following theorem explains why we call it "PCA tree" (in fact, we designed the form of the loss function and tree so that this would happen). Theorem 4.1 (optimality condition over the leaves). The optimal {U * j , µ * j } i∈L given the remaining parameters are fixed, i.e.:min {Uj ,µ j }i∈L E(Θ) s.t. U T j U j = I, ∀j ∈ L (2)corresponds to setting each {U * j , µ * j } to the L principal components of R j , the RS of leaf j.Proof. Because of the separability condition (theorem 4.2) and the fact that {w i , w i0 } i∈D are fixed, problem (2) is equivalent to min {Uj ,µ j }i∈L n∈Rjx ng j (x n ; U j , µ j ) which (since the objective and constraints separate) is also equivalent to solving this for each leaf j ∈ L separately:EQUATIONThe solution of the latter problem is well known to be given by PCA, i.e., µ * j = 1 |Rj | n∈Rj x n is the mean of the training instances in R j and U * j consists of the L eigenvectors of the covariance matrixΣ j = 1 |Rj | n∈Rj (x n -µ * j )(x n -µ * j )T associated with its largest L eigenvalues.We borrow the idea of alternating optimization over the tree nodes from [7] (Tree Alternating Optimization (TAO)), which is fundamental to be able to update the parameters so as to reduce the objective function. Firstly, at any decision tree making hard decisions, each input instance follows exactly one root-leaf path. This results in the following separability condition. Theorem 4.2 (Separability condition). Let T(x; Θ) be the autoencoder tree and S ⊂ N a nonempty set of nodes that are not descendant from each other. Call θ i the parameters in node i. Then, as a function of the parameters {θ i : i ∈ S} (i.e., fixing all other parameters Θ rest = Θ \ {θ i : i ∈ S}), the function E(Θ) of eq. ( 1) can be equivalently written asEQUATIONwhere {E i : i ∈ S} and E rest are certain functions.Proof. Analogous to that in [7] . It follows from the fact that the reduced sets of all nodes in S are disjoint, because the tree makes hard decisions.That is, optimizing over the parameters of any set of nodes which are not descendants of each other separates: we can equivalently optimize each node on its own over its parameters and using only its RS. This simplifies the problem considerably and introduces significant parallelism: for example, all nodes at the same depth can be optimized in parallel. Indeed, our overall algorithm will process nodes from the leaves towards the root, optimizing all nodes at the same depth at each step.Second, we still have to solve the problem of optimizing (1) over the parameters of one given node. This simplifies into a reduced problem (RP). If the node is a leaf j, theorem 4.1 gives us the answer: we set the leaf's U j , µ j to the PCA on R j . This is the exact solution to the optimization over the leaves, so the overall objective function value will either decrease or stay constant. If the node is a decision node i, optimizing (1) over w i , w i0 is equivalent to a RP as given by the following theorem. EQUATIONwhere: R i is the reduced set of node i; we define the weighted 0/1 loss L in (y in , •):C i → R + ∪ {0} for instance n ∈ R i as L in (y in , y) = l in (y) -l in (y in ) ∀y ∈ C i ;we define the pseudolabel y in = arg min y∈Ci l in (y) as the "best" child of i for n (or any y in ∈ arg min y∈Ci l in (y) in case of ties); and we define the function l in :C i → R as l in (z) = x n -T z (x n ; Θ z ) 2 2 for any z ∈ C i (child of i), where T z (•; Θ z ) is the predictive function for the subtree rooted at node z.Proof. Analogous to that for the decision node RP in [7] . A difference is that there the tree was a classification tree with the 0/1 loss, which resulted in an unweighted problem; while here we use the reconstruction error, which results in a weighted problem in (6) . This follows from the fact that all a decision node can do with an instance is send it down its left or right child, and the ideal choice is the one that results in the lowest reconstruction error downstream from that node.The reduced problem of eq. ( 6) is an ℓ 1 -regularized weighted 0/1 loss binary classification problem over a linear classifier h i with binary pseudolabels {y in }, defined as the child of i that gives the best reconstruction for x n under the current tree. This problem is NP-hard, but we can typically get a good approximate solution by solving an ℓ 1 -regularized surrogate loss instead. We use the logistic loss with instances weighted as in theorem 6. We can guarantee that the overall objective function value decreases or stays constant by accepting the surrogate solution only if it improves over the previous parameters2 , which empirically is usually the case. This means each node update, hence each overall iteration over all nodes, monotonically decreases (1). We stop when there is little change in the parameters or objective, or when we reach a set number of iterations. This concludes the algorithm, whose pseudocode is in fig. 6 . Essentially, it repeatedly updates the nodes in turn: at a leaf it solves a PCA, and at a decision node it solves an ℓ 1 -regularized, instanceweighted logistic regression problem (we use LIBLINEAR [10] ). The tree is initialized as a random median tree, i.e., at the root we pick a random direction for the weight vector and a bias so that half of the instances go to either child; and we repeat this recursively down the tree. 
computational complexity for training and inference: 
 Throughout, assume a complete tree of depth ∆; sparse decision weight vectors having sD nonzero values on average, with s ∈ [0, 1] (so s = 1 means dense vectors); and a latent dimension 1 ≤ L ≤ D at each leaf (L ≤ 3 for visualization). As cost we use the number of scalar multiplications.By inference we mean the time to map an input instance x ∈ R D to its leaf j ∈ L and latent vector z ∈ R L . This requires traversing a single root-leaf path of depth ∆, at a cost Θ(sD∆), and computing the leaf PCA projection, at a cost Θ(LD), total Θ((L + s∆)D) (which, for ∆ = 0, i.e., regular PCA, corresponds to Θ(LD)). This is very fast, since both L and ∆ are very small. Reconstructing a latent vector z ∈ R L at a leaf j ∈ L costs Θ(LD).The cost of one iteration, i.e., updating each node in the tree once, is as follows. For the decision nodes, this equals to setting up the RP (i.e., computing the pseudolabel losses) and solving it with one logistic regression in each node's RS. To work this out we note that the RSs of all nodes at the same depth total N instances, and we assume logistic regression to be linear in the dimensionality and sample size. Then, the total cost of all decision nodes has two terms. The first is the pseudolabel cost over all decision nodes, which is Θ( 1 2 sN D∆(∆ -1)). This is due to the fact that, at each decision node, we have to send each training instance down the left and right children to compute their losses (actually, we need only send it down one child if we precompute the losses), and the decision nodes span depths from 0 to ∆ -1. The second is the RP solution, whose cost is proportional to that of running ∆ logistic regressions on the whole dataset (at depths 0, 1, . . . , ∆ -1), or Θ(c 1 N D∆), where c 1 > 0 is a constant factor (which includes the average number of iterations of the logistic regression solver). This cost is as in the sparse oblique classification trees of [7] .For the leaves, we have a peculiar situation. Say a leaf has a RS with M instances. The cost of PCA 3 on that RS is Θ(M D 2 + c 2 D 3 ) if D ≤ M and Θ(DM 2 + c 2 M 3 ) if D ≥ M ,depending on whether we compute the eigenvectors of the D × D covariance matrix or of the N × N Gram matrix, respectively, where c 2 > 0 is a constant factor. Assume for simplicity that each RS has M = N 2 -∆ instances (balanced partition) and define the critical depth∆ * = log 2 N D . Then: cost, all leaves = ∆ ≤ ∆ * , shallow tree regime: Θ(N D 2 + c 2 D 3 2 ∆ ) ∆ ≥ ∆ * , deep tree regime: Θ(N 2 D2 -∆ + c 2 N 3 2 -2∆ ) ≤ Θ((1 + c 2 )N D 2 ).For a given dataset of fixed N and D, the cost depends on the tree depth ∆ and has a turning point when ∆ = ∆ * . For datasets having 4 N < D, we are always in the deep regime, but if N > D then we can be in the shallow or deep regime for small or large ∆, respectively. Interestingly, this leads to a non-monotonic cost as a function of ∆, as shown in fig. 3 , because the shallow regime cost increases with ∆ but the deep regime one decreases with ∆. This is surprising because the deeper the tree the more leaves it has, yet the faster they train. The reason is that the PCA cost is superlinear on the sample size, but the sample size per leaf decreases proportionally to the number of leaves.We emphasize that, as noted above, in both regimes the leaves' cost is strictly upper bounded by (1 + c 2 )N D 2 (with equality if ∆ = ∆ * ), hence this cost is at most linear in N and quadratic in D, just like in regular PCA. For fixed N and D, the leaves cost first dominates the decision nodes cost (shallow tree regime) because the former includes a term N D 2 that dominates the decision nodes' term c 1 N D∆, since c 1 ∆ < D in practical cases. But as ∆ grows (deep tree regime) the leaves cost decreases and reaches N (D+c 2 ) for ∆ = log 2 N (when each leaf constains a single instance), so the overall cost is eventually dominated by the decision nodes, Θ(c 1 N D∆). As a useful summary, the rough cost is Θ(N D 2 ) for shallow trees and O(N D) for deep trees-which is asymptotically faster than PCA! (although very deep trees having few instances per leaf are likely not practical). This makes the algorithm highly scalable to large sample sizes without the need for any approximation. This is a significant advantage over neighbor embedding algorithms such as t-SNE, which have a quadratic cost on N and require some approximation to reduce this [39, 42, 43] .Parallel training From the separability condition, all nodes at the same depth can be updated in parallel, which accelerates the training considerably. Assume for simplicity we have 2 ∆ processors (which is perfectly feasible for trees having depth 4, as in our experiments). Then the costs are as follows. For decision nodes, we have the cost of running a logistic regression on ∆ datasets of sizes N, N 2 , . . . , N 2 -∆ (from the root to the leaf parents), a geometric series which is upper bounded by 2N , hence a total cost of Θ(2c 1 N D), i.e., ∆ 2 times faster than the sequential computation. For the leaves, we have the cost of running PCA in a single leaf, i.e., 2 ∆ times faster than the sequential computation. Again, this is a significant advantage over t-SNE, which cannot be easily parallelized. 
experiments: 
 In summary, we show the following: we confirm our theoretical predictions about monotonic decrease of the objective function and training time; compare the reconstruction error with PCA; and demonstrate how PCA trees are highly interpretable and extract significant structure from complex datasets. We use several datasets of different types (images, documents), some of which appear in the appendix.Fig. 3 (right 2 plots) shows learning curves for different datasets and tree depths. As can be seen, the objective function decreases monotonically over iterations, and converges to a near-optimal solution in usually less than 10 iterations. Fig. 3 (plot 3) shows scalability to larger sample sizes using the Infinite MNIST dataset [27] (images of 28 × 28, so D = 784). We ran the PCA tree for 10 iterations, when it approximately converged. The linear cost of the PCA tree is clear as a slope 1 in the log-log plot. Training on 1M samples with depth 4 takes less than an hour. In contrast, t-SNE [39] and UMAP [28] are slower and have a superlinear cost, even though the implementations we used are not exact (they use approximations, such as Barnes-Hut trees or neighbor subsampling, to accelerate the computations, besides finding approximate rather than nearest neighbors).Infinite MNIST Fashion MNIST Amazon Reviews Fig. 4 shows the reconstruction error for PCA and PCA tree. Although the PCA tree uses L = 2 in each leaf, it is able to achieve a much lower error than PCA, which would need a much larger number of components to match it. This is the result of the PCA tree ability to learn a partition of the data and local PCAs that are jointly optimal. The appendix contains results for other datasets (images: MNIST, documents: Amazon Reviews).Even at a glance we can see lots of structure in the tree: many classes organize hierarchically in a meaningful way; decision weight vectors are sparse and highly meaningful, often separating groups of classes by focusing on the discriminant regions of the image; the PCs in the leaves usually are highly meaningful, indicating the change of some relevant degree of freedom; and the leaf scatterplots show clusters or class structure; among other things. We comment on some of these in detail.For reference, fig. 9 (appendix) shows the 2D global plot by PCA, t-SNE and other algorithms.Global tree structure Although our algorithm is unsupervised, it has the ability to separate different types of classes. For example, leaves L1 and L2 contain different types of shoes, while leaf L14 contains only bags. Some classes (e.g. Pullover and Shirt in L8) differ in tiny details and can only be separated if using the labels (remember that minimizing the reconstruction error seeks maximumvariance directions, which are not necessarily aligned with class variation). The coordinate axes in the leaves show the most variance in the data within the leaf's region. Since the RSs are nested along the tree hierarchy, we can also show a scatterplot at each decision node (appendix fig. 9 ), which shows how classes and other patterns are progressively separated as one descends deeper into the tree. However, note the optimal parameters if the tree was shallower would differ from those.Local tree structure The decision nodes' weight vectors select features sparsely, often focusing on parts of the image that can differentiate groups of images and send them selectively to the left or right subtree. The figure is annotated in some places to make this clear. For example, decision node D2 separates clothes "with" and "without" sleeves. In some cases, this results in the tree separating actual classes. For instance, node D14 separates "outerwear" type of clothes from "bags", which is explainable by simply looking at the decision node weights and the part of the image where they focus: blue values indicate "if the image has a neckline and shoulders" (like outwear clothes) and red values indicate "if it has wide frames" (like bags).PCA at the leaves Since each leaf applies 2D PCA on its corresponding reduced set, its linear projection scatterplot shows the most variance directions in the leaf region. As expected with PCA, amplified by the fact that it focuses on a region of the whole data, this often shows insights about the intrinsic local structure of the data. In particular, we can usually identify the coordinate axes (PCs) with meaningful quantities or degrees of freedom (labeled in the scatterplot below each leaf in fig. 5 ). For example, leaves L1, L2, L4, L7, L14, L15 and L16 contain objects of a specific type, such as L1 containing high-heel and high-ankle boots and sandals, or even isolating a class, such as L14 isolating bags. Interestingly, we find all these leaves often detect similar degrees of freedom: usually, PC1 is (overall) pixel intensity in the image and PC2 is object size (either height or width). Again, this is the result of minimizing reconstruction error, which projects along maximum-variance directions. Also, since overall intensity and object size are correlated, this is seen in the PCA plots as triangular shapes whose vertex corresponds to low-intensity images (caused by either overall low intensity or by a very small object). This can be clearly seen in leaf L4 (consisting of T-shirts and tops), as well as in other leaves: L7/L15 intensity-width; L1/L2/L16 intensity-height; L14 intensitysize of bags. The density of the projections in the 2D embedding is also informative. For example, in leaf L4, narrow tops are much more frequent than thin tops.Some leaves find degrees of freedom not aligned with a PC direction, but visible in the scatterplot, such as elongated clusters. These often correspond to different classes, where the direction of elongation is some meaningful degree of freedom within the cluster. For example, as shown in the bottom of fig. 5 , leaf L3 contains two types of clothes: "shirts" and "shoes". PC1 represents the class discrimination direction, while PC2 represents intensity. Local inspection of each cluster separately demonstrates that both clusters have maximum variation indicating pixel intensity. The shoe cluster also has variations in "weight" (lightweight shoes below and chunky shoes above). The "shirts" cluster has variation in T-shirt width. Leaves L4, L15, and L16 have almost a single class: shirts, dresses, and slim sneakers, respectively, and all of them have similar variations, i.e., intensity as PC1 and width or thickness as PC2. Finally, note that the appearance of clusters or complex structure within a leaf scatterplot suggests it would be worth either further expanding that leaf or using a deeper tree, which could be done in an interactive way.This dataset provides evidence of what patterns are real and whether they are detected by a given DR method. If clusters appear in a PCA plot, they are real (unlike in a t-SNE or UMAP embedding). This is because, in an orthogonal linear projection, distances either decrease or do not change, but cannot increase. Another example is the fact that the image overall intensity arises throughout the PCA tree leaves as a principal component. This is not surprising: intensity changes in this dataset are large, and thus carry considerable variance, which the local PCAs pick. Even the global PCA is able to pick that too and locate low-intensity images in the lower-left area of the 2D embedding (see appendix fig. 10 ). However, both t-SNE and UMAP do not; we have verified that low-intensity images are mapped all over the embedding in a haphazard way. A similar effect happens with the Amazon reviews document dataset (appendix section E): individual document classes appear in PCA plots as "streaks", caused by the document length (which changes the norm of the TF-IDF feature vector), and converging on the area occupied by short documents. Again, such an important pattern is lost in the t-SNE and UMAP embeddings.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Experiment section and supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 We have defined a new model for data visualization, the PCA tree, as a tree autoencoder that produces a set of hierarchical low-dimensional maps, and have given an algorithm to learn it by minimizing the reconstruction error, based on Tree Alternating Optimization (TAO). PCA trees are fast for training and out-of-sample inference, and highly interpretable. We hope they will provide a useful tool for data visualization. Beyond this, tree autoencoders have application in dimensionality reduction in general, as well as in clustering, subspace clustering, data compression, fast search and other problems that we are exploring. A Appendix / supplemental material A.1 Algorithm pseudocode input training set {x n } N n=1 ; sparsity hyperparameter λ ≥ 0; initial tree T(•; Θ) of depth ∆ with parameters Θ = {w i , w i0 } i∈D ∪ {U j , µ j } j∈L N 0 , . . . , N ∆ ← nodes at depth 0, . . . , ∆, respectively for each i ∈ N : generate R i (instances that reach node i) on tree T repeat for d = ∆ down to 0 The text is converted to lowercase, and all URLs, email addresses, punctuation, and numbers are removed. Stopwords are removed using NLTK [3] , and the dataset is lemmatized using SpaCy [18] . We then apply scikit-learn's TfIdfVectorizer with default parameters, resulting in a dimension of D = 72764.parfor i ∈ N d if i ∈ L then {U i , µ i } ← PCALetter Recognition task for classifying 26 capital letters in the English alphabet [25] . We use only the training set (N = 16000). Standardization is applied to all features.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Too much computation.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Experiment section and supplementary material Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: Generic algorithm for dimensionality reduction.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We subsample text reviews from the large collection in [15] (version from 2014). Reviews are selected from the following categories: Implementation We implement PCA tree using a combination of C++ and Python. Decision node optimization is implemented in C++ using LIBLINEAR [10] , and leaf optimization is implemented in Python using PCA from the scikit-learn library with svd_solver=arpack and n_components=2.We use the following hyperparameters: λ = 10 for MNIST and Fashion MNIST, λ = 1 for Letter, and λ = 0.01 for 20newsgroups and Amazon Reviews. The algorithm includes an early stopping criterion that terminates training if there is no decrease in the training error for 3 iterations with a change of less than 10 -3 .For t-SNE, we use the scikit-learn implementation (the default number of iterations is 1 000). For UMAP, we use the implementation from https://umap-learn.readthedocs.io/en/latest/ (the default number of iterations is 500 if the dataset size has fewer than 10 000 points and 200 otherwise).Hardware All experiments are conudcted on a Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz, 256GB RAM. The tree structure allows visualization of PCA scatterplots not only for the leaves but also for the decision nodes, as shown in Fig. 9 . Note how PCA projections of each class or group of classes separate distinctly from each other. EQUATIONWe evaluate our algorithm on the MNIST dataset. It is noticeable that almost half of the PCA tree leaves are nearly pure, containing mostly samples from a single class, while the other half contain classes that can be linearly transformed into each other. For example, leaf L16 contains three classes: 4, 9, and 7. By adding the top bar and removing the middle bar, we can achieve the following linear transformation: 4 → 9 → 7, as shown by the first principal component. The second component represents pixel intensity, so digits go from thin to thick from bottom to top. Now, consider the pure leaf L12. The digit 1 changes its inclination from "to the right" to "to the left" on the x-axis and its intensity from bottom to top.PCA overlaps nearly all data points except for those representing the digit "1".t-SNE distorts the global structure of the data. It forms clusters such that each cluster represents a separate class, which can be misleading in terms of distances. For example, the cluster for the digit "2" appears close to the cluster for "1" and far from "3", but in the high-dimensional space, the relationship is the opposite. A similar issue occurs with the digits "0" and "5".UMAP also distorts the positions of data points. It may show that the digit "3" is closer to "6" than to "4", while in the original high-dimensional space, the relationship is the opposite.Such distortions can result in wrong interpretations of the data's structure and cluster relationships.EQUATIONConsider now the dataset from a different domain, specifically text data. We use the Amazon Reviews dataset, represented as TF-IDF features, containing 20,000 samples and 9,430 features. The decision tree structure, illustrated in fig. 13 ) allows us to visualize the top words corresponding to the highest and lowest weight values, indicating important words at each decision node. Furthermore, using highest and lowest entries for both principal components, can be demonstrated variation across reviews, see table 1 By visually inspecting the leaves, we can identify similar reviews from different classes. For example, the first leaf contains many samples from different classes, but most of these samples are short reviews about "returns" and "refunds". Elongated clusters within the leaf reveal reviews about different topics, such as games (left cluster) and cosmetics (right cluster). Additionally, reviews from different classes may pertain to items with similar purposes. In leaf L3, there are four classes, but all are related to games: the top cluster is about "funny learning", the left cluster is about "teaching children to count using cards", and the right cluster is about "card games".Similar to previous examples, this tree has principal components that represent semantic or class separation. For instance, in leaf L5, the first principal component separates children's games from adult games, while in leaf L11, the first principal component distinguishes one class from the rest. Note that some leaves are pure, such as leaf L7 or L10, and within these leaves, clusters correspond to topics. Leaf L7, for example, is split into three topics: "cuticles" (left), "polishing" (top), and "nail growing" (right), although all samples are from the same class.t-SNE and UMAP significantly overlap most of the classes. While PCA projections present some cluster elongations, they also show overlapping classes and do not reveal the inner local structure. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).We could define the autoencoder so it uses a single, global latent space. However, this complicates the optimization (although it could be done with the method of auxiliary coordinates (MAC) of[8] by considering the problem as a nested function). Also, the per-leaf space arises naturally from the fact that the encoder partitions the space into leaf regions.s.t. U T j U j = I, ∀j ∈ L(3)In fact, in practice we find that always accepting the update (even if it increases the objective) works as well and is simpler. This makes the iterates slightly noisy, but over iterations the objective decreases steadily.It is possible to accelerate this via approximate SVD algorithms or matrix sketches to compute just the top eigenvectors, as well as using a good initialization, but here we consider the cost of a full eigendecomposition.If N < D we can always run PCA first and reduce the data to D < N without error.It is instructive to compare these results with those of training a sparse oblique classification tree on the Fashion MNIST dataset[14], as there are interesting similarities and differences. 


Decision: The majority of the reviewers lean towards accept. I also find the proposed method interesting and novel, which may spark further research ideas. Thus I also recommend the paper be accepted.

In the camera-ready version, the authors should revise their paper based on the discussions with the reviewers. Additionally, for future work, I suggest the authors expand their comparisons to other hierarchical methods, such as those suggested by reviewer 2dBg. While the authors provided extra comparisons during the discussion phase, the use of reconstruction error is somewhat unsatisfying as the proposed method is explicitly designed to optimize this error. Other metrics should also be included in future work, although admittedly good metrics are hard to come by in these problems.

Reviews: review: paper_summary: paper_summary: The authors nicely describe an extension of [5] to unsupervised dimensionality reduction.
The paradigm is quite different from regular DR (unique scatterplot) as it yields a tree and thus multiple leafs.
There are some similarities with hierarchical clustering (to some extent).
The authors also emphasize a connection with autoencoders (though readers might understand NN autoencoders while here these are rather linear auto-encoder, equivalent to PCA).
Attention is dedicated to the computational complexity and parallelization of the tree structure.
There is one illustrative experiment in the paper and some other classical benchmarks are reported in the suppl.mat.
strength_weakness: strength_weakness: The novelty with respect to [5] might seem a bit limited.
The proposed visualization paradigm is unconventional and probably shares the same shortcomings as those of a dendrogram in hierarchical clustering.
The global structure is probably hard to tell, which is not different than the usual drawback of t-SNE and such (although a user study might answer that question).
To some extent (again), the proposed visualization is reminiscent of what was done in the past with SOMs (here the leafs would resemble a 1D SOM). The vast literature about SOMs might contain some variants with hierarchies or PCAs in the grid nodes... (open question)
Another possible comparison is with some older spectral DR techniques, working by stitching local 2D projections (back to a unique scatterplot then...).
questions: questions: See above
limitations: limitations: The limitations appear quite clearly in the text, although the first limitation at the end of section 1 does not really make sense (t-SNE uses graph but is essentially DR and thus assumes/uses coordinates (or feature vectors)).
paper_summary: paper_summary: The paper proposes a hierarchical data visualisation method that is a combined use of oblique decision tree and PCA over leaf nodes. In experiments, the authors demonstrate the visualisation results of the proposed method for a few datasets and compare the reconstruction errors with standard PCA.
strength_weakness: strength_weakness: Very limited originality. The proposed method works by firstly training an oblique decision tree and then applying PCA over examples assigned to each leaf nodes. This is a straightforward use of very classical existing methods. It hardly qualifies as a new method…

Since this paper contributes to hierarchical data visualisation, their experiments should focus on comparison with the use of existing hierarchical DR for data visualisation.

Most theoretical results/analysis on PCA tree come from simple applications of classical PCA results. The three theorems do not seem to present any significant result, for which I invite the authors to comment on.

The method description is quite hard to follow, which appears like a piece of overly complicated writing for simple method.
questions: questions: How is the oblique tree trained in their experiments?

I invite the authors to comment on why the results from the three theorems are important.
limitations: limitations: No discussion on limitation.
paper_summary: paper_summary: This study proposes PCA trees as a hierarchical data visualization tool. PCA trees hierarchically partition the data space using hyperplanes, with the partition pattern represented by a binary tree. PCA is then applied to each leaf node of this binary tree. The parameters of a PCA tree consist of coefficients defining the hyperplanes for splitting at internal nodes and projection matrices for PCA at leaf nodes. The function representing the fitting of the PCA tree to the data satisfies a separability condition with respect to its parameters. Therefore, we can perform effective alternating optimization. The characteristics of PCA trees are also verified through visualization of real-world datasets such as Fashion MNIST.
strength_weakness: strength_weakness: PCA trees divide the data space into multiple disjoint subsets and project the data points in each subset into a different low-dimensional space. Therefore, it is debatable whether this truly achieves dimensionality reduction in the strict sense. The description of ''dimensionality reduction'' in the abstract and introduction may be misleading. It might be more appropriate to view this as a data visualization technique. Furthermore, there are doubts about whether the visualization provided by PCA trees is intuitively easy to understand. Analysts need to conceptualize and integrate complex hierarchical partitions of high-dimensional space along with projections in various directions. In particular, intuitively grasping the positional relationships of data points between different nodes might be challenging. However, given that the assessment of visualization quality is largely subjective, PCA trees may still have value when used complementarily with other visualization techniques.
questions: questions: What strategies or techniques are employed to avoid converging to local optima?
limitations: limitations: The paper acknowledges that the solution may converge to local optima. Further discussion on strategies to mitigate this issue would enhance the work.
paper_summary: paper_summary: This paper proposes a dimensionality reduction approach that comprises a tree of local PCA projections. The authors make the following contributions:
1. A mathematical description of their approach.
2. An optimisation algorithm and pseudocode to fit their method to a training set, together with the computational complexity of inference and training of the model giving evidence of the excellent properties of the approach.
3. Experiments with benchmark datasets of various types confirming the claims around scalability and computational complexity, the reconstruction error, and examples of using the approach to explore and interpret datasets.
strength_weakness: strength_weakness: 1. There are several hyperparameters for the approach, most notably $\lambda$ and $\Delta$. However, the paper does not describe how to set these parameters nor the sensitivity of the approach to different values.
2. At the moment the approach builds out a full tree to depth $\Delta$, but it would be better if the tree could be deeper in particular areas depending on the data in the associated region. In fact, several nodes would be improved in the example in fig. 4 if the tree were deeper - e.g. L3, L5, L8 and L11.
3. I couldn't see the parameter settings for the experiments with t-SNE and UMAP.
questions: questions: How to set the hyperparameters? What is the sensitivity to hyperparameter values? How significant is the effect of local minima to this optimization problem?

Graph-based similarity approaches have several advantages and disadvantages compared to distance-based methods like the approach in this paper. Given that the proposed approach sub-divides data space to explore local structure, is there an opportunity to unify the two approaches in the leaves?
limitations: limitations: The paper describes limitations of the proposed approach, although to some extent these are limitations for all dimensionality reduction / visualisation methods. The paper does not speak much to the effect of local minima. The paper does not explore the sensitivity to hyperparameters. It might also be interesting to explore dimensionality reduction in synthetic datasets where there is known structure, e.g., the noisy swiss roll datasets, box datasets and others, for example used in Lee and Verleysen, Nonlinear Dimensionality Reduction (2007) as an approach to compare a wide set of methods.

Comments: comment: Following your suggestion, we constructed some reasonable hierarchical DR baselines to compare with in the Fashion MNIST dataset (see PDF file). In all cases, we first construct a tree, then fix it and train PCA on each leaf (i.e., on the training points of each leaf). The tree is learned as follows:
- Via hierarchical clustering.
- As a "random median oblique tree", i.e., we construct a complete oblique tree of given depth where each decision node's hyperplane has a random direction and the bias selected to partition its dataset 50%/50%. This is actually how we initialize the tree in our algorithm.
- As a $k$-means recursive bipartition, i.e., we construct a complete oblique tree of given depth where each decision node's hyperplane is obtained by running $k$-means with $K=2$ clusters.
- We also tested the case $L=0$ of our PCA tree, i.e., to train a tree autoencoder of latent dimension zero, then fit PCA to the leaves (with $L=0$). This (not shown in the PDF file) is guaranteed to perform worse than the PCA tree, obviously. It performs similarly to the $k$-means recursive bipartition.

As we noted in our earlier reply, all these approaches perform considerably worse than our PCA tree, in both reconstruction error and 2D scatterplots. In the PDF file, among other things, we report the reconstruction error for the points in the training set (which is an objective measure), for different numbers of leaves, and also show the PCA scatterplots in the leaves of the tree. This confirms the fundamental novelty and advantage of our approach: the ability to optimize not just the PCA mapping at the tree leaves, but crucially the tree itself. This adjusts the set of training points in each leaf and its PCA so the error is minimized. Learning the tree on its own and fitting PCA afterwards results in a much worse partition.

The PDF is in the following anonymized link: https://anonymous.4open.science/api/repo/rebuttal-8298/file/rebuttal.pdf?v=f61e309a

Below, we include the tabular and text parts of it (but not the scatterplots).

| Number of leaves | PCA tree | agglomerative clustering | median tree | $k$-means bipartition |
| - | - | - | - | - |
| 8 | **22.07** | 36.22 | 30.80 | 24.19 |
| 16 | **19.57** | 29.87 | 28.72 | 21.43 |
| 128 | **15.40** | 23.98 | 23.49 | 16.38 |

We compare our PCA tree with several two-stage approaches, where we first learn a tree and then apply PCA to each leaf. The PCA tree consistently shows significantly lower reconstruction error across varying numbers of leaves (Table 1). 

**Hierarchical Clustering**. We use agglomerative clustering to build a dendrogram and then apply PCA within each cluster. This method has notable drawbacks: it lacks an out-of-sample mapping, requires substantial computation time due to the pairwise distance calculations, and results in a poor tree (Fig. 1): 39379 samples (more than half of the whole dataset) end in a leaf, while many other leaves are nearly empty, containing only 1-2 points. 

**Median tree**. We split the data into two halves via a random-direction hyperplane at each decision node, resulting in a balanced partition at the leaves. This tree serves as initialization for our algorithm, so it is predictably worse. 

**$k$-means bipartition**. We construct a binary oblique tree by recursively running the $k$-means algorithm at each decision node, partitioning the data into two child nodes. The decision node hyperplanes (not shown) are not sparse, so they are harder to interpret than in our PCA tree. This works better and is able to identify some of the classes (since, after all, it is a form of clustering), but from a DR point of view it is not clear what meaningful patterns it reveals, particularly when it mixes multiple classes in the same leaf (see L7, L8, L11, L15, L16 in Fig. 2). It is important to note that in an unsupervised dataset, class labels are not available to color data points, making these scatterplots less insightful.
Thank you for your response to the questions in my review and the clarification of parameter settings. What you say is all reasonable. After considering the response, I will maintain my current scoring.
Thank you for clarifying the method and providing other explanations.  I  misunderstood the part that you jointly learn the tree (decision nodes) and the projection parameters.  

Regarding to comparison with other hierarchical methods, the point is to show the difference between the visual patterns captured by those methods and yours, even the compared ones are “poor” or work by completely different ways from yours.  Intuitively, there are straightforward ways of producing a hierarchical visualization result, in a similar fashion to those in your Fig 4.  For instance, you can run a hierarchal clustering algorithm and then visualise each cluster. Also, you can obtain hierarchy by grouping feature dimensions. Another way is by what I thought you did, which is to train an oblique decision tree first and then run PCA.  There are works pursuing hierarchical visualisation aspects from different application domains, for instance,

*  https://www.sciencedirect.com/science/article/abs/pii/S2214579619302102?fr=RR-2&ref=pdf_download&rr=8b17b9a228b7776b

*  https://www.nature.com/articles/s41587-021-01186-x

I think more research effort on investigating whether your proposed approach could reveal more interesting or under explored data patterns than naïve or existing approaches is needed.

Given that I under-estimate the novelty but still don’t find it highly novel,  and that I find it lacks experiments on demonstrating advantage over other ways to produce a hierarchical data visualization, I will increase my score to 4, but I am reluctant to give a higher score.
Thank you for your response.

Regarding the initialization method to avoid local optima, while it doesn't seem particularly novel, I think it's a reasonable approach. If the paper is accepted, I believe this should be included in the final manuscript. However, I don't think it's significant enough to improve the overall evaluation of the paper.

Concerning the points I raised as Weaknesses, I understand your perspective, and I agree that there is some merit to what you're saying. However, this aspect inevitably involves subjectivity, making it difficult to draw a clear conclusion. That's precisely why I included these points under Weaknesses and not under Questions. As I mentioned in my initial review, I believe PCA trees may have value when used to complement other visualization techniques. Your definition of the latent space dimension was helpful in clarifying the discussion.

After considering all aspects, I've decided to maintain my original evaluation.