Unsupervised Manifold Alignment with Joint Multidimensional Scaling
Authors: Dexiong Chen
Bowen Fan
Carlos Oliver
Karsten Borgwardt
Venue: ICLR 2023 poster (2023)
Keywords: unsupervised manifold alignment
multidimensional scaling
optimal transport
graph matching

Abstract: We introduce Joint Multidimensional Scaling, a novel approach for unsupervised manifold alignment, which maps datasets from two different domains, without any known correspondences between data instances across the datasets, to a common low-dimensional Euclidean space. Our approach integrates Multidimensional Scaling (MDS) and Wasserstein Procrustes analysis into a joint optimization problem to simultaneously generate isometric embeddings of data and learn correspondences between instances from two different datasets, while only requiring intra-dataset pairwise dissimilarities as input. This unique characteristic makes our approach applicable to datasets without access to the input features, such as solving the inexact graph matching problem. We propose an alternating optimization scheme to solve the problem that can fully benefit from the optimization techniques for MDS and Wasserstein Procrustes. We demonstrate the effectiveness of our approach in several applications, including joint visualization of two datasets, unsupervised heterogeneous domain adaptation, graph matching, and protein structure alignment. The implementation of our work is available at https://github.com/BorgwardtLab/JointMDS.

Published as a conference paper at ICLR 2023
UNSUPERVISED MANIFOLD ALIGNMENT WITH JOINT MULTIDIMENSIONAL SCALING
Dexiong Chen, Bowen Fan, Carlos Oliver & Karsten Borgwardt
Department of Biosystems Science and Engineering, ETH Zürich, Switzerland SIB, Swiss Institute of Bioinformatics, Switzerland (firstname. astnaamae absse.ethz.
ABSTRACT
We introduce Joint Multidimensional Scaling, a novel approach for unsupervised manifold alignment, which maps datasets from two different domains, without any known correspondences between data instances across the datasets, to a com- mon low-dimensional Euclidean space. Our approach integrates Multidimen- sional Scaling (MDS) and Wasserstein Procrustes analysis into joint optimiza- tion problem to simultaneously generate isometric embeddings of data and learn correspondences between instances from two different datasets, while only quiring intra-datasee pairwise dissimilarities as input. This unique characteristic makes our approach applicable datasets without access to the input features, such as solving the inexact graph matching problem. We propose an alternating optimization scheme to solve the problem that can fully benefit from the opti- mization techniques for MDS and Wasserstein Procrustes. We demonstrate the effectiveness of our approach in several applications, including joint visualization of two datasets, unsupervised heterogeneous domain adaptation, graph matching, and protein structure alignment. The implementation of our work is available https: / ////ttttuust// com Btt///ogggwaattaee Lab/ Lttt//aaa///oo/iiio JointMDS.
INTRODUCTION
Many problems in machine learning require joint visual exploration and manipulation of multiple datasets from different (heterogeneous) domains, which is generally preferable first step prior to any further data analysis. These different data domains may consist of measurements for the same samples obtained with different methods technologies, such single-cell multi-omics data in bioinformatics (Demetci et al., 2022; Liu et al., 2019; Cao & Gao, 2022). Alternatively, the data could be comprised of different datasets of similar objects, such as word spaces of different languages in natural language modeling representing related objects such as procedure recommendation in biomedicine (Xu et al., 2019b). There are two main challenges in joint exploration of multiple datasets. First, the data from the heterogeneous domains may be high-dimensional or may not possess input features but rather only dissimilarities between them. Second, the correspondences between data instances across different domains may not be known a priori. We propose in this work to tackle both issues simultancously while making few assumptions on the data modality.
Livare--Melis et al., 2019; Grave et al., 2019), graphs
To address the first challenge, for several decades many dimensionality reduction methods have been proposed provide lower-dimensiona. embeddings of data. Among them, multidimensional scaling (MDS) (Borg & Groenen, 2005) and its extensions (Tenenbaum et al., 2000; Chen & Buja, 2009) are widely used ones. They generate low-dimensional data embeddings while preserving the local and global information about its manifold structure. One of the key characteristics of MDS is the fact that it only requires pairwise similaaiisss as input rather than specific data features, which makes it applicable to problems whose data does not have access to the input features, such as graph node embedding learning (Gansner et al., 2004). However, when it comes dealing with multiple datasets from different domains at the same time, the subspaces learned by MDS for different datasets are not naturally aligned, making it not directly applicable for joint exploration.
Dexiong Chen and Bowen Fan contributed equally.

---

Published as a conference paper at ICLR 2023
One well-known method for aligning data instances from different spaces is Procrustes analysis. When used together with dimensionality reduction, it results in a manifold alignment method (Wang & Mahadevan, 2008; Kohli et al., 2021; Lin et al., 2021). However, these approaches require prior knowledge about the correspondences between data instances across the domains, which limits their applicability in many real-world applications where this information is hard or expensive to obtain. Unsupervised manifold alignment approaches (Wang & Mahadevan, 2009; Cui et al., 2014) have been proposed overcome this limitation by aligning the underlying manifold structures of two datasets with unknown correspondences while projecting data onto a common low-dimensional space.
In this work, we propose to combine MDS with the idea of unsupervised manifold alignment to simultaneously embed data instances from two domains without known correspondences to a common low-dimensional space, while only requiring intra-dataset dissimilarities. We formulate the problem as a joint optimization problem, where we integrate the stress functions for each dataset that measure the distance deviations and adopt the idea of Wasserstein Procrustes analysis (Alvarez-Melis et al., 2019) to align the embedded data instances from two datasets in a fully unsupervised manner. We propose to solve the resulting optimization problem through an alternating optimization strategy, resulting in an algorithm that benefit from the optimization techniques for solving each individual sub- cuooproblem. Our approach, named Joint MDS, allows recovering the correspondences between instances across domains while also producing aligned low-dimensional embeddings for data from both domains, which is the main advantage compared to Gromov-Wasserstei (GW) optimal transport (Mémoli, 2011; Yan al., 2018) for only correspondence finding. We show the effectiveness of joint MDS in several machine learning applications, including joint visualization of two datasets, unsupervised heterogeneous domain adaptation, graph matching, and protein structure alignment.
2 RELATED WORK
We present here the work most related to ours, namely MDS,, unsupervised manifold alignment and optimal transport (OT) for correspondence finding.
Multidimensional scaling and extensions MDS is one of the most commonly used dimension- ality reduction methods that only require pairwise similariisssss between data instances as input. Classical MDS (Torgerson, 1965) was introduced under the assumption that the dissimilarity is an Euclidean distance, which has an analytic solution via SVD. As an extension of classic MDS, metric MDS consists in learning www-dimensional embeddings that preserve any dissimilarity by minimizing stress function. Several extensions of MDS have also been proposed for various practical reasons, such as non-metric MDS (Agarwal et al., 2007), Isomap (Tenenbaum et al., 2000), local MDS (Chen & Buja, 2009) and so on. MDS has also been used for graph drawing (Gansner et al., 2004) by producing node embeddings using shortest path distances on the graph. Our approach can be seen as an important extension of MDS to work with multiple datasets.
Unsupervised manifold alignment While (eemi-)supervised manifold alignment methods (Ham al., 2005; Wang & Mahadevan, 2008; Shon et al., 2005) require at least partial information about the correspondence across domains, unsupervised manifold alignment learns the correspondence directly from the underlying structures of the data. One of the earliest works for unsupervised manifold alignment was presented in (Wang& & Mahadevan, 2009), where a similarity metric based on the permutation of the local geometry was used to find cross-domain corresponding instances followed by a non-linear dimensionality reduction. A similar approach was also adopted in (Tuia et al., 2014) with a graph-based similarity metric. A more generalized framework, named GUMA (Cui al., 2014), was proposed as an optimization problem with three complex terms to project data instances via a linear transformation and match them simultaneously. As an extension of (Cui al., 2014), UnionCom (Cao et al., 2020) introduced geodesic distance matching instead of the kernel matrices, to deal with multi-mode data bioinformaties in particular. Additionally, generative adversarial networks and the maximum mean discrepancy have also been used to find correspondences jointly with dimensionality reduction for unsupervised manifold alignment (Amodio & Krishnaswamy, 2018; Liu et al., 2019). Our approach differs from these previous approaches as it makes few assumptions on the data modality but only requires intra-dataset dissimilarities as input.
Optimal transport for correspondence finding OT (Peyré et al., 2019) is powerful and flexible approach to compare two distributions, and has a strong theoretical foundation. It can find a soft-correspondence mapping between two sets of samples without any supervision. With the
2

---

Published as a conference paper at ICLR 2023
R"
Zi
dej
Figure 1: Joint Multidimensional Scaling maps data from two domains X' and 2' to a common low-dimensional space IR while preserving the pairwise intra-domain dissimilarities.
recent advances in computational algorithms for OT (Cuturi, 2013). it has become increasingly popular in various machine learning fields. However, one major drawback of OT is that it was originally designed compare two distributions in the same metric space: the cost function cannot be properly defined between samples in different spaces. This limitation makes it unsuitable for tasks involving data from different domains. As an important extension, the
(Mémoli, 2011) distance has been proposed to overcome this limitation, by comparing two pairwise intra- domain (dis)similarity matrices directly. This modification makes the GW distance invariant many geometric transformations on the data, such as rotation or translation. GW has thus been successfully applied several tasks involving datasets from heterogeneous domains, such as heterogencous domain adaptation (HDA) (Yan et al., 201 8; Titouan et al., 2020), graph matching and partitioning (Xu al., 201 2b; Vayer et al., 2019), and integration of single-cell multi-omics data (Demetei al., 2022). As an alternative of GW. OT methods with global invariances have also been recently proposed for unsupervised word translation tasks in NLP Selvarez-Melit et al., 2019; Grave et al., 2019), and for more general purpose (Jin et al., 2021). In addition to discovering correspondences, our approach additionally produce oiw-dimensional embeddings of data from both domains, which could be useful for visualization and further analysis.
BACKGROUND
In this section, revisit the metric MDS problem and Wasserstein Procrustes, two main building blocks of our method.
Notation We denote by D E Raxn the pairwise dissimilarities of dataset of elements (in an unknown space X') and by d: Rdd x Rd + RR+ a distance function which is typically the Euclidean distance. For two sets of data points Z E Raad and Z' E R xd of respective n and n' elements in R4, we denote by d(z, Z') the pairwise distance matrix in R2xx' such that its (i. entry is equal to dZz,, Zz). We denote by || ||r the Frobenius norm of matrix and (r,) the associated inner- mnoduct. We denote by A, the probability simplex with n bins A, == {a E IT E aj 11.
Metric multidimensional scaling
Given a dissimilarity matrix D E Rax= with (i. entry of samples in unknown space X,, MDS consists of finding
denoted as dzy. representing T e igaxd that best preserves the pairwise dissimilarities by solving coordinates == (Z),- .......... ),,) the following weighted stress minimization problem:
min stress (2, D, W) ZeRaxa
way weyldi d(zi, 2;))
for given weight matrix W == (w.)
RRTX" Despite being non-convex problem, it can be efficiently solved by an iterative majorization algorithm, named SMACOF (scaling by majorizing complicated function) (Borg & Groenen, 2005). This algorithm could be used individually to map two datasets to two fow-dimensional Euclidean spaces. However, both the alignment of the two subspaces and the correspondences between data instances are not known a priori. It is thus meaningless to work jointly with the embeddings of both datasets directly.
Wasserstein Procrustes OT is widely used approach for finding correspondences between feature vectors from the same space in an unsupervised fashion. Its strong theoretical foundations and fast algorithms make it a natural candidate align the distributions of the embedded data obtained with MDS. Specifically, let a E A,, and b E A,,' be the weights of the discrete measures Σ, a;,,, and
Σ, by0;; with respective location Z = (Z1, (2....., 2T.)T T E RRa and Z' = (2/...., 22,) E R
3

---

Published as a conference paper at ICLR 2023
For cost function R R + R4, Kantorovich's formulation of the OT problem between the two discrete measures is defined as
min (P, F, (ell(a,b)
(2)
where C E Raxn' represents the pairwise costs whose entries are Ciy := c(zi, zj) and II b) is the polytope of the admissible couplings between a and
(I b) (P IR P Pl-aaP b}.
In practice, a and b are uniform measures as we consider the mass be evenly distributed among the data points. In particular, if с is the squared Euclidean distance such that c(z, = Iz z the solution of Eq. (2) defines a squared distance on the set discrete measures, called the squared Wasserstein distance, which we denote by W (Z, Z') below.
Naive application of OT to find correspondences between the embedded data obtained with two MDSs for two datasets can easily fail since the two spaces of Z and Z' not coherent, in other words, there is no meaningful notion of distance between them. In order to jointly align the two spaces and find the correspondences, Alvarez-Melis al. (2019) propose to jointly optimize linear transformation f from a pre-defined invariance class F::
min W (fZ),, Z') = min min (P, (()() FEF
PE fEF
They provide solutions of this problem for invariances defined by linear operators with bounded norm := {0 E gdxd| ||O||p I kp), where || 1lp is the Schatten f--norm. In particular, when p = DO and hp = 1, we have = Od. the orthogonal group of matrices Rdxdd and one recovers the Wasserstein Procrustes problem (Grave et al., 2019).
4 JOINT MULTIDIMENSIONAL SCALING PROBLEM
We introduce our new joint MDS problem which can jointly embed two datasets in common space.
4.1 CORRESPONDENCE DISCOVERY WITH JOINT MULTIDIMENSIONAL SCALING
The Joint MDS problem consists of jointly mapping data points from two different domains into a common Euclidean space. Following the discussion in Section 3, the two spaces obtained with two individual MDSs for each dataset are not necessarily aligned and the correspondences between data instances are unknown. To address both issues, we leverage Wasserstein Procrustes to algin the embedded data from two domains.
Specifically, combine the stress minimization and the Wasserstein Procrustes to define the overall objective function for our Joint MDS problem. Given matrices D R= and D' R=''xw xn' representing the pairwise dissimilarities of two collections, we formulate the Joint MDS as the following optimization problem:
min ZER"Xa "xd
stress stress(Z.D, (Z.D, W) + stress '',,'' w') + 2XP,, & (ZO, (d0.)))). PeIl(a.)).0EO.
(5)
where W and W' weights for the pairwise dissimilarities generally equal 1/n² and 1/n²² and a and b refer to the sample weights as defined above. This objective function exhibits a simple interpretation: the two stress function terms measure the distance deviation for each domain while the last term measures the correspondences between instances from the two domains. It is worth noting that the stress functions are rotation-invariant i.e. if Z and Z' are solutions respectively minimizing the two stress functions, their rotations are also solutions. Thus, using the Wasserstein Procrustes with global geometric invariances (in O₂) instead of the original OT could find better correspondences. An illustration of Joint MDS is shown in Figure 1.
4.2 CHOICE OF THE DISSIMILARITIES
In general, any distance could be used to compute the dissimilarities if they are not provided directly with the dataset. For instance, the Fuclidean distance is natural choice. As extension of the
4

---

Published as a conference paper at ICLR 2023
Euclidean distance in Riemannian geometry, the geodesic distance (Tenenbaum et al., 2000) often captures better the local geometric structure of the data distribution. However, the geodesic distance is generally hard to compute exactly as it requires full knowledge about the data manifold. Fortunately, can be efficiently estimated approximately by constructing a k-nearest neighbor graph and computing the shortest path distance the graph. Similar Isomap (Tenenbaum et al., 2000), a variant of MDS computed on the geodesic distance, we also observe better performance for joint MDS when using geodesic distances, especially in unsupervised heterogeneous domain adaptation tasks.
4.3 OPTIMIZATION
The above optimization problem is non-convex, which is hard to solve in general. Here, we propose an alternating optimization scheme by solving two subproblems that have already been studied previously. Specifically, we show that when P and O are fixed, the Joint MDS problem (5) becomes a simple weighted MDS problem. On the other hand, when fixing and Z', the problem (5) is reduced Wasserstein Procrustes problem. As consequence, we can easily leverage optimization techniques developed for each of the sub-problems respectively.
Weighted MDS problem We first observe that the distance is invariant to the orthogonal transfor- mations from Oa. Thus, for fixed P and O, the problem amounts to minimizing the following stress function (with change of variable ZO):
n
min ZeRax4.Z'ER xd
wiylds - d(zi, 2;))
- 2(44, +X 2f (2,
v1===1
(6)
where the last term can be rewritten
n' Σ
n Σ
2A(P, (Z,
d(2;)'')))))) d( + Σ
XP
AT v.
(0-
d(z'= 2;))²
/==,==
This is equivalent to solving the MDS problem with a stress function stress D, W) where
Z Z'
D:
D
D = O D' W
W
AP
APT
W'
This problem can be solved with the SMACOF algorithm, which simply relies the majorization theory without making any assumption whether D has distance structure. The majorization principle essentially consists in minimizing a more manageable surrogate function g(X, Z) rather than directly minimizing the original complicated function h(X) == stress X, D. W). This surrogate function g(X, Z) is required to (i) be majorizing function of h(X), i.e. h(X) < gXX, Z) for any
X E nx and (ii) touch the surface at the so-called supporting point Z, i.e. h(Z) = (Z, Z). Now, let the minimum of gX, Z) over X be attained at X* The above assumptions imply the following chain of inequalities:
hZZ) = o(, g X' Z) 2 h(X').
called the sandwich inequality. By substituting with X and iterating this process, we obtain a sequence (Zo..... Ze) that yields non-increasee in-iiccreasing sequence of function values. (Borg & Groenen, 2005) showed that there is indeed such a majorizing function for the stress function, which is detailed in Section A of the Appendix. Through the lens of majorization, (De Leeuw, 1988) have shown that the above sequence converges to a local minimum of the stress minimization problem with linear convergence rate. Local minima are more likely occur in low-dimensional solutions while being less likely to happen in high dimensions (De Leeuw & Mair, 2009; Groenen & Heiser, 1996). The SMACOF algorithm based on stress majorization summarized in Section A of the Appendix.
Wasserstein Procrustes problem For fixed embeddings Z and Z', the first two terms are independent of P and O and therefore this step consists in solving a Wasserstein Procrustes problem:
min OcOa. (a)
(P, d (20, Z') & (P)),
(8)
where we introduced an entropic regularization term which makes the objective strongly convex over P and thus easier to solve. The resulting problem be solved with alternating optimization
5

---

Published as a conference paper at ICLR 2023
over P and O studied in (Alvarez-Melis et al., 2019). Specifically, when fixing O, the above problem is a classic discrete OT problem with entropic regularization that can be efficiently solved with, e.g., the Sinkhorn's 's algorithm. More details about Sinkhorn's 's algorithm can be found in the Appendix. When fixing P. the problem is equivalent to a classic orthogonal Procrustes problem: maxogo, (O, Z PZ')r, r. whose solution is simply given by the singular value decomposition of P'Z' Solving the Wasserstein Procrustes problem offers correspondences between two sets of embeddings of the same dimensions even their spaces are not aligned, which is particularly suitable here. By alternating between these two steps, we obtain our final algorithm for solving Joint MDS.
Overall algorithm Z and Z' could be simply initialized with two individual SMACOF algorithms with D and D'. However, for some hard matching problems, this might not offer good initialization of the Wasserstein Procrustes problem. In this case, GW could be used as a convex relaxation of the Wasserstein Procrustes problem (Alvarez-Melis et al., 2019; Grave et al., 2019), which provides better initialization for the coupling matrix P. In addition, similar to the observations in (A (Alvarez- Melis et al., 2019), we also empirically find that annealing the entropic regularization parameter in Wasserstein Procrustes leads to better convergence. When using the solution of GW as initialization, we also find that annealing is useful. The full algorithm summarized in Algorithm 1.
Algorithm 1 Joint Multidimensional Scaling
1: Input: distances D E R=*= and D' E 12n'xn', weights W E R2nx" and W E IR" matching penalty A, entropic regularization E, max iterations T.
2: Output: low-dimensional embeddings Z, Z' E RR*d optimal coupling P E Raxn'
3: Set Z MDS(D, W) and Z' MDS(DD, W') with SMACOF using random initialization.
4: Set D in Eq (7) with D and D'.
5: for / = 1, ,,...,,''' do
6: Update P and O by solving Wasserstein Procrustes in Eq. (8) between Z and Z' using s.
7: Update
ZO and Z in Eq (7).
8: Update W in Eq (7) using P, X W and W
9: Update Z, Z' MDS(D, W) using SMACOF with 2 initialization.
10:
end for
Complexity analysis Let us denote by N n + n'' the total number of data points of both domains. The complexity of SMACOF iteration is the complexity of Guttman Transform, which equals to O(N2) (De Leeuw, 1988). On the other hand, the complexity of one iteration of the alternating optimization method for solving the Wasserstein Procrustes is Odd + N2) (Alvarez-Melh) et al.. 2019) where the first term corresponds to the complexity of SVD and the second term corresponds to the Sinkhorn's ''s algorithm with fixed number of iterations. While a linear convergence of the SMACOF algorithm was shown in (De Leeuw, 1988), we empirically observe that fixed number of iterations is sufficient for the convergence of SMACOF as well for solving Wasserstein Procrustes at each iteration of the Joint MDS algorithm 1. Therefore, for a total number of iterations T,, the complexity of the full algorithm ( + 2)) Despite the quadratic complexity, our algorithm could benefit from the acceleration of GPUs most of the operations in SMACOF and Sinkhorn's 's algorithm are matrix multiplications.
APPLICATIONS
5.1 JOINT VISUAL EXPLORATION OF TWO DATASETS
We show that our Joint MDS can be used to jointly visualize two related datasets by simultaneously mapping two datasets to a common 2D space while preserving the original manifold structures.
Experimental setup Here, we consider 3 (pairs of) synthetic datasets and 3 (pairs of) real-world datasets. The synthetic datasets respectively consist of a bifurcation, a Swiss roll and a circular frustum from (Liu et al., 2019). Each synthetic dataset has 300 samples that have been linearly projected onto 1000 and 2000 dimensional feature spaces respectively and have been injected white noise (Liu et al., 2019). The 3 real-world datasets consist of 2 sets of single-cell multi-omics data (SNAREseq and scGEM) (Demetci et al., 2022) and a subsampled digit dataset (MNIST-USS USSP (Deng, 2012; Hull, 1994)). We compute the pairwise geodesic distances of each dataset as described in

---

Published as a conference paper at ICLR 2023
HAN 90S Corpseene)
JAKE TES Conpenee
JIREMOS COPEERNOT
2015
Jo18
Joint visual exploration of two datasets. Top row: 3 synthetic datasets (bifurcation, Swiss roll, and circular frustum). Bottom row: real-world datasets (SNAREseq. scGEM and MNIST-USPS) Different colors represent datasets and different markers represent different classes except for MNIST. USPS, in which different colors represent different classes of digits from 0 9.
Figure 2:
Section 4.2. We fix the number of components d to visualize datasets in R.2 We fix the matching penalization parameter À to 0.1 for all the datasets. More details about the datasets and parameters and additional results can found Section B.2 of the Appendix.
Results We show in Figure 2 the results for all the considered datasets. Joint MDS successfully embedded each pair of datasets to common 2D space while preserving the original manifold structures such as proximity of similar classes. In contrast, some other recently proposed methods such as (Demetci et al., 2022) only map one dataset onto the space of the other dataset through barycentric projection from GW, which could lose information about the manifold structure of the dataset before projection. As we also have access to the ground truth of the 1-to-1 correspondences for the synthetic and single-cell datasets, we can quantitatively assess the alignment learned by Joint MDS, by using the fraction of samples closer than the true match (FOSCTTM) metric (Liu et al., 2019). For given sample from dataset, compute the fraction of samples from the other dataset that are closer to it than its true match and we average the fraction values for all the samples in both datasets. lower average FOSCTTM generally indicates better alignment and FOSCTTM would be zero for a perfect alignment as all the samples are closest to their true matches. We show in Table the results FOSCTTM compared several baseline methods that are specifically proposed for aligning single-cell data. Joint MDS can also be used to align and visualize two sets of point clouds with a specific application for human pose alignment. provide more details and results in Section B.2.5 of the Appendix due to limited space.
Table 1: Average FOSCTTM comparison (lower is better).
Method
Bifurcation
Swiss roll
Circular frustum
SNAREseq
scGEM
MMD-MA (Liu et al., 2019)
12.44
3.27
1.25
15.00
20.14
UnionCom (Cao et al,, 2020)
15.20
SCOT (Demetei al., 2022)
8.66
2.16
0.88
15.00
19.80
Joint MDS (d=2)
11.15
0.96
0.90
17.18
20.42
Joint MDS (d=16)
7.56
0.58
0.87
15.59
18.54
5.2 UNSUPERVISED HETEROGENEOUS DOMAIN ADAPTATION
We use the same datasets as in Section 5.1. To solve the HDA problem, we first solve the joint MDS problem between the two pairwise geodesic distance matrices, which provides embeddings for instances from both domains in the same space. Then, we train a k-NN classifier (k is fixed to 5) on the source domain and estimate the labels on the target domain without any further adaptation. For the
7

---

Published as a conference paper at ICLR 2023
more complex MNIST-USPS dataset, we use linear SVM classifier with the regularization parameter set to instead, which results in better prediction accuracy. We follow the same parameter selection as in (Demetci et al., 2022) and compare
method to the state-of-the- including SCOT (Demetci et al., 2022) and EGW (Yan et al., 2018), which are both based on entropic regularized Gromov-Wasserstein The classification accuracies are shown in Table 2. While all the methods work well on the easier datasets, our Joint MDS outperforms GW-based baselines on some harder tasks such as MNIST-USPS.
unsupervised HDA methods
Table 2: Classification accuracy for unsupervised domain adaptation.
Method
Bifurcation
Swiss mll
Circular frustum
SNAREseq
scGEM
MSPST-USPS
SCOT (Demetci et al., 2022)
93.7
97.7
95.7
98.2
57.6
26.7
EGW (Yan al., 2018)
95.7
99.3
94.7
93.8
62.7
43.1
Joint MDS (d=2)
96.0
99.3
94.0
85.5
64.4
15.0
Joint MDS (d=16)
96.7
99.3
94.7
94.7
72.9
60.2
5.3 GRAPH MATCHING
Data We apply the proposed method on two real-world datasets for the graph matching task which were used in (Xu et al., 2019b;a), The first dataset was collected from MIMIC-III (Johnson et al., 2016). which is one of the most commonly used critical care databases in healthcare related studies. This dataset contains more than 50,000 hospital admissions for patients and each is represented as sequence of ICD (International Classification of Diseases) codes for different diseases and procedures. The interactions in the graph for disease (resp. procedure) procedures) appearing in the same admission. The final two constructed graphs involve 11,086 admissions, one with 56 nodes of diseases and the other with 25 nodes of procedures. The procedure recommendation could be formulated as the problem of matching these two graphs. The second dataset contains the yeast has 1,004 proteins together with 4,920 high-confidence interactions. The graph to match is its noisy version, which was built by adding 7% lower-confidence interactions with g E (5, 15, 253.
constructed with the diseases (resp.
interaction (PPI) networks of yeast. The original PPI network of
Experimental setup To solve the graph matching problem, we use the shortest path distances on the graph as the input of our Joint MDS similarly to graph drawing methods (Gansner al., 2004; Kamada et al., 1989), and use the coupling matrix P as the matching prediction. We compare our method with some recent graph matching methods, including MAGNA++ (Vijayan et al., 2015), HubAlign (Hashemifar & Xu, 2014), KerGM (Zhang et al., 2019) and GWL (Xu et al., 2019b). In particular, GWL is a powerful graph matching algorithm based on the Gromov- Wasserstein distance that outperforms many classic methods (Xu et al., 2019a; Liu et al., 2021). Note that Joint MDS differs from GWL in three aspects: i) Joint MDS relies on SMACOF which has convergence guarantees (De Leeuw, 1988), while GWL relies on SGD that does not necessarily converge for non-convex problems; ii) it generates isometric embeddings, which could be more beneficial for tasks require distance preservation such as protein structure alignment; iii) it uses OT instead of GW in the low-dimensional embedding space to learn the correspondences. For baseline methods, we use the authors' implementations with recommended hyperparameters. For our method. we follow the same hyperparameter selection procedure and report the average accuracy over 5 different runs. We do not perform MAGNA++ and HubAlign on MIMIC-III matching task since they can only produce one-to-one node pairs
graph matching task, we use 75% of admissions for training and 25% for testing. We use training data to respectively build disease graph and procedure graph, and use the test data to build the correspondences between disease and procedure as the true recommendation. For each node in the disease graph, caleulate top 3 and top 5 recommended procedure nodes based on the coupling matrix P. We consider as a correct match if the true closest procedure node appears in the predicted top-k recommendation list. For the PPI network matching task, we calculated the node correctness (NC) as the evaluation metrics: Σn1 Σ/= P.j if Tip = 1, where P and 7 respectively corresponds to the predicted matching probability matrix and the true matching matrix.
For the diseases
Results In Table 3, the results of the two graph matching tasks are presented. Our method achieves comparable performance to GWL, while both methods outperform other classic baselines. More results and experimental details are available in Section B.4 of the Appendix.
8

---

Published as a conference paper at ICLR 2023
Table 3: Graph matching performances PP[ networks and MIMIC-III disease-proceed graphs.
Method
5%
PP115%
25%
MIMIC top
MIMIC top
MAGNA++
50.00
35.16
12.85
HubAlign
46.06
32.47
27.39
KerGM
66.14
39.04
32.17
22.67
47.86 42.14
GWT.
84.31
74.35
67.42
27.98
Joint MDS
86.44_0.33
72.61.10.62
55.3100.8
30.24.1 1.66
46.28 L 1.51
RNSD-D (p-valuem 6.1 0.1
RMSDD- (p-value-5.36ee000
Method
RMSD-D:
12-
GUMA-22 -D
28.89
palnt MOS-
GUMA
26,04
Join
Joint MDS-I 2D
18.04=00.10
Joint MDS
13.11:0.02
GUMMA
60 2D
average RMSD-D over 58 pairs of protein models. Right: RMSD-D comparison of GUMA (Cui et al., 2014) and Joint MDS, with the difference significance measured by Wilcoxon signed-rank test.
Figure 3: Protein structure alignment results. Left:
5.4 PROTEIN STRUCTURE ALIGNMENT
Protein 3D structure alignment is an important sub-task in protein structure reconstruction (Wang & Mahadevan, 2009; Badaczewska-Dee -Dwid et al., 2020) and protein model quality assessment (Pereira et al., 2021). This task consists of structurally aligning two moderately accurate protein models estimated by different methods, either experimental or computational. and eventually integrating them to generate a more accurate and high-resolution 3D model of protein folding. Here, we consider more challenging setting where the correspondences across protein models are not known. In contrast to previous work (Wang& & Mahadevan, 2009; Cui et al., 2014) where only few examples are illustrated, we use a larger and more realistic dataset, and provide a quantitative way evaluate the unsupervised manifold alignment methods for this task. Specifically, we consider here the first domain of all the proteins from CASP14 (Pereira et al., 2021) from T1024 T1113 and use the protein models predicted by the two top-placed methods in the leaderboard, namely AlphaFold2 and Baker. This results a dataset of 58 pairs of protein models, with an average number of residues equal to 198. More details about the dataset can be found in Section B.5 of the Appendix. We compare our method to a statc-of-the-ar unsupervised manifold alignment method namely GUMA (Cui et al., 2014). The parameters are fixed across the dataset and the performance is measured the average of the RMSD-D obtained by 3 different runs, defined as the root-mean mean-square deviation (RMSD-D) of residue positions and distances, given by
RMSD-D=
(() V IDD' 2')l Ei. Pu,liz, P.
where P denotes the true permutation matrix with and 1 as entries. We respectively compute embeddings in the orignal 3D space and in the 2D space, and show the results in Figure 3. We also measure the significance of the difference with the v-luuee a Wilcoxon signed-rank test. While GUMA hardly works in this challenging task with few number of RMSD-D values being small, our Joint MDS manages to produce good alignment in a large number of proteins.
CONCLUSION
We introduced Joint MDS, a flexible and powerful analysis tool for joint exploration of two datasets from different domains without known correspondences between instances across the datasets. While only requires pairwise intra-dataset dissimilarities as input, we showcased its applicability and effectiveness in several applications. We see the main limitation of our approach in its quadratic complexity of number of samples, which makes it hard to scale to very large datasets. Investigation of faster stochastic and online optimization methods for solving either MDS Wasserstein Pro- crustes (Rajawat & Kumar, 2017; Pai et al., 2019; Shamai et al., 2015; Boyarski et al., 2017) would be interesting research directions.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 813533 (K.B.). The authors would also like to thank Leslie O' O'Bray for her insightful feedback on the manuscript, which greatly improved it.
REFERENCES
Sameer Agarwal, Josh Wills, Lawrence Cayton, Gert Lanckriet, David Kriegman, and Serge Belongie. Generalized non-metric multidimensional scaling. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2007.
David Alvarez-Melis, Stefanie Jegelka, and Tommi Jaakkola. Towards optimal transport with global invariances. In International Conference Artificial Intelligence and Statistics (AISTATS), 2019.
Matthew Amodio and Smita Krishnaswamy, Magan: Aligning biological manifolds. International Conference on Machine Learning (ICML), 2018.
Aleksandra Badaczewska- Andrzej Kolinski, and Scbastian Kmiccik, Computational reconstruction of atomistic protein structures from coarse-grained models. Computational and structural biotechnology journal, 18:162-176, 2020.
Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. FAUST: Dataset and evaluation for 3D mesh registration. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Piscataway, NJ, USA, 2014, IEEE.
Ingwer Borg and Patrick JF Groenen. Modern multidimensional scaling: Theory and applications. Springer Science & Business Media, 2005.
Amit Boyarski, Alex M Bronstein, and Michael M Bronstein. Subspace least squares multidimen- sional scaling. In International Conference on Scale Space and Variational Methods in Computer Vision, pp. 681-693. Springer, 2017.
Kai Cao, Xiangqi Bai, Yiguang Hong, and Lin Wan. Unsupervised topological alignment for single-cell multi-omies integration. Bioinformatics,
(atpppeemmnt__)xi8--ii.. 2020.
Zhi-Jie Cao and Ge Gao. Multi-omics single-cell data integration and regulatory inference with graph-linked embedding. Nature Biotechnology, pp. 1-9, 2022.
Lisha Chen and Andreas Buja. Local multidimensional scaling for nonlinear dimension reduction, graph drawing, and proximity analysis. Journal of the American Statistical Association, 104(485): 209-219, 2009.
Zhen Cui, Hong Chang, Shiguang Shan, and Xilin Chen. Generalized unsupervised manifold alignment. Advances in Neural Information Processing Systems (NeurIPS), 2014.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Jan De Leeuw. Convergence of the majorization method for multidimensional scaling. Journal of classification, 132) 1988.
Jan De Leeuw and Patrick Mair. Multidimensional scaling using majorization: Smacof in Journal of statistical software, 31:1-30, 2009.
Pinar Demetei, Rebecca Santorella, Björn Sandstede, William Stafford Noble, and Ritambhara Singh. Scot: Single-cell multi-omics alignment with optimal transport. Journal of Computational Biology. 1911)3318, 2022.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web). IEEE signal processing magazine, 191 2012. 2012.
10

---

Published as a conference paper at ICLR 2023
Emden R Gansner, Yehuda Koren, and Stephen North. Graph drawing by stress majorization. In International Symposium on Graph Drawing, pp. 239-250. Springer, 2004.
Edouard Grave, Armand Joulin, and Quentin Berthet. Unsupervised alignment of embeddings with wasserstein procrustes. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.
Patrick JF Groenen and Willem Heiser. The tunneling method for global optimization in multidi- mensional scaling. Psychometrika, 61(3):529-550, 1996.
Jihun Ham, Daniel Lee, and Lawrence Saul. Semisupervised alignment of manifolds. In International Workshop on Artificial Intelligence and Statistics, 2005.
Somaye Hashemifar and Jinbo Xu. Hubalign: an accurate and efficient method for global alignment of protein protein interaction networks. Bioinformatics. 30(17):i438 1444,,44444414 2014.
Jonathan Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-5544 1994.
Kun Jin, Chaoyue Liu, and Cathy Xia. Two-sided wasserstein procrustes analysis. In International Joint Conference on Artificial Intelligence (IJCAI), 2021.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1 2016.
Tomíhisa Kamada, Satoru Kawai, et al. An algorithm for drawing general undirected graphs. Information processing letters, 31(1):7-15, 1989.
Dhruv Kohli, Alexander Cloninger, and Gal Mishne. Ldle: Low distortion local eigenmaps. J. Mach. Learn. Res., 22:282-1, 2021.
Ya- Wei Eileen Lin, Yuval Kluger, and Ronen Talmon. Hyperbolic procrustes analysis using rieman- nian geometry. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
Jie Liu, Yuanhao Huang, Ritambhara Singh, Jean-Philippe Vert, and William Stafford Noble. Jointly embedding multiple single-cell omics measurements. In Workshop on Algorithms in Bioinformatics, 2019.
Linfeng Liu, Michael C Hughes, Soha Hassoun, and Liping Liu. Stochastic iterative graph matching. In International Conference on Machine Learning (ICML), 2021.
Facundo Mémoli. Gromov-wasserstin distances and the metric approach to object matching. Foundations of computational mathematics, 11(4):477-47, 2011.
Gautam Pai, Ronen Talmon. Alex Bronstein, and Ron Kimmel. Dimal: Deep isometric manifold learning using sparse geodesic sampling. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 2019.
Joana Pereira, Adam J Simpkin, Marcus D Hartmann, Daniel J Rigden, Ronan M Keegan, and Andrei N Lupas. High-accuracy protein structure prediction in casp14. Proteins: Structure, Function, and Bioinformatics, 19(12)166871199, 2021.
Gabriel Peyré, Marco Cuturi, et Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(--6)355-607, 2019.
Ketan Rajawat and Sandeep Kumar. Stochastic multidimensional scaling. IEEE Transactions Signal and Information Processing over Networks, 3(2):360-375, 2017.
Gil Shamai, Yonathan Aflalo, Michael Zibulevsky, and Ron Kimmel. Classical scaling revisited. In Proceedings of the International Conference Computer Vision (ICCV), 2015.
Aaron Shon, Keith Grochow, Aaron Hertzmann, and Rajesh P Rao. Learning shared latent structure for image synthesis and robotic imitation. In Advances in Neural Information Processing Systems (NeurIPS), 2005.
11

---

Published as a conference paper at ICLR 2023
Justin Solomon, Gabriel Peyré, Vladimir G Kim, and Suvrit Sra. Entropic metric alignment for correspondence problems. ACM Transactions on Craphics (ToG), 35(4):1-13, 2016.
Joshua B Tenenbaum, Vin de Silva, and John C Laagffrd. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500::219-23222 2000.
Vayer Titouan, levgen Redko, Rémi Flamary, and Nicolas Courty. Co-optimal transport. Advances in 2020.
Neural Information Processing Systems, 33:
Warren Torgerson. Multidimenniioall scaling of similarity. Psychometrika, 30(4):379-333 1965,
Devis Tuia, Michele Volpi, and Gustau Camps- Valls. Unsupervised alignment of image manifolds with centrality measures. In 2014 22nd International Conference on Pattern Recognition, pp. 912-917. IBEE, 2014.
Titouan Vayer, Rémi Flamary, Romain Tavenard, Laetitia Chapel, and Nicolas Courty. Sliced gromov-wasserstiin. arXiv preprint arXiv:1905.10122. 2019.
Vipin Vijayan, Vikram Saraph, and Tijana Milenković. Magna++: maximizing accuracy in global network alignment via both node and edge conservation. Bioinformatics, 31(14):24992441, 2015.
Chang Wang and Sridhar Mahadevan. Manifold alignment using procrustes analysis. In International Conference Machine Learning (ICML), 2008
Chang Wang and Sridhar Mahadevan. Manifold aligmment without correspondence, International Joint Conference on Artificial Intelligence (IJCAI), 2009.
Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable gromov-wasserstiin learning for graph partitioning and matching. Advances Neural Information Processing Systems (NeurIPS), 2019a.
Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Grasesewasserstein learning for graph matching and node embedding. In International Conference on Machine Learning (ICML), 2019b.
Yuguang Yan, Wen Li, Hanrui Wu, Huaqing Min, Mingkui Tan, and Qingyao Wu. Semi-supervised optimal transport for heterogeneous domain adaptation. In International Joint Conference on Artificial Intelligence (ICAA)), 2018.
Zhen Zhang, Yijian Xiang, Lingfei Wu, Bing Xue, and Arye Nehorai. Kergm: Kernelized graph matching. In Advances Neural Information Processing Systems (NeurIPS), 2019.
12

---

Published as a conference paper at ICLR 2023
Appendix
This appendix provides additional background and details about methods and experiments. It is organized as follows: Section A provides further background on optimization and Section B presents experimental details and additional results.
A FURTHER BACKGROUND ON OPTIMIZATION
A.1 MULTIDIMENSIONAL SCALING AND SMACOF
This section provides further background on the metric MDS problem and details about the SMACOF algorithm. The metric MDS problem consists in solving the minimization problem of the stress function defined in Eq. (I), equivalently
min stress D, W) = Σ (di - d(zi, 2,))²
ZeRuxd
for symmetric dissimilarities. We first give the explicit expression of the majorizing function for the above stress function.
Lemma ((Borg & Groenen, 2005)). Let us define
Wijdij (xi Xj, 2z)
(X,
Σ Wijdd + Σ (xi, xj) (x;,
x)) -2
d(z;, 2z)
Then we have stress X, D, W) S g(X, Z) and stress (Zrres(,,,,, W) = g)
Proof. First, we have by Cauchy-Schwaar Schwarz inequality:
(x x,, Zi 2,} < x x,|l|zz; zjll d(xi, , x,) )))((z; 2j).
By expanding the expression of the stress and using the above inequality, obtain
stress X.D. D.D. W)
Σ
wijde +
Σ
- Σ wijdkj((xi,
Kjj
(xi
2z3
Wijd
Σ
Wijd (x;, - > Wijrlij
d(z;,
(X, Z).
is also easy to show that when X Z. we have equality.
Now the following properties of g provides simple update at each iteration of the stress majorization:
Lemma 2 ((Borg & Groenen, 2005)), g( X, Z) is quadratic function in X whose minimum is at- tained through the Guttman transform at X V (()), where V Eies Wig (xxj(e;-;))) (c;)
V with being the canonical basis of IR"", ; denoting its pseudo-inverse, and B(Z) Eiej bis(es (c)(c: -c)) with
w. /dzd,,/((z,,2)) if d(z,,, 21) 0. otherwisc.
bij
0
Proof. Through simple calculus, we have
Σ way of x;)
Σ wiylix x, tr(x T VX).
13

---

Published as a conference paper at ICLR 2023
and Σ wiydey [ X,. x;: kj)
tr(X B())
d 2z))
Since g is quadratic function X, its minimum is simply attained at its stationary point, equal
x' ==BZZZZ
Based on these two Lemmas and the principle of the majorization, we can obtain the SMACOF algorithm for solving the metric MDS problem, summarized in Algorithm 2.
Algorithm 2 SMACOF algorithm
Input: distance D E I=*"" weight W R"x", initial embeddings Zo, tolerance E, max iterations T.
2: Output: low-dimensional embeddings Z E Raxd
3: for 1....,T' do
4: Compute the Guttman transform Z; V+B(Zi-1)
5: if stress (Z,-1, D. W) stress Z, D, W) then
6: Set =
7: break
8: end if
9:
end for
A.2 WASSERSTEIN PROCRUSTES
The section provides further background and details about the Wasserstein Procrustes prob- lem (Alvarez-Melis al., 2019).
A.2.1
SINKHORN'S s ALGORITHM
When fixing O in Eq. (8), the problem amounts solving the classic discrete OT problem with entropic regularization:
min (P, P Pell(a.b)
(10)
1) and C = dzO Z') the cost matrix. This problem can be efficiently solved by the Sinkhorm's 's algorithm, which an iterative matrix scaling method to approach the double stochastic matrix. Specifically, the f-th iteration of the algorithm performs the following updates:
where ((P)==->> Ploglog -
Kv(e) (1+1)
a
K (?)
where K K C/2 Then, the matrix diag(u Kdiag(V converges to the solution of Eq. (10) This algorithm converges faster with larger as strong regularization leads to
when & tends a more convex objective (Peyré et al., 2019). As a consequence, using an annealing scheme on has been observed useful for alleviating the initialization issue when jointly optimizing P and O in (Alvarez-Melis et al., 2019). Thus, we also adopt the same annealing scheme ét = act-1 in our experiments with decay factor 0.95.
A.2.2 ORTHOGONAL PROCRUSTES
When fixing P. the problem in Eq. (8) is reduced to
OEO, (0,M).
max
where M = Z PP'. The solution is given by the following Lemma
Lemma 3 (AAlvarez-Mells et al., 2019)), Let M be the matrix with SVD M UAVT, then arg magmax (0, M) UVT.
14

---

Published as a conference paper at ICLR 2023
Proof. The proof is simply based the fact that the set of orthogonal matrices is group. Specifically, we have
(0, M) LAV' > -UTOV oV, A)r;
which is maximized when UT OV is equal to the identity matrix, since it an orthogonal matrix (as the product of orthogonal matrices). As a consequence, O =VVV.
B EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
In this section, we provide implementation details and additional experimental results. Our code will be released upon publication.
B. COMPUTATION DETAILS
All experiments are run on single Macbook Pro 2020 laptop with 2 GHz Quad-core Intel core i5 CPU.
B.2
JOINT VISUAL EXPLORATION OF TWO DATASETS
We provide details about datasets, implementation and evaluation metric for joint visual exploration of two datasets.
B.2.1 DATASETS
The datasets used in joint visual exploration include 3 (pairs of) synthetic datasets and 3 (pairs of) real-world datasets.
Synthetic datasets. We use 3 synthetic datasets from (Liu et al., 2019)', including a bifurcation, Swiss roll, and circular frustum. All 3 synthetic datasets consist of two domains of 300 samples that have been linearly projected onto 1000 and 2000-dimensional feature spaces respectively where the mapping matrix is sampled from a Gaussian distribution. The transformed samples are also injected small white noise (Liu et al., 2019). There are respectively classes for each dataset. Each resulting dataset thus provides two data matrices X R 1000 and X' 300 2000 associated with their respective class labels and y' which will be used for the unsupervised heterogeneous domain adaptation (HDA) problem. All datasets were simulated with one-to-one sample-wise correspondences, which are solely used for evaluation purpose. Each domain is projected to different dimension, so there is no feature-wise correspondence either. In all simulations, we normalize the features with standardisation before running the Joint MDS as in (Liu et al., 2019).
Real-world datasets. We use 2 sets of single-cell multi-omics data from (Demetci et al., 2022) and digit dataset demonstrate the applicability of our method to real datasets.
The 2 single-cell datasets are generated by co-assays, which provides cell-level correspondence information for evaluation. The first dataset is generated by the scGEM assay, which simultaneously profiles gene expression and DNA methylation. It results in two data matrices X 1777x33 for the gene expression data and X 177227 for chromatin accessibility data associated with classes based on the cell types: iPS, d24T+, d16T+, d8 and BJ. The other dataset is generated by the SNAREseq assay, linking gene expression with chromatin accessibility. The data has been pre- processed in (Demetci et al., 2022) and results in two data matrices X E R 110f7x19 and X' E Te 10 respectively for ATAC-seq and RNA-seq, associated with classes based on the cell types: GM, BJ, K562 and HI. Both datasets are unit normalized as in (Demetci et al., 2022).
The digit dataset MNIST-USPS contain two subsets of MNIST and USPS respectively. Each subset include 1000 digit images with 100 images for each digit class, which results in two data matrices X E R 1000x 784 and X E R 1500x226 for subsets of MNIST and USPS respectively. There are 10 classes in total representing digits from 0 to 9. The data is not preprocessed but only raw pixels used to compute the distance matrices.
Thee datasets he downloaded from https: /ttp//.cle.ee///oe/ ge............... washt ngtton edu/projj ppmd:/mdd---po. ma/
15

---

Published as a conference paper at ICLR 2023
B.2.2 IMPLEMENTATION DETAILS
To compute the dissimilarities for data from each domain, we use the geodesic distances presented in Section 4.2. The resulting distance matrices is rescaled by the average of the shortest-path distances to match the distances scale the two domains. The number of nearest neighbors (NN) for each dataset is set to minimize the FOSCTTM via grid search similar to (Demetci et al., 2022), and is then fixed throughout the experiments. For MNIST-USPS, since we do not have 1-to-1 sample-wise correspondences, we fix number of NNs to 5. For synthetic datasets, the matching penalization parameter À is fixed to 0.1 and the entropic regularization parameter for OT is fixed to 1.0 with an annealing decay equal to 0.95. For real-worke datasets, select both parameters based either FOSCTTM if 1-to-1 correspondences are available or transfer accuracy otherwise. We run Joint MDS 4 times with different initializations and select the best embeddings determined by the run with the smallest final objective value to mitigate the local minima issues. We found our results generally stable and thus did not report the std for these datasets.
B.2.3 EVALUATION METRIC
As we also have access to the ground truth of the 1-to-1 correspondences for the synthetic datasets and the 2 single-cell datasets, can quantitatively assess the alignment learned by Joint MDS. This is achieved by using the fraction of samples closer than the true match (FOSCTTM) metric (Liu et al., 2019). For given sample from dataset, we compute the fraction of samples from the other dataset that are closer to it than its true match and we average the fraction values for all the samples in both datasets, A lower average FOSCTTM generally indicates a better alignment and FOSCTTM would be zero for a perfect alignment as all the samples are closest to their true matches.
B.2.4 ADDITIONAL VISUAL RESULTS
We show in Figure 4 visualization comparisons of MDS and Joint MDS on the 6 datasets. The MDS visualizations are obtained by running the SMACOF algorithm individually on each domain. While there's no alignment between the samples across the domains, our Joint MDS clearly aligns the samples in the 2D space as data points from the same class are located at similar positions across the two domains.
B.2.5 HUMAN BODY POSE ALIGNMENT
In addition to exploring two datasets, Joint MDS can also be used for comparing two sets of point clouds. In order to make our discussion visually more relatable, we apply Joint MDS to the task of aligning human body point clouds in two different poses. We consider here 3 pairs of human body poses, pairs from the MPI FAUST dataset (Bogo et al., 2014) (each pose has 3446 points) and 1 pair from Solomon et al. (2016) (has 1024 and 1502 points respectively) to visualize how Joint MDS works different cases. For the sake of simplicity, we only used Euclidean distances as the pairwise dissimilarities. We fixed À to 0.1 and entropic regularization to 1.0 with annealing. We respectively compute embeddings in 3D and 2D space, and visualize the correspondences found by Joint MDS. The results are shown in Figure 5 where each column represents example. Different colors indicate a different position i in the source set and the alpha value of the corresponding color in the target set is given by the correspondence vector P, returned by Joint MDS. Joint MDS is able to find correspondences between two poses, even if the target set of point clouds only shares semantic structure as shown in the right column.
B.3 UNSUPERVISED HETEROGENEOUS DOMAIN ADAPTATION
We use the same datasets as in Section B.2 as all the datasets also provide class labels. We use the same hyperparameters as in Section B.2, including the number of NNs for computing the geodesic distance matrices, the matching penalization parameter and the entropic regularization parameter. To handle the classification problem, we first use our Joint MDS to obtain two embeddings for both domains. Then, we train a simple k-NN classifier (with & fixed to 5) on the embeddings of domain I and evaluate the classifier ANNIST-USPS we use a linear SVM classifier with the regularization parameter C' 1, which we found substantially outperforming the k-NN classifier. In addition, we also observe that the transfer
domain 2 without any further adaptation. For the more complex dataset
16

---

Published as a conference paper at ICLR 2023
Coman:
Comaah
Comah1
Domal
JATEMIS
Comperent.
Compenen
Compenent
JAYEM25 Compurare
Doman:
Domeer 2
Denue
Rdt
VOS Conperent)
VBS Component
JAIEMOS COTEERAAEE
JAVENTS CONTERWNE
Demann:
Domaan
DoPOR
MOS
JAREMOS Darporeatt
JaYLMOS
Comporent33
Compenert
JATENES Comporart.
JOYENOS Camgonant
Figure 4: Comparison of MDS and Joint MDS. Each row presents dataset and from top to bottom datasets are respectively: bifurcation, Swiss roll, circular frustum, seGEM, SNAREseq and MNIST-USPS. Left columm: results obtained by MDS applied individually to each domain with the Euclidean distances. Right column: results obtained by Joint MDS. Each color represents a class.
17

---

Published as a conference paper at ICLR 2023
Figure 5: Joint MDS can find correspondences between a source set of point clouds and target set of point clouds with similar structure or with shared semantic structure (right column). as well as common embedding space. Note that here we plot the mesh surfaces in the 3D embeddings for visual purpose and we did not use any mesh information for correspondence finding but only point clouds.
Transfer accuracy
0.5
0.4
0.2
120 180 210 240
Number of components
Figure 6: Transfer accuracy number of components for Joint MDS on the MNIST-USPS dataset.
accuracy generally increases with the number of components d, as shown on MNIST-USPS as an example Figure 6.
B.4
GRAPH MATCHING
Here, we provide the dataset and implementation details for the experiments with respect to the graph matching task.
B.4.1 DATASETS AND BASELINES
We use graph matching datasets including PPI network of yeast (with its noisy versions) and the MIMIC-III disease- srreass-procedure interaction dataset. The two datasets can be downloaded at https: : / qithub. com/ cww/Hongt http//oongtttgeee swwll and https: github..bh//ggggggg com/ Hwng/tngggutttuun /ww re- spectively.
For each baseline method compare with, list its source and language below:
18

---

Published as a conference paper at ICLR 2023
HubAlign (C)
(Hashemifar
&
Xu, 2014):
httpa://hoeetticc.e
"hashemifar/
MAGNA++ (C) (Vijayan al., httpps//wwww3.ddddee///r///////s
GWL (Python) (Xu et al., 2019b): https: htpp:://githhle http://wwww..o.oogogggog////a///
All baselines are implemented under the recommended settings.
B.4.2 EVALUATION METRICS
As described in section 5.3, the PPI network matching task is one-to-one node matching task, where we can directly compute the node correctness (NC) based on the true matching matrix and predicted matching matrix. However, the number of diseases (56) and the number of procedures (25) do not match in the disease-procedure graph matching task. In contrast to the original implementation in (Xu al., 2019b) by Xu et al., which evaluates the learnt matching matrix by comparing the recommended procedures with the true procedures for each individual admission from the test set, we propose to build the disease and procedure graphs from these individuals and evaluate the recommended procedures for each disease directly. We compare the top 3 and top recommended procedures with the corresponding true procedures, which has the most interactions with the target disease in admissions from the test set, then we take the average accuracy for all disease nodes as the evaluation metric.
B.4.3 IMPLEMENTATION DETAILS
We first normalize the adjacency matrices for each graph via D-1/2AD where denotes the adjacency matrix and D denotes the diagonal matrix of degrees. Then, we compute the shortest- path distances as the dissimilarities as described Section B.2.2. Similar to stress majorization- based graph drawing algorithms (Gansner et al., 2004; Kamada al., 1989), we apply different weights W RRx= and W IT xn' on the two distance deviation terms in the weighted stress minimization problem (5) with Wiy = 1/d"; and dis is the shortest-path distance between node and node j (similarity, for w/y = 1/d Empirically, we find k = 4 leads to best performances.
B.4.4 RATIONALITY ANALYSIS
To further verify the rationality of the learnt matching matrix of the proposed method, we visualize the k-NN graph of disease nodes and procedure nodes in the common low-dimensional space in Figure 7 and we can further interpret the learned embeddings from clinical point of view. For example, in the selected region inside the red dash line box, we can see a cluster of cardiac-related diseases like congestive heart failure (d4280), pleural effusion (d5 155119 19), Percutancous transluminal coronary angioplasty (dV4582), and also cardiac-related procedures like extracorporeal circulation auxiliary open heart surgery (p3961), combined right and left heart cardiac catheterization (p3723) and single internal mammary-coronar artery bypass (p3615). For reference, the complete ICD codes for the diseases and procedures can be found in the supplementary material of (Xu et al., 2019b). This finding indicates that our method can learn clinically meaningful joint embeddings from the disease and procedure graphs.
B.5 PROTEIN STRUCTURE ALIGNMENT
Protein structure alignment takes two protein 3D structure models input, and outputs protein structure models that preserve the pairwise distances while being aligned.
Datasets and experimental setup. In contrast to the previous work (Wang Mahadevan, 2009; Cui et al., 2014) where only one protein was studied, we use much larger and more realistic dataset, and provide a quantitative way to evaluate the unsupervised manifold alignment methods for this task. Specifically, we consider here the first domain of all the proteins from CASP14 (Pereira et al., 2021) from T1024 to T1113 and use the protein models predicted by the two top-placed methods in the leaderboard, namely AlphaFold2 and Baker. This results in dataset of 58 pairs of protein models, with an average number of residues equal to 198. The dataset can be downloaded from https://preet //pped///ddddeeee ct ioncenter...... org/ casp1 4 4/results.cg.
19

---

Published as a conference paper at ICLR 2023
Disease-procedure interaction graph
disease procedure
Figure 7: Visualization of the NN graph of diseases and procedures in the common 2D space with
k 4.
20

---

Published as a conference paper at ICLR 2023
Table 4: Performance comparison on remove the orthogonal Procrustes matrix
Method scGEM
SNAREseq
PPI 5%
MIMIC top
MIMIC top
Removing O
61.8
94.1
85.25
14.48
35.48
Original 72.9
94.7
86.44
30.24
46.28
We compare Joint MDS to the state-of-the-ar unsupervised manifold alignment method GUMA (Cui et al., 2014). We use Joint MDS directly on the pairwise Euclidean distances between the residue 3D coordinates. The parameters for both our method and GUMA are fixed throughout the datasets, including À fixed to 0.1, entropic regularization & fixed 10. We the Gromov-Wand Wasserstein initialization as described in Section 4.3. As MDS could be affected by bad initialization, we run Joint MDS 3 times on each protein pair with different random seeds and take the embeddings with the smallest cost for evaluation. To measure the robustness of our method, we repeat this process 3 times again, and take the average and standard deviation of the RMSD-D across all repetitions. The small standard deviation in Figure 3 demonstrate the robustness of our method.
Additional alignment results. We showcase in Figure five examples of alignment (TS1024, TS1039, TS1050, TS1058 and TS1099) obtained by GUMA, Joint MDS, GUMA-2D and Joint MDS-2D. While Joint MDS outperforms GUMA in almost all datasets, we also find a few protein samples where both methods fail.
B.6 ADDITIONAL EXPERIMENTS
Runtime comparison. We use four datasets to compare the runtime of Joint MDS with other baseline methods. The four datasets include: scGEM and SNAREseq from (Demetci et al., 2022) for unsupervised heterogeneous domain adaptation. MIMIC (Xu et al., 2019b) and PPI network (5% noise) (Xu et al., 2019a) for graph matching. The baseline methods we compared against are SCOT (Demetci et al., 2022), UnionCom (Cao et al., 2020), EGW (Yan et al., 2018) (for unsupervised heterogeneous domain adaptation) and GWL (Xu et al., 2019b) (for graph matching). For all baseline methods, we used the original implementations for each task. For method, we used the same parameters which achieved the reported performance. We ran all the methods until convergence and repeated this three times compare the average consumed time. Figure 9 suggest that the runtime of Joint MDS is at the same order of magnitude compared to the best of other baseline methods while achieving the best performance in most of the tasks.
Convergence curves. Here, we provide the convergence curves (objective value versus number of iterations) of our method on the above four tasks. The curves in figure 10 show that the proposed optimization algorithm reached final convergence for all the tasks. Empirically, we also validated the convergence of the method by visualizing the node correctness of each iteration the PPI network matching task for the three noise levels respectively, in Figure 11. Even though the coupling matrix P is intialized with GW which already offers a not bad quality, its quality further improves with the number of iterations of Joint MDS, thanks to regularization through explicit learning of embeddings.
Removing the orthogonal Procrustes matrix. We further evaluate the necessity of adding the orthogonal Procrustes matrix O in our final objective function, as showed in Eq 5, by an ablation study comparing the performance of removing O and the performance of the original objective. Similarly the runtime amalysis, the ablation study was conducted on two unsupervised heterogeneous domain adaptation tasks and two graph matching tasks. In Table 4 we can see that the original Joint MDS has achieved consistent higher performance various tasks compared incomplete version with O removed, some tasks like scGEM and MIMIC, the performance gap is very large, which demonstrates the importance of incorporating O in the objective function.
21

---

Published as a conference paper at ICLR 2023
GUMAA-D (RMSDD D-22.87)
Joint MDS-22 (RMSSD 6.6555)
Fumsasg
wwww
GUMA-20 (RMSD D-25.65)
Joint MD5-20 RMMSDD-7.007
GUMA-20 JAMGD-D=34.0 Joint MDS-20 (RMSD-D-. D000D---:0029
(RMSD-DD36.64)
Joint (RMSD-D-100
GUMA IRMGD-0-28.80
Joint HDS (AMGD-Da1.6))
Input
GUMA RRMSD-D-42.00
Joint NDS IRMB D-056.57777
Figure 8: Comparison of GUMA and Joint MDS for protein structure alignment. The protein models for top to bottom are respectively TS1024, TS1039, TS1050, TS1058 and TS1099. Joint MDS generally outperforms GUMA except for the last case where both methods fail.
22

---

Published as a conference paper at ICLR 2023
Runtime comparison
SCOT
EGW
UnionCam
47.08
Joint MDS (d=2)
MDS (d=16)
33.36
30.17
28.67
22.81
21.55
12.1
6.05
3.88
2.36
ScGEM
SNAREseq
Runtime comparison
223.14
GWL Joint MDS
116.51
100
9.88
1.85
PPI 5% noise
MIMIC
Figure 9: Runtime comparison against baseline methods for heterogeneous domain adaptation (top) and graph matching (bottom).
23

---

Published as a conference paper at ICLR 2023
SCGEM
SNAREseq
0.054
0.0700
0.052
0.0675
Objective
0.0650
0.050
0.0600
0.046
0.0575
0.044
0.0550
0
160
200
300
400
20
40
80
100
Iteration
Iteration
MIMIC
PPI
0.00380
0.0030
Objective
0,00370
Objective
0.0028
0.003
0.0024
0,00355
0.0022
0.00350
200
Iteration
Iteration
Figure 10: Convergence curves for scGEM (top left), SNAREseq (top right), MIMIC (bottom left) and PPI (bottom right).
5%
15%
0.86
0.72
0.85
0.70
0.84
0.62
0.82
0.80
0.58
50
125
200
75
100
150
176
lteration
lleration
25%
0.57
0.58
2 0.53
0.51
0.50
0.49
25
50
75
100
125
150
175
200
Bleraion
Figure 11: Node correctness with respect to number of iterations for the PPI matching task with noise level 5 5% (top left), 15% (top left) and 25% (bottom).
24


Reviews:
[{"review_id": "zVW_tQENLgq", "reviewer": "Reviewer_jgCM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a practical method to jointly estimate the low dimension embedding in Euclidean space and the correspondences between data sets, which can be viewed as the alignment extension of classical embedding work by Multidimensional Scaling (MDS). Unlike some previous works in this domain, the proposed method is unsupervised by nature without given any known subset of alignment.  \n\nThis joint MDS framework is formulated as a combined stress objective function plus the additional matching penalty term from Wasserstein Procrustes analysis. Given this proposed join objective function is non-convex (same as similar problems), this paper proposed an alternative optimization algorithm (similar to EM type), which can be viewed as the combination of weighted MDS problem and Wasserstein Procrustes problem in each iteration. \n\nExperimental results on both synthetic data and real-world data (e.g., cell, MNIST, human body data, protein, etc) on a number of tasks from joint 2D visualization, domain adaptation, graph matching and protein alignment. ", "summary_of_the_review": "The overall contribution seems marginal but given the solid technical work behind this alternative optimization algorithm, and the rich empirical evaluation to this joint alignment problem, my recommendation for this paper is 6 \"marginally above the acceptance threshold\". ", "strength_and_weaknesses": "Strengths:\n\nThe problem itself is interesting and the article is pleasant to read too, i.e., join alignment and embedding from 2 different data sets without supervision can be find a lot of application across NLP, computer vision, healthcare, biology, etc. \n\nThe optimization part is also solid as with enough considerations of previous works to joint optimize this non-convex problem by SMACOF algorithm and Wasserstein optimizations from (Alvarez-Melis et al., 2019), and with detailed analysis of computational complexity (Section 4.3), which is quite helpful to see. \n\nExperimental results also can be considered solid as across a number of synthetic or real data coupled with different tasks (with more results in Appendix B), and the proposed method often show (not always) the relative best performance among existing methods. \n\nWeaknesses:\n\nOne of the key concerns is, the proposed joint MDS method is based on MDS as the starting point, which utilize the (dis) similarities from Euclidean space and this may conflict with the nature of manifold embedding when data with strong nonlinear intrinsic geometry structure. Indeed, there are some discussions provided in Section 4.2, which is good to see, but still some unclear remains. As ISOMAP is essentially MDS coupled with geodesic distance, and not sure this is applied in which part of results in Section 5? (Maybe I missed).  Also, outside MDS and ISOMAP, there are other graph spectral works in this direction such as Laplacian Eigenmaps (and others), should be helpful to have some judgment for the choice in this work. \n\nAdditionally, it seems using GW as the initialization step (mentioned in Section 4.3) to provide a better coupling matrix P in the algorithm, make the contribution from this proposed iterative optimization less clear as maybe the non-convex formulation is good in theory but less practical in the numerical way. \n\nReference:\n1. D. Alvarez-Melis, S. Jegelka, and T. S Jaakkola. Towards optimal transport with global invariances. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2019", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "t_YhcqY2ygf", "reviewer": "Reviewer_rQ9i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to embed two sets of data into a common low dimensional space and at the same time estimate the correspondences between the data points of two sets.  The proposed method combines Multidimensional Scaling (MDS) and Wasserstein Procrustes analysis into a joint optimization problem, which is solved using alternating updates. The authors demonstrate the effectiveness of the proposed method on a variety of datasets.", "summary_of_the_review": "In summary, the idea of the paper is very natural for solving the proposed task, although its novelty looks limited. In general, I enjoyed reading the paper. Given both of its limitations and advantages, this work has a potential to be published at ICLR.", "strength_and_weaknesses": "Strength:\n\n1. The paper is very well-written and technically sound. The idea of the paper is natural and clearly motivated.\n\n2. The proposed method is effective on a variety of datasets.\n\nWeakness and questions:\n\n1. The novelty of the paper seems limited to me. It looks to me that this work simply combines the methods in two regimes and implement a joint optimization. \n\n2. From the optimization perspective, the proposed method is essentially solving the two problems (MDS and Wasserstein Procrustes) in an alternating way. It is hard to tell whether the resulting heuristic algorithm will lead to convergence to a local/global min.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GTlLjcKJeAT", "reviewer": "Reviewer_DPfi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework for unsupervised manifold alignment with joint multidimensional scaling. It combines a MDS method with a optimal transport to join the embedding. The main contribution is the combination of these two methods forming a general framework and a optimization scheme to solve both problems jointly. The experiments shows interesting results on two different tasks (visual exploration and graph matching).  ", "summary_of_the_review": "This article proposes a new joint optimization problem for both multidimensional scaling with unsupervised manifold alignment. It relies on two well founded methods: multidimensional scaling  with SMACOF as solver and optimal transport with Sinkhorn's solver. The complete framework is general enough to be applied on different kind of tasks and the experiments shows promising results. Despite some weakness, this framework may interest a large part of the community.", "strength_and_weaknesses": "## Strength\n\nThe proposed framework is interesting on several points,\n1. It relies on tools with good mathematical properties. Optimal transport has now effective methods for solving the problem with appropriate regularization and MDS can be approximated with the SMACOF algorithm.Thus, both sub-optimization problems are easily solved.\n2. The experiments are strong with very good results for graph matching leading to potentially new methods.\n3. The generality of the framework opens it to many problems that occurs in different community (especially all the problematic around matching).\n4. The method seems easy to code and then enjoy a good reproducibility.\n\n## Weaknesses\n\nHowever I see several issues in the current work,\n1. While the state-of-art is mostly complete, the authors forget some recent methods that may be competititve [1].\n2. For graph matching, a comparison against KerGM [2] would be welcomed.\n3. The method does not scale to large dataset.\n4. There is no analysis of the convergence of Algorithm 1. Since it is a classical two stage optimization scheme, with appropriate hypothesis, convergence toward a stationary point should be possible [3].\n5. There is no comments about the initialization. How dependent are the results to the initialization? Is it stable enough?\n\n[1] Jin, K., Liu, C., & Xia, C. (2021). Two-Sided Wasserstein Procrustes Analysis. In IJCAI (pp. 3515-3521).\n[2] Zhang, Z., Xiang, Y., Wu, L., Xue, B., & Nehorai, A. (2019). KerGM: Kernelized graph matching. Advances in Neural Information Processing Systems, 32.\n[3] Bezdek, J. C., & Hathaway, R. J. (2003). Convergence of alternating optimization. Neural, Parallel & Scientific Computations, 11(4), 351-368.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5McsBHin1Fs", "reviewer": "Reviewer_hbpW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper generalizes the classical  multidimensional scaling approach for unsupervised manifold alignment. This enables the mapping of , datasets from two different domains, without requiring correspondences across the datasets, to a common low-dimensional Euclidean space. This approach generalizes approaches based on dictionaries that map the source and target data onto a common latent space. Results on a few benchmark datasets are proposed.", "summary_of_the_review": "I will encourage the authors to do more experiments with real and challenging datasets.", "strength_and_weaknesses": "Strengths: A useful extension of MDS with applications in unsupervised domain adaptation (UDA) is proposed. This approach enables the mapping of datasets from two different domains onto a latent space.\nWeaknesses: I am not impressed with papers that have reasonable theory but demonstrate results on synthetic data, MNIST etc. Results in table 3 are not impressive either.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster