# Geometric Multimodal Learning Based on Local Signal Expansion for Joint Diagonalization

Maysam Behmanesh, Peyman Adibi, Jocelyn Chanussot, Christian Jutten, Sayyed Mohammad Saeed Ehsani

<figure><img src="image_1.png" /></figure>

To cite this version:

Maysam Behmanesh, Peyman Adibi, Jocelyn Chanussot, Christian Jutten, Sayyed Mohammad Saeed Ehsani. Geometric Multimodal Learning Based on Local Signal Expansion for Joint Diagonalization. IEEE Transactions on Signal Processing, 2021, 69, pp.1271-1286. 10.1109/TSP.2021.3053513. hal-03429695

HAL Id: hal-03429695

https://hal.science/hal-03429695v1

Submitted on 26 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Geometric Multimodal Learning Based on Local Signal Expansion for Joint Diagonalization

Maysam Behmanesh, *Graduate Student Member, IEEE*, Peyman Adibi, Jocelyn Chanussot<sup>1, ∗</sup>, Fellow, IEEE, Christian Jutten<sup>1, †</sup>, Fellow, IEEE, and Sayyed Mohammad Saeed Ehsani

**Abstract**—Multimodal learning, also known as multi-view learning, data integration, or data fusion, is an emerging field in signal processing, machine learning, and pattern recognition domains. It aims at building models, learned from several related and complementary modalities, in order to increase the generalization performances of a predictive learning model. Multimodal manifold learning extends spectral or diffusion geometry-aware data analysis to multiple modalities. This can be performed through the definition of undirected graph Laplacian matrices in different modalities. However, finding common eigenbasis of multiple Laplacians is not always a relevant solution for multimodal manifold learning problems. As a matter of fact, the Laplacians of all modalities are not simultaneously diagonalizable in many real-world problems due to the major differences between the different modalities. In this paper, we propose a multimodal manifold learning approach based on intrinsic local tangent spaces of underlying data manifolds in order to discover the local geometrical structure around matching and mismatching samples in different modalities in sparse diagonalization problems. This approach searches for approximate common eigenbasis of Laplacian matrices by expanding the signal of limited existing information about matching and mismatching samples of different modalities to their on-manifold neighbors. Experiments on synthetic and real-world datasets in supervised, unsupervised, and semi-supervised problems demonstrate the superiority of our proposed approach over existing state-of-the-art related methods.

**Index Terms**—Dimensionality reduction, intrinsic tangent space, manifold learning, multimodal signal processing, Laplacian matrices joint diagonalization.

## I. INTRODUCTION

INFORMATION on many natural phenomena or systems can be acquired from various source domains, using different types of sensors with different characteristics and properties.

Manuscript received April 25, 2020; revised August 25, 2020, October 27, 2020, and December 30, 2020; accepted January 6, 2021. Date of publication January 22, 2021; date of current version February 26, 2021. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Syed Mohsen Naqvi. This work was supported in part by MIAI@Grenoble Alpes, under Grant ANR-19-P3IA-0003. The work of Peyman Adibi was supported by the Campus France organization and French embassy in Iran to visit GIPSA-lab for 6 months in 2017–2018. (Corresponding authors: Maysam Behmanesh; Peyman Adibi.)

Maysam Behmanesh, Peyman Adibi, and Sayyed Mohammad Saeed Ehsani are with the Artificial Intelligence Department, Faculty of Computer Engineering, University of Isfahan, Isfahan 8174673441, Iran (e-mail: mbehmanesh@eng.ui.ac.ir; adibi@eng.ui.ac.ir; ehsani@eng.ui.ac.ir).

Jocelyn Chanussot and Christian Jutten are with the CNRS, Grenoble Institute of Technology, GIPSA-Lab, University of Grenoble Alpes, Saint Martin d'Hères Cedex 38402, Grenoble, France (e-mail: jocelyn.chanussot@gipsa-lab.grenoble-inp.fr; christian.jutten@gipsa-lab.grenoble-inp.fr).

Digital Object Identifier 10.1109/TSP.2021.3053513

The term “modality” refers to each type of acquired signal. Data acquired using a single modality usually only contain partial information of an object. Multimodal offers complementary information from different modalities [1]. For example, natural language understanding can involve written data, spoken language as well as video recording of the speaker. Remote sensing scene interpretation can include Lidar, radar or optical acquisitions of one given scene, with different properties [2].

During the last decade, studying multimodal machine learning emerged and attracted considerable attention in signal processing, pattern recognition and machine learning communities [3]–[6]. Multimodal learning fuses the encoded complementary information from multiple modalities to discover a latent representation of data and consists in developing inference models to learn a unique function to model each modality and find optimal functions simultaneously to improve the generalization performances.

Due to the ability of neural network-based methods in learning the latent representations in an unsupervised manner, they are extensively used in multimodal learning. For example, the variational autoencoders (VAE) are used to learn a shared latent representation in multi-view problems [7]. Several other famous methods are based on Bayesian inference techniques. Because these methods are able to explore the spatial and temporal structures of data, they are well used to combine the modalities into a joint representation in temporal modeling problems [2], such as audio-visual speech recognition (AVSR) using deep Bayesian belief networks [8].

One of the most important categories of multimodal learning methods are based on extending spectral or diffusion geometry-aware data analysis to multiple modalities that are known as multimodal manifold learning. These methods involve various issues in various fields such as domain adaptation [9], [10], medical and biological multimodal data processing [11], [12], remote sensing [13], [14], or hyperspectral signal processing [15].

In the literature, various methods have been developed to analyze multimodal problems considering data manifolds in different modalities. Some of these methods try to learn a common low-dimensional latent space from different modalities such as, SCVM [16], MVML-LA [17], and LeMA [18]. The other category of multimodal manifold learning methods tries to learn an objective function that projects the original data into a low-dimensional space based on separability between manifolds, such as KEMA [19], NDML [20], and MVL [21], which is different from typical multiple kernel learning (MKL)
approaches [22]. All of these methods assume that the data samples in various modalities are completely paired and have the same number of data samples. Also in some of them, data samples in various modalities are assumed to be homogeneous. These assumptions significantly limit their application scope.

More practical assumptions in many multimodal problems are: 1) there are not the same number of samples in each modality, and 2) there are no explicit sample correspondence information between modalities, because they are collected independently.

By considering these assumptions, some other multimodal manifold learning methods have been developed to discover a common eigenbasis of Laplacians among all modalities by extending different spectral and diffusion methods to the multimodal setting [23]–[25].

However, due to the differences between modalities, the Laplacians of all modalities are typically diagonalizable only between matching samples. In these problems, which are called sparse diagonalization problems, many matching samples from different modalities are required to find approximate common eigenbasis in Laplacian matrices. The determination of these samples in each modality is one of the drawbacks of the sparse diagonalization problems.

In order to tackle this limitation, it is essential to synthesize and extend the geometrical relationship information of data manifolds in different modalities. In a recent paper [25], we used functional mapping to extend limited supervised information on modalities manifolds. Conversely, in the current paper, we propose another technique, called local signal expansion for joint diagonalization (LSEJD), that uses the intrinsic local tangent spaces for signal expansion of multimodal problems. Local tangent spaces are used to discover the local geometrical structure of manifolds, and determine the new proper matching and mismatching samples in each modality and provide a powerful and flexible tool to extend little known related information. Given the signal of limited matching and mismatching information between different modalities as several matrices, we locally extend these signals by inserting new informative columns to contain more geometrical knowledge of data cloud, using manifold tangent spaces in different modalities.

In summary, this method uses the same regularization framework given in most of the sparse diagonalization problems to approximate common eigenbasis for Laplacian matrices of all modalities. But, unlike [24] and the most state-of-the-art methods, in which the correspondence information of samples are manually selected based on the expert knowledge, the proposed method introduces a new framework to find automatically the additional matching and mismatching samples by locally extending the existing correspondence information in different modalities, while respecting their geometric structures.

The main contributions of this work are summarized as follows:

* The proposed method extends manifold learning to multimodal scenarios in a semi-supervised setting by the propagation of the existing supervised information to their local neighborhoods. We use a multimodal manifold learning framework by approximation of common Laplacian eigenbasis. Different from the methods in which all matching

<table><caption>TABLE I
DESCRIPTION OF ALL VARIABLES</caption><thead><tr><th>Variable</th><th>Description</th></tr></thead><tbody><tr><td><i>m</i></td><td>number of modalities</td></tr><tr><td><i>d</i><sub>i</sub></td><td>input dimension of modality <i>i</i></td></tr><tr><td><i>n</i><sub>i</sub></td><td>number of samples in modality <i>i</i></td></tr><tr><td><i>M</i><sup><i>i</i></sup></td><td>underlying manifold of modality <i>i</i></td></tr><tr><td><b>X</b><sup><i>i</i></sup>(<i>n</i><sub>i</sub> × <i>d</i><sub>i</sub>)</td><td>data points matrix in modality <i>i</i></td></tr><tr><td><i>q</i></td><td>number of matching samples</td></tr><tr><td><i>u</i></td><td>number of mismatching samples</td></tr><tr><td><i>t</i><sub>c</sub></td><td>number of selected nearest neighbors samples</td></tr><tr><td><i>t</i><sub>b</sub></td><td>number of selected farthest neighbors samples</td></tr><tr><td><i>ξ</i><sub>i</sub></td><td>intrinsic dimension of manifold <i>M</i><sup><i>i</i></sup></td></tr><tr><td><b>L</b><sub><i>i</i></sub>(<i>n</i><sub>i</sub> × <i>n</i><sub>i</sub>)</td><td>Laplacian matrix of modality <i>i</i></td></tr><tr><td><b>A</b><sub><i>i</i></sub>(<i>n</i><sub>i</sub> × <i>n</i><sub>i</sub>)</td><td>eigenvalues of modality <i>i</i></td></tr><tr><td><b>V</b><sub><i>i</i></sub>(<i>n</i><sub>i</sub> × <i>n</i><sub>i</sub>)</td><td>eigenvectors of modality <i>i</i></td></tr><tr><td><b>C</b><sub><i>i,j</i></sub>(<i>n</i><sub>i</sub> × <i>q</i>)</td><td>matching matrix between two modalities <i>i</i> and <i>j</i>, respectively</td></tr><tr><td><b>c</b><sub><i>i,j</i></sub><sup><i>j</i></sup>(<i>n</i><sub>i</sub> × 1)</td><td>the column <i>j</i> of matrix <b>C</b><sub><i>i,j</i></sub></td></tr><tr><td><b>c</b><sub><i>i,j</i></sub><sup><i>i,j</i></sup></td><td>the entry <i>t</i> of the vector <b>c</b><sub><i>i,j</i></sub><sup><i>i,j</i></sup></td></tr><tr><td><b>B</b><sub><i>i,j</i></sub>(<i>n</i><sub>i</sub> × <i>q</i>)</td><td>mismatching matrix between two modalities <i>i</i> and <i>j</i>, respectively</td></tr><tr><td><b>T</b><sub><b>x</b><sub><i>k</i></sub></sub>(<i>d</i><sub><i>i</i></sub> × <i>ξ</i><sub><i>i</i></sub>)</td><td>tangent space projection matrix of manifold <i>M</i><sup><i>i</i></sup> at sample <b>x</b><sub><i>k</i></sub></td></tr><tr><td><b>C</b><sub><i>i,j</i></sub><sup>'</sup>(<i>n</i><sub>i</sub> × (<i>q</i> × <i>t</i><sub>c</sub>))</td><td>extended matching matrix between two modalities <i>i</i> and <i>j</i>, respectively</td></tr><tr><td><b>B</b><sub><i>i,j</i></sub><sup>'</sup>(<i>n</i><sub>i</sub> × (<i>q</i> × <i>t</i><sub>b</sub>))</td><td>extended mismatching matrix between two modalities <i>i</i> and <i>j</i>, respectively</td></tr></tbody></table>

samples are determined with expert knowledge, we use the local alignment of tangent spaces to discover the local geometrical structure of matching and mismatching manifold parts in each modality.

* Unlike most multimodal learning methods that need an accurate determination of many matching and mismatching samples, our proposed method is able to achieve better results than other methods using only a few samples.

* We show the generalization ability of both the supervised and the unsupervised versions of the proposed method, by applying it to various tasks such as manifold learning, multimodal clustering, and multimodal classification.

The remainder of the paper is organized as follows. In Section II, we review the basic formulation of the used multimodal manifold analysis framework and outline its learning paradigm in full matching and sparse matching Laplacian diagonalization. Following this, our proposed framework is introduced in Section III. Experiments are presented and discussed in Section IV. Finally, Section V concludes the paper.

## II. PROBLEM FORMULATION

The main purpose of multimodal manifold learning is trying to learn the common low-dimensional embedding for all modalities. Suppose that multimodal data are acquired from *m* different spaces {$S^t$}$_{i=1}^m$ with different dimensions {$d_i$}$_{i=1}^m$. Also, suppose that there is an underlying manifold $M^i \subset \mathbb{R}^d$ for data in modality *i*, where $d_i$ is the space dimension of modality *i*. Let also $n_i$ be the number of samples in modality *i*.

All variables used in this paper and their descriptions are summarized in Table I.

The manifold is often modeled by an undirected weighted graph and is more precisely analyzed with the Laplacian matrix
of this graph. The undirected weighted graph corresponding to the manifold $M^i$ can be constructed by defining the data points $\{\mathbf{x}_1^i, \dots, \mathbf{x}_{n_i}^i\} \subset S^i$ as vertices with the adjacency matrix $\mathbf{W}$, whose elements $w_{k,l} = K(\mathbf{x}_k, \mathbf{x}_l)$ contain the neighborhood relations between samples and their measure of similarity using kernel $K(\cdot, \cdot)$.

The Laplacian matrix $\mathbf{L}$ is defined as $\mathbf{L} = \mathbf{D} - \mathbf{W}$ where $\mathbf{D} = \text{diag}(\sum_{k \neq l} w_{k,l})$ is the diagonal matrix of nodes degree. For all $m$ Laplacian matrices $\{\mathbf{L}_i \in \mathbb{R}^{n_i \times n_i}\}_{i=1}^m$, there are unitary eigenspaces $\mathbf{U}_i$'s, where $\mathbf{L}_i = \mathbf{U}_i \mathbf{\Lambda}_i \mathbf{U}_i^T$, and $\mathbf{U}_i$ is a matrix with eigenvectors of $\mathbf{L}_i$ as its columns, and $\mathbf{\Lambda}_i$ is the diagonal matrix of the corresponding eigenvalues. The eigenbasis $\mathbf{U}_i$ of the Laplacian matrix $\mathbf{L}_i$ can represent data in modality $i$.

### A. Approximate Common Eigenbasis of Laplacian Matrices

A multimodal manifold learning algorithm tries to learn a common space of data based on geometric structure of each modality (within-modality structure), as well as semantic relationship between all modalities (between-modality correlation). The main challenge in a multimodal scenario is representing data from various modalities in a common space by finding a common eigenbasis of all $m$ Laplacian matrices $\{\mathbf{L}_i\}_{i=1}^m$.

In a particular case, if all $m$ neighborhood graphs of Laplacian matrices $\{\mathbf{L}_i\}_{i=1}^m$ are isomorphic, i.e. 1) they have the same number of samples, 2) they have the same structure (isometric), and 3) the smallest eigenvalue of each Laplacian matrix has no multiplicity (connected graph), they are jointly diagonalizable and there exists a common set of orthonormal vectors $\mathbf{V}$ (joint eigenvectors) for all Laplacian $\{\mathbf{L}_i\}_{i=1}^m$, such that $\mathbf{V}^T \mathbf{L}_i \mathbf{V} = \mathbf{\Lambda}_i$, where $\mathbf{\Lambda}_i = \text{diag}(\lambda_{i,1}, \dots, \lambda_{i,n_i})$ is a diagonal matrix of the eigenvalues of $\mathbf{L}_i$.

In practice, because there are different sensing conditions and different number of samples in each modality, their eigenbasis are different, and we need to find an approximate eigenbasis in each modality. These approximate common eigenbases $\mathbf{V}_1, \dots, \mathbf{V}_m$ must have three characteristics [24]:

1) For each Laplacian matrix $\mathbf{L}_i$, there is an eigenbasis $\mathbf{V}_i$ that diagonalizes it, i.e., $\mathbf{V}_i^T \mathbf{L}_i \mathbf{V}_i = \mathbf{\Lambda}_i$, where $\mathbf{\Lambda}_i = \text{diag}(\lambda_{i,1}, \dots, \lambda_{i,n_i})$ are the eigenvalues of $\mathbf{L}_i$. In this case, the eigenvalues $\lambda_{i,1}, \dots, \lambda_{i,n_i}$ and eigenbasis $\mathbf{V}_i$ can be regarded as frequencies and Fourier basis, respectively. In the approximate common eigenbasis, using the first $k$ eigenvectors in $\mathbf{V}_i$ corresponding to the $k$ greatest corresponding eigenvalues ensures that the embedding preserves the neighborhood structures well [26].

2) Approximate common eigenbasis should be orthonormal, i.e. $\mathbf{V}_i^T \mathbf{V}_i = \mathbf{I}$. This ensures an efficient embedding because the dimensions of the embedding are uncorrelated.

3) Approximate common eigenbasis should be consistent based on given matching and mismatching samples (as discussed in the next subsection).

### B. Matching and Mismatching Samples

The samples correspondence information, based on the semantic relationship that they provide between modalities, are

<table><caption>Fig. 1. An example of matching and mismatching samples in a dataset with two modalities, visual and textual.</caption><thead><tr><th></th><th>visual modality</th><th></th><th></th></tr></thead><tbody><tr><th rowspan="5">textual modality</th><td>A</td><td>B</td><td>C</td></tr><tr><td>'water'</td><td>'nature'</td><td>'red'</td></tr><tr><td>'cat'</td><td>'water'</td><td>'macro'</td></tr><tr><td>'sea'</td><td>'explore'</td><td>'flower'</td></tr><tr><td>'washington'</td><td>'sea'</td><td>'flowers'</td></tr><tr><td>'tiger'</td><td>'fish'</td><td>'rose'</td></tr><tr><th></th><td>D</td><td>E</td><td>F</td></tr></tbody></table>

divided into two categories, matching and mismatching information.

Matching samples are data points in the feature space where the manifolds obtained from different modalities are in accordance with similar data samples, representing the same objects. They represent redundant information between the different modalities. On the contrary, mismatching samples represent data points that are related to dissimilar data, hence representing specific information brought by the different modalities.

Applying consistency in computing common eigenbasis by using matching samples is required. However, considering mismatching samples constraints can also be useful in the multimodal problems in order to better approximate the common eigenbasis. Using these constraints is helpful especially in the situations where there are missing data in some modalities. In such cases, there may be a modality with very few or even no matching samples with other modalities. Since a point in a modality is typically mismatched with many other points in another modality, there can exist much more mismatching sample pairs than the matching ones between modalities.

For example, consider a problem with two modalities, such as visual and textual, respectively. As can be seen in Fig. 1, matching samples are $\{A, D\}$, $\{B, E\}$, and $\{C, F\}$ because they provide the most semantic commonalities, and $\{A, F\}$, $\{B, F\}$, $\{C, D\}$, and $\{C, E\}$ are mismatching samples because they have the least semantic commonality. Samples $\{A, E\}$, and $\{B, D\}$ do not offer any strong semantic relationship. Also, as can be seen, it is possible that one given sample in a given modality be matched or mismatched with multiple samples in another modality.

In order to determine matching samples between two modalities $i$ and $j$ with a set of $q$ matching samples, two matching matrices $\mathbf{C}_{i,j}$ and $\mathbf{C}_{j,i}$ are defined. For example $\mathbf{C}_{i,j}$ is defined as follows:

$$ \mathbf{C}_{i,j} = \begin{bmatrix} c_{1,1}^{i,j} & c_{1,2}^{i,j} & \cdots & c_{1,q}^{i,j} \\ c_{2,1}^{i,j} & c_{2,2}^{i,j} & \cdots & c_{2,q}^{i,j} \\ \vdots & \vdots & \ddots & \vdots \\ c_{n_i,1}^{i,j} & c_{n_i,2}^{i,j} & \cdots & c_{n_i,q}^{i,j} \end{bmatrix}_{n_i \times q} $$

The $l$-th column of the matrix $\mathbf{C}_{i,j}$ is $c_i^{i,j} = [c_{1,l}^{i,j} c_{2,l}^{i,j} \cdots c_{n_i,l}^{i,j}]^T$.

If there is a set of $q$ matching samples $\{k_1^{i,j}, \dots, k_q^{i,j}\}$ and $\{k_1^{j,i}, \dots, k_q^{j,i}\}$ in modalities $i$ and $j$, respectively, for the $l$-th
matching consistency ($1 \le l \le q$), $k_l^{i,j}$ is matched with $k_l^{j,i}$, where $k_l^{i,j} \in \{1, 2, \dots, n_i\}$ and $k_l^{j,i} \in \{1, 2, \dots, n_j\}$. In this coding, the value of entry $c_{k_l^{i,j},l}^{i,j}$ in column $l$ of $C_{i,j}$ is set to one and other entries are set to zero. The corresponding column $l$ of $C_{j,i}$ contains one at entry $c_{k_l^{j,i},l}^{j,i}$ and zero for all other entries. This special case is referred to as sparse matching.

For example, suppose that modalities $i$ and $j$ contain data samples $\{a_1, \dots, a_3\}$ and $\{b_1, \dots, b_4\}$ respectively (thus $n_i = 3$ and $n_j = 4$), and there are $q = 2$ matching samples $\{1, 3\}$ and $\{2, 3\}$ between them, which means sample $a_1$ in modality $i$ is matched with sample $b_2$ in modality $j$, and sample $a_3$ in modality $i$ is matched with sample $b_3$ in modality $j$. The coding of these matching samples are defined as follows:

$$C_{i,j} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \\ 0 & 1 \end{bmatrix}_{3 \times 2} \qquad C_{j,i} = \begin{bmatrix} 0 & 0 \\ 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{bmatrix}_{4 \times 2}$$

When all modalities have an equal number of samples $n_i = n$ and data in each modality are sorted similarly, and $C_{i,j} = I$ for all $i, j = 1, \dots, m$, we have the simplest case of correspondences, where we simply have a single basis $V_i = V$ for all modalities. This case is referred to as full matching.

By specifying matching matrices $C_{i,j}$ and $C_{j,i}$ in modalities $i$ and $j$, since the basis for all modalities must be consistent, their corresponding eigenbasis are expected to be as similar as possible, i.e. $C_{i,j}^T V_i = C_{j,i}^T V_j$. This condition is called Fourier coupling in [24].

Similar to the matching matrix, two mismatching matrices $B_{i,j}$ and $B_{j,i}$ in modalities $i$ and $j$ are defined, respectively. If there is a set of $u$ mismatching samples $\{g_1^{i,j}, \dots, g_u^{i,j}\}$ and $\{g_1^{j,i}, \dots, g_u^{j,i}\}$ in these two modalities, for the $l$-th mismatching consistency ($1 \le l \le u$), $g_l^{i,j}$ is mismatched with $g_l^{j,i}$, where $g_l^{i,j} \in \{1, 2, \dots, n_i\}$ and $g_l^{j,i} \in \{1, 2, \dots, n_j\}$. In this coding, the value of entry $b_l^{i,j}$ is set to one and other entries are set to zero. The corresponding column $l$ of $B_{i,j}$ contains one at entry $b_l^{i,j}$ and zero for all other entries.

In order to make consistency in the eigenbases $\{V_i\}_{i=1}^m$ based on mismatching samples, their respective Fourier coefficients, $B_{i,j}^T V_i$ and $B_{j,i}^T V_j$, are required to be as different as possible.

### C. Full Matching Simultaneous Diagonalization of Laplacians

Nonlinear dimensionality reduction methods try to represent high-dimensional data with an intrinsic low-dimensional manifold $\mathcal{M}$. Finding a neighborhood preserving $k$-dimensional embedding of $\mathcal{M}$ can be posed as the following minimum eigenvalue problem [26]:

$$\min_{\mathbf{U} \in \mathbb{R}^{n \times k}} \mathrm{tr}(\mathbf{U}^T \mathbf{L} \mathbf{U}) \quad \text{s.t.} \quad \mathbf{U}^T \mathbf{U} = \mathbf{I} \qquad (1)$$

which has an analytical solution $\mathbf{U} = [\mathbf{u}_1, \dots, \mathbf{u}_k]$ containing the first $k$ eigenvectors of Laplacian matrix $\mathbf{L}$ corresponding to its smallest eigenvalues. Thus, by using the eigenvectors of the Laplacian matrix, the data are embedded effectively. The

problem (1) can be optimized with eigensolvers based on the Jacobi iteration [27] and formulated as follows:

$$\min_{\mathbf{U}^T \mathbf{U}=\mathbf{I}} \mathrm{off}(\mathbf{U}^T \mathbf{L} \mathbf{U}) \qquad (2)$$

where $\mathrm{off}(\mathbf{X}) = \|\mathbf{X} - \mathrm{Diag}(\mathbf{X})\|_F^2$ is an off-diagonality criterion of squared matrices $\mathbf{X}$ and $\mathrm{Diag}(\mathbf{X})$ contains only the diagonal values of $\mathbf{X}$. For a symmetric matrix $\mathbf{L}$, when $\mathbf{U}$ contains the eigenvectors of $\mathbf{L}$ as its column, a solver of the optimization problem (2) achieves the minimum value of zero.

**JADE method:** In this method, under the full matching assumption, the approximate joint eigenvectors $\mathbf{V}$ of the Laplacian matrices are found by solving the following optimization problem [28]:

$$\min_{\mathbf{V}^T \mathbf{V}=\mathbf{I}} \sum_{i=1}^{m} \mathrm{off}(\mathbf{V}^T \mathbf{L}_i \mathbf{V}) \qquad (3)$$

### D. Sparse Matching Simultaneous Diagonalization of Laplacians

In the full matching simultaneous diagonalization problem, we assumed that there are full matching samples, which is feasible if the numbers of data in all modalities are equal. However, this assumption is too restrictive.

In most multimodal problems, there are different numbers of samples in each modality (e.g. due to different space or time samplings, or different frame rates) and we use sparse point-wise correspondence information that is called sparse matching case. Sparse matching simultaneous diagonalization of the Laplacian problem uses a set of matrices $\{C_{i,j} \in \mathbb{R}^{n_i \times q} : \sum_{i,j=1, i \neq j}^m \text{ whose columns contain the corresponding vectors in the respective modalities.}$

The purpose of sparse matching simultaneous diagonalization is to find a set of corresponding basis $\{V_i \in \mathbb{R}^{n_i \times n_i} : V_i^T V_i = I\}_{i=1}^m$ such that $V_i^T L_i V_i$ are approximately diagonal for $i = 1, \dots, m$, and $V_1, \dots, V_m$ behave consistently. To ensure this, following definitions of subSection II-B, we define a matching matrix $C_{i,j}$ on manifold $\mathcal{M}^i$ and a matching matrix $C_{j,i}$ on manifold $\mathcal{M}^j$ such that their corresponding eigenbasis are required to coincide, i.e. $C_{i,j}^T V_i = C_{j,i}^T V_j$ [29]. Similarly, we introduce a mismatching constraint between mismatching matrices $B_i$ and $B_j$ such that their corresponding eigenbasis are required to be as different as possible. Finding the sparse matching joint Laplacian diagonalization can be formulated as the following optimization problem [24]:

$$\min_{\mathbf{V}_i^T \mathbf{V}_i=\mathbf{I}} \sum_{i=1}^{m} \mathrm{off}(\mathbf{V}_i^T \mathbf{L}_i \mathbf{V}_i) + \mu_c \sum_{i,j=1}^{m} \| C_{i,j}^T \mathbf{V}_i - C_{j,i}^T \mathbf{V}_j \|_F^2 \qquad (4) \\ - \mu_b \sum_{i,j=1}^{m} \| \mathbf{B}_{i,j}^T \mathbf{V}_i - \mathbf{B}_{j,i}^T \mathbf{V}_j \|_F^2$$

### E. Local Tangent Space of the Intrinsic Manifold

Assume $n$ data samples $\{\mathbf{x}_i\}_{i=1}^n$ residing on a smooth manifold $\mathcal{M} \subset \mathbb{R}^d$. There is a tangent space at each point $\mathbf{x}_i$ of $\mathcal{M}$ that is a good local approximation of the manifold at this point [30]. This
<figure><img src="image_3.png" /><figcaption>Fig. 2. The local tangent space $T_{\mathbf{x}_i}$ of a $2-D$ manifold $\mathcal{M}$ in $3-D$ space at point $\mathbf{x}_i$, and the projection of its $j$-th on-manifold neighbor $\mathbf{x}_{i,j}$ onto this tangent space.</figcaption></figure>

local tangent space, $T_{\mathbf{x}_i}$, is a linear manifold that locally approximates the nonlinear manifold $\mathcal{M}$ at $\mathbf{x}_i$. In practice, if the manifold is smooth enough, the PCA-based subspace constructed on the neighborhood of $\mathbf{x}_i$ can be a good approximation of the tangent space at $\mathbf{x}_i$ [31]. To acquire the local tangent space at point $\mathbf{x}_i$, Principal Component Analysis (PCA) is first performed on its neighboring data samples $\mathcal{N}(\mathbf{x}_i) = \{\mathbf{x}_{i1}, \dots, \mathbf{x}_{ik}\}$ that is the local patch built by the $\mathbf{x}_i$ and its $k$ nearest neighbors. Then, the best $\xi$ eigenvectors $\mathcal{R}^i = \{\mathbf{r}_1^i, \dots, \mathbf{r}_\xi^i\}$ of the local covariance, where $\xi$ is equal to the dimensionality of the intrinsic manifold, are extracted to estimate from $\mathcal{N}(\mathbf{x}_i)$, which constitute an orthogonal basis $T_{\mathbf{x}_i}$.

In high-dimensional data space, the coordinate of the neighborhood $\mathbf{x}_{i,j}$ in the approximated tangent space can be obtained as follows:

$$ \tau_{i,j} = T_{\mathbf{x}_i}^T (\mathbf{x}_{i,j} - \mathbf{x}_i) \quad (5) $$

where $T_{\mathbf{x}_i} = [\mathbf{r}_1^i, \dots, \mathbf{r}_\xi^i]$ is the projection matrix on tangent space at $\mathbf{x}_i$, and $\mathbf{x}_{i,j}$ is $j$-th nearest neighbor of $\mathbf{x}_i$, $1 \le j \le k$.

In the tangent space, the distance between $\mathbf{x}_i$ and $\mathbf{x}_{i,j}$ can be approximated by the Euclidian distance, i.e. by the norm of $\tau_{i,j}$ (see equation (6) below) instead of using a more complex geodesic metrics on the manifold.

An illustration of the local tangent space of manifold $\mathcal{M}$ at $\mathbf{x}_i$ and the projection of $\mathbf{x}_{i,j}$ on this subspace corresponding tangent coordinates system, $\tau_{i,j}$, are shown in Fig. 2.

### III. THE METHODOLOGY OF SPARSE MATCHING SIMULTANEOUS LAPLACIAN DIAGONALIZATION BASED ON LOCAL SIGNAL EXPANSION

In this section, we first introduce the motivation and theoretical analysis of using local tangent space in expanding the matching and mismatching samples. Then, we present our proposed approach for sparse simultaneous Laplacian diagonalization. Finally, the computational complexity of this method is presented.

#### A. Motivation

A drawback of the sparse diagonalization case is the limited existing knowledge given by matching and mismatching constraints between modalities, determined for example by a

domain expert. Actually, having matching or mismatching regions instead of sample points on different manifolds in different modalities is more reasonable and useful for real-world problems. Therefore, we try to propagate instance-level knowledge to their intrinsic neighbors to obtain patch-level knowledge about the correspondence of manifolds.

In most of the machine learning problems, data samples are represented in a high-dimensional space, while the intrinsic dimension of them, i.e. the degrees of freedom or the number of underlying data variations in the dataset, is often much smaller than the input space dimension. The intrinsic dimension $\xi$ can be interpreted as the dimensionality of the manifold $\mathcal{M}$ embedded within the $d$-dimensional ambient space $\mathcal{S}$ where $\xi \ll d$.

Although the idea of simultaneously diagonalizing the Laplacian matrices provides a powerful and flexible tool to discover the underlying data manifold, it may not be able to capture the local geometrical structure of data and loses much useful information, especially when data are sparsely distributed in the original space. Since the graph Laplacian constructed by sparsely distributed data in the high-dimensional space can be hardly connected to a smooth manifold, it may not be able to discover the correct geometrical relationship between data samples and their underlying manifold [32].

The local tangent spaces of the underlying manifold $\mathcal{M}$ can reflect the manifold structure in each region of data space. By representing each sample based on the intrinsic dimensions on a local tangent space of $\mathcal{M}$, the local variations become more important in discovering the geometrical structure at each sample while it avoids unwanted noise and global variation. Since the local tangent space of $\mathcal{M}$ at sample $\mathbf{x}_i$, $T_{\mathbf{x}_i}$, has the same dimension as $\mathcal{M}$, by mapping its neighboring data samples $\mathcal{N}(\mathbf{x}_i)$ on $T_{\mathbf{x}_i}$, 1) data is mapped to a subspace with intrinsic dimension, 2) the role of local variations becomes prominent due to the local nature of the tangent spaces, and 3) index of the most similar sample to $\mathbf{x}_i$ among its $k$ Euclidean nearest neighbors, shown by $s_i$, is found more effectively as follows:

$$ s_i = \underset{1 \le j \le k}{\operatorname{argmin}} \| T_{\mathbf{x}_i}^T (\mathbf{x}_{i,j} - \mathbf{x}_i) \|^2. \quad (6) $$

This equation constitutes the basis for computing equations (21) and (22), as will be explained in subSection III-C.

Inspired by these characteristics of local tangent spaces, we developed the idea of simultaneous diagonalization of Laplacian matrices of each modality by expansion of the local signal of relevant information among different modalities.

#### B. Theoretical Analysis

In this section, we propose a theoretical analysis of using local tangent space in extending knowledge of matching/mismatching samples. We first show why the PCA-based subspace constructed on the neighborhood of each sample can be an accurate tangent space approximation at that sample. Then, we explain why approximated tangent coordinates can find more similar/dissimilar samples compared to Euclidean distance by analyzing the error between the approximated tangent coordinates and their true values.
For the first part, we find a subspace within each local neighborhood of the data manifold by minimizing the sum of the projecting distances between the data samples of the neighborhood and their orthogonal projections in the subspace. This subspace is used as an approximation to the tangent space of the neighborhood.

Let {$\mathbf{x}_i$}$_{i=1}^n \in \mathbb{R}^d$ be the data samples drawn from a $\xi$-dimensional manifold $\mathcal{M}$, and $\mathcal{N}(\mathbf{x}_i) = \{\mathbf{x}_{i_1}, \dots, \mathbf{x}_{i_k}\}$ be the $k$ nearest neighbors of $\mathbf{x}_i$. The goal is to find a set of $n$ orthonormal bases $\mathbf{T}_i = [\mathbf{t}_{i,1}, \dots, \mathbf{t}_{i,\xi}] \in \mathbb{R}^{d \times \xi}$, $i = 1, \dots, n$, by solving the following optimization problem:

$$ \min_{\mathbf{T}_i^T \mathbf{T}_i = \mathbf{I}_\xi} \sum_{j=1}^k \| \mathbf{x}_{ij} - \mathbf{x}_i - \mathbf{T}_i \mathbf{T}_i^T (\mathbf{x}_{ij} - \mathbf{x}_i) \|^2, \quad (7) $$

where $\mathbf{I}_\xi$ indicates the identity matrix of size $\xi \times \xi$ and $\mathbf{T}_i \mathbf{T}_i^T (\mathbf{x}_{ij} - \mathbf{x}_i)$ is the orthogonal projection of difference between $\mathbf{x}_{ij}$ and $\mathbf{x}_i$ in the subspace. The problem (7) can be written in the following form:

$$ \min_{\mathbf{T}_i^T \mathbf{T}_i = \mathbf{I}_\xi} \mathrm{tr} \left( \times \left( \sum_{j=1}^k \left[ (\mathbf{x}_{ij} - \mathbf{x}_i)^T (\mathbf{I}_\xi - \mathbf{T}_i \mathbf{T}_i^T) (\mathbf{x}_{ij} - \mathbf{x}_i) \right] \right) \right). \quad (8) $$

Problem (8) is transformed to:

$$ \min_{\mathbf{T}_i^T \mathbf{T}_i = \mathbf{I}_\xi} \sum_{j=1}^k \| \mathbf{x}_{ij} - \mathbf{x}_i \|^2 - \mathrm{tr} \left( \times \left( \sum_{j=1}^k \left[ (\mathbf{x}_{ij} - \mathbf{x}_i)^T \mathbf{T}_i \mathbf{T}_i^T (\mathbf{x}_{ij} - \mathbf{x}_i) \right] \right) \right), \quad (9) $$

which is equivalent to the following problem:

$$ \max_{\mathbf{T}_i^T \mathbf{T}_i = \mathbf{I}_\xi} \mathrm{tr} \left[ \sum_{j=1}^k (\mathbf{x}_{ij} - \mathbf{x}_i)^T \mathbf{T}_i \mathbf{T}_i^T (\mathbf{x}_{ij} - \mathbf{x}_i) \right]. \quad (10) $$

By substituting $\tilde{\mathbf{X}}_i = [\mathbf{x}_{i_1}, \dots, \mathbf{x}_{i_k} - \mathbf{x}_i]$ in equation (10), the optimization problem is transferred to the following form:

$$ \max_{\mathbf{T}_i^T \mathbf{T}_i = \mathbf{I}_\xi} \mathrm{tr}(\mathbf{T}_i^T \tilde{\mathbf{X}}_i \tilde{\mathbf{X}}_i^T \mathbf{T}_i). \quad (11) $$

If the $\xi$-dimensional manifold $\mathcal{M}$ is smooth and sampled uniformly, such that $\mathbf{x}_i \approx \tilde{\mathbf{x}} = \frac{1}{k} \sum_{j=1}^k \mathbf{x}_{ij}$, equation (11) is just the optimization problem in PCA. Then, vectors $\mathbf{t}_{i,1}, \dots, \mathbf{t}_{i,\xi}$ will be the eigenvectors of $\tilde{\mathbf{X}}_i \tilde{\mathbf{X}}_i^T$ corresponding to its $\xi$ largest eigenvalues and the projection of $\mathbf{x}_{ij}$ onto the approximated tangent space is given by $\boldsymbol{\tau}_{ij} = \mathbf{T}_i^T (\mathbf{x}_{ij} - \mathbf{x}_i)$, $j = 1, \dots, \xi$.

Assume that the data samples {$\mathbf{x}_i$}$_{i=1}^n \in \mathbb{R}^d$ in the $d$-dimensional ambient space $\mathcal{S}$ ($\xi \ll d$) are sampled from $\mathcal{M}$ by using generative model $\mathbf{x}_i = F(\mathbf{y}_i) + \varepsilon_i$, $i = 1, \dots, n$, where $F = [F^1 \dots F^d]^T$ is a smooth isometry with $F^l: \mathbb{R}^\xi \to \mathbb{R}, l = 1, \dots, d$, $\mathbf{x}_i \in \mathbb{R}^d$, $\mathbf{y}_i \in \mathbb{R}^\xi$, and $\varepsilon_i$ indicates a random noise vector. By using the first-order Taylor expansion of $F$ at $\mathbf{x}_i$, a neighbor $\mathbf{x}_{ij} \in \mathbf{X}_i$ can be represented by:

$$ \mathbf{x}_{ij} = \mathbf{x}_i + \mathbf{J}_F(\mathbf{y}_i)(\mathbf{y}_{ij} - \mathbf{y}_i) + \varepsilon_i + \mathbf{e}_{ij}, \quad (12) $$

where $\mathbf{e}_{ij} = O(\|\mathbf{y}_{ij} - \mathbf{y}_i\|^2)$ contains second or higher-order sentences of Taylor expansion and $\mathbf{J}_F(\mathbf{y}_i) \in \mathbb{R}^{d \times \xi}$ is the Jacobian matrix of $F$ at $\mathbf{y}_i$, which $\mathbf{J}_F(\mathbf{y}_i)^T \mathbf{J}_F(\mathbf{y}_i) = \mathbf{I}_\xi$, as proven in [33].

By considering the tangent coordinate of $\mathbf{x}_{ij}$ approximated by local PCA, $\boldsymbol{\tau}_{ij} = \mathbf{T}_i^T (\mathbf{x}_{ij} - \mathbf{x}_i)$, and its true tangent coordinate, $\mathbf{J}_F(\mathbf{y}_i)^T \mathbf{J}_F(\mathbf{y}_i)(\mathbf{y}_{ij} - \mathbf{y}_i) = \mathbf{y}_{ij} - \mathbf{y}_i$, the error between them is given by:

$$ E(\boldsymbol{\tau}_{ij}) = (\| \mathbf{T}_i^T (\mathbf{x}_{ij} - \mathbf{x}_i) \| - \| \mathbf{y}_{ij} - \mathbf{y}_i \|)^2, \quad (13) $$

which gives the following inequality:

$$ E(\boldsymbol{\tau}_{ij}) \le \| \mathbf{T}_i^T (\mathbf{x}_{ij} - \mathbf{x}_i) \|^2 + \| \mathbf{y}_{ij} - \mathbf{y}_i \|^2. \quad (14) $$

By substituting from equation (12) and according to Cauchy-Schwarz inequality, which will be proved in the appendix, relation (14) results in:

$$ E(\boldsymbol{\tau}_{ij}) \le 3 \left[ \| \mathbf{T}_i^T \mathbf{J}_F(\mathbf{y}_i)(\mathbf{y}_{ij} - \mathbf{y}_i) \|^2 + \| \mathbf{T}_i^T \varepsilon_i \|^2 + \| \mathbf{T}_i^T \mathbf{e}_{ij} \|^2 \right] + \| \mathbf{y}_{ij} - \mathbf{y}_i \|^2. \quad (15) $$

Similarly, without using tangent coordinate, the error of distance based on the coordinate on the manifold, $\tilde{\boldsymbol{\tau}}_{ij} = (\mathbf{x}_{ij} - \mathbf{x}_i)$, and its true tangent coordinate is given by:

$$ \begin{aligned} E(\tilde{\boldsymbol{\tau}}_{ij}) &= (\| \mathbf{x}_{ij} - \mathbf{x}_i \| - \| \mathbf{y}_{ij} - \mathbf{y}_i \|)^2 \\ &\le 3 \left[ \| \mathbf{J}_F(\mathbf{y}_i)(\mathbf{y}_{ij} - \mathbf{y}_i) \|^2 + \| \varepsilon_i \|^2 + \| \mathbf{e}_{ij} \|^2 \right] \\ &\quad + \| \mathbf{y}_{ij} - \mathbf{y}_i \|^2. \end{aligned} \quad (16) $$

For each $\mathbf{v} \in \mathbb{R}^d$:

$$ \mathbf{v}^T \mathbf{T}_i \mathbf{T}_i^T \mathbf{v} \le \lambda_d \mathbf{v}^T \mathbf{v}, \quad (17) $$

where $\lambda_d$ denotes the largest eigenvalue of $\mathbf{T}_i \mathbf{T}_i^T$.

Since the non-zero eigenvalues of $\mathbf{T}_i \mathbf{T}_i^T$ are the same as those of $\mathbf{T}_i^T \mathbf{T}_i$ that all of them are equal to one, and by using $\| \mathbf{u} \|^2 = \mathrm{tr}(\mathbf{u}^T \mathbf{u})$ for any typical vector $\mathbf{u}$, relation (17) results in:

$$ \| \mathbf{T}_i^T \mathbf{v} \|^2 \le \| \mathbf{v} \|^2. \quad (18) $$

According to (15) and (18):

$$ \begin{aligned} & \| \mathbf{T}_i^T \mathbf{J}_F(\mathbf{y}_i)(\mathbf{y}_{ij} - \mathbf{y}_i) \|^2 + \| \mathbf{T}_i^T \varepsilon_i \|^2 + \| \mathbf{T}_i^T \mathbf{e}_{ij} \|^2 \\ & \le \| \mathbf{J}_F(\mathbf{y}_i)(\mathbf{y}_{ij} - \mathbf{y}_i) \|^2 + \| \varepsilon_i \|^2 + \| \mathbf{e}_{ij} \|^2. \end{aligned} \quad (19) $$

Finally, it can be concluded from relations (15), (16), and (19) that when data are mapped on the local tangent space, compared with the case that tangent coordinates are not used, the upper bound of $E(\boldsymbol{\tau}_{ij})$ is smaller than the upper bound of $E(\tilde{\boldsymbol{\tau}}_{ij})$. Therefore, the nearest neighbors are likely to be found more accurately, using projection on tangent spaces.

### C. Sparse Matching Simultaneous Diagonalization of Laplacians Using Local Tangent Spaces

Assume that the data samples in modalities $i$ and $j$ are located close to $\xi_i$-dimensional manifold $\mathcal{M}^i \subset \mathbb{R}^{d_i}$ and $\xi_j$-dimensional manifold $\mathcal{M}^j \subset \mathbb{R}^{d_j}$, respectively. At each point $\mathbf{x}_i \in \mathcal{M}^i$ on modality $i$, we define a $d_i \times \xi_i$ matrix $\mathbf{T}_i^x$ indicating the tangent space at $\mathbf{x}_i$, that all its neighboring points, $\mathcal{N}(\mathbf{x}_i)$, would be
represented by projecting on it. For example $\tau_{lr} = T_{x_l}^T (x_{lr} - x_l)$ is the projection of r-th neighbor of $x_l$ to its tangent space. As mentioned in the subSection II-E, local PCA is used to approximate $T_{x_l}^i$, such that $x_l$ and its k nearest neighbors are used to compute a local $d_i \times d_i$ covariance matrix, and the top $\xi_i$ eigenvectors of this matrix are used as the columns of matrix $T_{x_l}^i$.

Concluding from subSection III-A and as analyzed theoretically in subSection III-B, for two matching samples a and b in modalities i and j, respectively, samples whose projections on tangent space of a are among the nearest neighbors within all its neighboring projected samples on this tangent space are expected to have the most consistency with b. Similarly, the projected samples on tangent space of b that are in nearest neighbors of b among all its neighboring projected samples on tangent space of b are expected to have the most consistency with a.

Now, we can simultaneously diagonalize graph Laplacian matrices in modalities i and j by considering the limited given information and the additional obtained extended information.

The limited given information is the initial knowledge of problem that contains the sets of matching and mismatching samples between modalities. These information for each two modalities (e.g. modalities i and j) are coded in matrices $C_{i,j}$, $C_{j,i}$, $B_{i,j}$, and $B_{j,i}$ as explained in the Section II-B.

In order to obtain the extended matching information between modalities i and j, we define matrices $C'_{i,j}$ with dimensions $n_i \times q_{C'}$ and $C'_{j,i}$ with dimensions $n_j \times q_{C'}$, respectively. In these two matrices, $q_{C'} = q \times t_c$ is the number of extended matching samples and $t_c$ is the number of nearest neighbors in the tangent space used to extend matching information.

For each matching sample $l$ ($1 \le l \le q$) in modality i, indexed by $k_l^{i,j}$ in this modality, with $k_l^{i,j} \in \{1, 2, ..., n_i\}$, the values of $t_c$ columns of $C'_{i,j}$ are determined with $t_c$ nearest neighbors on the tangent space of sample $k_l^{i,j}$. For example, if $t_c = 3$, we consider the first, the second, and the third nearest neighbors in the tangent space of the matching sample $l$ as the extended matching samples and thus the values of 3 columns of $C'_{i,j}$ are determined. The values of the $C'_{j,i}$ are determined similarly.

Also, to obtain the extended mismatching information between modalities i and j, we use the $t_b$ farthest neighbors in the tangent space of matching samples in each modality and define matrices $B'_{i,j}$ with dimensions $n_i \times q_{B'}$ and $B'_{j,i}$ with dimensions $n_j \times q_{B'}$, where $q_{B'} = q \times t_b$. These two matrices require their respective low-dimensional embedding $B'_{i,j}V_i$ and $B'_{j,i}V_j$ to be as different as possible.

Having the above mentioned limited given information propagation idea, we can extend the problem (4) to express the problem of simultaneous approximate diagonalization of graph Laplacian matrices in m modalities as follows:

$$ \min_{V_i^T V_i = 1} \sum_{i=1}^{m} \text{off}(V_i^T L_i V_i) + \mu_c \left( \sum_{i,j=1}^{m} \| C_{i,j}^T V_i - C_{j,i}^T V_j \|_F^2 \right) $$

$$ \begin{aligned} & + \delta \sum_{i,j=1}^{m} \| C_{i,j}^{T} V_i - C_{j,i}^{T} V_j \|_F^2 \\ & - \mu_b \left( \sum_{i,j=1}^{m} \| B_{i,j}^T V_i - B_{j,i}^T V_j \|_F^2 \right) \\ & + \gamma \sum_{i,j=1}^{m} \| B_{i,j}^{T} V_i - B_{j,i}^{T} V_j \|_F^2 \end{aligned} \quad (20) $$

where $\delta$ and $\gamma$ are proper regularization coefficients between zero and one that reflect the importance rate of new matching and mismatching samples compared to original ones in the sparse simultaneous diagonalization.

Matrices $C'_{i,j}$ and $B'_{i,j}$ used in problem (20) are computed as follows. In modality i, for each sample $x_k$ ($1 \le k \le n_i$), we construct the tangent spaces projection matrices $T_{x_k}^i$ with dimension $d_i \times \xi_i$, as explained in Section II-E. For all samples in modality i, these matrices are collected in tensor $\mathcal{T}^i = [T_{x_1}^i \dots T_{x_{n_i}}^i]^T$. By using each column $l$ of matrix $C_i(c_l^i)$, we can construct columns $l, l+q, l+2q, \dots$, and $l+(t_c-1) \times q$ of matrix $C'_{i,j}$.

We use l-th column of matrix $C_{i,j}$ ($c_l^{i,j} = [c_{1,l}^{i,j} \dots c_{2,l}^{i,j} \dots c_{n_i,l}^{i,j}]^T$), which contains one at index $k_l^{i,j}$ and zero elsewhere, to construct the $t_c$ columns $c_l^{i,j}$, $c_{l+q}^{i,j}$, $c_{l+2q}^{i,j}$, ..., and $c_{l+(t_c-1) \times q}^{i,j}$ of $C'_{i,j} = [c_1^{i,j} \dots c_{q_{C'}}^{i,j}]$.

Inspired by equation (6), we obtain equation (21) to set the value of each entry of column $c_l^{i,j}$, denoted by $c_{l,i,j}^{i,j}$ ($1 \le t \le n_i$), and other columns mentioned above, by finding r-th nearest neighbors ($1 \le r \le t_c$) of $k_l^{i,j}$-th sample in each column $l$ ($1 \le l \le q$) of $C_i$.

For simplicity of understanding, we temporarily suppose that $t_c = 2$. If the indices of two nearest neighbors on tangent space of $k_l^{i,j}$-th sample are $s_1$ and $s_2$, respectively, then $c_{s_1,l}^{i,j} = \alpha$, $c_{s_2,l+q}^{i,j} = \alpha$ and other entries of columns $l$ and $l+q$ are set to zero. In general:

$$ c_{t_c,(l+(r-1)q)}^{i,j} = \begin{cases} \alpha & t = \underset{1 \le f \le k_c}{\operatorname{argmin}} \| \mathcal{T}^i c_l^{i,j} (\mathbf{x}_{l_f}^i - \mathbf{X}^i \mathbf{c}_l^{i,j}) \|_F^2 \\ f \notin \bigcup_{s=1}^{r-1} A(c_l^{i,j} + (s-1)q) & 0 \le f \le k_c \\ 0 & \text{otherwise} \end{cases} \quad (21) $$

where $0 \le \alpha \le 1$, $\mathbf{X}^i$ is data points matrix in modality i with dimension $n_i \times d_i$, $\mathbf{x}_{l_f}^i$ represents the f-th nearest neighbor of the sample coded in column $l$, $c_{t,b}^{i,j}$ is the entry in row $t$ and column $b$ of $C'_{i,j}$, $1 \le t \le n_i$ and $1 \le b \le q_{C'}$, and $A(\mathbf{y})$ represents non-zero elements $y_k$ of vector $\mathbf{y}$, i.e. $A(\mathbf{y}) = \{k|y_k \ne 0\}$. In equation (21), suppose $k_l^{i,j} = e$ ($1 \le e \le n_i$), then $\mathcal{T}^T c_l^{i,j} = T_{x_e}^i$, $\mathbf{X}^i \mathbf{c}_l^{i,j} = \mathbf{x}_e^i$, and the f-th nearest neighbor of e-th sample is $\mathbf{x}_{e_f}^i$. In order to find r-th nearest neighbor of e-th sample, $A(\cdot)$ contains all r-1 nearest neighbors which have already been selected, and $\operatorname{argmin} \| \cdot \|^2$ computes the r-th nearest neighbor of e-th sample. How to choose the value of $\alpha$ will be discussed in the following paragraphs.
<figure><img src="image_4.png" /><figcaption>Fig. 3. Graphical illustration of how the knowledge of the matching samples is expanded. Red dashed lines and blue dotted lines show distances on the manifold and tangent space, respectively. Sample with index $k_l^{i,j}$ in modality i is matching with sample with index $k_l^{j,i}$ in modality j. By assuming $t_c = 2$, if $s_1$ and $s_2$ are the indices of the two nearest neighbors on the tangent space of $k_l^{i,j}$-th sample, then these samples are matched with sample with index $k_l^{j,i}$ in modality j. Also, by assuming $t_b = 2$, if $f_1$ and $f_2$ are the indices of the two farthest neighbors on the tangent space of $k_l^{i,j}$-th sample, then, these samples are mismatched with sample with index $k_l^{j,i}$ in modality j. In the bottom, the procedure of assigning the entry of matrices $C'_{i,j}$ and $\mathbf{B}'_{i,j}$ is illustrated according to equations (21) and (22), respectively. In order to define new matching samples in matrix $C'_{i,j}$, in l-th column, entry in $s_1$-th row is set to $\alpha$ ($c_{s_1,l}^{i,j} = \alpha$), and other rows in this column are set to zero. Also, in $(l+q)$-th column of $C'_{i,j}$, entry in $s_2$-th row is set to $\alpha$ ($c_{s_2,l+q}^{i,j} = \alpha$), and other rows in this column are set to zero. Each column of the matching and mismatching matrices generates $t_c = 2$ and $t_b = 2$ shown columns of the extended matching and mismatching matrices, respectively. Defining extended mismatching samples in matrix $\mathbf{B}'_{i,j}$ is performed similarly.</figcaption></figure>

Similarly, we construct extended mismatching matrix $\mathbf{B}'_{i,j} = [\mathbf{b}'_{1,1} \dots \mathbf{b}'_{q_{B'}}]$ by using the original matching matrix $\mathbf{C}_i$. We use equation (22) to set the value of each entry of $\mathbf{b}'_{i,j}$, denoted by $b_{t,l}^{i,j}$ ($1 \le t \le n_i$), and other related columns (as mentioned before for extended matching matrix), by finding the p-th farthest neighbor ($1 \le p \le t_b$) of $k_l^{i,j}$-th sample in each column $l$ ($1 \le l \le q$) of matrix $\mathbf{C}_{i,j}$.

For simplicity of understanding, we temporarily suppose that $t_b = 2$. If the indices of the two farthest neighbors on tangent space of $k_l^{i,j}$-th sample are $f_1$ and $f_2$, respectively, then $b_{f_1,l}^{i,j} = \beta$, $b_{f_2,l+q}^{i,j} = \beta$ and other entries of columns $l$ and $l+q$ are set to zero. In general:

$$b_{t,(l+(p-1)q)}^{i,j} = \begin{cases} \beta & t=\underset{1 \le f \le k, f \notin \cup_{i=1}^{p-1} A(\mathbf{b}'_{l+(s-1)q})}{\operatorname{argmax}} \|\mathbf{T}^l \mathbf{c}_l^{i,j}(\mathbf{x}_f^l - \mathbf{X}^l \mathbf{c}_i^{j})\|^2} \\ 0 & \text{otherwise} \end{cases} \quad (22)$$

Parameters $\alpha$ and $\beta$ ($0 \le \beta \le 1$) are the impact coefficients of additional obtained matching and mismatching information. If $\alpha = 1$ the impact of additional obtained matching information and the impact of limited given matching information are equal. By substituting $\alpha$ with $(1 - \frac{r}{l_c})$ in equation (21), and $\beta$ with $(1 - \frac{p}{t_b})$ in equation (22), respectively, the impact of these two parameters can be reduced monotonically, concerning the distance from original samples.

Fig. 3 shows a graphical illustration of using equations (21) and (22) to expand the knowledge stemming from the matching samples.

Solving problem (20) simultaneously finds all $n_i$ approximate eigenvectors of the Laplacians, while in many problems we need only their first $k$ eigenvectors. Therefore, we solve the problem with a smaller matrix $\bar{\mathbf{V}}_i = [\mathbf{v}_{i,1} \dots \mathbf{v}_{i,k}]$ of size $n_i \times k$. To find this subset of columns of eigenvectors $\mathbf{V}_i$ that approximately diagonalize $\mathbf{L}_i$, we use the approach of [34], and reformulate the problem to (23), where $\bar{\mathbf{\Lambda}}_i = \operatorname{diag}(\lambda_{i,1}, \dots, \lambda_{i,k})$ contains the $k$ smallest eigenvalues of $\mathbf{L}_i$, and $\bar{\mathbf{V}}_i$ contains the approximate corresponding eigenvectors.

$$\begin{aligned} \min_{\bar{\mathbf{V}}_i^T \bar{\mathbf{V}}_i = \mathbf{I}} & \sum_{i=1}^{m} \| \bar{\mathbf{V}}_i^T \mathbf{L}_i \bar{\mathbf{V}}_i - \bar{\mathbf{\Lambda}}_i \|_F^2 \\ & + \mu_c \left( \sum_{i,j=1}^{m} \| \mathbf{C}_{i,j}^T \bar{\mathbf{V}}_i - \mathbf{C}_{j,i}^T \bar{\mathbf{V}}_j \|_F^2 \right. \\ & \qquad \left. + \delta \sum_{i,j=1}^{m} \| \mathbf{C}_{i,j}^T \bar{\mathbf{V}}_i - \mathbf{C}_{j,i}^T \bar{\mathbf{V}}_j \|_F^2 \right) \\ & - \mu_b \left( \sum_{i,j=1}^{m} \| \mathbf{B}_{i,j}^T \bar{\mathbf{V}}_i - \mathbf{B}_{j,i}^T \bar{\mathbf{V}}_j \|_F^2 \right. \\ & \qquad \left. + \gamma \sum_{i,j=1}^{m} \| \mathbf{B}_{i,j}^T \bar{\mathbf{V}}_i - \mathbf{B}_{j,i}^T \bar{\mathbf{V}}_j \|_F^2 \right) \end{aligned} \quad (23)$$

The number of variables in the above problem is $\sum_{i=1}^{m} n_i k$. That is one of the drawbacks of this framework if dealing with
very large datasets. To overcome this problem, we follow the subspace parametrization method proposed in [24]. It has been proven that the joint approximate eigenbasis can be written as the combination of the first k eigenvectors of Laplacian matrix [29].

We can approximate the first k vectors of the joint common eigenbase combining the first $k' \ge k$ eigenvectors of $L_i$, denoted $\overline{U}_i = [u_{i,1}...u_{i,k'}]$. By this approximation, $\overline{V}_i = \overline{U}_i A_i$, where $A_i$ is $k' \times k$ matrix of linear combination coefficients and $A_i^T A_i = I$. By substitution of the approximated eigenbase $\overline{U}_i$ in problem (23), we have:

$$ \begin{aligned} \min_{A_i^T A_i = I} & \sum_{i=1}^{m} \| A_i^T \overline{\Lambda}_i A_i - \overline{\Lambda}_i \|_F^2 \\ & + \mu_c \left( \sum_{i,j=1}^{m} \| C_{i,j}^T \overline{U}_i A_i - C_{j,i}^T \overline{U}_j A_j \|_F^2 \right. \\ & \qquad \left. + \delta \sum_{i,j=1}^{m} \| C_{i,j}^T \overline{U}_i A_i - C_{j,i}^T \overline{U}_j A_j \|_F^2 \right) \\ & - \mu_b \left( \sum_{i,j=1}^{m} \| B_{i,j}^T \overline{U}_i A_i - B_{j,i}^T \overline{U}_j A_j \|_F^2 \right. \\ & \qquad \left. + \gamma \sum_{i,j=1}^{m} \| B_{i,j}^T \overline{U}_i A_i - B_{j,i}^T \overline{U}_j A_j \|_F^2 \right) \end{aligned} \quad (24) $$

where $\overline{\Lambda}_i = \text{diag}(\lambda_{i,1},...,\lambda_{i,k'})$ is the diagonal matrix that contains the first $k'$ eigenvectors of $L_i$. The number of variables in this formulation is $m \times k' \times k$ that is independent of $n_i$.

**Optimization:** Problem (24) is a constrained optimization problem. In order to calculate the solution of this problem, we apply the standard constrained optimization tools such as fmincon in MATLAB used in [24]. The gradients of the objective function and the constraints given by the following equations are required for the optimization:

$$ \nabla_{A_i} \| A_i^T \overline{\Lambda}_i A_i - \overline{\Lambda}_i \|_F^2 = 4(\overline{\Lambda}_i A_i A_i^T \overline{\Lambda}_i A_i - \overline{\Lambda}_i A_i \overline{\Lambda}_i) \quad (25) $$

$$ \begin{aligned} & \nabla_{A_i} \| C_{i,j}^T \overline{U}_i A_i - C_{j,i}^T \overline{U}_j A_j \|_F^2 \\ & + \nabla_{A_i} \| C_{i,j}^T \overline{U}_i A_i - C_{j,i}^T \overline{U}_j A_j \|_F^2 \\ & = 2 \overline{U}_i^T C_{i,j} (C_{i,j}^T \overline{U}_i A_i - C_{j,i}^T \overline{U}_j A_j) \\ & + 2 \overline{U}_i^T C_{i,j}^T (C_{i,j}^T \overline{U}_i A_i - C_{j,i}^T \overline{U}_j A_j) \end{aligned} \quad (26) $$

$$ \begin{aligned} & \nabla_{A_i} \| B_{i,j}^T \overline{U}_i A_i - B_{j,i}^T \overline{U}_j A_j \|_F^2 \\ & + \nabla_{A_i} \| B_{i,j}^T \overline{U}_i A_i - B_{j,i}^T \overline{U}_j A_j \|_F^2 \\ & = 2 \overline{U}_i^T B_{i,j} (B_{i,j}^T \overline{U}_i A_i - B_{j,i}^T \overline{U}_j A_j) \\ & + 2 \overline{U}_i^T B_{i,j}^T (B_{i,j}^T \overline{U}_i A_i - B_{j,i}^T \overline{U}_j A_j) \end{aligned} \quad (27) $$

**Complexity analysis:** As discussed in Section II-B, by using q and u matching and mismatching samples, respectively, the computational cost contains the computing of $q \times k'$ matching

Fourier coefficients matrices and $u \times k'$ mismatching Fourier coefficients matrices between two modalities i and j. After extending the limited information and obtaining $t_c$ and $t_b$ additional matching and mismatching samples, the computational cost for each extended matching and mismatching samples are $t_c \times k'$ and $t_b \times k'$, respectively.

As a result, by assuming that the number of modalities m is small enough, since $t_c > q$ and $t_b > u$, computing the cost function and its gradient in each iteration of the optimization algorithm in both LSEJD (matching), and LSEJD (matching+mismatching), have a complexity of $O(t_c k'k)$ and $O((t_c + t_b)k'k)$, respectively.

**Summary:** The proposed procedure for approximating the common eigenbasis is summarized into the following main steps:

1) Compute the graph Laplacian matrix $L_i$, $1 \le i \le m$, in each modality.

2) Compute the tensor $T^T$, $1 \le i \le m$, which contains tangent space projection matrix for all samples in each modality.

3) Compute matching matrix $C'_{i,j}$, $1 \le i, j \le m$, between every two modalities i and j according to (21).

4) Compute mismatching matrix $B'_{i,j}$, $1 \le i, j \le m$, between every two modalities i and j according to (22).

5) Compute the gradients of the objective function and constraints according to (25), (26), and (27).

6) Compute the approximated eigenbases $\overline{V}_i = \overline{U}_i A_i$, $1 \le i \le m$, by solving (24), given the gradients of step (5).

## IV. EXPERIMENTAL RESULTS

In this section, we investigate the effectiveness and efficiency of our proposed LSEJD method in unsupervised and semi-supervised learning problems. We compare its performances with different state-of-the-art related multimodal learning methods. In the first subsection, we give a brief description of the datasets used for the validation of our proposed methods.

In the next subsections, we adopt two evaluation problems: multimodal unsupervised learning, and multimodal semi-supervised learning.

Since most recent related works on unsupervised learning are conducted on multi-view datasets, we compare our method with the latest work in multi-view clustering as a special kind of multimodal clustering.

### A. Description of the Datasets

We consider two categories: multimodal and multi-view datasets, respectively.

#### Multimodal datasets:

* Circle and Text datasets are two synthetic datasets with overlapping clusters. Each modality is corrupted with some noise and clusters which are close in one modality are far apart in the other one [24].

* NUS dataset is a web image dataset which includes images and their tags [35]. In our multimodal experiments, two defined modalities based on NUS are used [24].

* Caltech dataset is a subset of the Caltech-101 dataset [36]. This dataset considers seven classes that contain two
modalities. The first modality has images represented by bio-inspired features and their annotations in the 4 × 4 pyramid histogram of visual words (PHOW) constitute the second modality as mentioned in [24].

**Multi-view datasets:**

* Caltech-101 dataset in multi-view experiments is divided into two categories, Caltech-101-7, and Caltech-101-20. In Caltech-101-7, we select the seven widely used classes as in the multimodal experiments and get 1474 images. In Caltech-101-20, we select a larger set, which contains a total of 2386 images of 20 classes. Six types of features are extracted from all the images as mentioned in [37].

* Handwritten dataset consists of 2000 images of handwritten digits 0 to 9 with 200 samples in each category. We use 6 sets of features in multi-view experiments. These features are extracted from all the images and contain 240 pixel averages in 2 × 3 windows (PIX), 76 Fourier coefficients of the character shapes (FOU), 216 profile correlations (FAC), 64 Karhunen-love coefficients (KAR), 47 Zernike moment (ZER) and 6 morphological (MOR) features [37].

## B. Multimodal Unsupervised Learning

We evaluate our approach to the application of unsupervised learning via multimodal spectral clustering and compare the performance of this method with state-of-the-art multimodal unsupervised representation learning. In the unimodal spectral clustering method, the multiplicity of null eigenvalues of the Laplacian matrix is equal to the number of connected components in the data. The eigenvectors of the Laplacian matrix represent the indicator functions of these components. Then, K-means is used as a simple clustering method to perform clustering on each component. For all multimodal datasets, Gaussian weights with self-tuning widths are used to construct a graph whose Laplacian matrix is obtained for each modality [39].

We apply our approach to approximate a common eigenbasis for the Laplacian matrices of each dataset and apply spectral clustering using this approximate common eigenbasis for data clustering.

The micro-averaged accuracy (Accuracy) and the normalized mutual information (NMI), used in [24], are two standard criteria to evaluate the performance of model in multimodal unsupervised learning.

There are two versions of our proposed model:

1) $m$-LSEJD, which only contains matching matrix, with $\mu_c = 1$, and $\mu_b = 0$,

2) $m^2$-LSEJD, which contains matching and mismatching matrices, with $\mu_c = 1$, and $\mu_b = 1$.

Parameters $\delta$ and $\gamma$ are the coefficient of new matching and mismatching samples, respectively. All of these parameters are set to 0.9 (i.e. impact coefficients of new matching and mismatching samples are 90% of the original matching and mismatching samples). Also, parameters $\alpha$ and $\beta$ are set to 1.

In each experiment, a fraction of the samples in one modality is first randomly selected, and their matching samples in the other modalities are determined, in order to extend this sparse matching sampling. In each experiment, for a fair comparison, the average of ten runs is reported as the result.

Tables II and III show the performances of our proposed LSEJD approach regarding the different multimodal and multi-view clustering methods, respectively. In both tables, the best result is shown in bold.

As we can see from Table II in real-world datasets (Caltech and NUS), our proposed LSEJD approach performs better than full matching methods, especially the JADE method [28], because it tries to approximate a common set of eigenvectors for all Laplacian matrices while these two datasets have different number of samples and, unlike synthetic datasets, the explicit correspondence between the samples are not specified. Also, in comparison with other sparse matching methods, it performs better in most of the fractions of samples than CD sparse method [24], because it discovers matching and mismatching samples well and uses them to approximate the common eigenbasis. Laplacian averaging unimodal approaches (Harmonic Mean and Arithmetic Mean) [24], and non-spectral multimodal approaches (Comraf [39], MVSC [38], MultiNMF [36], and SC-ML [40]) have low performances on the two synthetic datasets (Circle and Text) due to the characteristics of those datasets. As we can see, multimodal clustering approaches provide better performances for those datasets, and among them, JADE [28] method has the best results because it is a full-matching approach and those datasets have full-matching modalities (unlike real-world datasets).

For the multi-view datasets, unlike the multimodal datasets, the numbers of samples in all modalities are equal. They have full matching samples. In order to demonstrate the efficiency of our proposed approach to expanding the knowledge of the matching samples, we use 60% of the existing limited information about matching samples in multi-view clustering and compare this with the state-of-the-art method. The results are reported in Table III. We can see that although several competing multimodal and multi-view spectral clustering methods (containing CD, SwMPC, DwMPC, SwMPC, MSC, and MLAN algorithms) achieve good performances in some experiments, our proposed LSEJD approach is able to achieve better results than most of them. This is especially remarkable since it is using much less samples. Our proposed approach can be compared to neural networks-based methods, e.g. SGLRL [7]. These methods often lead to very good performances in largescale datasets. However, with moderate-sized datasets, our proposed LSEJD method provides better results with less complexity.

In this experiment, our LSEJD method uses the first nearest/farthest samples in the tangent space of each matching sample pair, respectively. We also evaluate the effect of other nearest and farthest samples in the tangent space of each matching sample pairs. The results of these experiments are summarized in Table IV. The results confirm that the nearest and farthest projected samples on the tangent space are the most meaningful ones for correspondence and non-correspondence propagation, respectively.

We observe that in all the experiments, using the first nearest and farthest samples in the tangent space provides the best performance for multimodal clustering. These results show the positive impact of using the intrinsic local tangent spaces of underlying data manifolds in order to discover the local geometrical structures of matching and mismatching samples and
<table><caption>TABLE II
PERFORMANCES OF DIFFERENT MULTIMODAL CLUSTERING METHODS (ACCURACY(%)/NMI(%))</caption><thead><tr><th rowspan="2">Method</th><th colspan="4">Multimodal datasets</th></tr><tr><th>Circle</th><th>Text</th><th>Caltech [23],[38]</th><th>NUS [23],[35]</th></tr></thead><tbody><tr><td>Harmonic Mean</td><td>95.6/90.1</td><td>97.2/91.0</td><td>84.8/79.2</td><td>89.0/83.8</td></tr><tr><td>Arithmetic Mean</td><td>96.5/91.2</td><td>96.9/89.6</td><td>88.6/83.1</td><td>95.2/92.1</td></tr><tr><td>Comraf [39]</td><td>40.8/16.9</td><td>60.8/41.7</td><td>-</td><td>86.9/84.3</td></tr><tr><td>MVSC [38]</td><td>95.6/90.1</td><td>97.2/91.2</td><td>85.7/80.8</td><td>89.0/83.8</td></tr><tr><td>MultiNMF [36]</td><td>41.1/14.2</td><td>50.5/23.2</td><td>-</td><td>77.4/79.3</td></tr><tr><td>SC-ML [40]</td><td>98.2/94.6</td><td>97.8/93.1</td><td>88.6/81.6</td><td>94.5/90.7</td></tr><tr><td>JADE [23]</td><td><b>100/100</b></td><td><b>98.4/94.1</b></td><td>86.7/80.6</td><td>93.1/87.5</td></tr><tr><td rowspan="3">CD (pos) [24]</td><td>10%*</td><td>52.5/26.0</td><td>54.5/26.2</td><td>78.7/75.3</td><td>78.6/77.9</td></tr><tr><td>20%</td><td>61.3/40.2</td><td>60.0/41.9</td><td>80.8/76.0</td><td>82.9/78.2</td></tr><tr><td>60%</td><td>93.7/85.4</td><td>86.5/69.7</td><td>87.0/80.0</td><td>87.2/78.9</td></tr><tr><td rowspan="3">CD (pos+neg) [24]</td><td>100%</td><td>98.9/95.5</td><td>96.8/89.4</td><td>89.5/83.3</td><td>94.5/90.6</td></tr><tr><td>10%</td><td>67.3/46.5</td><td>63.6/42.1</td><td>86.5/80.9</td><td>92.7/86.2</td></tr><tr><td>20%</td><td>69.6/50.2</td><td>67.8/50.0</td><td>87.9/81.2</td><td>93.3/87.0</td></tr><tr><td rowspan="3">m-LSEJD</td><td>60%</td><td>95.2/87.9</td><td>87.0/68.5</td><td>89.2/84.0</td><td>94.5/88.5</td></tr><tr><td>10%</td><td>67.7/47.7</td><td>74.2/55.2</td><td>81.3/75.7</td><td>82.0/75.6</td></tr><tr><td>20%</td><td>72.6/67.9</td><td>77.0/61.3</td><td>83.5/79.0</td><td>80.6/74.6</td></tr><tr><td rowspan="3">m²-LSEJD</td><td>60%</td><td>78.8/67.5</td><td>82.8/66.3</td><td>84.5/79.5</td><td>91.0/87.7</td></tr><tr><td>100%</td><td>81.5/70.4</td><td>82.9/65.3</td><td>90.5/85.2</td><td><b>96.6/93.1</b></td></tr><tr><td>10%</td><td>79.8/64.5</td><td>79.4/54.2</td><td>85.7/77.0</td><td>86.3/83.2</td></tr><tr><td rowspan="2">m²-LSEJD</td><td>20%</td><td>92.9/84.5</td><td>85.9/65.9</td><td>88.6/81.7</td><td>87.0/82.7</td></tr><tr><td>60%</td><td>97.2/91.2</td><td>88.9/72.0</td><td><b>92.2/92.8</b></td><td>93.7/90.7</td></tr></tbody></table>

<table><caption>TABLE III
PERFORMANCES OF DIFFERENT MULTI-VIEW CLUSTERING METHODS
(ACCURACY(%)/NMI(%))</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">Multi-view datasets</th></tr><tr><th>Caltech101-7 [37]</th><th>Caltech101-20 [37]</th><th>Handwritten [37]</th></tr></thead><tbody><tr><td>MVSC [37]</td><td>53.4/52.9</td><td>51.9/53.8</td><td>81.0/83.4</td></tr><tr><td>AMGC [41]</td><td>66.0/56.1</td><td>52.4/54.4</td><td>87.5/87.5</td></tr><tr><td>MLAN [41]</td><td>78.0/63.0</td><td>53.3/47.6</td><td>97.3/93.9</td></tr><tr><td>SwMC [42]</td><td>64.8/55.9</td><td>41.9/30.8</td><td>84.5/89.4</td></tr><tr><td>MSC [43]</td><td>65.7/43.5</td><td>51.0/33.0</td><td>58.7/45.0</td></tr><tr><td>CSMSC [44]</td><td>63.6/54.7</td><td>44.0/56.4</td><td>90.9/84.1</td></tr><tr><td>DwMPC [45]</td><td>69.4/63.3</td><td>50.2/62.2</td><td>97.0/93.4</td></tr><tr><td>SwMPC [45]</td><td>64.8/54.5</td><td><b>57.3/68.3</b></td><td>88.1/91.2</td></tr><tr><td>SCMV-3DT [46]</td><td>62.4/60.3</td><td>-</td><td>93.0/86.0</td></tr><tr><td>SGLRL [7]</td><td>80.1/85.3</td><td>-</td><td>95.7/91.6</td></tr><tr><td>CD (pos) [24]</td><td>71.4/61.4</td><td>48.0/35.9</td><td>97.3/94.0</td></tr><tr><td>CD (pos+neg) [24]</td><td>79.8/64.1</td><td>49.6/37.3</td><td>97.7/94.4</td></tr><tr><td>m-LSEJD</td><td>76.7/70.1</td><td>54.5/36.2</td><td>96.8/92.4</td></tr><tr><td>m²-LSEJD</td><td><b>81.7/72.7</b></td><td>56.3/39.7</td><td><b>97.8/94.9</b></td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Fig. 4. Performance variation by considering a relaxed constraint among far samples with respect to $k_1$ (number of selected farthest neighbors), and $k_2$ (number of samples randomly selected among $k_1$ samples) on two datasets Caltech (left) and NUS (right).</figcaption></figure>

prove that the first nearest sample in the tangent space of each sample is indeed its most similar one.

Also, in this experiment, our LSEJD approach uses only one nearest/farthest sample in the tangent space of each matching sample pair. The effect of using more than one nearest sample or one farthest sample in the tangent space of matching sample pairs are shown in Table V. The results show that with each fraction of data on most of the datasets, using more than one nearest/farthest sample in the tangent space improves the accuracy of spectral clustering, although this improvement is not monotonic.

Although using more samples in the tangent space of each matching sample pair provides more accurate results, due to the use of matching and mismatching matrices with larger dimensions, it has a higher computational complexity.

To further study the effect of considering far samples from a given point, we relax the constraint by replacing the farthest samples with a set of far enough ones. In this experiment, the first $k_1 \in \{1, \dots, 10\}$ farthest neighbors are selected. Then, in each

group of selected samples, the effect of random selection of $k_2 \in \{1, \dots, 5\}$ samples among them is evaluated. Fig. 4 shows the results of this experiment in the Caltech and NUS datasets. It can be seen that by relaxing the range of selecting the farthest neighbors between 1 and 10, the efficiency gradually decreases.

In order to demonstrate the advantage of using the intrinsic tangent spaces of the manifold to locally extend the correspondence information of matching samples, we compare this with a simpler idea of using the Euclidean distance on the nearest neighbor graph of the data to find the best matching/mismatching samples. The results of this experiment, summarized in Tables VI and VII, show that in all datasets, using nearest neighbors on the intrinsic tangent spaces of the manifold is more effective.

Fig. 5 shows the results of using the two above approaches to find the best matching/mismatching samples on several samples of the NUS dataset. For each of 4 sample images, the first row shows the best matching/mismatching samples found using the local tangent space of manifold and the second row shows the best matching/mismatching samples found using the Euclidean distance on the nearest neighbor graph of data. Since the first

<footnote>* the percentage of samples randomly selected for matching and/or mismatching in each modality</footnote>
<table><caption>TABLE IV
EFFECT OF USING ONLY *k*-TH NEAREST AND FARTHEST SAMPLES (*k* = 1, 2, 3, 4, 5), FOR CORRESPONDENCE AND NON-CORRESPONDENCE PROPAGATION (ACCURACY(%)/NMI(%))</caption><thead><tr><th rowspan="2"></th><th colspan="5">m-LSEJD</th><th colspan="5">m²-LSEJD</th></tr><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th></tr></thead><tbody><tr><td rowspan="4">Circle</td><td>10%</td><td><b>67.7/47.7</b></td><td>60.5/40.6</td><td>57.3/36.8</td><td>57.1/34.7</td><td>58.7/37.0</td><td><b>79.8/64.5</b></td><td>62.3/41.0</td><td>60.9/39.4</td><td>58.0/35.7</td><td>57.6/37.3</td></tr><tr><td>20%</td><td><b>72.6/67.9</b></td><td>66.1/47.0</td><td>64.2/44.5</td><td>63.7/44.3</td><td>63.4/45.1</td><td><b>92.9/84.5</b></td><td>69.4/49.0</td><td>66.7/46.1</td><td>66.0/45.2</td><td>66.4/45.5</td></tr><tr><td>60%</td><td><b>78.8/67.5</b></td><td>77.6/64.4</td><td>78.4/65.2</td><td>78.3/65.6</td><td>77.2/64.6</td><td><b>97.2/91.2</b></td><td>95.3/86.8</td><td>95.9/88.2</td><td>95.1/86.2</td><td>95.2/86.6</td></tr><tr><td>100%</td><td><b>81.5/70.4</b></td><td>81.5/70.4</td><td>81.5/70.4</td><td>81.5/70.4</td><td>81.5/70.4</td><td><b>79.4/54.2</b></td><td>49.3/26.4</td><td>64.9/45.9</td><td>61.8/36.1</td><td>63.9/42.9</td></tr><tr><td rowspan="4">Text</td><td>10%</td><td><b>74.2/55.2</b></td><td>63.2/42.3</td><td>55.6/31.2</td><td>60.6/39.3</td><td>55.0/33.1</td><td><b>85.9/65.9</b></td><td>70.7/43.1</td><td>77.3/51.6</td><td>78.2/55.4</td><td>71.5/46.0</td></tr><tr><td>20%</td><td><b>77.0/61.3</b></td><td>63.8/43.8</td><td>67.0/48.5</td><td>66.3/48.2</td><td>65.9/48.1</td><td><b>88.9/72.0</b></td><td>88.5/71.9</td><td>88.1/70.3</td><td>88.1/70.9</td><td>88.1/70.9</td></tr><tr><td>60%</td><td><b>82.8/66.3</b></td><td>82.1/65.0</td><td>82.2/65.2</td><td>82.2/65.4</td><td>82.2/65.2</td><td><b>85.7/77.0</b></td><td>82.4/74.7</td><td>78.4/71.7</td><td>77.1/71.2</td><td>78.1/73.6</td></tr><tr><td>100%</td><td><b>82.9/65.3</b></td><td>82.9/65.3</td><td>82.9/65.3</td><td>82.9/65.3</td><td>82.9/65.3</td><td><b>88.6/81.7</b></td><td>88.6/81.2</td><td>87.1/80.3</td><td>76.7/70.0</td><td>76.7/71.5</td></tr><tr><td rowspan="4">Caltech</td><td>10%</td><td><b>81.3/75.7</b></td><td>79.7/72.0</td><td>80.3/73.6</td><td>80.6/74.9</td><td>77.1/71.6</td><td><b>92.2/92.8</b></td><td>90.0/84.2</td><td>87.1/79.1</td><td>87.1/80.5</td><td>88.6/82.1</td></tr><tr><td>20%</td><td><b>83.5/79.0</b></td><td>79.7/73.8</td><td>79.0/73.5</td><td>80.0/75.2</td><td>76.8/69.8</td><td><b>86.3/83.2</b></td><td>85.3/80.0</td><td>84.6/79.5</td><td>83.0/78.4</td><td>83.2/78.0</td></tr><tr><td>60%</td><td><b>84.5/79.5</b></td><td>79.7/74.8</td><td>81.0/77.7</td><td>79.7/74.6</td><td>81.0/77.9</td><td><b>87.0/82.7</b></td><td>84.4/79.2</td><td>84.0/77.5</td><td>84.8/78.7</td><td>84.6/78.6</td></tr><tr><td>100%</td><td><b>90.5/85.2</b></td><td>90.5/85.2</td><td>90.5/85.2</td><td>90.5/85.2</td><td>90.5/85.2</td><td><b>89.7/83.2</b></td><td>84.3/78.3</td><td>85.1/78.5</td><td>86.1/79.2</td><td>83.4/77.5</td></tr><tr><td rowspan="4">NUS</td><td>10%</td><td><b>82.0/75.6</b></td><td>81.3/75.5</td><td>78.9/73.8</td><td>80.6/75.2</td><td>80.9/77.0</td><td><b>86.3/83.2</b></td><td>85.3/80.0</td><td>84.6/79.5</td><td>83.0/78.4</td><td>83.2/78.0</td></tr><tr><td>20%</td><td><b>80.6/74.6</b></td><td>72.6/65.4</td><td>75.3/68.1</td><td>74.4/66.8</td><td>76.4/69.4</td><td><b>87.0/82.7</b></td><td>84.4/79.2</td><td>84.0/77.5</td><td>84.8/78.7</td><td>84.6/78.6</td></tr><tr><td>60%</td><td><b>91.0/87.7</b></td><td>89.9/84.8</td><td>89.7/84.0</td><td>89.0/83.7</td><td>88.5/83.4</td><td><b>89.7/83.2</b></td><td>84.3/78.3</td><td>85.1/78.5</td><td>86.1/79.2</td><td>83.4/77.5</td></tr><tr><td>100%</td><td><b>96.6/93.1</b></td><td>96.6/93.1</td><td>96.6/93.1</td><td>95.9/91.7</td><td>96.6/93.1</td><td><b>94.8/85.0</b></td><td>92.9/80.8</td><td>96.8/90.4</td><td>97.0/91.3</td><td>97.1/91.3</td></tr></tbody></table>

<table><caption>TABLE V
EFFECT OF USING A DIFFERENT NUMBERS OF NEAREST AND FARTHEST SAMPLES (*k*) FOR CORRESPONDENCE AND NON-CORRESPONDENCE PROPAGATION, RESPECTIVELY (ACCURACY(%)/NMI(%)) (*k* = 1, 2, 3, 4, 5)</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="5">m-LSEJD</th><th colspan="5">m²-LSEJD</th></tr><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th></tr></thead><tbody><tr><td rowspan="4">Circle</td><td>10%</td><td>67.7/47.7</td><td>67.2/47.8</td><td>72.6/55.9</td><td>72.2/57.2</td><td><b>73.7/58.9</b></td><td>79.8/64.5</td><td>74.8/53.8</td><td>82.2/65.5</td><td>92.9/80.8</td><td><b>94.8/85.0</b></td></tr><tr><td>20%</td><td>72.6/67.9</td><td>75.2/62.8</td><td>75.8/63.4</td><td>78.2/66.5</td><td><b>80.1/68.2</b></td><td>92.9/84.5</td><td>95.0/85.5</td><td>96.2/88.5</td><td>96.8/90.4</td><td><b>97.0/91.0</b></td></tr><tr><td>60%</td><td>78.8/67.5</td><td>80.8/70.7</td><td>81.1/71.5</td><td>82.3/69.3</td><td><b>84.2/70.6</b></td><td>97.2/91.2</td><td>97.3/91.5</td><td><b>97.3/91.7</b></td><td>97.0/91.1</td><td>97.1/91.3</td></tr><tr><td>10%</td><td>74.2/55.2</td><td>79.3/58.5</td><td>79.2/61.5</td><td>80.0/61.7</td><td><b>81.3/63.2</b></td><td>79.4/54.2</td><td>83.9/62.8</td><td>85.0/64.4</td><td>86.6/68.8</td><td><b>88.0/71.2</b></td></tr><tr><td rowspan="4">Text</td><td>20%</td><td>77.0/61.3</td><td>82.4/63.8</td><td>81.7/64.0</td><td>82.2/63.6</td><td><b>81.9/63.4</b></td><td>85.9/65.9</td><td>87.1/69.2</td><td>87.5/70.5</td><td>89.0/73.0</td><td><b>89.1/73.8</b></td></tr><tr><td>60%</td><td>82.8/66.3</td><td><b>82.5/64.6</b></td><td>82.4/64.4</td><td>82.4/64.4</td><td>82.3/64.4</td><td>88.9/72.0</td><td>89.6/74.1</td><td><b>90.1/75.4</b></td><td>89.1/74.3</td><td>89.1/74.1</td></tr><tr><td>10%</td><td>81.3/75.7</td><td>83.8/78.3</td><td>81.3/75.2</td><td>81.9/78.3</td><td><b>90.2/84.0</b></td><td>85.7/77.0</td><td>84.3/75.0</td><td>84.8/78.3</td><td>85.7/78.6</td><td><b>89.5/82.1</b></td></tr><tr><td>20%</td><td>83.5/79.0</td><td>82.5/77.6</td><td>84.1/79.3</td><td>88.6/83.5</td><td><b>90.2/83.8</b></td><td>88.6/81.7</td><td>89.5/82.7</td><td>89.5/83.8</td><td><b>93.8/90.3</b></td><td>90.0/84.5</td></tr><tr><td rowspan="4">Caltech</td><td>60%</td><td>84.5/79.5</td><td>92.1/86.7</td><td>91.4/87.0</td><td>94.0/91.4</td><td><b>94.3/91.6</b></td><td>92.2/92.8</td><td>91.4/86.6</td><td>97.1/95.7</td><td><b>100.0/100.0</b></td><td>100.0/100.0</td></tr><tr><td>10%</td><td>82.0/75.6</td><td>76.6/69.4</td><td>79.3/71.2</td><td>81.9/75.9</td><td><b>84.4/79.5</b></td><td>86.3/83.2</td><td>86.1/81.0</td><td>85.2/78.7</td><td>85.4/79.8</td><td><b>86.4/81.3</b></td></tr><tr><td>20%</td><td>80.6/74.6</td><td>85.5/81.4</td><td>88.6/83.7</td><td>87.1/82.9</td><td><b>91.0/86.5</b></td><td>87.0/82.7</td><td>85.8/79.8</td><td>86.7/81.7</td><td>88.1/82.7</td><td><b>88.9/82.8</b></td></tr><tr><td>60%</td><td>91.0/87.7</td><td>93.8/89.7</td><td>93.6/90.6</td><td>95.0/90.8</td><td><b>95.0/91.6</b></td><td>89.7/83.2</td><td>91.3/87.9</td><td>90.5/86.3</td><td>90.8/84.5</td><td><b>91.5/85.8</b></td></tr></tbody></table>

<table><caption>TABLE VI
COMPARISON OF THE EFFECT OF TWO APPROACHES IN *m*-LSEJD METHOD: FIRST NEAREST NEIGHBOR ON THE LOCAL TANGENT SPACE OF THE MANIFOLD AND *k* NEAREST NEIGHBORS BASED ON THE EUCLIDEAN DISTANCE ON THE NEAREST NEIGHBOR GRAPH OF THE DATA (*k* = 1, 2, 3, 4, 5) (ACCURACY(%)/NMI(%))</caption><thead><tr><th rowspan="2">m-LSEJD</th><th rowspan="2">First nearest neighbor on the local tangent space</th><th colspan="5">k nearest neighbors based on Euclidean distance on the nearest neighbor graph of the manifold</th></tr><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th></tr></thead><tbody><tr><td rowspan="4">Circle</td><td>10%</td><td><b>67.7/47.7</b></td><td>63.0/45.8</td><td>62.3/45.8</td><td>63.4/46.3</td><td>62.5/45.9</td><td>65.5/48.8</td></tr><tr><td>20%</td><td><b>72.6/67.9</b></td><td>67.0/52.0</td><td>68.2/51.9</td><td>67.4/51.3</td><td>69.8/53.7</td><td>69.2/54.7</td></tr><tr><td>60%</td><td><b>78.8/67.5</b></td><td>77.0/64.5</td><td>77.7/65.0</td><td>77.6/64.9</td><td>78.2/66.4</td><td>77.0/63.6</td></tr><tr><td>100%</td><td><b>81.5/70.4</b></td><td>78.0/65.9</td><td>77.3/65.6</td><td>78.4/66.2</td><td>78.1/65.6</td><td>78.0/65.4</td></tr><tr><td rowspan="4">Text</td><td>10%</td><td><b>74.2/55.2</b></td><td>65.4/43.4</td><td>64.8/43.9</td><td>64.3/41.2</td><td>67.3/44.0</td><td>64.8/43.1</td></tr><tr><td>20%</td><td><b>77.0/61.3</b></td><td>66.9/45.8</td><td>69.9/47.6</td><td>66.7/47.3</td><td>67.2/47.9</td><td>69.4/49.7</td></tr><tr><td>60%</td><td><b>82.8/66.3</b></td><td>67.2/48.6</td><td>67.7/49.0</td><td>66.2/47.6</td><td>66.3/49.3</td><td>64.7/49.2</td></tr><tr><td>100%</td><td><b>82.9/65.3</b></td><td>62.5/55.3</td><td>62.5/55.6</td><td>62.5/55.6</td><td>62.5/55.3</td><td>62.5/55.3</td></tr><tr><td rowspan="4">Caltech</td><td>10%</td><td><b>81.3/75.7</b></td><td>79.1/73.9</td><td>79.8/73.6</td><td>79.8/73.3</td><td>77.2/71.3</td><td>79.9/72.8</td></tr><tr><td>20%</td><td><b>83.5/79.0</b></td><td>79.0/73.3</td><td>79.9/74.2</td><td>79.8/73.4</td><td>78.9/72.8</td><td>81.6/75.4</td></tr><tr><td>60%</td><td><b>84.5/79.5</b></td><td>82.6/76.4</td><td>83.3/77.0</td><td>83.3/77.0</td><td>84.1/77.3</td><td>83.6/76.9</td></tr><tr><td>100%</td><td><b>90.5/85.2</b></td><td>87.6/80.0</td><td>88.6/82.9</td><td>88.6/82.9</td><td>83.8/76.2</td><td>88.6/82.9</td></tr><tr><td rowspan="4">NUS</td><td>10%</td><td><b>82.0/75.6</b></td><td>81.3/75.5</td><td>79.4/74.3</td><td>78.0/73.0</td><td>77.2/72.5</td><td>77.1/73.2</td></tr><tr><td>20%</td><td>80.6/74.6</td><td><b>81.8/75.2</b></td><td>80.2/73.0</td><td>80.5/74.2</td><td>80.8/74.5</td><td>79.2/72.9</td></tr><tr><td>60%</td><td><b>91.0/87.7</b></td><td>84.7/81.2</td><td>88.5/83.1</td><td>85.9/80.9</td><td>86.8/82.7</td><td>87.2/82.8</td></tr><tr><td>100%</td><td><b>96.6/93.1</b></td><td>90.3/84.8</td><td>89.7/83.3</td><td>90.3/84.2</td><td>89.7/83.3</td><td>91.7/85.4</td></tr></tbody></table>
<table><caption>TABLE VII
COMPARISON OF THE EFFECT OF THE TWO APPROACHES IN $m^2$-LSEJD METHOD: FIRST NEAREST NEIGHBOR ON THE LOCAL TANGENT SPACE OF THE MANIFOLD AND $k$ NEAREST/FARTHEST NEIGHBORS BASED ON THE EUCLIDEAN DISTANCE ON NEAREST NEIGHBOR GRAPH OF THE DATA ($k = 1, 2, 3, 4, 5$), RESPECTIVELY (ACCURACY(%)/NMI(%))</caption><thead><tr><th rowspan="2">m<sup>2</sup>-LSEJD</th><th rowspan="2">First nearest/farthest neighbor on the local tangent space</th><th colspan="5">k nearest neighbors based on Euclidean distance on the nearest neighbor graph of the manifold</th></tr><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th></tr></thead><tbody><tr><td rowspan="3">Circle</td><td>10%</td><td><b>79.8/64.5</b></td><td>78.9/66.9</td><td>68.5/59.1</td><td>78.0/63.6</td><td>77.9/63.5</td><td>78.9/61.0</td></tr><tr><td>20%</td><td><b>92.9/84.5</b></td><td>89.9/73.3</td><td>78.6/54.2</td><td>74.8/56.3</td><td>70.5/59.1</td><td>78.8/67.0</td></tr><tr><td>60%</td><td><b>97.2/91.2</b></td><td>76.6/57.1</td><td>78.5/63.3</td><td>73.5/57.2</td><td>79.1/66.8</td><td>88.2/70.2</td></tr><tr><td rowspan="3">Text</td><td>10%</td><td><b>79.4/54.2</b></td><td>68.8/53.2</td><td>76.2/65.8</td><td>73.6/61.6</td><td>72.3/58.2</td><td>70.3/55.7</td></tr><tr><td>20%</td><td><b>85.9/65.9</b></td><td>83.8/73.5</td><td>69.9/54.4</td><td>73.3/57.2</td><td>73.1/62.8</td><td>74.7/70.7</td></tr><tr><td>60%</td><td><b>88.9/72.0</b></td><td>73.0/60.9</td><td>75.8/64.3</td><td>72.7/60.3</td><td>72.9/66.1</td><td>72.7/60.0</td></tr><tr><td rowspan="3">Caltech</td><td>10%</td><td><b>85.7/77.0</b></td><td>84.8/76.1</td><td>82.9/75.7</td><td>82.9/73.4</td><td>77.1/73.1</td><td>80.0/71.2</td></tr><tr><td>20%</td><td><b>88.6/81.7</b></td><td>88.6/81.3</td><td>75.2/63.9</td><td>87.6/80.2</td><td>77.1/71.7</td><td>83.8/72.8</td></tr><tr><td>60%</td><td><b>92.2/92.8</b></td><td>69.5/67.1</td><td>72.4/69.7</td><td>77.1/66.2</td><td>89.5/84.4</td><td>81.9/71.6</td></tr><tr><td rowspan="3">NUS</td><td>10%</td><td><b>86.3/83.2</b></td><td>81.3/76.6</td><td>79.5/76.0</td><td>81.4/76.7</td><td>81.7/76.2</td><td>9.6/73.8</td></tr><tr><td>20%</td><td><b>87.0/82.7</b></td><td>84.7/78.5</td><td>82.7/76.3</td><td>84.2/77.5</td><td>84.8/78.0</td><td>81.2/75.0</td></tr><tr><td>60%</td><td><b>93.7/90.7</b></td><td>88.8/84.6</td><td>89.1/84.8</td><td>90.0/85.8</td><td>90.3/86.4</td><td>88.7/84.0</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Fig. 5. Comparison of the two approaches to find the best matching/mismatching samples on 4 images of the NUS dataset. In the first column, the shown image samples in modality 1 are matching ones with the shown samples in modality 2, (each image in modality 1 is matching with its tags represented by 1000-dimensional bag of words in modality 2). The best matching/mismatching samples of each 4 images are found in modality 1. The first row of each example shows the best matching/mismatching samples found using the local tangent space of the manifold and the second row shows the best matching/mismatching samples found using the Euclidean distance on the nearest neighbor graph of data. It can be seen that the best matching/mismatching samples of modality 1 found using the local tangent space of the manifold are the most similar/different samples with this counterpart sample in modality 2.</figcaption></figure>

nearest/farthest neighbor on local tangent space of manifold of each sample is its most similar/different sample, one can easily see that the nearest/farthest neighbors of each sample expand well the local information of matching samples to their neighbors. According to this figure, the first nearest neighbor on local tangent space of each sample in modality 1 is matched with its corresponding samples in modality 2, and the first farthest neighbor of local tangent space of each sample in modality 1 is mismatched with its corresponding samples in modality

2. For the first sample in this figure (the flower), image 1 in the best matching samples found using local tangent space manifold in modality 1 is matched correctly with tags 'red,' 'macro,' 'flower,' 'flowers,' 'holiday,' 'rose' in modality 2, and image 1 in the best mismatching samples found using local tangent space is mismatched correctly with these tags. It is evident that this is not the case for best matching/mismatching samples found using Euclidean distance on the nearest neighbor graph of data. We can also see from this figure that the first
<table><caption>TABLE VIII
CLASSIFICATION RESULTS (MEAN ACCURACY (%) ± STANDARD DEVIATION)</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">Multimodal datasets</th></tr><tr><th>Caltech [23],[38]</th><th>NUS [23],[35]</th></tr></thead><tbody><tr><td>unimodal1</td><td>77.3±0.6</td><td>81.5±0.5</td></tr><tr><td>unimodal2</td><td>82.1±0.3</td><td>70.0±0.5</td></tr><tr><td>JADE [23]</td><td>86.7±0.2</td><td>89.8±0.3</td></tr><tr><td rowspan="5">CD (pos) [24]</td><td>10%</td><td>76.9±1.1</td><td>81.8±0.8</td></tr><tr><td>20%</td><td>76.8±0.7</td><td>81.4±0.5</td></tr><tr><td>60%</td><td>78.4±0.5</td><td>81.9±0.1</td></tr><tr><td>100%</td><td>80.1±0.1</td><td>82.2±0.5</td></tr><tr><td>10%</td><td>73.4±0.8</td><td>80.3±0.4</td></tr><tr><td rowspan="5">CD (pos+neg) [24]</td><td>20%</td><td>73.7±0.3</td><td>78.9±0.1</td></tr><tr><td>60%</td><td>83.3±0.8</td><td>86.1±0.2</td></tr><tr><td>10%</td><td>84.1±1.4</td><td>83.2±1.3</td></tr><tr><td>20%</td><td>82.2±2.4</td><td>87.1±1.5</td></tr><tr><td>60%</td><td>82.7±1.1</td><td>90.2±0.6</td></tr><tr><td rowspan="3">m-LSEJD</td><td>100%</td><td>89.5±0.2</td><td>90.9±0.3</td></tr><tr><td>10%</td><td>88.5±1.6</td><td>87.2±1.1</td></tr><tr><td>20%</td><td>89.1±0.4</td><td>89.6±0.4</td></tr><tr><td rowspan="2">m²-LSEJD</td><td>60%</td><td><b>90.3±0.5</b></td><td><b>91.6±0.1</b></td></tr><tr><td></td><td></td><td></td></tr></tbody></table>

matching/mismatching sample found using local tangent space are the most meaningful one for correspondence/non-correspondence propagation, respectively, as shown in Tables VI and VII.

### C. Multimodal Semi-Supervised Learning

For more experiments, we use our proposed approach in a semi-supervised learning framework and evaluate it in classification tasks. We use the diffusion distance computed using the approximate common eigenvectors. We compute the diffusion distance by using the first t approximate common eigenvectors of the Laplacians, which are the columns of matrices $V'_i$ in problem (20) as follows:

$$d^2(x_a, x_b) = \sum_{l=1}^{t} K(\Lambda_{i,l}) (v_{i,l}(a) - v_{i,l}(b))^2 \\ = \| \Psi(x_a) - \Psi(x_b) \|_2^2 \quad (28)$$

where $\lambda_{i,l}$ is the l-th eigenvalue and $v_{i,l}(a)$ is the a-th entry in l-th approximate eigenvector of Laplacian in modality i, $K(\lambda)$ is the diffusion kernel which is selected as $e^{-5\lambda}$, and $\Psi(x)$ is the embedding of data point x obtained by diffusion map ($t=100$ is selected in equation (28) for our experiments).

After calculating the diffusion distances, we apply a k NN classifier based on the diffusion distance through a 10-fold cross-validation process. The standard criterion used to measure the classification performances is the averaged accuracy of 10 runs, for each dataset.

In this experiment, the effect of expanding the limited information about matching samples is reported in Table VIII.

As can be seen, with the small number of labeled samples, the proposed LSEJD approaches significantly improve classification performances compared with each modality on its own (unimodal 1 and unimodal 2) and perform better than other methods

that use only limited information about matching samples [24], and the method that uses joint eigenspace (JADE) [28].

### V. CONCLUSION AND DISCUSSION

Finding a common eigenbasis of multiple Laplacians of acquired data or sensed signals in multiple modalities can be a suitable method for multimodal manifold learning problems. However, in most of the real-world applications, due to intermodel differences, the Laplacians of all modalities are not jointly diagonalizable. In this paper, we proposed an innovative approach for multimodal manifold learning, which tries to find approximate common eigenbasis of Laplacian matrices of all modalities using local tangent spaces of the intrinsic manifold.

We performed three experiments to evaluate our approach with synthetic and real-world problems. Comparison of the obtained results with several state-of-the-art multimodal and multi-view learning approaches demonstrated the advantages and effectiveness of our approach in unsupervised, and semi-supervised scenarios. These results prove the advantage of using intrinsic local tangent space of underlying signal or data manifolds on matching samples to discover the local geometrical structures in each modality.

In this paper, the intrinsic dimension of data in each modality is supposed to be known in advance. Although so far several methods are designed to automatically estimate the intrinsic dimension of data [47], designing the multimodal manifold learning methods that simultaneously estimate the intrinsic dimension of data in each modality needs further consideration, which can be considered as a future work.

## APPENDIX

According to Cauchy-Schwarz inequality, for any two vectors $\mathbf{u} \in \mathbb{R}^\eta$ and $\mathbf{v} \in \mathbb{R}^\eta$, we have $\langle \mathbf{u}, \mathbf{v} \rangle \le \| \mathbf{u} \| \| \mathbf{v} \|$, and thus:

$$(u_1 v_1 + \cdots + u_\eta v_\eta)^2 \le (u_1^2 + \cdots + u_\eta^2) (v_1^2 + \cdots + v_\eta^2). \quad (29)$$

where $u_i$ indicates the i-th element of vector $\mathbf{u}$.

If $\mathbf{u} = \mathbf{o}_1$, where $\mathbf{o}_1 \in \mathbb{R}^\eta$ is a vector of ones with dimension $\eta$, relation (29) can be written as follows:

$$(v_1 + \cdots + v_\eta)^2 \le \eta (v_1^2 + \cdots + v_\eta^2). \quad (30)$$

For $\eta$ vectors $\{\mathbf{w}_i\}_{i=1}^\eta$, where $\mathbf{w}_i \in \mathbb{R}^\rho$ and $\| \cdot \|^2$ indicates $L_2$ norm:

$$\left\| \sum_{i=1}^{\eta} \mathbf{w}_i \right\|^2 = (w_{1,1} + \cdots + w_{\eta,1})^2 + \cdots + (w_{1,\rho} + \cdots + w_{\eta,\rho})^2. \quad (31)$$

where $w_{i,j}$ indicates the j-th element of vector $\mathbf{w}_i$.

According to (30), equation (31) entails the following inequality:

$$\left\| \sum_{i=1}^{\eta} \mathbf{w}_i \right\|^2 \le \eta (w_{1,1}^2 + \cdots + w_{\eta,1}^2) + \cdots + \eta (w_{1,\rho}^2 + \cdots + w_{\eta,\rho}^2). \quad (32)$$
By changing the order of the right components in (32):

$$ \left\| \sum_{i=1}^{\eta} \mathbf{w}_i \right\|^2 \le \eta (w_{1,1}^2 + \cdots + w_{1,\rho}^2) + \cdots + \eta (w_{\eta,1}^2 + \cdots + w_{\eta,\rho}^2), \quad (33) $$

which is equivalent to the following inequality:

$$ \left\| \sum_{i=1}^{\eta} \mathbf{w}_i \right\|^2 \le \eta \sum_{i=1}^{\eta} \| \mathbf{w}_i \|^2 . \quad (34) $$

ACKNOWLEDGMENT

Authors would like to thank Prof. Eynard and his colleagues for sharing the codes and datasets of their work presented in [24] with us during this research.

REFERENCES

[1] D. Lahat, T. Adali, and C. Jutten, "Multimodal data fusion: An overview of methods, challenges and prospects," in Proc. IEEE, vol. 103, no. 9, pp. 1449–1477, Sep. 2015.

[2] T. Baltrušaitis, C. Ahuja, and L. Morency, "Multimodal machine learning: A survey of taxonomy," IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 2, pp. 423–443, Feb. 2019.

[3] T. Wimalajeewa and P. K. Varshney, "Compressive sensing-based detection with multimodal dependent data," IEEE Trans. Signal Process., vol. 66, no. 3, pp. 627–640, Feb. 2018.

[4] F. Sedighin, M. Babaie-Zadeh, B. Rivet, and C. Jutten, "Multimodal soft nonnegative matrix co-factorization for convolutive source separation," IEEE Trans. Signal Process., vol. 65, no. 12, pp. 3179–3190, Jun. 2017.

[5] S. Hou, L. Chen, D. Tao, S. Zhou, W. Liu, and Y. Zheng, "Multi-layer multi-view topic model for classifying advertising video," Pattern Recognit., vol. 68, pp. 66–81, 2017.

[6] Z. Yuan, T. Lu, and C. L. Tan, "Learning discriminated and correlated patches for multi-view object detection using sparse coding," Pattern Recognit., vol. 69, pp. 26–38, 2017.

[7] W. H. Ming Yin and J. Gao, "Shared generative latent representation learning for multi-view clustering," in Proc. 34th AAAI Conf. Artif. Intell., New York, NY, USA, Feb. 2020, doi: 10.1609/aaai.v34i04.6146.

[8] J. Huang and B. Kingsbury, "Audio-visual deep learning for noise robust speech recognition," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 7596–7599.

[9] O. Yair, M. Ben-Chen, and R. Talmon, "Parallel transport on the cone manifold of spd matrices for domain adaptation," IEEE Trans. Signal Process., vol. 67, no. 7, pp. 1797–1811, Apr. 2019.

[10] M. Pilanci and E. Vural, "Domain adaptation via transferring spectral properties of label functions on graphs," in Proc. IEEE 12th Image, Video, Multidimensional Signal Process. Workshop, 2016, pp. 1–5

[11] T. Cao, C. Zach, S. Modla, D. Powell, K. Czymmek, and M. Niethammer, "Multi-modal registration for correlative microscopy using image analogies," Med. Image Anal., vol. 18, no. 6, pp. 914–926, 2014.

[12] C. Wachinger and N. Navab, "Manifold learning for multi-modal image registration," in Proc 11th Brit. Mach. Vis. Conf., 2010, pp. 1–12.

[13] D. Hong, N. Yokoya, J. Chanussot, and X. Zhu, "Cospace: Common subspace learning from hyperspectral-multispectral correspondences," IEEE Trans. Geosci. Remote Sens., vol. 57, no. 7, pp. 4349–4359, Jul. 2019.

[14] J. Hu, D. Hong, and X. X. Zhu, "Mima: Mapper-induced manifold alignment for semi-supervised fusion of optical image and polarimetric sar data," IEEE Trans. Geosci. Remote Sens., vol. 57, no. 11, pp. 9025–9040, Nov. 2019.

[15] Z. Wu, W. Zhu, J. Chanussot, Y. Xu, and S. Osher, "Hyperspectral anomaly detection via global and local joint modeling of background," IEEE Trans. Signal Process., vol. 67, no. 14, pp. 3858–3869, Jul. 2019.

[16] H. Zhang, T. Wang, and G. Dai, "Semi-supervised cross-modal common representation learning with vector-valued manifold regularization," Pattern Recognit. Lett., vol. 130, pp. 335–344, 2020.

[17] Y. Zhao et al., "Multi-view manifold learning with locality alignment," Pattern Recognit., vol. 78, pp. 154–166, 2018.

[18] D. Hong, N. Yokoya, N. Ge, J. Chanussot, and X. X. Zhu, "Learnable manifold alignment (lema): A semi-supervised cross-modality learning framework for land cover and land use classification," ISPRS J. Photogrammetry Remote Sens., vol. 147, pp. 193–205, 2019.

[19] D. Tuia and G. Camps-Valls, "Kernel manifold alignment for domain adaptation," PLOS One, vol. 11, no. 2, pp. 1–25, Feb. 2016.

[20] B. Li, J. Li, and X.-P. Zhang, "Nonparametric discriminant multi-manifold learning for dimensionality reduction," Neurocomputing, vol. 152, pp. 121–126, 2015.

[21] H. Q. Minh, L. Bazzani, and V. Murino, "A unifying framework in vector-valued reproducing kernel hilbert spaces for manifold regularization and co-regularized multi-view learning," J. Mach. Learn. Res., vol. 17, no. 1, pp. 769–840, Jan. 2016.

[22] A. Nazarpour and P. Adibi, "Two-stage multiple kernel learning for supervised dimensionality reduction," Pattern Recognit., vol. 48, no. 5, pp. 1854–1862, 2015.

[23] D. Eynard, K. Glashoff, M. M. Bronstein, and A. M. Bronstein, "Multimodal diffusion geometry by joint diagonalization of laplacians," 2012, arXiv:abs/1209.2295.

[24] D. Eynard, A. Kovnatsky, M. M. Bronstein, K. Glashoff, and A. M. Bronstein, "Multimodal manifold analysis by simultaneous diagonalization of laplacians," IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 12, pp. 2505–2517, Dec. 2015.

[25] A. Pournemat, P. Adibi, and J. Chanussot, "Semisupervised charting for spectral multimodal manifold learning and alignment," Pattern Recognit., vol. 111, Art. no. 107645, 2021.

[26] M. Belkin and P. Niyogi, "Laplacian eigenmaps for dimensionality reduction and data representation," Neural Comput., vol. 15, no. 6, p. 1373–1396, Jun. 2003.

[27] J. Jacob, "Über ein leichtes verfahren die in der theorie der säkularstörungen vorkommenden gleichungen numerisch aufzulösen," J. Für Reine Und Angew. Math., vol. 30, pp. 51–95, 1846.

[28] J.-F. Cardoso and A. Souloumiac, "Jacobi angles for simultaneous diagonalization," SIAM J. Matrix Anal. Appl., vol. 17, pp. 161–164, 1996.

[29] A. Kovnatsky, M. M. Bronstein, A. M. Bronstein, K. Glashoff, and R. Kimmel, "Coupled quasi-harmonic bases," Comput. Graph. Forum, vol. 32, pp. 439–448, 2013.

[30] X. Xing, K. Wang, and W. Feng, A Fusion Scheme of Local Manifold Learning Methods, P. Bracken, Ed. Rijeka: IntechOpen, 2017.

[31] H. Tyagi, E. Vural, and P. Frossard, "Tangent space estimation for smooth embeddings of riemannian manifolds," Inf. Inference: J. IMA, vol. 2, no. 1, pp. 69–114, Jun. 2013.

[32] Y. Zhou and S. Sun, "Local tangent space discriminant analysis," Neural Process. Lett., vol. 43, no. 3, p. 727–744, Jun. 2016.

[33] P. Zhang, H. Qiao, and B. Zhang, "An improved local tangent space alignment method for manifold learning," Pattern Recognit. Lett., vol. 32, no. 2, pp. 181–189, 2011.

[34] A. Yeredor, "Non-orthogonal joint diagonalization in the least-squares sense with application in blind source separation," IEEE Trans. Signal Process., vol. 50, no. 7, pp. 1545–1553, Jul. 2002.

[35] T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng, "Nus-wide: A real-world web image database from national university of Singapore," in Proc. ACM Int. Conf. Image Video Retrieval, Ser., New York, NY, USA: Association for Computing Machinery, 2009, pp. 1–9.

[36] Fei-Fei R. Fergus, and P. Perona, "One-shot learning of object categories," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 4, pp. 594–611, Apr. 2006.

[37] Y. Li, F. Nie, H. Huang, and J. Huang, "Large-scale multi-view spectral clustering via bipartite graph," in Proc. 29th AAAI Conf. Artif. Intell., Ser., 2015, pp. 2750–2756.

[38] X. Cai, F. Nie, H. Huang, and F. Kamangar, "Heterogeneous image feature integration via multi-modal spectral clustering," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 6, 2011, pp. 1977–1984.

[39] L. Zelnik-Manor and P. Perona, "Self-tuning spectral clustering," in Proc. 17th Int. Conf. Neural Inf. Process. Syst., Ser., Cambridge, MA, USA: MIT Press, 2004, pp. 1601–1608.

[40] X. Dong, P. Frossard, P. Vandergheynst, and N. Nefedov, "Clustering on multi-layer graphs via subspace analysis on grassmann manifolds," IEEE Trans. Signal Process., vol. 62, no. 4, pp. 905–918, Feb. 2013.

[41] F. Nie, G. Cai, J. Li, and X. Li, "Auto-weighted multi-view learning for image clustering and semi-supervised classification," IEEE Trans. Image Process., vol. 27, no. 3, pp. 1501–1511, Mar. 2018.

[42] F. Nie, J. Li, and X. Li, "Self-weighted multiview clustering with multiple graphs," in Proc. 26th Int. Joint Conf. Artif. Intell., Ser., 2017, pp. 2564–2570.
[43] H. Gao, F. Nie, X. Li, and H. Huang, "Multi-view subspace clustering," in *Proc. IEEE Int. Conf. Comput. Vis.*, Dec. 2015, pp. 4238-4246.

[44] S. Luo, C. Zhang, W. Zhang, and X. Cao, "Consistent and specific multi-view subspace clustering," in *Proc. 30th AAAI Conf. Artif. Intell.*, 2018, pp. 3730-3737.

[45] R. Wang, F. Nie, Z. Wang, H. Hu, and X. Li, "Parameter-free weighted multi-view projected clustering with structured graph learning," *IEEE Trans. Knowl. Data Eng.*, vol. 32, no. 10, pp. 2014-2025, Oct. 2020.

[46] S. X. Ming Yin, J. Gao, and Y. Guo, "Multiview subspace clustering via tensorial t-product representation," *IEEE Trans. Neural Netw. Learn. Syst.*, vol. 30, pp. 851-864, Mar. 2019.

[47] F. Cammastra, "Data dimensionality estimation methods: A survey," *Pattern Recognit.*, vol. 36, no. 12, pp. 2945-2954, 2003.

<figure><img src="image_7.png" /></figure>

**Maysam Behmaenesh** (Graduate Student Member, IEEE) received the M.Sc. degree in artificial intelligence from the Engineering Department, Shahid Bahonar University of Kerman, Kerman, Iran, in 2014. He is currently working toward the Ph.D. degree with the Department of Artificial Intelligence, Faculty of computer engineering, University of Isfahan, Isfahan, Iran. From 2019 to 2020, he has been a Visiting Scholar with GIPSA-lab, Grenoble Institute of Technology, Grenoble, France. His research interests mainly include multimodal processing, machine

learning, pattern recognition, and deep neural networks.

<figure><img src="image_8.png" /></figure>

**Peyman Adibi** received the Ph.D. degree from Faculty of Computer Engineering, Amirkabir University of Technology, Tehran, Iran in 2009. He is currently an Assistant Professor with the Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran, where he is the Head of Artificial Intelligence Department. His current research interests include machine learning and pattern recognition, computer vision and image processing, computational intelligence and soft computing, statistical signal processing, and their applications.

<figure><img src="image_9.png" /></figure>

**Jocelyn Chanussot** (Fellow, IEEE) received the M.Sc. degree in electrical engineering from the Grenoble Institute of Technology, Grenoble, France, in 1995, and the Ph.D. degree from the Université de Savoie, Annecy, France, in 1998. In 1999, he was with Geography Imagery Perception Laboratory for the Delegation Generale de l'Armement, DGA-French National Defense Department, Arcueil, France. From 2015 to 2017, he was a Visiting Professor with the University of California, Los Angeles, Los Angeles, CA, USA. Since 1999, he has been with the Grenoble

Institute of Technology, where he is currently a Professor of signal and image processing. He also conducted his research with the Grenoble Images Speech Signals and Automatics Laboratory. He was a Visiting Scholar with Stanford University, Stanford, CA, USA, KTH Royal Institute of Technology, Stockholm, Sweden, the National University of Singapore, Singapore. Since 2013, he has been an Adjunct Professor with the University of Iceland, Reykjavik, Iceland, and with the Chinese Academy of Sciences, Beijing, China, since 2019. From 2006 to 2008, he was a Member of the Machine Learning for Signal Processing Technical Committee of the IEEE Signal Processing Society and the Institut Universitaire de France, Paris, France, from 2012 to 2017. He was the General Chair of the first IEEE GRSS Workshop on Hyperspectral Image and Signal Processing, Evolution in Remote sensing (WHISPERS). He was the Program Chair of the IEEE International Workshop on Machine Learning for Signal Processing, (2009). He was an Associate Editor for the IEEE GEOSCIENCE AND REMOTE SENSING LETTERS from 2005 to 2007 and *Pattern Recognition* from 2006 to 2008. Since 2007, he has been an Associate Editor for the IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, for the IEEE TRANSACTIONS ON IMAGE PROCESSING, since 2017 and for the Proceedings of the IEEE, since 2020. He was the Guest Editor for the IEEE Signal Processing Magazine and for the IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING.

<figure><img src="image_10.png" /></figure>

**Christian Jutten** (Fellow, IEEE) received the Ph.D. and Doctor es Sciences degrees in signal processing from the Grenoble Institute of Technology (GIT), Grenoble, France, in 1981 and 1987, respectively. From 1982, he was an Associate Professor with GIT, before being a Full Professor with University Grenoble Alpes, Grenoble, France, in 1989. He has coauthored more than 105 papers in international journals, four books, 27 keynote plenary talks and more than 230 communications in international conferences. Since 80's, his research interests have in-

cluded machine learning and source separation, including theory separability, nonlinear mixtures, sparsity, multimodality and applications brain and hyperspectral imaging, chemical sensor array, and speech. He was a Visiting Professor with École polytechnique fédérale de Lausanne, Lausanne, Switzerland, 1989, Riken labs, Japan, 1996, and Campinas University, Brazil, 2010. He was the Director or the Deputy Director of his lab from 1993 to 2010, especially the Head of the Signal Processing Department consisting of 120 people and the Deputy Director of GIPSA-lab consisting of 300 people from 2007 to 2010. He was a Scientific Advisor for signal and images processing at the French Ministry of Research (1996-1998) and CNRS (2003-2006 and since 2012). He was an Organizer or the Program Chair of many international conferences, especially of the 1st International Conference on Blind Signal Separation and Independent Component Analysis in 1999 (ICA'99). He was a Member of a few of the IEEE Technical Committees. He was the recipient of many awards, including Best Paper Awards of EURASIP (1992), the IEEE GRSS (2012), and the Medal Blondel (1997) from the French Electrical Engineering Society, and one Grand Prix of the French Académie des Sciences (2016). He was elected the IEEE Fellow (2008), the EURASIP Fellow (2013) and a Senior Member of Institut Universitaire de France, Paris, France, since 2008. He was the recipient of the 2012 ERC Advanced Grant for the project Challenges in Extraction and Separation of Sources (CHESS).

<figure><img src="image_11.png" /></figure>

**Sayyed Mohammad Saeed Ehsani** received the B.Sc. and M.Sc. degrees in communication engineering from the School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran, the Ph.D. degree in computer science from the School of Engineering, Kennedy Western University, Cheyenne, WY, USA, and the Hon. D.L. degree from Cambridge University, Cambridge, U.K. He is currently a Member of the Artificial Intelligence Department, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran. His

research interests include pattern recognition, neuro-fuzzy, rule-base control, next generation networks, big data and cybernetics. He is a Fellow Member of ABI or IBC and a Member of the IEEE's Technical Committees.