<figure><img src="image_1.png" /></figure>

HAL
open science

# Efficient embedding initialization via dominant eigenvector projections

Quentin R Petit, Chong Li, Nahid Emad, Jack Dongarra

► To cite this version:

Quentin R Petit, Chong Li, Nahid Emad, Jack Dongarra. Efficient embedding initialization via dominant eigenvector projections. SC Workshops '25 - Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, Nov 2025, Saint-Louis, United States. pp.1790-1799, 10.1145/3731599.3767541. hal-05369672

HAL Id: hal-05369672

https://hal.science/hal-05369672v1

Submitted on 17 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Efficient Embedding Initialization via Dominant Eigenvector Projections

Quentin R. Petit

Centre de Recherche en Informatique
 Mines Paris - PSL University France
 Fontainebleau, France
 quentin.petit@minesparis.psl.eu

Nahid Emad

Maison de la Simulation & LI-PaRAD
 Université Paris-Saclay
 Saclay, France
 nahid.emad@uvsq.fr

Chong Li

Distributed and Parallel Technologies Lab
 Huawei Paris Research Center
 Boulogne-Billancourt, France
 ch.l@huawei.com

Jack Dongarra

University of Tennessee
 Knoxville, USA
 dongarra@icl.utk.edu

## Abstract

The embedding layer is essential in deep learning, transforming high-dimensional data into compact representations. However, growing datasets and model sizes pose challenges in training time, memory, and generalization. We propose a scalable method for embedding initialization via spectral dimensionality reduction using dominant eigenvector projections.

The proposed approach leverages on MIRAMns, multiple implicitly restarted Arnoldi method with nested subspaces, to extract most informative directions from large and potentially sparse data representations. Unlike traditional embeddings or autoencoders, this proposed approach requires few tunable parameters and is inherently parallel. We apply MIRAMns to matrix representations such as covariance and co-occurrence matrices to compute low-dimensional embeddings that preserve data structure and variance. Experiments across diverse datasets show that the proposed method achieves comparable or better accuracy with significantly reduced dimensionality, enabling smaller, faster deep networks. Additionally, our parallel implementation scales efficiently on HPC platforms, making it well-suited for large-scale scientific and AI workloads.

## CCS Concepts

* **Computing methodologies → Spectral methods; Parallel algorithms; Machine learning.**

## Keywords

Spectral dimensionality reduction, Unsupervised embedding initialization, Nested Krylov subspace, Dominant eigenvectors, MIRAMns, Arnoldi method, Scalable Eigensolver, High-performance computing

## ACM Reference Format:

Quentin R. Petit, Chong Li, Nahid Emad, and Jack Dongarra. 2025. Efficient Embedding Initialization via Dominant Eigenvector Projections. In *Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC Workshops '25), November 16–21, 2025, St Louis, MO, USA*. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3731599.3767541

# 1 Introduction

As datasets grow in scale and complexity, modern deep learning systems face significant challenges in efficiency, scalability, and memory usage. A key bottleneck lies in representing high-dimensional, sparse, or categorical input data in a form that neural networks can effectively process. Embedding layers, while powerful, can result in models with large parameter counts, especially when dealing with high-cardinality inputs. This increases memory consumption, training time and the risk of overfitting.

To address these challenges, embedding layers are commonly employed to transform complex input features into dense, low-dimensional representations. These layers act as a bridge between raw input data and the rest of the neural network, enabling it to process structured, categorical, numerical, or sequence-based inputs effectively. By mapping inputs into a continuous vector space, embedding layers help to expose relationships and patterns that are more amenable to learning by subsequent layers. They are thus critical components in a wide range of applications, including natural language processing, recommender systems, and classification tasks in vision or signal processing domains. Traditional approaches to embedding initialization, with random weights or task-specific pretrained embeddings, often require substantial training or fail to exploit the underlying structure of the data in an unsupervised manner.

While embedding layers have proven effective, their typical initialization strategies do not explicitly exploit the statistical structure of the input data. As a result, these methods often produce unnecessarily large embedding spaces, increasing model size and training cost. Dimensionality reduction techniques are interesting to address this limitation and to ensure that only the most informative directions are retained. However, applying the dimensionality reduction

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

SC Workshops '25, St Louis, MO, USA

© 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.

ACM ISBN 979-8-4007-1871-7/2025/11

https://doi.org/10.1145/3731599.3767541
techniques directly to modern large-scale or sparse datasets is challenging. Classical full decomposition methods are only applicable to matrices of moderate size due to their high computational cost. To process larger matrices or datasets, restarting projection methods must be used. These techniques (Figure 1), as their name suggests, project the original large-size problem into a lower-dimensional subspace where classical methods can be applied. They provide an approximation of the dominant eigenvalues and their corresponding eigenvectors. However, the stability and convergence of such methods are highly dependent on the spectral properties of the matrix and the conditions under which the projection is performed.

<figure><img src="image_3.png" /><figcaption>Figure 1: Overview of restarted projection methods</figcaption></figure>

The Explicitly Restarted Arnoldi Method (ERAM) [36] and the Implicitly Restarted Arnoldi Method (IRAM) [39] are considered as reference techniques for large-size eigenvalue problems. ERAM, although conceptually simple, suffers from significant numerical instability, and its high memory and computational time requirements limit its applicability to these systems. IRAM, on the other hand, achieves superior numerical robustness by combining Arnoldi iterations with implicit QR shifts, which selectively filter out unwanted spectral components [38]. This implicit restart makes IRAM more efficient in terms of memory and computation time. However, a well-known limitation of IRAM is its difficulty in converging when eigenvalues are closely clustered. The Multiple Implicitly Restarted Arnoldi Method with nested subspaces (MIRAMns) [37] effectively overcomes this issue through the use of nested subspace hierarchies and adaptive restart strategies. As a variant of IRAM, MIRAMns is designed to robustly extract dominant eigenpairs even in challenging spectral conditions. Because of its power and scalability, we adopt this method for constructing compact embeddings in large-scale deep learning applications.

In this work, we propose a scalable, robust, and unsupervised approach for embedding initialization based on the dominant eigenvectors. This approach is based on MIRAMns, which enables efficient and reliable computation of dominant eigenvectors even in challenging spectral conditions. These eigenvectors form the basis of compact embedding that preserves the key structure of the input data while significantly reducing dimensionality. The experimental results show that MIRAMns-based embeddings not only reduce model complexity and training time but also retain or improve classification accuracy across a variety of datasets. Furthermore, the method exhibits strong scalability, positioning it as a valuable

tool for embedding generation in large-scale high performance environment.

## 2 Background and Related Work

Pretrained embedding methods, such as Word2Vec [30], GloVe [33] and FastText [5] are widely used to initialize embedding layers by leveraging co-occurrence statistics or subword structures from large unlabeled corpora. Graph-based variants like DeepWalk [34] and Node2Vec [14] apply similar principles in network contexts. They exploit co-occurrence statistics or topological structures to produce meaningful vector representations in an unsupervised manner. These approaches, however, exhibit several well-known limitations. First, they rely on external data and large corpora, which may not be available or representative for scientific or domain-specific tasks. Second, they are fragile to preprocessing choices such as tokenization or subword segmentation, which strongly affect the quality of the learned embeddings. Finally, pretrained embeddings are usually fixed in size and content, offering limited control over dimensionality and making adaptation to new tasks or resource-constrained models expensive and sometimes ineffective [6, 9]. Other embedding compression as Neural Architecture Search (NAS)-based approaches-based approaches [26, 28, 46] require a considerable additional costs that have a significant impact on the training time.

Moreover, a common practice in deep learning is to bypass pretraining and randomly initialize embedding matrices [21], typically from uniform or normal distributions and optimize them during supervised training [4]. While this approach is simple and flexible, it introduces two major limitations: (1) it relies entirely on supervised gradients to shape the embedding, which can lead to slow convergence and overfitting when data is insufficient and (2) it often results in unnecessarily large embedding matrices, increasing memory footprint and computation cost [25].

Various post-hoc compression techniques have been explored to mitigate these issues after training, including quantization [15, 44], pruning [8, 27, 35], decomposition [11, 45] and knowledge distillation [43]. While often effective, these approaches are applied after embeddings have already been trained, meaning they do not reduce the computational cost of the initial training phase. In addition, they may require additional fine-tuning and sometimes lead to suboptimal trade-offs between compression and accuracy.

A common way to control embedding size from the start is to exploit dimensionality reduction techniques. Methods such as Principal Component Analysis (PCA) [13, 19] and truncated Singular Value Decomposition (SVD) [40] provide compact representations by projecting data into subspaces that capture the most significant variance or structure. Autoencoders [18] have also been used for unsupervised feature learning. These approaches can be viewed as forms of low-rank approximation, where the goal is to represent the original high-dimensional data with a lower-rank matrix that preserves as much of the important information as possible. This perspective highlights their ability to control embedding size while leveraging the statistical structure of the data, offering more interpretable initializations than random methods. However, their practical use in deep learning remains limited for several reasons: (1) the convergence and approximation of these methods can be
unstable when eigenvalues are closely clustered [41] and (2) autoencoders introduce additional training complexity and hyperparameter tuning. For these reasons, dimensionality reduction is rarely applied directly to initialize embedding layers.

Complementing these techniques, some previous work explores spectral approaches as a direct means of initializing the integration layer. Although spectral embeddings such as Laplacian eigemaps [3] or spectral clustering [32] are common in other contexts, they are typically designed for specific tasks such as manifold learning or clustering. They are not intended for general-purpose embedding initialization that is compatible with large-scale deep learning. The appeal of spectral methods lies in their unsupervised nature and strong theoretical foundation in linear algebra [2]. Spectral methods also face practical challenges in this setting, including the high computational cost for large matrices and the sensitivity to numerical issues when eigenvalues are clustered [41].

# 3 Embedding Initialization via MIRAMns

Existing methods for embedding initialization leave an underexplored space between pretrained, random, and classical spectral approaches. Pretrained embeddings provide rich semantic structure but offer little control over dimensionality and depend on large, domain-specific corpora. Random initialization, while simple, ignores statistical information in the input data, leading to large and redundant embeddings.

To address the limitations mentioned above, we propose a scalable and unsupervised embedding initialization method based on dominant eigenvector projections computed with the Multiple Implicitly Restarted Arnoldi Method with nested subspaces (MIRAMns). By extending Arnoldi-based eigensolvers with nested subspace exploration and adaptive restart strategies, MIRAMns efficiently extracts the most informative spectral components, even when eigenvalues are closely clustered. This enables the construction of compact embeddings that preserve essential data structure while being suitable for high-performance computing environments.

## 3.1 Motivation & Overview

The objective of the proposed approach is to initialize embedding by explicitly exploiting the statistical structure of the input data. Reducing the dimensionality of the input while retaining its most informative directions can significantly decrease the size of the embedding matrix, resulting in lower memory usage, reduced training cost, and potentially improved generalization. Unlike random initialization, which ignores available correlations, and pretrained embeddings, which depend on external corpora and lack dimensionality control, a spectral approach provides a principled way to capture global data structure before training.

An initialization of embedding that already reflects the statistical structure of the input data provides several advantages. First, by projecting inputs into a subspace aligned with the dominant variance or structural relationships, the embedding layer can start from a representation that is closer to the optimal solution, potentially accelerating convergence during training. Second, compact embeddings reduce the number of trainable parameters, making models

lighter and more suitable for resource-constrained or real-time applications. Third, embeddings derived from a well-defined spectral basis are more interpretable and reproducible than those learned entirely in a supervised manner, which is particularly relevant in scientific and regulated domains.

<figure><img src="image_4.png" /><figcaption>Figure 2: Overview of our method to initialize embedding from eigenvectors associated to dominant eigenvalues.</figcaption></figure>

To achieve this, the method extracts dominant eigenvectors from matrix representations of the data, such as covariance or co-occurrence matrices. These eigenvectors form a compact basis that captures intrinsic patterns in the data, making them well suited for initializing embedding layers.

Moreover, the use of dominant eigenvectors ensures that the resulting embedding space retains the global structure of the data. Extraction should be possible on matrices where the non-clustering of eigenvalues is not guaranteed. As the datasets for possible applications are vast, we need to ensure that the proposed approach is robust enough to deal with matrices whose eigenvalues are potentially close.

These considerations motivate the use of a spectral approach for embedding initialization. To be applicable to a wide variety of datasets, we propose a compact embedding technique based on the MIRAMns eigensolver. MIRAMns addresses this generalization challenge by enabling the efficient computation of dominant eigenvectors under conditions where traditional eigensolvers struggle.

The proposed method follows three key steps as seen in Figure 2: (1) We construct a square matrix representation that encodes statistical relationships within the data (subsection 3.2). (2) We apply MIRAMns to compute a high-quality approximation of the dominant eigenvectors of the matrix (subsection 3.3). (3) These eigenvectors are used to initialize the embedding matrix, which is then used to project the input data into a reduced-dimensional space (Section 3.4).

## 3.2 Matrix Representation Construction

To apply spectral methods for embedding, we first transform the input dataset into a square matrix $A \in \mathbb{R}^{n \times n}$ that captures relationships or statistical structure among features or elements. This matrix can take various forms, depending on the nature of the data.

For numerical datasets, a covariance matrix is used:

$$A_{ij} = \frac{1}{m-1} \sum_{k=1}^{m} (x_{k,i} - \bar{x}_i)(x_{k,j} - \bar{x}_j) \quad (1)$$
**Algorithm 1** Approximation of dominant eigenvectors based on MIRAMns

**Require:** A the initial matrix, $V_{m_i}$ the matrix of orthogonal basis issued from the Arnoldi factorization, $H_{m_i}$ the upper Hessenberg matrix projection of A onto the Krylov subspace, $f_{m_i}$ the residual vector with $AV_{m_i} = V_{m_i}H_{m_i} + f_{m_i}e_{m_i}^T$ the $m_i$-steps Arnoldi factorization with $i = 1, 2, ..., l$

**Ensure:** The approximate k dominant eigen-elements of A

1. **for** i until convergence **do**

2. **for** i ← 1, 2, ..., l **do**

3. Compute $\sigma(H_{m_i})$ and their associated eigenvectors

4. Compute k desired eigenpairs and their corresponding residual norms.

5. **if** convergence of Ritz residual **then** stop all processes.

6. **end for**

7. Select the best results in these subspaces and the associated best subspace size $m_{best}$. Set $m = m_{best}$, $H_m = H_{m_{best}}$, $V_m = V_{m_{best}}$ and $f_m = f_{m_{best}}$.

8. Select set of $p = m - k$ shifts ($\mu_1^{(m)}, ..., \mu_p^{(m)}$), based upon $\sigma(H_m)$ or perhaps other information and set $q^T \leftarrow e_m^T$.

9. **for** j ← 1, 2, ..., p **do**

10. Factor $[Q_j, R_j] = qr(H_m - \mu_j^{(m)}I)$

11. $H_m \leftarrow Q_j^H H_m Q_j$

12. $V_m \leftarrow V_m Q_j$

13. $q \leftarrow q^H Q_j$

14. **end for**

15. Set $f_k \leftarrow v_{k+1} \hat{\beta}_k + f_m \sigma_k$, $V_k \leftarrow V_m(1: k, 1: k)$, $H_k \leftarrow H_m(1: k, 1: k)$

16. Beginning with the k-step Arnoldi factorization $AV_k = V_k H_k + f_k e_k^T$. Apply $p_i = m_i - k$ additional steps of the Arnoldi process to obtain l new $m_i$-step Arnoldi factorization $AV_{m_i} = V_{m_i} H_{m_i} + f_{m_i} e_{m_i}^T$ (for $i = 1, ..., l$)

17. **end for**

where m is the number of instances, $x_{k,i}$ the value of the feature i for sample k and $\tilde{x}_i$ its empirical mean. This representation emphasizes linear correlations and is suitable when variance preservation is the primary objective.

For structured or categorical data, a co-occurrence matrix [29] is often more appropriate. It measures how frequently pairs of elements appear together:

$$A_{i,j} = \text{freq}(e_i, e_j) \qquad (2)$$

where freq(eᵢ, eⱼ) denotes the co-occurrence count of elements eᵢ and eⱼ within the same context (for example in the same sequence, graph or categorical group).

Alternatively, a similarity matrix may be constructed to quantify the degree of association between feature pairs using metrics such as cosine similarity or Pearson correlation [12]:

$$A_{i,j} = \frac{x_i \cdot x_j}{\|x_i\| \|x_j\|} \qquad (3)$$

Each of these matrices serves to represent the structure or statistical properties of the data in a form that is suitable for spectral analysis. The resulting matrix, which is square and typically symmetric, becomes the input to the MIRAMns algorithm.

## 3.3 Dominant Eigenvectors Extraction

The core of this method lies in computing the k dominant eigenvectors of the matrix A that capture the directions of maximum variance or strongest structural relationship in the data. Formally, we aim to find k dominant eigenpairs ($\lambda_i, u_i$) that satisfy

$$Au_i = \lambda_i u_i \qquad (4)$$

with $|\lambda_1| \ge |\lambda_2| \ge \dots \ge |\lambda_k| \ge \dots \ge |\lambda_n|$.

The eigenvectors {$u_1, u_2, ..., u_k$} define the subspace used to initialize the embedding layer. Selecting dominant eigenvectors ensures that the resulting embedding preserves the global structure of the data while discarding noise and redundant dimensions.

We propose using MIRAMns to calculate the approximation of the dominant eigenvectors. MIRAMns extends IRAM by constructing a sequence of nested Krylov subspaces $K_{m_1} \subset K_{m_2} \subset \dots \subset K_{m_l}$ and adaptively selecting at each restart the subspace $K_{best}$ that minimizes the residual norm. By adaptively selecting the optimal subspace at each iteration, MIRAMns improves convergence even in the presence of noise or near-degenerate spectra. MIRAMns uses shifted QR iterations to concentrate desired spectral information into a small leading subspace, improving stability and avoiding numerical degradation. Algorithm 1 outlines the main steps of the proposed approach to extract dominant eigenvectors with MIRAMns. The convergence criteria are the maximum number of iterations or the Ritz residual. In experiments, we use respectively the 500 and $10^{-8}$ default values.

## 3.4 Embedding Initialization

Once the k dominant eigenvectors {$u_1, u_2, ..., u_k$} of the matrix A are computed, they are assembled into an embedding matrix

$$E = [u_1 u_2 \dots u_k] \in \mathbb{R}^{n \times k} \qquad (5)$$

This matrix defines a linear transformation from the original input space $\mathbb{R}^n$ to a reduced k-dimensional space:

$$\tilde{x} = E^T x, \text{ with } x \in \mathbb{R}^n \text{ and } \tilde{x} \in \mathbb{R}^k \qquad (6)$$

where $\tilde{x}$ serves as the compact representation of the input x (vector of features). Because the eigenvectors are orthogonal by construction, this transformation preserves the dominant variance or structural information captured by the selected eigenpairs. It is linear, fast and retains the global structure of the data captured by the dominant eigenvectors. The resulting embedded vectors are then used as inputs to the rest of the model. The choice of subspace sizes was discussed by Shahzadeh et al. in [37].

## 3.5 Parallelization Strategy

The method focuses on the dominant eigenpairs, significantly reducing both memory footprint and computational overhead while still capturing the essential structure of the data. During each cycle of the algorithm, we compute several Krylov subspaces of increasing size and evaluate their approximation quality in parallel. Although this nested subspace strategy improves convergence robustness, it introduces a degree of sequential overhead, as the subspaces must be constructed and analyzed one after another. The computational cost associated with this sequential exploration will be discussed in section 4. However, MIRAMns remains very well suited to a parallel
environment since it has the same parallelism as IRAM, making it well suited for large-scale HPC environments. By focusing only on a small number of dominant eigenpairs rather than performing full matrix factorizations or complete spectral decompositions, the algorithm significantly reduces both memory requirements and communication overhead. Its dominant operations are inherently parallelizable and efficiently implemented in block-distributed settings. These operations are those involving matrices of which at least one of the dimensions is the large size of the problem. Limited reliance on global synchronization further improves scalability, enabling strong performance across hundreds or even thousands of cores. Concentrating computations in small projected subspaces also ensures good scalability for very large matrices.

# 4 Experiments and Results

We evaluate the effectiveness of the proposed embedding initialization strategy, built upon the MIRAMns algorithm, across various datasets, learning tasks, and computational configurations. Rather than focusing on improving task-specific accuracy metrics, our primary goal is to validate the effectiveness and practicality of this embedding approach. Specifically, we aim to demonstrate its capacity to enable compact and efficient low-dimensional representations for downstream learning. The evaluation focuses on two key aspects: (1) the scalability of the parallel implementation of MIRAMns on a high-performance computing (HPC) cluster, and (2) the training efficiency gained by using lower-dimensional representations derived from the proposed embedding approach.

We conducted experiments on heterogeneous datasets covering different input dimensions and structural characteristics. This includes low-dimensional datasets with dense features, as well as high-dimensional structured inputs such as images. This diversity enables us to evaluate the generality and scalability of the proposed embedding approach across a broad range of real-world scenarios. A summary of the datasets is provided in Table 1.

<table><caption>Table 1: Datasets Overview</caption><thead><tr><td>Name</td><td>Features</td><td>Instances</td><td>Classes</td></tr></thead><tbody><tr><td>Fetal Health [24]</td><td>21</td><td>2126</td><td>3</td></tr><tr><td>Date fruit [22]</td><td>34</td><td>898</td><td>7</td></tr><tr><td>Radar [20]</td><td>175</td><td>325834</td><td>7</td></tr><tr><td>MNIST [7]</td><td>784</td><td>70000</td><td>10</td></tr><tr><td>CIFAR-10 [23]</td><td>3072</td><td>60000</td><td>10</td></tr></tbody></table>

The experiments were conducted in two distinct hardware platforms tailored to the different stages of our pipeline:

* Embedding construction via eigenvectors extraction was performed on the *Ruche* CPU-based HPC cluster [31]. The cluster comprises 25 compute nodes, each equipped with two CPUs (20 cores per CPU, 2.1 GHz, Cascade Lake), yielding up to 1,000 parallel processing cores in total. Each node is provisioned with 192 GB of RAM and interconnected via a 100 Gbit/s Omni-Path Architecture network.

* Neural network training and evaluation were carried out on a Huawei Atlas cluster equipped with 8x Ascend NPUs and a GPU workstation, respectively with an implementation in MindSpore [1] and in TensorFlow.

All matrix computations for the eigenvectors extraction are implemented using the Eigen C++ open-source library [16], selected for its low-level optimized linear algebra routines. To enable distributed-memory parallelism, we employed the MPI standard [42], which facilitates inter-node communication with minimal synchronization costs.

The complete source code and results used in the experiments is publicly available at: https://github.com/petitquentin/E2IDEP.

## 4.1 Scalability and Performance

To evaluate the scalability of the MIRAMns-based dominant eigenvector extraction method, experiments were conducted in a distributed parallel computing environment. Scalability tests focused on datasets with more than 100 input features, as these generated larger covariance matrices and more clearly reveal the algorithm's computational behavior. For each dataset, the number of processors was varied, and the average execution time required to reach convergence in the dominant eigenvectors was recorded. Figure 3 presents the observed execution times along with the corresponding speedup values.

<figure><img src="image_5.png" /><figcaption>Figure 3: Execution time (left axis) and speedup (right axis) of the MIRAMns-based dominant eigenvectors extraction method as a function of the number of processors.</figcaption></figure>

Across all tested datasets, we observe an initial increase in speedup as more processors are added. This behavior is consistent with the expected benefits of parallelism, where distributing the computational workload across multiple nodes reduces overall execution time. However, beyond a certain threshold which is specific to each dataset, the speedup begins to decline. This is due to the fact that the datasets are not large enough to maintain high computational load per processor as the parallelism increases. As the number of processors grows, the cost of inter-node communication and synchronization begins to outweigh the benefits of parallel execution. Although the proposed approach is based on MIRAMns and therefore appears capable of effectively exploiting parallel resources, optimal performance is achieved only when the computational workload per node remains sufficiently high to offset communication overhead. It is worth noting that we opted for a MIRAMns implementation for general matrices. The execution times reported in this paper are overestimated due to the symmetry of the input matrices used with MIRAMns.

These results confirm that the proposed approach is scalable up to a certain level of parallelism, after which communication costs dominate.

## 4.2 Embedding Impact on Learning

The influence of the proposed eigenvector-based embedding initialization is assessed in terms of both predictive performance and
<figure><img src="image_6.png" /><figcaption>Figure 4: Comparison of accuracy with different model sizes as a function of embedding size (dashed lines: without embedding)</figcaption></figure>

computational efficiency of downstream learning models. To provide a comprehensive analysis, the discussion first focuses on model accuracy, reflecting the quality of predictions using the embedded features. The impact on training time and model size is then examined.

<table><caption>Table 2: Hidden sizes and estimation of number of parameters used for MLP models across datasets with varying input characteristics.</caption><thead><tr><th>Dataset class</th><th>Model</th><th>Hidden sizes</th><th>Est. params #</th></tr></thead><tbody><tr><td rowspan="4">Compact</td><td>★ Large</td><td>512, 512, 512, 512, 256</td><td>≈ 1 × 10<sup>6</sup></td></tr><tr><td>★ Medium</td><td>256, 256, 256, 256, 128</td><td>≈ 2.5 × 10<sup>5</sup></td></tr><tr><td>★ Low</td><td>64, 64, 64, 64, 32</td><td>≈ 2 × 10<sup>4</sup></td></tr><tr><td>★ Very low</td><td>32, 32, 32, 32, 16</td><td>≈ 7 × 10<sup>3</sup></td></tr><tr><td rowspan="4">High-dim</td><td>★ Large</td><td>512, 512</td><td>≈ 1 × 10<sup>6</sup></td></tr><tr><td>★ Medium</td><td>256, 256</td><td>≈ 5 × 10<sup>5</sup></td></tr><tr><td>★ Low</td><td>64, 64</td><td>≈ 1 × 10<sup>5</sup></td></tr><tr><td>★ Very low</td><td>32, 32</td><td>≈ 5 × 10<sup>4</sup></td></tr></tbody></table>

### 4.2.1 Accuracy analysis.
To explore the interaction between model complexity and embedding size, we built several multi-layer perceptron (MLP) models [17] variants by varying the number of perceptrons in each hidden layer. This allowed us to control the total number of trainable parameters. The layer configurations used in the experiments are detailed in Table 2. To account for differences in input dimensionality and structural complexity across datasets, we designed MLP architectures with varying depth and width. Datasets with compact or tabular feature sets benefit from deeper MLPs to progressively extract and combine features. In contrast, for high-dimensional datasets, shallower models are sufficient due to the richness and expressiveness of the input representations, which require fewer hierarchical transformations. The number of layers is therefore not tied to the data modality itself, but rather to the representational needs of the dataset. All hyperparameters were

held constant across experiments. No additional tuning was performed for the models using reduced-dimensional inputs or smaller architectures.

For each dataset, we compared the classification accuracy of models trained on the original input space to those trained on the dominant eigenvectors-based embedded representations. Figure 4 reports test accuracies as a function of embedding size and model configuration. For reference, we also include the baseline performance of models trained without any dimensionality reduction.

Across all datasets, we observe a consistent improvement in classification accuracy as the first few embedding dimensions are introduced. This trend reflects the fact that extremely low-dimensional embeddings fail to capture sufficient information from the original feature space, thereby limiting the model's ability to learn effective decision boundaries, regardless of the model size.

The rate of accuracy improvement gradually diminishes as more embedding dimensions are added, eventually reaching a threshold beyond which additional dimensions yield minimal gain. The number of dimensions required to reach this saturation point varies by dataset. For example, accuracy tends to stabilize around 20 dimensions for the Radar and Date Fruit datasets (Figures 4c and 4d), while approximately 30 dimensions are required for CIFAR-10 (Figure 4e).

In all cases, models trained on embedding from eigenvectors achieve accuracy levels that approach, match, or even surpass those trained on the original input space. This shows that the proposed embedding approach effectively preserves the essential structure of the data. Notably, for certain datasets the proposed approach leads to substantial improvements in accuracy for smaller models. In these cases, small-capacity models trained on embedded inputs achieve accuracy comparable to that of larger models trained on the full input space. For example, with the Radar and CIFAR-10 datasets (Figures 4c and 4e), very low model with embedding has equivalent accuracy to the original medium-sized model. This suggests that the proposed embedding not only reduces input dimensionality
<figure><img src="image_7.png" /><figcaption>Figure 5: Comparison of training time with different model sizes as a function of embedding size (same legends as Fig. 4)</figcaption></figure>

but also enhances the expressive efficiency of the model, enabling
competitive performance with reduced architectural complexity.

These results show that MIRAMns-based integration not only
maintains high classification accuracy, but also produces more com-
pact models that perform as well as larger models. They also suggest
(Figure 4) that our approach minimizes the risk of overfitting, lead-
ing to more robust and generalizable models. Indeed, we observe
that reduced models can achieve higher accuracy than full models,
indicating that the latter have captured noise or random variations.

4.2.2 *Impact on training time.* Beyond improvements in classification accuracy, the proposed embedding approach significantly reduces training time and enhances convergence behavior. As shown in Figure 5, training times decrease consistently across all datasets. This reduction is directly linked to the lower input dimensionality, which reduces the number of trainable parameters and the overall computational load during backpropagation.

For datasets with relatively low input dimensionality, such as
Date Fruits (Figure 5b) and Fetal Health (Figure 5a), the reduction in
training time is modest, around 10%. However, the benefits become
more pronounced for high-dimensional data. In the case of CIFAR-
10 (Figure 5e), training time is reduced by nearly 40% when using
the embedded input. The greater the reduction in size, the greater
the impact embedding has on training time.

The use of the embedding also enables significantly smaller
models to achieve similar predictive performance. For example, on
MNIST (Figure 5d), a model trained with a 30-dimensional embed-
ding contains 278k parameters, which is approximately 2.4× fewer
than the 670k parameters in the baseline model without embedding.
Despite this reduction, both models reach comparable accuracy.
The effect is even more pronounced on CIFAR-10. There, a very
compact model with a 30-dimensional embedded input contains
only 2.3k parameters, compared to 855k in the baseline medium-
sized model. Both achieve similar classification performance. This
corresponds to a model size reduction of more than 300 times.

These results show that the MIRAMns-based embedding not
only reduces training time but also enables the use of significantly
smaller models without compromising classification accuracy. This
dual benefit is particularly valuable in scenarios where computa-
tional resources are limited or fast training is critical, such as in
real-time systems or edge computing environments. By effectively

compressing the input space while preserving task-relevant infor-
mation, the proposed approach supports more efficient and scalable
learning across a range of data types and model configurations.

5 Conclusion and Future Work

In this paper, we proposed a novel method for embedding layer
initialization in deep learning models using spectral dimensionality
reduction via dominant eigenvector projections. The proposed ap-
proach is based on an eigensolver using nested Krylov subspaces,
which offers improved robustness and convergence relatively to the
comparable spectral techniques [37]. The method is unsupervised
and model-agnostic, making it broadly applicable across diverse
data types and tasks.

Through experiments on different type of datasets, we show
that embeddings generated by the proposed approach can achieve
significant dimensionality reduction while preserving the mean-
ingful information and variance of the original data, and avoiding
overfitting. This enables the training of smaller and faster neural
networks without compromising accuracy. The results show that
proposed embedding technique is especially beneficial for smaller
models, where the reduction in input dimensionality helps to mit-
gigate the impact of limited model capacity. The method remains
applicable and effective for larger architectures as well. We provide
empirical evidence of its scalability through experiments conducted
in a distributed high-performance computing environment. These
experiments highlight the method suitability for large-scale dataset
computations.

The experiments presented are based on a general implemen-
tation that is designed to validate the fundamental principles of
the proposed approach. However, the current performance of our
implementation can be improved, particularly by taking into ac-
count matrix specifications such as symmetry. The use of MLPs
has highlighted the potential of the proposed approach in a con-
trolled setting. However, this configuration may not be applicable
for generalization of certain complex models.

Several directions for future work emerge. First, we plan to ex-
tend the current implementation to fully leverage heterogeneous
architectures, particularly AI-specific accelerators, to further re-
duce computation time and improve scalability. The fundamental
numerical operations (primarily matrix-vector and matrix-matrix
multiplications) of this method are naturally compatible with GPU acceleration. Preliminary results on GPU platforms have already shown promising acceleration [10]. Second, we aim to develop solution to update in a dynamic way embedding during training. Currently, MIRAMns generates embedding as a static pretraining step. A promising direction is to explore how MIRAMns can support dynamic updates of embedding as the model learns, especially in streaming or continually learning systems. Finally, we intend to explore integration of MIRAMns-based initialization into other neural architectures, such as convolutional neural networks or LLMs, and evaluate its impact on training dynamics and generalization in more complex tasks.

## References

[1] 2025. MindSpore. https://gitee.com/mindspore/mindspore.

[2] Yossi Azar, Amos Fiat, Anna Karlin, Frank McSherry, and Jared Saia. 2001. Spectral analysis of data. In *Proceedings of the thirty-third annual ACM symposium on Theory of computing*. 619–626.

[3] Mikhail Belkin and Partha Niyogi. 2001. Laplacian eigenmaps and spectral techniques for embedding and clustering. *Advances in neural information processing systems* 14 (2001).

[4] Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. *IEEE transactions on pattern analysis and machine intelligence* 35, 8 (2013), 1798–1828.

[5] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. *Transactions of the association for computational linguistics* 5 (2017), 135–146.

[6] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. *arXiv preprint arXiv:2108.07258* (2021).

[7] Li Deng. 2012. The mnist database of handwritten digit images for machine learning research. *IEEE signal processing magazine* 29, 6 (2012), 141–142.

[8] Wei Deng, Junwei Pan, Tian Zhou, Deguang Kong, Aaron Flores, and Guang Lin. 2021. Deeplight: Deep lightweight feature interactions for accelerating ctr predictions in ad serving. In *Proceedings of the 14th ACM international conference on Web search and data mining*. 922–930.

[9] Bhuwan Dhingra, Hanxiao Liu, Ruslan Salakhutdinov, and William W Cohen. 2017. A comparative study of word embeddings for reading comprehension. *arXiv preprint arXiv:1703.00993* (2017).

[10] Alexandre Fender, Nahid Emad, Serge Petiton, and Joe Eaton. 2016. Leveraging accelerators in the multiple implicitly restarted Arnoldi method with nested subspaces. In *2016 IEEE International Conference on Emerging Technologies and Innovative Business Practices for the Transformation of Societies*. IEEE, 389–394.

[11] Benjamin Ghaemmaghami, Zihao Deng, Benjamin Cho, Leo Orshansky, Ashish Kumar Singh, Mattan Erez, and Michael Orshansky. 2020. Training with multi-layer embeddings for model reduction. *arXiv preprint arXiv:2006.05623* (2020).

[12] John C Gower. 1971. A general coefficient of similarity and some of its properties. *Biometrics* (1971), 857–871.

[13] Michael Greenacre, Patrick JF Groenen, Trevor Hastie, Alfonso Iodice d'Enza, Angelos Markos, and Elena Tuzhilina. 2022. Principal component analysis. *Nature Reviews Methods Primers* 2, 1 (2022), 100.

[14] Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In *Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining*. 855–864.

[15] Hui Guan, Andrey Malevich, Jiyan Yang, Jongsoo Park, and Hector Yuen. 2019. Post-Training 4-bit Quantization on Embedding Tables. *CoRR* (2019).

[16] Gaël Guennebaud, Benoit Jacob, et al. 2010. Eigen. http://eigen.tuxfamily.org 3, 1 (2010), 8.

[17] Trevor Hastie. 2009. The elements of statistical learning: data mining, inference, and prediction.

[18] Geoffrey E Hinton and Ruslan R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. *science* 313, 5786 (2006), 504–507.

[19] Ian Jolliffe. 2005. Principal component analysis. *Encyclopedia of statistics in behavioral science* (2005).

[20] Iman Khosravi. 2020. Crop mapping using fused optical-radar data set. UCI Machine Learning Repository.

[21] Tom Kocmi and Ondrej Bojar. 2017. An Exploration of Word Embedding Initialization in Deep-Learning Tasks. In *14th International Conference on Natural Language Processing*. 56.

[22] Murat Koklu, Ilkay Cinar, and Yavuz Selim Taspinar. 2021. Classification of rice varieties with deep learning methods. *Computers and electronics in agriculture* 187 (2021), 106285.

[23] Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. *Master's thesis, University of Tront* (2009).

[24] Adem Kuzu and Yunus Santur. 2023. Early Diagnosis and Classification of Fetal Health Status from a Fetal Cardiotocography Dataset Using Ensemble Learning. *Diagnostics* 13, 15 (2023), 2471.

[25] Jeongmin Lim, Young Geun Kim, Sung Woo Chung, Farinaz Koushanfar, and Joonho Kong. 2023. Near-Memory Computing With Compressed Embedding Table for Personalized Recommendation. *IEEE Transactions on Emerging Topics in Computing* 12, 3 (2023), 938–951.

[26] Haochen Liu, Xiangyu Zhao, Chong Wang, Xiaobing Liu, and Jiliang Tang. 2020. Automated embedding size search in deep recommender systems. In *Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval*. 2307–2316.

[27] Siyi Liu, Chen Gao, Yihong Chen, Depeng Jin, and Yong Li. 2021. Learnable Embedding Sizes for Recommender Systems. In *International Conference on Learning Representations*.

[28] Fuyuan Lyu, Xing Tang, Hong Zhu, Huifeng Guo, Yingxue Zhang, Ruiming Tang, and Xue Liu. 2022. Optembed: Learning optimal embedding table for click-through rate prediction. In *Proceedings of the 31st ACM International Conference on Information & Knowledge Management*. 1399–1409.

[29] Christopher D Manning. 2009. *An introduction to information retrieval*. Cambridge university press.

[30] Tomas Mikolov. 2013. Efficient estimation of word representations in vector space. *arXiv preprint arXiv:1301.3781* 3781 (2013).

[31] Mésocentre. Accessed: 2025-08-05. Computational resources from Mésocentre of Université Paris-Saclay, CentraleSupélec and ENS Paris-Saclay supported by CNRS and Région Ile-de-France. https://mesocentre.universite-paris-saclay.fr/.

[32] Andrew Ng, Michael Jordan, and Yair Weiss. 2001. On spectral clustering: Analysis and an algorithm. *Advances in neural information processing systems* 14 (2001).

[33] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In *Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)*. 1532–1543.

[34] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In *Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining*. 701–710.

[35] Liang Qu, Yonghong Ye, Ningzhi Tang, Lixin Zhang, Yuhui Shi, and Hongzhi Yin. 2022. Single-shot embedding dimension search in recommender system. In *Proceedings of the 45th International ACM SIGIR conference on research and development in Information Retrieval*. 513–522.

[36] Yousef Saad. 2011. *Numerical methods for large eigenvalue problems: revised edition*. SIAM.

[37] SA Shahzadeh Fazeli, Nahid Emad, and Zifan Liu. 2015. A key to choose subspace size in implicitly restarted Arnoldi method. *Numerical Algorithms* 70 (2015), 407–426.

[38] Danny C Sorensen. 1992. Implicit application of polynomial filters in a k-step Arnoldi method. *Siam journal on matrix analysis and applications* 13, 1 (1992), 357–385.

[39] Danny C Sorensen. 1997. Implicitly restarted Arnoldi/Lanczos methods for large scale eigenvalue calculations. In *Parallel Numerical Algorithms*. Springer, 119–165.

[40] Gilbert W Stewart. 1998. *Matrix algorithms: volume 1: basic decompositions*. SIAM.

[41] Lloyd N Trefethen and Manfred R Trummer. 1987. An instability phenomenon in spectral methods. *SIAM J. Numer. Anal.* 24, 5 (1987), 1008–1023.

[42] David W Walker and Jack J Dongarra. 1996. MPI: a standard message passing interface. *Supercomputer* 12 (1996), 56–68.

[43] Xin Xia, Hongzhi Yin, Junliang Yu, Qinyong Wang, Guandong Xu, and Quoc Viet Hung Nguyen. 2022. On-device next-item recommendation with self-supervised knowledge distillation. In *Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval*. 546–555.

[44] Zhiqiang Xu, Dong Li, Weijie Zhao, Xing Shen, Tianbo Huang, Xiaoyun Li, and Ping Li. 2021. Agile and accurate CTR prediction model training for massive-scale online advertising systems. In *Proceedings of the 2021 international conference on management of data*. 2404–2409.

[45] Chunxing Yin, Bilge Acun, Xing Liu, and Carole-Jean Wu Tt-rec. 2021. Tensor train compression for deep learning recommendation models. In *Conference on Machine Learning and Systems (MLSys)*.

[46] Xiangyu Zhaok, Haochen Liu, Wenqi Fan, et al. 2021. Autoemb: Automated embedding dimensionality search in streaming recommendations. In *2021 IEEE International Conference on Data Mining (ICDM)*. IEEE, 896–905.
# Artifact Description (AD)

## Relation To Contributions

This artifact contains the complete source code, datasets, scripts, and trained models used in the paper "Efficient Embedding Initialization via Dominant Eigenvector Projections". The artifact supports reproducibility of the experimental results presented in the paper, including eigenvector extraction, embedding initialization, and performance benchmarking of deep learning models.

The artifact is publicly available at: https://github.com/petitquentin/E2IDEP. The DOI of this artifact is 10.5281/zenodo.16781370.

## Expected Results

The expected results are:

* building the square representation of datasets

* successful extraction of dominant eigenvectors

* initialization of embeddings

* reproduction of the performance figures reported in the paper

## Expected Reproduction Time

The entire pipeline (configuration + experiments) takes several hours. The speed will largely depend on the number of processors used in parallel and the performance of the accelerators for training Deep Learning models.

## Artifact Setup (incl. Inputs)

**Dependencies.** CMake, Eigen, MPI, Protobuf, MindSpore, TensorFlow.

Hardware tested: Ruche SLURM cluster, Huawei Atlas 900, Nvidia RTX 2060 workstation.

**Data.** Small datasets are included. Large datasets (Radar, MNIST, CIFAR-10) are downloadable via scripts or `keras datasets` python library.

**Contents.** `code/` folder contains all implementation code to prepare data before model training. In particular, the `build_matrix/` folder contains python scripts to prepare each dataset (only take training data set to extract information for embedding). The folder `lib_MIRAMns/` contains the C++ implementation of the dominant eigenvectors extraction method based on MIRAMns.

`data/` folder includes small example dataset and the README of the folder gives instruction to download the large dataset Radar. For MNIST and CIFAR-10 datasets, `keras datasets` is used to load the data directly within the code.

`scripts/` folder contains pipeline scripts to automate data preprocessing, compilation and eigenvector extraction. It also includes scripts to test scalability of MIRAMns. Please note that all scripts are made to run with `bash` from the project's root directory, where the main README is.

`models/` folder includes training code to build models using TensorFlow and MindSpore. It also contains figures generated for the paper and the post-processing scripts used to improve clarity and readability of figures. In order to facilitate reading and to contain the logs of our executions, the models were created in Jupyter Notebook. (`.ipynb`).

<table><caption>Table 3: Artifact check-list (Meta-information)</caption><thead><tr><td>Item</td><td>Description</td></tr></thead><tbody><tr><td>Algorithm</td><td>Covariance square matrix construction, dominant eigenvectors computation, model training</td></tr><tr><td>Programming languages</td><td>C++, Python</td></tr><tr><td>Dependencies</td><td>CPMake, Eigen, MPI, Protobuf, MindSpore, TensorFlow</td></tr><tr><td>Hardware</td><td>C++ scripts: tested on multi-core CPU cluster with MPI (e.g., Ruche SLURM cluster)<br/>- Mindspore scripts: tested on Huawei Atlas 900 Cluster (192 CPUs, 8x Ascend 910 NPUs, 1.5To RAM)<br/>- TensorFlow scripts: tested on GPU workstation (6-core Intel i5-9600K CPU, Nvidia RTX 2060 GPU, 64Go RAM)</td></tr><tr><td>Execution</td><td>Scripts provided. Alternative scripts given for non-SLURM systems</td></tr><tr><td>Data</td><td>Includes small datasets. Large datasets downloadable information given in README</td></tr><tr><td>Output</td><td>Embeddings, trained models, reproducible figures</td></tr><tr><td>License</td><td>MIT</td></tr><tr><td>Used in paper figures</td><td>Yes, pgf figures into the folder models/tensorflow/figures/ Post-processed figures for easier understanding with scripts in models/tensorflow/postprocessfigure</td></tr><tr><td>Reproducibility Level</td><td>Functional: The artifact reproduces all key results with provided scripts<br/>Determinism enabled in TensorFlow to ensure reproducibility of model accuracy and figures</td></tr></tbody></table>

Precomputed results are included in the folder `data/output/` to allow skipping the extraction of eigenvectors with the C++ library and to enable the training of deep learning models.

**MIRAMns library.** We present here the detail about the structure and operation of our C++ library located in `code/lib_MIRAMns/`. This library contains all the components and functions for calculating eigenvectors and eigenvalues of large matrices for embedding based on these vectors. The library also contains functions for constructing representations of datasets in the form of covariance matrices.

Key folders of the library are:

* `/algorithms/` folder contains the core implementation of the algorithms such as Arnoldi, IRAM and MIRAMns.

* `/data/` folder contains sample test matrices.

* `/include/` folder contains header files for the project's components (algorithms, I/O, tools, and utilities).

* `/iofiles/` folder contains functions for reading and writing matrix files, including support for `.mtx` files and `protobuf`.

* `/proto/` folder contains protocol buffer files used for data serialization and communication.
* `/test/` folder contains tests and example for the different modules and algorithms within the project.

* `/tools/` folder contains additional utility functions and Python scripts, such as those for calculating averages of matrix values.

* `/utils/` folder contains general-purpose utility functions for matrix transformations and other operations.

This library depends on several prerequisites: C++ compiler, CMake, MPI, Eigen, and Protocol Buffers. Instructions for installing these dependencies are available in the README file at the root of the project. A `make_MIRAMns_lib` script in the `scripts/` folder is available to install the library in a `build/` folder so that it can be used during experiments. Like all scripts in this folder, it must be run from the main project folder.

## Artifact Execution

All software requirements and installation instructions are listed in the repository README. The full pipeline is modular and divided into four main stages: data preparation, MIRAMns library compilation, eigenvector extraction, and model training. Reproducibility is supported both with and without access to SLURM.

*Installation and reproduction of experiments results.* Complete installation instructions are available in the project README. Here is a list of the main steps for installing and running the algorithms:

1. Install all dependencies (C++ and python environment)

2. Download and preprocess datasets to extract only training subsets

3. Deploy the CMake library MIRAM used to extract dominant eigenvectors for embedding

4. Build square matrix representation and for each dataset extract dominant eigenvectors

5. Training Deep Learning models thanks to extracted information

*Usage of the C++ lib_MIRAMns library.* The programs used to interact with the library in scripts are located in the folder `test/` of the library.

The `buildCovariance` program computes the covariance matrix from an input dense matrix in parallel with MPI. It is intended for data preprocessing. This tool reads a dense input matrix (e.g., features × samples), computes the covariance matrix in a distributed way, and saves the result to disk in a Matrix Market format (`.mtx`) file.

The `eigenvectors_save` executable computes the dominant eigenvalues and eigenvectors of a matrix in parallel using MPI, and saves the resulting eigenvectors in a defined Matrix Market format (`.mtx`) file.

The `scalability_eigenvectors/scalability_speigenvectors` programs compute the dominant eigenvalues and eigenvectors of a matrix in parallel using MPI. This is part of the scalability benchmarking of the library. The only outputs (if -debug not activated) of these programs are execution time. The difference between the two files is whether the matrix will be considered sparse or dense during calculations.

*Command-line parameters.* A set of parameters can be used to interact with the executables mentioned in the previous subsection. For

all these programs, it is possible to obtain the list of command-line parameters with the parameter -help or -h. Table 4 lists the main common parameters available for interacting with these programs.

<table><caption>**Table 4: Common command-line parameters used for executables in the C++ library.**</caption><tr><td>-i or -input</td><td>Path to the input file.</td></tr><tr><td>-o or -output</td><td>Path to the output file.</td></tr><tr><td>-k or -kdesired</td><td>Number of dominant eigenvalues and eigenvectors to compute.</td></tr><tr><td>-m or -max</td><td>Maximum number of iterations.</td></tr><tr><td>-l or -lmax</td><td>Maximum size of the subspace interval for the eigensolver. Must be at least equal to -k.</td></tr><tr><td>-t or -tolerance</td><td>Convergence tolerance (acceptable error).</td></tr><tr><td>-d or -debug</td><td>Enable debug mode for detailed output.</td></tr><tr><td>-h or -help</td><td>Show help message and exit.</td></tr></table>

## Artifact Analysis (incl. Outputs)

Outputs include:

* performance logs for scalability benchmarks

* approximations of eigenvectors (in `data/output/`)

* reproducible figures (in `models/tensorflow/figures`)

* embeddings and trained models

*Relation to Paper.* The artifact supports all the experiments presented in the paper. The figure generation uses `.pgf` files in the folder `models/tensorflow/figures`. The execution time required to extract the dominant eigenvectors was calculated from the averages observed over a minimum set of ten executions. These performance tests were performed on the Ruche cluster. Performance results are obtained by training standard deep models initialized using the extracted eigenvectors.

*Limitations.* The results of approximations of the dominant eigenvectors are available in the `data/output/` folder. These are the results obtained with the Ruche Mésocentre CPU cluster, with the SLURM scripts included in `scripts/extract_eigen/` folder. The approximations of the dominant eigenvectors may vary slightly depending on the approximations used in the calculations. The number of computing nodes and the way collective communications are managed can also have an impact on the approximations of these eigenvectors.

While all core components are included, performance may vary across hardware setups. SLURM scripts are provided but not mandatory, alternative local execution scripts using `mpirun` are included. Some datasets must be downloaded due to size constraints, it required an internet connection. Information to download the datasets is available in the README in the `data/` folder.

In order to ensure good reproducibility of the results obtained with the Deep Learning models, we used TensorFlow's experimental functionality to make the calculations deterministic. However, this results in a loss of performance compared to training without determinism.