<figure><img src="image_1.png" /></figure>

HAL
open science

# From Plain to Sparse Correspondence Analysis: a Generalized SVD Approach

Hervé Abdi, Vincent Guillemot, Ruiping Liu, Ndèye Niang, Gilbert Saporta, Ju-chi Yu

► To cite this version:

Hervé Abdi, Vincent Guillemot, Ruiping Liu, Ndèye Niang, Gilbert Saporta, et al. From Plain to Sparse Correspondence Analysis: a Generalized SVD Approach. Statistica Applicata - Italian Journal of Applied Statistics, 2024, 35 (3), pp.301-338. 10.26398/IJAS.0035-014. hal-04467589

HAL Id: hal-04467589

https://cnam.hal.science/hal-04467589v1

Submitted on 20 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
FROM PLAIN TO SPARSE CORRESPONDENCE ANALYSIS:
A GENERALIZED SVD APPROACH†

Hervé Abdi

ORCID: 0000-0002-9522-1978

School of Behavioral and Brain Sciences, The University of Texas at Dallas,
Richardson, TX, USA

Vincent Guillemot

ORCID: 0000-0002-7421-0655

Institut Pasteur, Université Paris Cité, Bioinformatics and Biostatistics Hub, Paris,
France,

Ruiping Liu

ORCID: 0000-0001-8591-7712

School of Applied Science, Beijing Information Science and Technology Univer-
sity, Beijing, China,

Ndèye Niang

ORCID: 0000-0002-6109-9935

Cedric Lab, Conservatoire national des arts et métiers, Paris, France,

Gilbert Saporta*

ORCID: 0000-0002-3406-5887

Cedric Lab, Conservatoire national des arts et métiers, Paris, France,

Ju-Chi Yu

ORCID: 0000-0002-6360-1861

Campbell Family Mental Health Research Institute, Centre for Addiction and
Mental Health, Toronto, Canada,

<footnote>*Corresponding author: Gilbert Saporta, gilbert.saporta@cnam.fr</footnote>

<footnote>† The order of the authors reflects only the alphabetical order. All authors contributed equally to this paper.</footnote>
Abstract

Correspondence Analysis (CA)—the method of choice to analyze contingency
tables—is widely applied in text analysis, psychometrics, chemometrics, etc. But
CA becomes difficult to interpret when items load on several dimensions, when
dimensions comprise items whose loadings are of intermediate values, or when
the number of rows or columns is large—a configuration routinely found in con-
temporary statistical practice. For principal component analysis (PCA), this inter-
pretation problem has been traditionally handled with rotation and more recently
with sparsification methods often inspired by the LASSO. Curiously, despite the
strong connections between CA and PCA, sparsifying correspondence analysis re-
mains essentially unexplored.

In this paper, we extend the Penalized Matrix Decomposition (a relatively recent
method based on the singular value decomposition) to sparsify CA. We present
some theoretical results and properties of the resulting sparse correspondence
analysis and illustrate this method with the analysis of a large textual data set.

**Keywords:** Sparsity, Correspondence Analysis, Generalized singular value decomposition, LASSO, Penalized matrix decomposition.
# From Plain to Sparse Correspondence Analysis:
A Generalized Singular Value Decomposition Approach

## Abstract

Correspondence Analysis (CA)—the method of choice to analyze contingency tables—is widely applied in text analysis, psychometrics, chemometrics, etc. But CA becomes difficult to interpret when items load on several dimensions, when dimensions comprise items whose loadings are of intermediate values, or when the number of rows or columns is large—a configuration routinely found in contemporary statistical practice. For principal component analysis (PCA), this interpretation problem has been traditionally handled with rotation and more recently with sparsification methods often inspired by the LASSO. Curiously, despite the strong connections between CA and PCA, sparsifying correspondence analysis remains essentially unexplored.

In this paper, we extend the Penalized Matrix Decomposition (a relatively recent method based on the singular value decomposition) to sparsify CA. We present some theoretical results and properties of the resulting sparse correspondence analysis and illustrate this method with the analysis of a large textual data set.

**Keywords:** Sparsity, Correspondence Analysis, Generalized singular value decomposition, LASSO, Penalized matrix decomposition.
# 1. Introduction

Correspondence analysis (CA)—the method of choice to analyze contingency tables—becomes difficult to interpret when 1) the data structure is complex as opposed to the *simple structure* (formalized by early psychometricians, such as, e.g., Thurstone, 1935, 1947) where each component is characterized by few items and each item contributes only to few—ideally one—components) or 2) when the number of rows or columns is large—a configuration routinely found in contemporary statistical practice. This interpretation problem, not specific to CA, also occurs in related multivariate methods such as principal component analysis (PCA) where it has been traditionally addressed with methods such as rotation and more recently with sparsification methods mostly derived from the LASSO (Hastie et al., 2001; Tibshirani, 1996). These sparsification methods are also commonly used in fields where the data comprise large numbers of variables (Jenatton et al., 2011) or observations that can include tens of thousands (e.g., in genomics, Chun and Keleş, 2010) to millions (as in neuroimaging, see, e.g., Le Floch et al., 2012; Silver et al., 2012).

But these recent sparsification methods have not yet been widely adapted for CA and its variants. In fact, so far, mostly multiple correspondence analysis (MCA)—which can be seen as an extension of PCA for qualitative variables, as well as an extension of CA to more than two qualitative variables—has benefited from such a (precious) few of these approaches (specifically, see Bernard et al., 2012; Guillemot et al., 2020; Mori et al., 2016).

It is only recently that sparsification for CA per se has been proposed (see, Liu et al., 2023). This approach uses the fact that CA can be interpreted 1) as a double weighted PCA of both rows and columns of the data matrix, or, equivalently, 2) as a generalized singular value decomposition (GSVD, see, e.g., Abdi, 2007; Greenacre, 1984) that incorporates metric constraints on the rows and columns of the data matrix. Within this framework, sparsification is implemented by adding additional constraints on the optimization problem solved by the singular value decomposition (SVD). This constrained SVD still decomposes the data matrix into (“pseudo”) singular vectors and (“pseudo”) singular values, but this decomposition seeks a compromise between concurrently maximizing explained variance and sparsity. Liu et al. (2023) distinguish two cases depending on whether sparsity is required for either rows or columns, or both.

This paper replicates and extends the approach of Liu et al. (2023) in particular by proposing in lieu of their sequential algorithm, a global algorithm for the simultaneous optimization of the dimensionality of the sparsified space and the
sparsification parameters of the rows and columns of the data.

Although the theory of sparsity-inducing constraints is well documented (especially for PCA), the extension to CA is not as straightforward given its special properties. In this paper, we introduce a general formulation of sparsification which can generalize PCA to other related multivariate methods and, specifically, to CA.

We begin with the definition and main properties of CA, followed by a short exposition of the relevant approaches to sparsify PCA. We then show how to extend the concepts from sparse PCA to obtain a sparse version of CA, and describe how sparsifying CA conflicts with some of its key properties that are therefore lost in the process. Finally, we illustrate sparse CA with an analysis of an example of textual analysis extracted from Project Gutenberg (Gerlach and Font-Clos, 2020).

## 2. Background

### 2.1. Notations

Matrices are denoted in upper case bold letters, vectors are denoted in lowercase bold letters, and their elements are denoted in lowercase italic letters (note that, by default, vectors are column vectors). Matrices, vectors and elements from the same matrix all use the same letter (e.g., **A**, **a**, **a**). The transpose operation is denoted by the superscript $^T$, the inverse operation is denoted by $^{-1}$. The identity matrix is denoted **I**, vectors or matrices of ones are denoted **1**, matrices or vectors of zeros are denoted **0** (by default, **I**, **0**), and **1** are conformable with the other terms in a formula). The standard product between two matrices is indicated by juxtaposition (i.e., **XY** means **X** times **Y**); the Hadamard product (i.e., element-wise) is denoted by $\odot$ (e.g., **X** $\odot$ **Y**), note that the Hadamard product is defined only between matrices with the same dimensions.

When provided with a square matrix, the diag operator gives a vector that contains the diagonal elements of this matrix. When provided with a vector, the diag operator gives a diagonal matrix with the elements of the vector as the diagonal elements of this matrix. A diagonal matrix is denoted **D**, when a subscript is attached, it denotes the vector that stores the diagonal elements of the matrix; for example, **D**<sub>a</sub> = diag(a). When provided with a square matrix, the trace operator gives the sum of the diagonal elements of this matrix. For an **I** by **J** matrix **X** and for **M** being a **J** by **J** symmetric positive definite matrix, the squared **M**-norm of **X** is denoted $||X||_M^2$ and is computed as:

$$ ||X||_M^2 = \text{trace}(XMX^T). \qquad (1) $$
When **M** is the identity matrix, the **M**-norm is equal to the square root of the sum of squares of the entries of the matrix and is called the Frobenius norm denoted $L_2 = \|\mathbf{X}\|_2^2$. Another useful norm is the sum of the absolute values of the matrix called the $L_1$ norm.

A probabilistic matrix (i.e., a matrix with non-negative elements whose sum is equal to 1) is denoted **Z**, its row (respectively column) sums are stored in vector **r** (respectively **c**): $\mathbf{r} = \mathbf{Z1}$ (respectively $\mathbf{c} = \mathbf{Z}^\top\mathbf{1}$). The matrix of row (respectively column) profiles is denoted $\mathbf{R} = \mathbf{D}_r^{-1}\mathbf{Z}$ (respectively $\mathbf{C} = \mathbf{ZD}_c^{-1}$).

When describing an optimization problem, the operator $\arg\min_{\mathbf{x}} f(\mathbf{x})$ [respectively $\arg\max_{\mathbf{x}} f(\mathbf{x})$] gives the value of $\mathbf{x}$ that minimizes (respectively maximizes) the function $f(\mathbf{x})$.

## 2.2. SVD and Generalized SVD

The singular value decomposition (SVD) and its extension the generalized singular value decomposition (GSVD, see, e.g., Abdi, 2007; Allen et al., 2014; Greenacre, 1984; Holmes, 2008; Takane, 2002) are the foundations of most contemporary multivariate statistical approaches.

The SVD of an $I \times J$ matrix $\mathbf{X}$ solves the following maximization problem (Eckart and Young, 1936): Find a matrix (denoted $\hat{\mathbf{X}}_L$) of rank $L$ [with $L \le \min(I,J)$], computed as

$$ \hat{\mathbf{X}}_L = \sum_{\ell=1}^{L} \delta_{\ell} \mathbf{u}_{\ell} \mathbf{v}_{\ell}^{\top} = \mathbf{U}_L \mathbf{\Delta}_L \mathbf{V}_L^{\top} \text{ with } \mathbf{U}_L^{\top} \mathbf{U}_L = \mathbf{V}_L^{\top} \mathbf{V}_L = \mathbf{I} \text{ and } \mathbf{\Delta}_L = \text{diag}(\delta_L) \quad (2) $$

such that $\hat{\mathbf{X}}_L$ is the matrix of $L$ rank closest to $\mathbf{X}$ (in the metric defined by the $L_2$ norm):

$$ \arg\min_{\mathbf{U}_L, \mathbf{\Delta}_L, \mathbf{V}_L} \|\mathbf{X} - \hat{\mathbf{X}}_L\|_2^2 \quad (3) $$

The SVD of a matrix can be computed by first computing its rank one approximation [i.e., the singular triplet $(\delta_1, \mathbf{u}_1, \mathbf{v}_1)$] and then subtracting this rank one approximation from $\mathbf{X}$—a procedure called deflation. The first singular triplet of the deflated matrix $\mathbf{X}$ is then the second singular triplet of $\mathbf{X}$, etc.

The generalized SVD (GSVD), differs from the plain SVD by incorporating different orthogonality constraints on the singular vectors. Specifically, with **M** being an $I \times I$ positive definite matrix (called the row metric matrix) and **W** a $J \times J$ positive definite matrix (called the column metric matrix), the GSVD of $\mathbf{X}$ solves
the following optimization problem (compare with Equation 3):

$$ \arg\min_{\mathbf{P}_L, \mathbf{\Delta}_L, \mathbf{Q}_L} \|\mathbf{X} - \hat{\mathbf{X}}_L\|_2^2 = \arg\min_{\mathbf{P}_L, \mathbf{\Delta}_L, \mathbf{Q}_L} \|\mathbf{X} - \mathbf{P}_L \mathbf{\Delta}_L \mathbf{Q}_L^T\|_2^2 \quad (4) $$

with

$$ \mathbf{P}_L^T \mathbf{M} \mathbf{P}_L = \mathbf{Q}_L^T \mathbf{W} \mathbf{Q}_L = \mathbf{I}, \quad \text{and} \quad \mathbf{\Delta}_L = \text{diag}(\delta_L), \quad (5) $$

where $\mathbf{P}_L$ is the $I \times L$ matrix containing the generalized left singular vectors and $\mathbf{Q}_L$ the $J \times L$ matrix containing the generalized right singular vectors. In practice, the GSVD of a matrix $\mathbf{X}$ can be obtained from the plain SVD of a matrix denoted $\tilde{\mathbf{X}}$ obtained by pre- and post-multiplying $\mathbf{X}$ by the square root of the row and column metric matrices:

$$ \tilde{\mathbf{X}} = \mathbf{M}^{\frac{1}{2}} \mathbf{X} \mathbf{W}^{\frac{1}{2}}. \quad (6) $$

More details are given in Appendix A.

## 2.3. Basics of Plain Correspondence Analysis

Correspondence analysis was originally developed to analyze the pattern of deviations from independence (as measured by a $\chi^2$ statistic) in a contingency table (see Abdi and Béra, 2018). CA provides, for both rows and columns, a set of factor scores whose total inertia is proportional to the independence $\chi^2$ computed on the original contingency table. The factor scores are obtained from the following generalized singular value decomposition (cf. Equation 4) where $\mathbf{D}_r^{-1}$ and $\mathbf{D}_c^{-1}$ are called $\chi^2$-metric matrices (Greenacre, 2010):

$$ \mathbf{Z} - \mathbf{r}\mathbf{c}^T = \mathbf{P}\mathbf{\Delta}\mathbf{Q}^T \quad \text{with} \quad \mathbf{P}^T\mathbf{D}_r^{-1}\mathbf{P} = \mathbf{Q}^T\mathbf{D}_c^{-1}\mathbf{Q} = \mathbf{I}. \quad (7) $$

Correspondence analysis can also be obtained from the plain SVD of:

$$ \tilde{\mathbf{Z}} = \mathbf{D}_r^{-\frac{1}{2}} (\mathbf{Z} - \mathbf{r}\mathbf{c}^T) \mathbf{D}_c^{-\frac{1}{2}} \quad (8) $$

(For further properties refer to Appendix C).

## 3. Sparse SVD with the Projected Penalized Matrix Decomposition

Because correspondence analysis is a particular PCA (and therefore a specific SVD, see Equations 8, above, as well as Equations 45 to 47 in Appendix B) a straightforward approach to the sparsification of CA is to adapt an already sparse version of PCA or SVD.
PCA being the oldest and most well-known multivariate method, it is no surprise that several sparse methods have been developed for PCA since the pioneering papers of Vines (2000) and Jolliffe et al. (2003). Case in point, in their— already old—review paper, Ning-min and Jing (2015) count about twenty algorithms for sparsifying PCA.

Recently, several authors have proposed sparse variants of the SVD (see, for reviews, e.g., Allen et al. 2014; Guillemot et al. 2019; Hastie et al. 2015; Jolliffe and Cadima 2016; Witten et al. 2009; Zou et al. 2006), or, specifically, of PCA (Benidis et al. 2016; Mattei et al. 2016). For most of these sparse variants, sparsification is obtained by adding sparsity constraints on both **P** and **Q**, or on **Q** alone. We decided to use the penalized matrix decomposition method (PMD) developed by Witten et al. (2009) because it is well-known and is implemented in R (with the PMA package).

### 3.1. Penalized Matrix Decomposition: Background

The penalized matrix decomposition (PMD) method (Witten et al., 2009) generalizes the plain SVD by adding sparsification constraints on the right and left singular vectors. Specifically, the PMD methods solves the following optimization problem:

$$ \arg\min_{\substack{\delta_{\ell, \mathbf{u}_{\ell}, \mathbf{v}_{\ell}} \\ \ell=1,...,L}} \left\| \mathbf{X} - \sum_{\ell=1}^{L} \delta_{\ell} \mathbf{u}_{\ell} \mathbf{v}_{\ell}^{\top} \right\|_{2}^{2} \quad \text{subject to} \quad \left\{ \begin{array}{l} \mathbf{u}_{\ell}^{\top} \mathbf{u}_{\ell} = 1 \\ \mathbf{v}_{\ell}^{\top} \mathbf{v}_{\ell} = 1 \\ \|\mathbf{u}_{\ell}\|_{1} \le s_{1,\ell} \\ \|\mathbf{v}_{\ell}\|_{1} \le s_{2,\ell} \end{array} \right. \qquad (9) $$

where $s_{1,\ell}$ and $s_{2,\ell}$ are positive constants, provided by the user as two vectors of length $L$ denoted $\mathbf{s}_1$ and $\mathbf{s}_2$ that will drive the sparsity of the solution. The solution to this optimization problem denoted $(\dot{\delta}, \dot{\mathbf{U}}, \dot{\mathbf{V}})$ is called a pseudo-singular triplet (containing respectively the pseudo-singular values, left pseudo-singular vectors, and right pseudo-singular vectors).

In PMD, the first pseudo-singular triplet is estimated by solving Equation 9 for $\ell = 1$. The next pseudo-singular triplets are estimated by approximating each subsequent deflated matrix by a rank one matrix. At each iteration $\ell > 1$, the deflated matrix is equal to

$$ \mathbf{X}_{\ell} = \mathbf{X}_{\ell-1} - \dot{\delta}_{\ell-1} \dot{\mathbf{u}}_{\ell-1} \dot{\mathbf{v}}_{\ell-1}^{\top}, \qquad (10) $$

where, by convention, $\mathbf{X}_1 = \mathbf{X}$. This procedure is very similar to the standard (i.e., Hotelling's) deflation for the SVD, but in Equation 10, the deflated matrix is not
guaranteed to be orthogonal to the previous rank one optimal matrix (as noted, e.g., by Mackey, 2009).

To (partially) palliate this problem, Witten et al. (2009) and Mackey (2009) independently proposed a heuristic to handle the non-orthogonality of the row (i.e., left) factor scores (in the context of sparse PCA) where the left pseudo-singular vectors are not required to be sparse. In this case, Hotelling's deflation is replaced by the so-called projection deflation

$$X_{\ell} = (I - \dot{u}_{\ell-1} \dot{u}_{\ell-1}^{\top}) X_{\ell-1}. \qquad (11)$$

## 3.2. Projected Penalized Matrix Decomposition

In our case, we want to be able to obtain both sparse left and right singular vectors. To do so, we propose to extend the updating step from Equation 11 to the left and right pseudo-singular vectors. This way, for each Dimension $\ell$, the projected deflated matrix is obtained as:

$$X_{\ell} = (I - \dot{u}_{\ell-1} \dot{u}_{\ell-1}^{\top}) \cdots (I - \dot{u}_{1} \dot{u}_{1}^{\top}) X (I - \dot{v}_{1} \dot{v}_{1}^{\top}) \cdots (I - \dot{v}_{\ell-1} \dot{v}_{\ell-1}^{\top}). \qquad (12)$$

With this deflation scheme, PMD is applied iteratively to the data matrix after the projection deflation. Combining PMD and the projected deflation, gives the projected Penalized Matrix Decomposition (pPMD, see Liu et al. 2023). It should be noted, however, that pPMD does not yield perfect orthogonality but (according to Witten et al., 2009) as for projection deflation, the solutions are unlikely to be highly correlated.

# 4. Sparse Correspondence Analysis

In this section, we present a new way to select optimal values for the sparsity parameters, as well as choosing the optimal number of dimensions for sparse CA. Finally, we discuss the effect of introducing sparsity on the properties of CA.

## 4.1. Sparse CA with pPMD

Because CA is obtained from the plain SVD of $\tilde{Z}$ (see Equation 8), the current version of sparse CA is obtained by applying pPMD to $\tilde{Z}$. This procedure generates (see Equation 9) pseudo-singular values (denoted $\delta$), left pseudo-singular vectors (denoted $\dot{U}$), and right pseudo-singular vectors (denoted $\dot{V}$). These pseudo vectors and values are then used to compute sparse weight and contribution matrices for rows and columns as:
• row weight matrix $\dot{\mathbf{P}} = \mathbf{D}_r^{\frac{1}{2}} \dot{\mathbf{U}}$,

• column weight matrix $\dot{\mathbf{G}} = \mathbf{D}_c^{\frac{1}{2}} \dot{\mathbf{V}}$,

Contribution matrices are obtained from the weight matrices:

• row contributions $\dot{\mathbf{T}}_I = \dot{\mathbf{U}} \odot \dot{\mathbf{U}}$,

• column contributions $\dot{\mathbf{T}}_J = \dot{\mathbf{V}} \odot \dot{\mathbf{V}}$.

Note that a zero weight implies a null contribution.

In this paper, we define, (in a manner reminiscent of the transition formula from CA), the factor scores of sparse CA as linear combinations of the profiles with the (sparse) weights:

• row factor scores $\dot{\mathbf{F}} = \mathbf{R}\mathbf{D}_c^{-1}\dot{\mathbf{Q}}$,

• column factor scores $\dot{\mathbf{G}} = \mathbf{C}\mathbf{D}_r^{-1}\dot{\mathbf{P}}$,

and (just like in plain CA) for each dimension, the variance of the factor scores is equal to the squared pseudo-singular value. Note, also, that, while weights are sparse, factor scores may not be sparse.

Even though the computation of the factor scores for sparse CA bears some resemblance to the transition formula (and would be equivalent to the transition formula in plain CA), the transition formulas (from Equation 43 in the Appendix) no longer hold: row (respectively column) factor scores are not barycenters anymore of the column (respectively row) factor scores: Transition formulas hold only for plain CA.

## 4.2. Two Types of Sparsity

Using a sparse version of CA is especially useful when the data are high-dimensional. However, large data sets come in two types: 1) both row and column sets are high-dimensional and sparsifying both dimensions makes sense, or 2) only one of the row or column sets is high-dimensional—and the data table is a “flat” or a “tall” contingency table—and then, it makes more sense to sparsify only the larger set of items. Therefore, two types of sparsity need to be considered for the sparse solution of CA: 1) both-side (or double) sparse CA or 2) one-side (or simple) sparse CA.

Both-side sparsity looks for underlying dimensions that are explained by sparse combinations of both rows and columns. Sparse SVD provides this solution. The easiest way to implement both-side sparsity is to sparsify rows and
columns weights in the same proportion, an approach which leads to choose similar degrees of sparsity for $\dot{\mathbf{P}}$ and $\dot{\mathbf{Q}}$. Following Witten et al. (2009), for $\dot{\mathbf{P}}$ and $\dot{\mathbf{Q}}$ to have a similar level of sparsity, $\eta$ is set constant (with $\eta < 1$) and the sparsity parameters are obtained as $s_{1,\ell} = \eta \sqrt{I}$, and $s_{2,\ell} = \eta \sqrt{J}$ (with $I$ and $J$ being the number of rows and columns of the data matrix). But, if the rows and columns contingency table correspond to essentially different types of variables, then it makes more sense to choose different degrees of sparsity for rows and columns. In this case, the parameter settings “grid” can be used, in order to restrict the $L_1$ norms of $\dot{\mathbf{P}}$ and $\dot{\mathbf{Q}}$ at different values: $s_{1,\ell}$ and $s_{2,\ell}$ will be restricted to take values (respectively) in the intervals $[1, \sqrt{I}]$ and $[1, \sqrt{J}]$.

One-side sparsity is suitable in asymmetrical situations when, for example, only the rows (or the columns) of the contingency table are relevant. In this case, only the relevant set needs to be sparsified. Interestingly, one-way sparsity is a special case of both-side sparsity when there is no penalty on one side (which is then left un-sparsified) and setting the sparsity parameter of the side to be sparsified equal to the square root of the cardinal of this set (e.g., to sparsify only the rows, set $s_{1,\ell} = \sqrt{I}$).

### 4.3. Choosing an Appropriate Value for the Sparsity Parameters: the Sparsity Index

An essential decision when using sparse CA is the choice of the values for the sparsity parameters $s_1$ and $s_2$ and the number of dimensions $L$. Various methods have been proposed: cross-validation (Witten et al., 2009), AIC or BIC (Shen et al. 2013; Zou et al. 2006), and compromise between the goodness of fit and sparsity (see, e.g., Trendafilov 2014; Trendafilov et al. 2017).

Among these procedures, we chose the sparsity index presented by Trendafilov et al. (2017). This sparsity index denoted, here, $\zeta(s_1, s_2, L)$, is the product of a “fit ratio” and a “zero ratio.”

The fit ratio is computed as the ratio of the sum of the pseudo-eigenvalues to the sum of the eigenvalues of the non-sparse solution, specifically

$$ \text{fit ratio} = \frac{\sum_{\ell=1}^{L} \dot{\lambda}_{\ell}}{\sum_{\ell=1}^{L} \lambda_{\ell}}. \qquad (13) $$

The fit ratio takes values between 0 and 1 with larger values indicating a better fit. The zero ratio is the ratio of the number of zero weights to the total number of
weights, specifically:

$$ \text{zero ratio} = \frac{\#0(\dot{\mathbf{P}}) + \#0(\dot{\mathbf{Q}})}{(I+J)L}, \qquad (14) $$

where $\#0(\dot{\mathbf{P}})$ [resp. $\#0(\dot{\mathbf{Q}})$] is the total number of zeros in $\dot{\mathbf{P}}$ (resp. $\dot{\mathbf{Q}}$). The zero ratio takes values between 0 and 1 with larger values indicating a sparser solution. The sparsity index $\zeta(\mathbf{s}_1, \mathbf{s}_2, L)$ is obtained as the product of the fit and the zero ratios, namely:

$$ \zeta(\mathbf{s}_1, \mathbf{s}_2, L) = \frac{\sum_{\ell=1}^{L} \dot{\lambda}_{\ell}}{\sum_{\ell=1}^{L} \lambda_{\ell}} \underbrace{\frac{\#0(\dot{\mathbf{P}}) + \#0(\dot{\mathbf{Q}})}{(I+J)L}}_{\text{"fit ratio"}}, \qquad (15) $$

To sum up, the sparsity index is a compromise between maximizing the explained variance (i.e., the fit ratio) and sparsifying the results (i.e., the zero ratio). In our application of sparse CA, we will therefore seek for the value(s) of $L$, $\mathbf{s}_1$, and $\mathbf{s}_2$ that maximize $\zeta(\mathbf{s}_1, \mathbf{s}_2, L)$.

Our global optimization algorithm differs from the sequential algorithm of Liu et al. (2023) which searches for the optimal sparsity level for each dimension conditional on the sparsity levels obtained for the previous dimensions, but without searching for an optimal value of $L$ (i.e., the dimensionality of the space). In contrast, we obtain a global optimum in a space of Dimension $L$ with the additional constraints that all dimensions have identical levels of sparsity : $s_{11} = s_{12} = \dots = s_{1L}$.

### 4.4. Lost Properties and other Issues

In addition to the usual (but still open) question of “How many components to keep?” sparse exploratory methods raise new specific issues such as—among others—loss of orthogonality and choice of the sparsity level.

The simultaneous orthogonality of the weight vectors and of the factor scores characterizes PCA (and SVD) because weight vectors and factor scores are both true eigenvectors. But this simultaneous orthogonality is lost in sparse PCA and similar methods: One cannot have both orthogonality for the weights and for the factor scores. For example, if we force successive sparse weight vectors to be orthogonal, as in SCoTLaSS (Jolliffe et al., 2003), the associated factor scores are no longer orthogonal.
This lack of orthogonality makes the interpretation of the factor scores some-
what difficult (in a way reminiscent of the issues linked to oblique rotation in
traditional factor analysis) because conclusions about one dimension involve all
correlated dimensions and because the same information is explained (to differ-
ent degrees) by all correlated dimensions. When interpreting the factor scores,
one could erroneously find the same information in different dimensions. In ad-
dition, with non-orthogonal factor scores, the variances explained by different
dimensions are no longer additive (i.e., the sum of the variances explained by a
set of non-orthogonal dimensions will over-estimatethe variance of the sub-space
spanned by these dimensions).

As we have noticed before, the simultaneous pseudo-barycentric transition
formulas (from Equation 43 in Appendix B) do not hold anymore because these
formulas are a characteristic property of plain CA. Here **F** is not proportional to
**P** and **G** is not proportional to **Q**: In other words, the relationship between the
weights and the factor scores is not linear anymore. As a consequence, graphics
should be drawn using the factor scores **F** and **G** rather than the weights **P** and
**Q**, because a graphic drawn from the weights is likely to have too many points
stuck to the axes (these will be the items with zero weights). However, a graph
drawn from the weights or even from the signed contributions could be of interest
in some applications.

**5. A Real Data Example**

We applied sparse CA to a data set—obtained from the Project Gutenberg
(Gerlach and Font-Clos, 2020)—compiling common words used in 100 books
each from 5 book categories: Biographies, Love stories, Mystery, Philosophy,
and Science Fiction. This created a contingency table (counting the number of
occurrences of words per book) with 1502 rows (words) and 500 columns (books).

**5.1. Plain CA Results**

Factor scores maps for plain CA for Dimensions 1 and 2 are shown in Fig-
ures 1 (for the words) and 2 (for the books). The word factor map shows only a
few words, whereas the book factor map does not show the names of the books
but color them by genre and add, for each type of book, a 70% tolerance convex
hulls (a *K% tolerance* interval comprises *K%* of a sample or a population, see,
e.g., Abdi et al. 2009)<fnref n="1" />. Parallel to the partition of the vocabulary, Dimension 1

<footnote n="1">¹The graph and convex hulls were created using functions ggConvexHull and CreateFactorMaps4CA from the R-package PTCA4CATA.</footnote>
(see Figure 2) differentiates the Philosophical genre from Love stories, Mystery, and Science fiction.

<figure><img src="image_3.png" /><figcaption>**Figure 1: Plain CA: Row factor scores**</figcaption></figure>

The second dimension of plain CA explains 7% of the inertia. As shown in Figure 1 and Table 1b), the row factor scores are characterized by the opposition of (on the negative side) words related to war (enemy, battle, war, government) or geography (e.g., south, north, west, east, miles, city), and verbs in the past tense (e.g., ordered, united, received, sent, arrived) versus on the negative side, words related to thoughts (e.g., understanding, experience, reason, meaning, conscious) and verbs in the present tense (e.g., does, miss, mean, thinks, makes, is, can).
Parallel to the partition of the vocabulary, Dimension 2 (see vertical axis Figure 2) differentiates the Biography genre from books of Philosophy.

<figure><img src="image_4.png" /><figcaption>Figure 2: Plain CA: Column factor scores</figcaption></figure>

In general, the Philosophy genre uses language closely aligned with scientific writing. In contrast, the Biography genre typically uses language focused on the events of a male character's past, often involving war. Meanwhile, the other three genres of fiction are more concerned with describing emotions and the experiences of female characters.

The Philosophy genre is particularly distinctive in its use of language, because it preferably uses highly specialized terminology not commonly found in
other genres. However, interpreting factor scores for this genre is challenging given the presence of many words with close-to-zero weights that are difficult to integrate in a coherent framework.

## 5.2. Choosing an Optimal Value of the Constraints Parameters with the Sparsity Index

Recall that the sparsity index $\zeta(s_1, s_2, L)$ is a function of $s_1$, $s_2$, and $L$. To find the sparsity index optimal value, we explored the (3-dimensional) space spanned by these parameters. We chose $L$ (i.e., the number of dimensions) from all integers between 2 and 20, and we chose $s_1$ (resp. $s_2$) from the 20 possible values evenly distributed between 1 and $\sqrt{I}$ (resp. $\sqrt{J}$). To speed up computations and provide the same sparsity value for each dimension, we decided to have identical values for $s_1$ and $s_2$. We then iteratively applied sparse CA to the Gutenberg Project data set with a number of dimensions equal to $L$ (with $L$ varying from 1 to 20). Figure 3 shows the scatterplot of all combinations of the parameters with zero ratio (see Equation 15) on the horizontal axis and the fit ratio on the vertical axis. The points in the figure are colored according to the number of dimensions of these solutions. The isolines correspond to a fixed value of the sum of squared fit and zero ratios, for example, the thick isoline going from a fit ratio of 1 to the zero ratio of 1 is the locus of the sum of these two squared ratios equal to 1. The closest solutions to the upper right corner (which matches a fit and a zero ratio of 1) is the optimal one with the largest sparsity index. In Figure 3, this solution is indicated by the arrow and the value of its sparsity index. In this analysis, the optimal sparsity index is equal to .47 and occurs for an $L = 2$ factor solution with a fit ratio equal to .72, a zero ratio equals .66, and sparsity parameters for rows being $s_1 \approx (10.94, 10.94)$ and for columns being $s_2 \approx (13.37, 13.37)$.

Figure 4 shows the values taken by the sparsity parameter on a map where the horizontal axis corresponds to the 20 values chosen between $\frac{1}{I}$ and $s_1$ and the vertical axis corresponds to the 20 values chosen between $\frac{1}{J}$ and $s_2$. The values of $s_1$ (resp. $s_2$) are scaled by $I$ (resp. $J$) so that the range of these possible sparsity parameters becomes between 0 and 1. In Figure 4, the optimal solution, which has the largest sparsity index is identified by the star.

## 5.3. Sparse CA Results

The results from sparse CA are shown in Figures 5 to 8. With sparsification, the words that have small contributions in plain CA now have zero contribution and the words with large contributions now have even larger contributions—a
<figure><img src="image_5.png" /><figcaption>Figure 3: Fit ratio to zero ratio graph.</figcaption></figure>

pattern shown in Figure 5 which plots, for the rows, the plain CA contributions (ordered from left to right by their factor scores) versus on the bottom the sparse CA.

A similar pattern, but with a smaller effect of sparsity, is found for the
contributions of the books (see Figure 6). The factor scores are shown in Figures 7
and 8 with words and books that have zero contributions on both dimensions indi-
cated by hollow dots. The first dimension of sparse CA explains 17% of the inertia.
Similar to plain CA, the first dimension differentiates neutral pronouns (e.g., itself,
their, us, human, this) and words (e.g., understanding, pure, science, ideas) from
<figure><img src="image_6.png" /><figcaption>Figure 4: Sparsity Index map.</figcaption></figure>

words of feminine figures (e.g., girl, she, herself, lady) and words describing emotions (e.g., miss, smiled, laughed, sorry; see Table 2a and the horizontal axis in Figure 7).

Patterns similar to plain CA are also found for the column factor scores of sparse CA, which differentiates the Philosophy genre from Love stories, Mystery, and Science fiction (see the horizontal axis in Figure 8). Books with a small contribution in plain CA have a zero contribution with sparse CA and books with a large contributions in CA now have even larger contributions in sparse CA—a pattern shown in Figure 6 which plots, for the books, the plain CA contributions
(ordered from left to right by their factor scores) versus, on the bottom, the sparse CA.

The second dimension of sparse CA explains 5% of the inertia. The row factor scores are characterized by the opposition (on the positive side the dimension) of words related to war (e.g., enemy, battle, attack, war), geography (e.g., south, north, east, west, miles, city), and verbs in the past tense (e.g., ordered, united, received) to (on the negative side the dimension) words related to thoughts (e.g., understanding, experience, reason, meaning, think) and feminine figures (e.g., she, girl, herself, lady, her), and verbs in the present tense (e.g., miss, is, does, say, be, can, am) (see the vertical axis in Figure 7 and Table 2b).

The column factor scores differentiate books from the Biography genre from books of Philosophy, Love stories, and Mystery (see the vertical axis in Figure 8 and bottom panel of Figure 6).

Compared to the results from plain CA, because sparse CA shrunk some words while emphasizing others, the pattern opposing neutral versus feminine pronouns along Dimension 1 becomes more noticeable. Compared to the results from plain CA, because the transition formula is no longer valid in sparse CA, the sparsity of the contributions (derived from loadings; see Figures 5 and 6) does not propagate to give sparse factor scores. But, as demonstrated, the sparsity of contributions can be integrated to facilitate the interpretation of factor scores. Moreover, although the results from sparse CA do not have the optimal proportion of explained inertia, sparse CA gives the solution with the optimal trade-off between the inertia explained and sparsity. Finally, it is worth noting that because the sparse CA factor scores of the two dimensions are not orthogonal, their percentages of explained inertia are not additive and need to be considered separately. However, here although the components from sparse CA are not orthogonal, the two selected dimensions are close-to-orthogonal with a correlation coefficient of .02.
<table><caption>(a) Dimension 1</caption><thead><tr><th>words</th><th>loadings</th><th>contributions<sup>a</sup></th><th>factor scores</th></tr></thead><tbody><tr><td>science</td><td>-4.22</td><td>60</td><td>-1.40</td></tr><tr><td>ideas</td><td>-3.80</td><td>38</td><td>-1.26</td></tr><tr><td>understanding</td><td>-3.71</td><td>21</td><td>-1.23</td></tr><tr><td>existence</td><td>-3.68</td><td>39</td><td>-1.22</td></tr><tr><td>system</td><td>-3.54</td><td>28</td><td>-1.17</td></tr><tr><td>pure</td><td>-3.47</td><td>20</td><td>-1.15</td></tr><tr><td>ii</td><td>-3.43</td><td>27</td><td>-1.14</td></tr><tr><td>experience</td><td>-3.31</td><td>36</td><td>-1.10</td></tr><tr><td>physical</td><td>-3.30</td><td>16</td><td>-1.09</td></tr><tr><td>material</td><td>-3.30</td><td>15</td><td>-1.09</td></tr><tr><td>nature</td><td>-3.25</td><td>79</td><td>-1.08</td></tr><tr><td>knowledge</td><td>-3.24</td><td>49</td><td>-1.07</td></tr><tr><td>iii</td><td>-3.24</td><td>15</td><td>-1.07</td></tr><tr><td>itself</td><td>-3.17</td><td>64</td><td>-1.05</td></tr><tr><td>according</td><td>-3.16</td><td>22</td><td>-1.05</td></tr><tr><td>example</td><td>-3.09</td><td>14</td><td>-1.02</td></tr><tr><td>parts</td><td>-3.06</td><td>17</td><td>-1.01</td></tr><tr><td>series</td><td>-3.05</td><td>10</td><td>-1.01</td></tr><tr><td>progress</td><td>-3.02</td><td>15</td><td>-1.00</td></tr><tr><td>object</td><td>-2.94</td><td>26</td><td>-0.98</td></tr><tr><td>stepped</td><td>2.11</td><td>3</td><td>0.70</td></tr><tr><td>sat</td><td>2.12</td><td>14</td><td>0.70</td></tr><tr><td>box</td><td>2.13</td><td>5</td><td>0.70</td></tr><tr><td>sorry</td><td>2.13</td><td>6</td><td>0.71</td></tr><tr><td>smile</td><td>2.13</td><td>9</td><td>0.71</td></tr><tr><td>cried</td><td>2.13</td><td>16</td><td>0.71</td></tr><tr><td>door</td><td>2.15</td><td>27</td><td>0.71</td></tr><tr><td>window</td><td>2.17</td><td>10</td><td>0.72</td></tr><tr><td>yes</td><td>2.18</td><td>30</td><td>0.72</td></tr><tr><td>guess</td><td>2.25</td><td>5</td><td>0.74</td></tr><tr><td>nice</td><td>2.25</td><td>4</td><td>0.74</td></tr><tr><td>laughed</td><td>2.31</td><td>9</td><td>0.77</td></tr><tr><td>shook</td><td>2.33</td><td>8</td><td>0.77</td></tr><tr><td>she</td><td>2.35</td><td>380</td><td>0.78</td></tr><tr><td>ca</td><td>2.36</td><td>14</td><td>0.78</td></tr><tr><td>whispered</td><td>2.36</td><td>6</td><td>0.78</td></tr><tr><td>oh</td><td>2.38</td><td>31</td><td>0.79</td></tr><tr><td>miss</td><td>2.42</td><td>36</td><td>0.80</td></tr><tr><td>girl</td><td>2.48</td><td>31</td><td>0.82</td></tr><tr><td>smiled</td><td>2.50</td><td>9</td><td>0.83</td></tr></tbody></table>

<table><caption>(b) Dimension 2</caption><thead><tr><th>words</th><th>loadings</th><th>contributions<sup>a</sup></th><th>factor scores</th></tr></thead><tbody><tr><td>understanding</td><td>-5.40</td><td>4</td><td>-0.99</td></tr><tr><td>pure</td><td>-4.54</td><td>3</td><td>-0.83</td></tr><tr><td>experience</td><td>-4.07</td><td>5</td><td>-0.75</td></tr><tr><td>existence</td><td>-4.05</td><td>4</td><td>-0.74</td></tr><tr><td>ideas</td><td>-3.77</td><td>3</td><td>-0.69</td></tr><tr><td>space</td><td>-3.57</td><td>2</td><td>-0.65</td></tr><tr><td>reason</td><td>-3.38</td><td>6</td><td>-0.62</td></tr><tr><td>sense</td><td>-3.19</td><td>4</td><td>-0.58</td></tr><tr><td>object</td><td>-3.16</td><td>2</td><td>-0.58</td></tr><tr><td>science</td><td>-3.10</td><td>3</td><td>-0.57</td></tr><tr><td>physical</td><td>-3.08</td><td>1</td><td>-0.56</td></tr><tr><td>material</td><td>-3.01</td><td>1</td><td>-0.55</td></tr><tr><td>itself</td><td>-2.99</td><td>5</td><td>-0.55</td></tr><tr><td>meaning</td><td>-2.94</td><td>1</td><td>-0.54</td></tr><tr><td>conscious</td><td>-2.94</td><td>0</td><td>-0.54</td></tr><tr><td>does</td><td>-2.91</td><td>5</td><td>-0.53</td></tr><tr><td>merely</td><td>-2.87</td><td>2</td><td>-0.53</td></tr><tr><td>soul</td><td>-2.76</td><td>2</td><td>-0.50</td></tr><tr><td>nature</td><td>-2.75</td><td>5</td><td>-0.50</td></tr><tr><td>absolutely</td><td>-2.65</td><td>0</td><td>-0.49</td></tr><tr><td>post</td><td>3.74</td><td>0</td><td>0.69</td></tr><tr><td>hundred</td><td>3.77</td><td>4</td><td>0.69</td></tr><tr><td>report</td><td>3.95</td><td>1</td><td>0.72</td></tr><tr><td>english</td><td>3.99</td><td>4</td><td>0.73</td></tr><tr><td>city</td><td>4.11</td><td>4</td><td>0.75</td></tr><tr><td>miles</td><td>4.71</td><td>4</td><td>0.86</td></tr><tr><td>east</td><td>4.79</td><td>2</td><td>0.88</td></tr><tr><td>orders</td><td>4.80</td><td>2</td><td>0.88</td></tr><tr><td>united</td><td>4.98</td><td>2</td><td>0.91</td></tr><tr><td>west</td><td>5.11</td><td>2</td><td>0.94</td></tr><tr><td>government</td><td>5.34</td><td>5</td><td>0.98</td></tr><tr><td>ordered</td><td>5.36</td><td>2</td><td>0.98</td></tr><tr><td>war</td><td>5.72</td><td>9</td><td>1.05</td></tr><tr><td>attack</td><td>5.83</td><td>3</td><td>1.07</td></tr><tr><td>command</td><td>6.05</td><td>4</td><td>1.11</td></tr><tr><td>river</td><td>6.27</td><td>8</td><td>1.15</td></tr><tr><td>north</td><td>6.47</td><td>5</td><td>1.19</td></tr><tr><td>battle</td><td>6.61</td><td>5</td><td>1.21</td></tr><tr><td>south</td><td>6.72</td><td>5</td><td>1.23</td></tr><tr><td>enemy</td><td>8.42</td><td>13</td><td>1.54</td></tr></tbody></table>

<caption>Table 1: The 20 most extreme words from each dimension of Plain CA.</caption>

*Note:* The contributions shown as 0 in 1b were too small to be displayed as integers; it is worth noting that this value does not indicate zero contributions. * indicates that the original values were multiplied by 10,000 and rounded to the nearest integer for display purposes.
<table><caption>(a) Dimension 1</caption><thead><tr><th>words</th><th>loadings*</th><th>contributions*</th><th>factor scores</th></tr></thead><tbody><tr><td>understanding</td><td>-0.67</td><td>2</td><td>-1.45</td></tr><tr><td>science</td><td>-1.75</td><td>9</td><td>-1.44</td></tr><tr><td>ideas</td><td>-1.16</td><td>5</td><td>-1.32</td></tr><tr><td>pure</td><td>-0.63</td><td>2</td><td>-1.30</td></tr><tr><td>existence</td><td>-1.24</td><td>5</td><td>-1.29</td></tr><tr><td>experience</td><td>-1.29</td><td>5</td><td>-1.19</td></tr><tr><td>system</td><td>-0.79</td><td>2</td><td>-1.18</td></tr><tr><td>ii</td><td>-0.81</td><td>2</td><td>-1.16</td></tr><tr><td>physical</td><td>-0.42</td><td>1</td><td>-1.13</td></tr><tr><td>knowledge</td><td>-1.82</td><td>7</td><td>-1.12</td></tr><tr><td>material</td><td>-0.36</td><td>0</td><td>-1.11</td></tr><tr><td>series</td><td>-0.25</td><td>0</td><td>-1.11</td></tr><tr><td>itself</td><td>-2.58</td><td>10</td><td>-1.11</td></tr><tr><td>nature</td><td>-3.09</td><td>12</td><td>-1.10</td></tr><tr><td>iii</td><td>-0.37</td><td>0</td><td>-1.09</td></tr><tr><td>object</td><td>-0.99</td><td>3</td><td>-1.07</td></tr><tr><td>space</td><td>-0.59</td><td>1</td><td>-1.06</td></tr><tr><td>according</td><td>-0.65</td><td>1</td><td>-1.05</td></tr><tr><td>parts</td><td>-0.49</td><td>1</td><td>-1.04</td></tr><tr><td>example</td><td>-0.33</td><td>0</td><td>-1.02</td></tr><tr><td>sat</td><td>0.44</td><td>0</td><td>0.64</td></tr><tr><td>lips</td><td>0.18</td><td>0</td><td>0.64</td></tr><tr><td>dear</td><td>0.71</td><td>1</td><td>0.65</td></tr><tr><td>smiling</td><td>0.00</td><td>0</td><td>0.65</td></tr><tr><td>sorry</td><td>0.04</td><td>0</td><td>0.65</td></tr><tr><td>yes</td><td>1.17</td><td>2</td><td>0.65</td></tr><tr><td>shook</td><td>0.07</td><td>0</td><td>0.65</td></tr><tr><td>ca</td><td>0.30</td><td>0</td><td>0.65</td></tr><tr><td>whispered</td><td>0.02</td><td>0</td><td>0.66</td></tr><tr><td>smile</td><td>0.20</td><td>0</td><td>0.66</td></tr><tr><td>lady</td><td>1.01</td><td>1</td><td>0.67</td></tr><tr><td>nice</td><td>0.00</td><td>0</td><td>0.68</td></tr><tr><td>laughed</td><td>0.15</td><td>0</td><td>0.68</td></tr><tr><td>her</td><td>22.60</td><td>67</td><td>0.70</td></tr><tr><td>herself</td><td>0.86</td><td>1</td><td>0.70</td></tr><tr><td>smiled</td><td>0.13</td><td>0</td><td>0.73</td></tr><tr><td>oh</td><td>1.17</td><td>2</td><td>0.73</td></tr><tr><td>girl</td><td>1.09</td><td>2</td><td>0.75</td></tr><tr><td>she</td><td>23.05</td><td>77</td><td>0.78</td></tr><tr><td>miss</td><td>1.65</td><td>4</td><td>0.82</td></tr></tbody></table>

<table><caption>(b) Dimension 2</caption><thead><tr><th>words</th><th>loadings*</th><th>contributions*</th><th>factor scores</th></tr></thead><tbody><tr><td>understanding</td><td>-0.95</td><td>5</td><td>-0.93</td></tr><tr><td>pure</td><td>-0.78</td><td>3</td><td>-0.76</td></tr><tr><td>experience</td><td>-1.58</td><td>7</td><td>-0.67</td></tr><tr><td>space</td><td>-0.75</td><td>2</td><td>-0.64</td></tr><tr><td>existence</td><td>-1.28</td><td>5</td><td>-0.63</td></tr><tr><td>ideas</td><td>-1.01</td><td>3</td><td>-0.57</td></tr><tr><td>reason</td><td>-2.28</td><td>9</td><td>-0.57</td></tr><tr><td>object</td><td>-0.97</td><td>3</td><td>-0.53</td></tr><tr><td>sense</td><td>-1.31</td><td>4</td><td>-0.52</td></tr><tr><td>miss</td><td>-1.46</td><td>3</td><td>-0.51</td></tr><tr><td>does</td><td>-2.27</td><td>7</td><td>-0.49</td></tr><tr><td>mean</td><td>-0.93</td><td>2</td><td>-0.48</td></tr><tr><td>meaning</td><td>-0.17</td><td>0</td><td>-0.48</td></tr><tr><td>conscious</td><td>-0.05</td><td>0</td><td>-0.46</td></tr><tr><td>physical</td><td>-0.23</td><td>0</td><td>-0.46</td></tr><tr><td>merely</td><td>-0.57</td><td>1</td><td>-0.45</td></tr><tr><td>material</td><td>-0.17</td><td>0</td><td>-0.45</td></tr><tr><td>things</td><td>-2.88</td><td>8</td><td>-0.44</td></tr><tr><td>nice</td><td>-0.00</td><td>0</td><td>-0.44</td></tr><tr><td>absolutely</td><td>-0.05</td><td>0</td><td>-0.43</td></tr><tr><td>advance</td><td>0.17</td><td>0</td><td>0.67</td></tr><tr><td>report</td><td>0.16</td><td>0</td><td>0.72</td></tr><tr><td>city</td><td>0.97</td><td>3</td><td>0.73</td></tr><tr><td>general</td><td>3.26</td><td>16</td><td>0.76</td></tr><tr><td>english</td><td>1.48</td><td>7</td><td>0.77</td></tr><tr><td>miles</td><td>0.92</td><td>4</td><td>0.80</td></tr><tr><td>east</td><td>0.36</td><td>1</td><td>0.88</td></tr><tr><td>orders</td><td>0.57</td><td>2</td><td>0.92</td></tr><tr><td>west</td><td>0.54</td><td>2</td><td>0.94</td></tr><tr><td>united</td><td>0.56</td><td>2</td><td>0.95</td></tr><tr><td>ordered</td><td>0.51</td><td>2</td><td>1.01</td></tr><tr><td>government</td><td>1.43</td><td>9</td><td>1.05</td></tr><tr><td>war</td><td>2.34</td><td>18</td><td>1.12</td></tr><tr><td>attack</td><td>0.69</td><td>4</td><td>1.13</td></tr><tr><td>river</td><td>1.65</td><td>13</td><td>1.17</td></tr><tr><td>command</td><td>1.05</td><td>8</td><td>1.22</td></tr><tr><td>north</td><td>1.04</td><td>8</td><td>1.23</td></tr><tr><td>south</td><td>1.11</td><td>9</td><td>1.29</td></tr><tr><td>battle</td><td>0.93</td><td>7</td><td>1.29</td></tr><tr><td>enemy</td><td>2.32</td><td>28</td><td>1.67</td></tr></tbody></table>

<caption>Table 2: The 20 most extreme words from each dimension of Sparse CA.</caption>

*Note:* The loadings and contributions shown as 0 in these tables were too small to be displayed as integers; it is worth noting that these values do not indicate sparsity. * indicates that the original values were multiplied by 10,000 and rounded to the nearest integer for display purposes.
<figure><img src="image_7.png" /><figcaption>Figure 5: Plain CA vs. Sparse CA: Row contributions</figcaption></figure>

# 6. Conclusion and Perspectives

In this paper, we extended sparse correspondence analysis developed by Liu et al. (2023) by adding a new global algorithm that identifies the optimal sparsity solution by determining both the optimal sparsity tuning parameters and the optimal number of kept dimensions. Specifically, by integrating this global algorithm, this new version of sparse CA estimates the optimal solution in a more analytic and objective way. Sparse correspondence analysis simplifies the interpretation in the analysis of large tables by highlighting important categories and obtaining
<figure><img src="image_8.png" /><figcaption>Figure 6: Plain CA vs. Sparse CA: Column contributions</figcaption></figure>

simple successive dimensions in the spirit of the simple structure of factor analysis. Its practical application raises new problems such as the choice of the optimal level of sparsity for rows and or columns, which could be different according to each dimension.

Another concern is the loss of orthogonality of successive dimensions—An issue that should be explored in future work.

Sparse CA remains basically a symmetrical method where rows and columns play the same role. In future work, we also plan to develop sparse variants of the non symmetric correspondence analysis introduced by Lauro and D'Ambra
<figure><img src="image_9.png" /><figcaption>Figure 7: Sparse CA: Row factor scores</figcaption></figure>

(1984) and explored by Balbi (1998).

Code and data are available at:

https://github.com/vguillemot/sparseCorrespondenceAnalysis.
<figure><img src="image_10.png" /><figcaption>Figure 8: Sparse CA: Column factor scores</figcaption></figure>

References

Abdi, H. (2007). Singular value decomposition (SVD) and generalized singular value decomposition (GSVD). In N. Salkind, ed., *Encyclopedia of Measurement and Statistics*, 907–912. Sage Publications, Thousand Oaks.

Abdi, H. and Béra, M. (2018). Correspondence analysis. In R. Alhajj and J. Rokne, eds., *Encyclopedia of Social Networks and Mining (2nd Edition)*, 1–12. Springer, New York. doi:10.1007/978-1-4614-7163-9_140-2.
Abdi, H., Dunlop, J., and Williams, L.J. (2009). How to compute reliability estimates and display confidence and tolerance intervals for pattern classifiers using the bootstrap and 3-way multidimensional scaling (DISTATIS). In *NeuroImage*, 45 (1): 89–95. doi:10.1016/j.neuroimage.2008.11.008.

Abdi, H. and Williams, L.J. (2010). Principal component analysis. In *WIREs Computational Statistics*, 2 (4): 433–459. doi:https://doi.org/10.1002/wics.101.

Allen, G.I., Grosenick, L., and Taylor, J. (2014). A generalized least-square matrix decomposition. In *Journal of the American Statistical Association*, 109 (505): 145–159. doi:10.1080/01621459.2013.852978.

Balbi, S. (1998). Graphical displays in non-symmetrical correspondence analysis. In J. Blasius and M.J. Greenacre, eds., *Visualization of Categorical Data*, 297–309. Academic Press, San Diego. doi:10.1016/B978-012299045-8/50023-1.

Beaton, D. (2020). Generalized eigen, singular value, and partial least squares decompositions: The GSVD package. doi:10.48550/ARXIV.2020.14734.

Beh, E.J. and Lombardo, R. (2021). *An Introduction to Correspondence Analysis*. Wiley, London. doi:10.1002/9781119044482.

Benidis, K., Sun, Y., Babu, P., and Palomar, D.P. (2016). Orthogonal sparse PCA and covariance estimation via procrustes reformulation. In *IEEE Transactions on Signal Processing*, 64 (23): 6211–6226. doi:10.1109/TSP.2016.2605073.

Bernard, A., Guinot, C., and Saporta, G. (2012). Sparse principal component analysis for multiblock data and its extension to sparse multiple correspondence analysis. In A. Colubi and al., eds., *Proceedings of the 20th International Conference on Computational Statistics (COMPSTAT 2012)*, 99–106. International Association for Statistical Computing.

Chun, H. and Keleş, S. (2010). Sparse partial least squares regression for simultaneous dimension reduction and variable selection. In *Journal of the Royal Statistical Society. Series B, Statistical Methodology*, 72 (1): 3–25. doi: 10.1111/j.1467-9868.2009.00723.x.

Dray, S. (2014). Analysing a pair of tables: coinertia analysis and duality. In J. Blasius and M.J. Greenacre, eds., *Visualization and Verbalization of Data*, 289–300. CRC Press, London.
Eckart, C. and Young, G. (1936). The approximation of one matrix by another of lower rank. In *Psychometrika*, 1 (3): 211–218. doi:10.1007/BF02288367.

Escoufier, Y. (2006). Operators related to a data matrix: a survey. In A. Rizzi and M. Vichi, eds., *Proceedings of the 17th International Conference on Computational Statistics (COMPSTAT 2006)*, 285–297. Physica-Verlag, Heidelberg. doi:10.1007/978-3-7908-1709-6_22.

Gerlach, M. and Font-Clos, F. (2020). A standardized project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics. In *En-tropy*, 22 (1). doi:10.3390/e22010126.

Greenacre, M.J. (1984). *Theory and Applications of Correspondence Analysis*. Academic Press, New York.

Greenacre, M.J. (2010). Correspondence analysis. In *Wiley Interdisciplinary Re-views: Computational Statistics*, 2 (5): 613–619.

Guillemot, V., Beaton, D., Gloaguen, A., Löfstedt, T., Levine, B., Raymond, N., Tenenhaus, A., and Abdi, H. (2019). A constrained singular value decomposi-tion method that integrates sparsity and orthogonality. In *PLOS ONE*, 14 (3): e0211463. doi:10.1371/journal.pone.0211463.

Guillemot, V., Le Borgne, J., Gloaguen, A., Tenenhaus, A., Saporta, G., Chollet, S., Beaton, D., and Abdi, H. (2020). Sparse multiple correspondence analysis. In *52èmes Journées de Statistique*, 830–835. URL https://pasteur.hal.science/pasteur-03037346/.

Hastie, T., Tibshirani, R., and Friedman, J. (2001). *The Elements of Statistical Learning*. Springer, New York. doi:10.1007/978-0-387-84858-7.

Hastie, T., Tibshirani, R., and Wainwright, M. (2015). *Statistical Learning with Sparsity: the Lasso and Generalizations*. CRC Press, Boca Raton. doi:10.1201/b18401.

Holmes, S. (2008). Multivariate data analysis: The French way. In *Probability and Statistics: Essays in Honor of David A. Freedman*, vol. 2, 219–234. Institute of Mathematical Statistics. doi:10.1214/193940307000000455.

Jenatton, R., Audibert, J.Y., and Bach, F. (2011). Structured variable selection with sparsity-inducing norms. In *The Journal of Machine Learning Research*, 12: 2777–2824. doi:10.5555/1953048.2078194.
Jolliffe, I.T. and Cadima, J. (2016). Principal component analysis: a review and recent developments. In *Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences*, 374 (2065). doi: 10.1098/rsta.2015.0202.

Jolliffe, I.T., Trendafilov, N.T., and Uddin, M. (2003). A modified principal component technique based on the LASSO. In *Journal of Computational and Graphical Statistics*, 12 (3): 531–547. doi:10.1198/1061860032148.

Lauro, N. and D'Ambra, L. (1984). L'analyse non symétrique des correspondances. In E. Diday, ed., *Data Analysis and Informatics III*, 433–446. Elsevier, North-Holland.

Le Floch, E., Guillemot, V., Frouin, V., Pinel, P., Lalanne, C., Trinchera, L., Tenenhaus, A., Moreno, A., Zilbovicius, M., Bourgeron, T., Dehaene, S., Thirion, B., Poline, J.B., and Duchesnay, E. (2012). Significant correlation between a set of genetic polymorphisms and a functional brain network revealed by feature selection and sparse partial least squares. In *NeuroImage*, 63 (1): 11–24. doi:10.1016/j.neuroimage.2012.06.061.

Lebart, L., Morineau, A., and Warwick, K. (1984). *Multivariate Descriptive Statistical Analysis: Correspondence Analysis and Related Techniques for Large Matrices*. Wiley, New York.

Liu, R., Niang, N., Saporta, G., and Wang, H. (2023). Sparse correspondence analysis for large contingency tables. In *Advances in Data Analysis and Classification*, 17: 1–20. doi:10.1007/s11634-022-00531-5.

Mackey, L. (2009). Deflation methods for sparse PCA. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, eds., *Advances in Neural Information Processing Systems*, vol. 21, 1017–1024. Curran Associates, Inc.

Mattei, P.A., Bouveyron, C., and Latouche, P. (2016). Globally sparse probabilistic pca. In A. Gretton and C.C. Robert, eds., *Proceedings of the 19th International Conference on Artificial Intelligence and Statistics*, vol. 51 of *Proceedings of Machine Learning Research*, 976–984. PMLR, Cadiz, Spain.

Mori, Y., Kuroda, M., and Makino, N. (2016). Sparse multiple correspondence analysis. In Y. Mori, M. Kuroda, and N. Makino, eds., *Nonlinear Principal Component Analysis and its Applications*, 47–56. Springer, New York. doi: 10.1007/978-981-10-0159-8_5.
Ning-min, S. and Jing, L. (2015). A literature survey on high-dimensional sparse
principal component analysis. In *International Journal of Database Theory and
Application*, 8 (6): 57–74. doi:10.14257/ijdta.2015.8.6.06.

Saporta, G. and Niang-Keita, N. (2006). Correspondence analysis and classi-
fication. In M.J. Greenacre and J. Blasius, eds., *Multiple Correspondence
Analysis and Related Methods*, 371–392. CRC Press, London. doi:10.1201/
9781420011319-19.

Shen, D., Shen, H., and Marron, J.S. (2013). Consistency of sparse PCA in high dimension, low sample size contexts. In *Journal of Multivariate Analysis*, 115: 317–333. doi:10.1016/j.jmva.2012.10.007.

Silver, M., Janousova, E., Hua, X., Thompson, P.M., Montana, G., and Alzheimer's Disease Neuroimaging Initiative, T.A.D.N. (2012). Identification of gene pathways implicated in Alzheimer's disease using longitudinal imaging phenotypes with sparse regression. In *NeuroImage*, 63 (3): 1681–1694. doi:10.1016/j.neuroimage.2012.08.002.

Takane, Y. (2002). Relationships among various kinds of eigenvalue and singular value decompositions. In H. Yanai, A. Okada, K. Shigemasu, Y. Kano, and J. Meulman, eds., *New Developments in Psychometrics*, 45–56. Springer Verlag, Tokyo.

Thurstone, L.L. (1935). *The Vectors of Mind: Multiple Factor Analysis for the Isolation of Primary Traits*. University of Chicago Press. doi:10.1037/10018-000.

Thurstone, L.L. (1947). *Multiple Factor Analysis*. University of Chicago Press.

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. In *Journal of the Royal Statistical Society. Series B (Methodological)*, 58 (1): 267–288. URL http://www.jstor.org/stable/2346178.

Trendafilov, N.T. (2014). From simple structure to sparse components: a review. In *Computational Statistics*, 29 (3-4): 431–454. doi:10.1007/s00180-013-0434-5.

Trendafilov, N.T., Fontanella, S., and Adachi, K. (2017). Sparse exploratory factor analysis. In *Psychometrika*, 82 (3): 778–794. doi:10.1007/s11336-017-9575-8.
Vines, S. (2000). Simple principal components. In *Journal of the Royal Statistical Society: Series C (Applied Statistics)*, 49 (4): 441–451. doi:10.1111/1467-9876.00204.

Witten, D.M., Tibshirani, R., and Hastie, T. (2009). A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. In *Biostatistics*, 10 (3): 515–534. doi:10.1093/biostatistics/kxp008.

Zou, H., Hastie, T., and Tibshirani, R. (2006). Sparse principal component analysis. In *Journal of Computational and Graphical Statistics*, 15 (2): 265–286. doi:10.1198/106186006X113430.
# Appendix

## A. Notations

Matrices are denoted in upper case bold letters, vectors are denoted in lowercase bold letters, and their elements are denoted in lowercase italic letters (note that, by default, vectors are column vectors). Matrices, vectors and elements from the same matrix all use the same letter (e.g., **A**, **a**, **a**). The transpose operation is denoted by the superscript T, the inverse operation is denoted by $^{-1}$. The identity matrix is denoted **I**, vectors or matrices of ones are denoted **1**, matrices or vectors of zeros are denoted **0** (by default, **I**, **0**), and **1** are conformable with the other terms in a formula). The standard product between two matrices is indicated by juxtaposition (i.e., **XY** means **X** times **Y**); the Hadamard product (i.e., element-wise) is denoted by $\odot$ (e.g., **X** $\odot$ **Y**), note that the Hadamard product is defined only between matrices with the same dimensions.

When provided with a square matrix, the diag operator gives a vector that contains the diagonal elements of this matrix. When provided with a vector, the diag operator gives a diagonal matrix with the elements of the vector as the diagonal elements of this matrix. A diagonal matrix is denoted **D**, and the subscript denotes the vector that stores the diagonal elements, for example, $\mathbf{D}_a = \text{diag}(\mathbf{a})$. When provided with a square matrix, the trace operator gives the sum of the diagonal elements of this matrix. For an **I** by **J** matrix **X** and for **M** being a **J** by **J** symmetric positive definite matrix, the squared **M**-norm of **X** is denoted $||\mathbf{X}||_M^2$ and is computed as:

$$ ||\mathbf{X}||_M^2 = \text{trace}(\mathbf{X}\mathbf{M}\mathbf{X}^T). \qquad (16) $$

When **M** is the identity matrix, the **M**-norm is equal to the square root of the sum of squares of the entries of the matrix and is called the Frobenius norm denoted $L_2 = ||\mathbf{X}||_2^2$. Another useful norm is the sum of the absolute values of the matrix called the $L_1$ norm.

When describing an optimization problem, the operator $\arg\min_{\mathbf{x}} f(\mathbf{x})$ searches for the value of $\mathbf{x}$ that minimizes the function $f(\mathbf{x})$, and the operator $\arg\max_{\mathbf{x}} f(\mathbf{x})$ searches for the value of $\mathbf{x}$ that maximizes the function $f(\mathbf{x})$.

## B. The Plain and Generalized Singular Value Decompositions

The singular value decomposition (SVD) and its extension—the generalized singular value decomposition (GSVD, for details on the generalized singular value
decomposition see Abdi 2007; Greenacre 1984; Takane 2002)—are the foundations of most contemporary multivariate statistical approaches.

## B.1. The (Plain) Singular Value Decomposition

The SVD of an $I \times J$ matrix $X$ solves the following maximization problem (Eckart and Young, 1936): Find a matrix, denoted $\hat{X}_L$, of rank $L$ [with $L < \min(I,J)$] equal to

$$ \hat{\mathbf{X}}_L = \sum_{l=1}^{L} \delta_l \mathbf{u}_l \mathbf{v}_l^\top = \mathbf{U}_L \mathbf{\Delta}_L \mathbf{V}_L^\top \text{ with } \mathbf{U}_L^\top \mathbf{U}_L = \mathbf{V}_L^\top \mathbf{V}_L = \mathbf{I} \text{ and } \mathbf{\Delta}_L = \text{diag}(\delta_L) \quad (17) $$

where $U$ is the $I \times L$ matrix containing the left singular vectors, $V$ is the $J \times L$ matrix containing the right singular vectors, and $\Delta$ the $L \times L$ diagonal matrix containing the singular values $\delta_1 \ge \dots \ge \delta_L \ge 0$, and such that $\hat{X}_L$ is the $L$ rank matrix closest to $X$. Specifically, $\hat{X}_L$ solves the following minimization problem:

$$ \arg\min_{\mathbf{U}_L, \mathbf{\Delta}_L, \mathbf{V}_L} \|\mathbf{X} - \hat{\mathbf{X}}_L\|_2^2 = \arg\min_{\mathbf{U}_L, \mathbf{\Delta}_L, \mathbf{V}_L} \|\mathbf{X} - \mathbf{U}_L \mathbf{\Delta}_L \mathbf{V}_L^\top\|_2^2 \quad \text{with} \quad \mathbf{U}_L^\top \mathbf{U}_L = \mathbf{V}_L^\top \mathbf{V}_L = \mathbf{I}, \quad (18) $$

When $L$ is equal to the rank of $X$, the SVD of $X$ is called the *complete SVD* (when unspecified, the SVD is the complete SVD), in this case, matrices $U$ and $V$ are written without their $L$ index. When $L$ is smaller than the rank of $X$, its SVD is called the *truncated SVD* of $X$.

The SVD of a matrix can be computed by first computing its rank one approximation [i.e., the singular triplet $(\delta_1, \mathbf{u}_1, \mathbf{v}_1)$] and then subtracting this rank one approximation from $X$—a procedure called a *deflation*. The first singular triplet of the deflated matrix $X$ is obtained then the second singular triplet of $X$. These procedure can then be continued till completion of the SVD of $X$.

## B.2. Generalized Singular Value Decomposition

The generalized SVD (GSVD), differs from the plain SVD by incorporating different orthogonality constraints on the singular vectors. Specifically, with $M$ being an $I \times I$ positive definite matrix (called the row metric matrix) and $W$ a $J \times J$ positive definite matrix (called the column metric matrix); the GSVD of $X$ solves the following problem (compare with Equation 18): Specifically, $\hat{X}_L$ solves

$$ \arg\min_{\mathbf{P}_L, \mathbf{\Delta}_L, \mathbf{Q}_L} \|\mathbf{X} - \hat{\mathbf{X}}_L\|_2^2 = \arg\min_{\mathbf{P}_L, \mathbf{\Delta}_L, \mathbf{Q}_L} \|\mathbf{X} - \mathbf{P}_L \mathbf{\Delta}_L \mathbf{Q}_L^\top\|_2^2 \quad (19) $$
with

$$ \mathbf{P}_L^\mathrm{T} \mathbf{M} \mathbf{P}_L = \mathbf{Q}_L^\mathrm{T} \mathbf{W} \mathbf{Q}_L = \mathbf{I}, \quad \Delta_L = \mathrm{diag}(\delta_L). \qquad (20) $$

where $\mathbf{P}_L$ is the $I \times L$ matrix containing the generalized left singular vectors, $\mathbf{Q}_L$ is the $J \times L$ matrix containing the generalized right singular vectors, and $\Delta_L$ is the diagonal matrix of the generalized singular values.

Similarly to the plain SVD, the optimal rank-$L$ approximation of $\mathbf{X}$ is obtained by $\tilde{\mathbf{X}}_L$ (i.e., the $L$-truncated GSVD of $\mathbf{X}$) as:

$$ \tilde{\mathbf{X}}_L = \sum_{\ell=1}^{L} \delta_{\ell} \mathbf{p}_{\ell} \mathbf{q}_{\ell}^{\mathrm{T}} = \mathbf{P}_L \Delta_L \mathbf{Q}_L^{\mathrm{T}}. \qquad (21) $$

### B.3. Generalized SVD from Plain SVD

In practice, the GSVD matrix $\mathbf{X}$ can be obtained from a plain SVD of a matrix denoted $\tilde{\mathbf{X}}$ obtained by first pre- and post-multiplying $\mathbf{X}$ by the square root of the row and column metric matrices:

$$ \tilde{\mathbf{X}} = \mathbf{M}^{\frac{1}{2}} \mathbf{X} \mathbf{W}^{\frac{1}{2}}. \qquad (22) $$

Matrix $\tilde{\mathbf{X}}$ is then decomposed with a plain SVD as:

$$ \tilde{\mathbf{X}} = \mathbf{U} \Delta \mathbf{V}^{\mathrm{T}} \quad \text{such that} \quad \mathbf{U}^{\mathrm{T}} \mathbf{U} = \mathbf{V}^{\mathrm{T}} \mathbf{V} = \mathbf{I}. \qquad (23) $$

The generalized singular vectors of $\mathbf{X}$ are then obtained from the (plain) singular vectors of $\tilde{\mathbf{X}}$ as

$$ \mathbf{P} = \mathbf{M}^{-\frac{1}{2}} \mathbf{U} \quad \text{and} \quad \mathbf{Q} = \mathbf{W}^{-\frac{1}{2}} \mathbf{V}. \qquad (24) $$

The constraints from Equations 18 and 19 are equivalent because

$$ \begin{aligned} \mathbf{P}^{\mathrm{T}} \mathbf{M} \mathbf{P} &= \mathbf{U}^{\mathrm{T}} \mathbf{M}^{-\frac{1}{2}} \mathbf{M} \mathbf{M}^{-\frac{1}{2}} \mathbf{U} \\ &= \mathbf{U}^{\mathrm{T}} \mathbf{U} \\ &= \mathbf{I}, \end{aligned} \quad \text{and} \quad \begin{aligned} \mathbf{Q}^{\mathrm{T}} \mathbf{W} \mathbf{Q} &= \mathbf{V}^{\mathrm{T}} \mathbf{W}^{-\frac{1}{2}} \mathbf{W} \mathbf{W}^{-\frac{1}{2}} \mathbf{V} \\ &= \mathbf{V}^{\mathrm{T}} \mathbf{V} \\ &= \mathbf{I}. \end{aligned} \qquad (25) $$

Finally, the decomposition of $\mathbf{X}$ from Equations 18 and 19 are also equivalent because

$$ \mathbf{P} \Delta \mathbf{Q}^{\mathrm{T}} = \mathbf{M}^{-\frac{1}{2}} \mathbf{U} \Delta \mathbf{V}^{\mathrm{T}} \mathbf{W}^{-\frac{1}{2}} = \mathbf{M}^{-\frac{1}{2}} \mathbf{M}^{\frac{1}{2}} \mathbf{X} \mathbf{W}^{\frac{1}{2}} \mathbf{W}^{-\frac{1}{2}} = \mathbf{X}. \qquad (26) $$
C. Plain Correspondence Analysis

Just like most multivariate methods, CA can be interpreted as an optimization problem (actually, as several equivalent optimization problems). But, in order to sparsify CA, we will need to add more constraints to its standard GSVD optimization problem. These new constraints can, in some cases, conflict with the original optimization problem and therefore, as a trade-off, some of the essential properties of CA could be relaxed or even lost. To facilitate the evaluation of this trade-off, we list below the relevant basic equations for CA along with its essential properties (for more details see, e.g., Abdi and Béra, 2018; Abdi and Williams, 2010; Beh and Lombardo, 2021; Greenacre, 1984; Lebart et al. 1984; or Saporta and Niang-Keita, 2006).

C.1. The Basic Equations of Correspondence Analysis

Correspondence analysis was originally developed to analyze the pattern of
deviations from independence (as measured by a χ² statistic) in a contingency
table (see Abdi and Béra, 2018). CA provides, for both rows and columns, a set of
factor scores whose total inertia is proportional to the independence χ² computed
on the original contingency table.

The contingency table to be analyzed is stored in an *I* rows by *J* columns matrix denoted **X**, whose generic element $x_{i,j}$ gives the number of observations that belongs to the *i*th level of the first nominal variable (i.e., the rows) and the *j*th level of the second nominal variable (i.e., the columns). The grand total of the table is denoted *N*.

The matrix **X** is first transformed into a probability matrix (i.e., a matrix comprising non-negative numbers and whose sum is equal to one) denoted **Z** and computed as **Z** = **N**⁻¹**X**. We denote: **r** the *I* by 1 vector of the row totals of **Z** and by $r_i$ the *i*th element of **r** (i.e., **r** = **Z**1, with **1** being a conformable vector of 1's); **c** the *J* by 1 vector of the columns totals, by $c_j$ the *j*th element of **c** (i.e., **c** = **Z**T**1**); and **D**_c = diag(**c**), **D**_r = diag(**r**) the diagonal matrices obtained from (respectively) **r** and **c**; these two diagonal matrices are called (respectively) row and column *mass* matrices. We denote by **R** = **D**_r⁻¹**Z** (respectively **C** = **D**_c⁻¹**Z**T) the row (respectively column) profile matrix (i.e., all elements are not negative, rows of **R** and columns of **C** sum to 1).

The factor scores are obtained from the following generalized singular value
decomposition (cf. Equation 19) where the metric matrices $\mathbf{D}_r^{-1}$ and $\mathbf{D}_c^{-1}$ are
called $\chi^2$-metric matrices (Greenacre, 2010)

$$ Z - rc^T = P\Delta Q^T \quad \text{with} \quad P^T D_r^{-1} P = Q^T D_c^{-1} Q = I. \qquad (27) $$

Here, by subtracting the (rank one) matrix $rc^T$ from the probability matrix $Z$, the decomposed matrix: $(Z - rc^T)$ is *double-centered* because now all rows and columns have zero means. In addition, this double centering of matrix $Z - rc^T$ propagates to the singular vectors.

For example, to show that matrix $Q$ has zero mean, we compute the column means of $Z - rc^T$ and replace it by its GSVD from Equation 27 to get

$$ (Z - rc^T) 1 = 0 = P\Delta Q^T 1 \implies P^T D_r^{-1} P\Delta Q^T 1 = 0 \implies \Delta Q^T 1 = 0 \implies Q^T 1 = 0, \qquad (28) $$

where $1$ is a $J$ by $1$ vector of $1$s.

The squared singular values are called *eigenvalues* (denoted $\lambda_k$) and are stored into the diagonal matrix $\Lambda$. The sum of the eigenvalues gives the total inertia (denoted $\mathcal{I}$ or $\phi^2$ and equal to $\chi^2/N$) of $(Z - rc^T)$. With the so-called “triplet notation,” (Dray, 2014; Escoufier, 2006; Holmes, 2008)—sometimes used as a general framework to formalize multivariate techniques—CA is equivalent to the analysis of the triplet $(Z - rc^T, D_c^{-1}, D_r^{-1})$. From this GSVD, the principal row and (respectively) column factor scores are obtained as

$$ F = D_r^{-1} P \Delta \quad \text{and} \quad G = D_c^{-1} Q \Delta. \qquad (29) $$

Note that the inertia of each dimension (i.e., each column of $F$ and $G$) is equal to its eigenvalue and that factor scores corresponding to different eigenvalues are orthogonal (under the constraints imposed by their masses). Specifically:

$$ \lambda_l = \sum_{i=1}^{I} r_i f_{i,l}^2 = \sum_{j=1}^{J} c_j g_{j,l}^2 \quad \text{and} \quad \sum_{i=1}^{I} r_i f_{i,l} f_{i,l'} = \sum_{j=1}^{J} c_j g_{j,l} g_{j,l'} = 0 \quad \forall l \neq l' \qquad (30) $$

or, in matrix notations:

$$ F^T D_r F = \Lambda \quad \text{and} \quad G^T D_c G = \Lambda, \qquad (31) $$

where $D_r$ and $D_c$ are called (respectively) the row and column *mass* matrices. This equality can be directly derived from Equations 27 and 29 (here illustrated for $F$):

$$ F^T D_r F = \Delta P^T D_r^{-1} D_r D_r^{-1} P \Delta = \Delta P^T D_r^{-1} P \Delta = \Delta^2 = \Lambda. \qquad (32) $$

By contrast with the *principal factor scores* whose $D_r$ and $D_c$ norms are equal to the singular values, the *standard factor scores* (indicated by a superscript *) have $D_r$ and $D_c$ norms equal to one, and are computed as

$$ F^* = D_r^{-1} P \quad \text{and} \quad G^* = D_c^{-1} Q. \qquad (33) $$
## C.2. Correspondence Analysis from a Plain SVD

Correspondence analysis can also be obtained from both the plain SVD and the GSVD (for details, see, e.g., Abdi 2007, and Beaton 2020). Specifically, generalized singular vectors and values and factor scores can be obtained by the following plain SVD:

$$ \tilde{Z} = D_r^{-\frac{1}{2}} (Z - rc^T) D_c^{-\frac{1}{2}} = U \Delta V^T \quad (34) $$

which, in turn, gives the generalized singular vectors as

$$ P = D_r^{\frac{1}{2}} U \quad \text{and} \quad Q = D_c^{\frac{1}{2}} V. \quad (35) $$

Finally, transposing this last equation in Equation 29 gives:

$$ F = D_r^{-1} P \Delta = D_r^{-\frac{1}{2}} U \Delta \quad \text{and} \quad G = D_c^{-1} Q \Delta = D_c^{-\frac{1}{2}} V \Delta. \quad (36) $$

As indicated by Equation 30, the inertia of a dimension is the sum of the inertia of either all the rows or all the columns, therefore a convenient way of evaluating the importance of a row (respectively a column) is to compute the proportion accounted by a given row (respectively column) into this total. This index, called the *contribution* of a row (respectively a column) is denoted $t_{i,\ell}$ (respectively $t_{j,\ell}$ for a column) and is computed as

$$ t_{i,\ell} = \frac{r_i f_{i,\ell}^2}{\sum_{i'=1}^{I} r_{i'} f_{i',\ell}^2} = r_i f_{i,\ell}^2 \lambda_{\ell}^{-1} \quad \text{and} \quad t_{j,\ell} = \frac{c_j g_{j,\ell}^2}{\sum_{j'=1}^{J} c_{j'} g_{j',\ell}^2} = c_j g_{j,\ell}^2 \lambda_{\ell}^{-1} \quad (37) $$

In matrix notations, the row (respectively columns) contributions are stored in the matrix $T_I$ (respectively $T_J$) computed as:

$$ T_I = D_r (F \odot F)^{-1} = D_r^{-1} (P \odot P) \quad \text{and} \quad T_J = D_c (G \odot G)^{-1} = D_c^{-1} (Q \odot Q). \quad (38) $$

Note that contributions can be obtained in two equivalent ways: from the factor scores or from the generalized singular vectors.

To facilitate the interpretation of a given dimension, to interpret a dimension we, traditionally, use only the items whose contribution is larger than their mass (i.e., $r_i$ or $c_j$). The contributions are also often plotted according to the sign of their corresponding factor scores and are then called *signed* contributions.
### C.3. Important Properties of Correspondence Analysis

In this section we list some important properties of correspondence analysis relevant for sparsification.

#### C.3.1. Inertia and $\chi^2$

The inertia (i.e., $\mathcal{I}$ or equivalently $\phi^2$) of the centered matrix $(\mathbf{Z} - \mathbf{rc}^T)$—as obtained from Equation 27—is equal to the independence $\chi^2$ divided by $N$. Recall that, with the present notations, $\chi^2/N = \phi^2$ is computed as

$$ \phi^2 = \sum_{i=1}^{I} \sum_{j=1}^{J} \frac{(z_{i,j} - r_i c_j)^2}{r_i c_j} = \text{trace} \left( \mathbf{D}_c^{-\frac{1}{2}} (\mathbf{Z} - \mathbf{rc}^T)^T \mathbf{D}_r^{-1} (\mathbf{Z} - \mathbf{rc}^T) \mathbf{D}_c^{-\frac{1}{2}} \right). \quad (39) $$

To show that $\phi^2$ is equal to the sum of the eigenvalues from Equation 27, suffice to plug the singular values decomposition from Equation 27 into Equation 39 to get:

$$ \phi^2 = \text{trace} \left( \mathbf{D}_c^{-\frac{1}{2}} (\mathbf{P}\Delta\mathbf{Q}^T)^T \mathbf{D}_r^{-1} (\mathbf{P}\Delta\mathbf{Q}^T) \mathbf{D}_c^{-\frac{1}{2}} \right). \quad (40) $$

Using the properties of the trace operator and re-arranging shows that $\phi^2$ is equal to the sum of the eigenvalues of $\mathbf{Z} - \mathbf{rc}^T$, namely that:

$$ \phi^2 = \text{trace} (\Delta \mathbf{P}^T \mathbf{D}_c^{-1} \mathbf{P} \Delta \mathbf{Q}^T \mathbf{D}_c^{-1} \mathbf{Q}) = \text{trace} (\Delta^2) = \text{trace} (\Lambda) = \sum_{\ell=1}^{L} \lambda_{\ell}. \quad (41) $$

which shows, as stated, that $\phi^2 = \sum \lambda_{\ell}$.

#### C.3.2. Factors are Centered

The centering of the singular vectors propagates to the factor scores when the means are computed using the masses stored in the diagonal matrices $\mathbf{D}_r$ (for the rows) and $\mathbf{D}_c$ (for the columns). So, $\bar{\mathbf{F}}$ (respectively $\bar{\mathbf{G}}$), denoting the average row (respectively column) factor scores, is computed as

$$ \bar{\mathbf{F}} = \mathbf{1}\mathbf{D}_r\bar{\mathbf{F}} = \mathbf{1}\mathbf{D}_r\mathbf{D}_r^{-1}\mathbf{P}\Delta = \mathbf{0} \quad \text{and} \quad \bar{\mathbf{G}} = \mathbf{1}\mathbf{D}_c\bar{\mathbf{G}} = \mathbf{1}\mathbf{D}_c\mathbf{D}_c^{-1}\mathbf{Q}\Delta = \mathbf{0} \quad (42) $$

(where $\mathbf{1}$ and $\mathbf{0}$ are conformable vectors of 1s and 0s).
### C.3.3. Transition Formulas: from Row to Column Factor Scores and Back

In CA the factor scores of one set (e.g., the rows) can be obtained from the profiles of this set and the factor scores of the other set (e.g., the columns). Specifically we have

$$F = D_r^{-1} P \Delta = R G \Delta^{-1} \quad \text{and} \quad G = D_c^{-1} Q \Delta = C F \Delta^{-1}. \quad (43)$$

These formulas called transition formulas are obtained from Equations 27 and 29; for example, the transition formula for the row factor scores (i.e., from the column factor scores) is derived as (taking into account that $Q$ is centered)

$$ \begin{align} F &= D_r^{-1} P \Delta = D_r^{-1} (Z - r c^T) D_c^{-1} Q \quad [\text{because } P \Delta = (Z - r c^T) D_c^{-1} Q] \nonumber \\ &= D_r^{-1} Z D_c^{-1} Q - D_r^{-1} r c^T D_c^{-1} Q \nonumber \\ &= D_r^{-1} Z D_c^{-1} Q \quad [\text{because } c^T D_c^{-1} Q = I^T Q = 0] \nonumber \\ &= D_r^{-1} Z D_c^{-1} D_c G \Delta^{-1} \quad [\text{because } Q = D_c G \Delta^{-1}] \nonumber \\ &= D_r^{-1} Z G \Delta^{-1} \nonumber \\ &= R G \Delta^{-1}. \tag{44} \end{align} $$

Note that, together, the two transition formulas imply that the eigenvalues in CA cannot be larger than 1.

The transition formulas can be interpreted as a two step process. Using the formula above (for computing $F$ from $G$) The first step corresponds to the term $RG$ and computes the row factor scores as the weighted average (i.e., the barycenter) of the column factor scores; the second step corresponds to the term $\Delta^{-1}$ and is an expansion that is inversely proportional to the singular value of each factor (this is an expansion because the singular values being no larger than 1, their inverse is no smaller than 1).

### C.3.4. Correspondence Analysis as a Double Principal Component Analysis

The row and column factor scores of CA can also be obtained from two different GSVD (or equivalently two weighted PCA), one performed on the row profiles (i.e., the matrix $R$) and the other one on the column profiles (i.e., the matrix $C$).

This way, the factor scores are obtained from the GSVD of the matrix of the row profiles matrix (i.e., $R$) as:

$$D_r^{-1} (Z - r c^T) = (R - I c^T) = P_R \Delta Q^T \quad \text{with} \quad P_R^T D_r P_R = Q^T D_c^{-1} Q = I \quad (45)$$
where $\mathbf{P_R}$ contains the left generalized singular vectors of the row profile matrix $\mathbf{R}$. We can link the decomposition of the row profile matrix to the centered data as:

$$ \mathbf{P_R} = \mathbf{D_r^{-1}P}, \quad \mathbf{F} = \mathbf{P_R\Delta} = \mathbf{D_r^{-1}P\Delta}, \quad \text{and} \quad \mathbf{G} = \mathbf{D_c^{-1}Q\Delta}; \quad (46) $$

But these factor scores can also be obtained from the GSVD of the matrix of the column profiles (i.e., matrix $\mathbf{C}$) as:

$$ (\mathbf{Z} - \mathbf{r}\mathbf{c}^{\mathrm{T}}) \mathbf{D_c^{-1}} = (\mathbf{C} - \mathbf{1}\mathbf{c}^{\mathrm{T}}) = \mathbf{P}\mathbf{\Delta}\mathbf{Q_c^{\mathrm{T}}} \text{ with } \mathbf{P}^{\mathrm{T}}\mathbf{D_r^{-1}P} = \mathbf{Q_c^{\mathrm{T}}D_cQ_c} = \mathbf{I} \quad (47) $$

where $\mathbf{Q_c}$ contains the right generalized singular vectors of the column profile matrix $\mathbf{C}$. We can link the decomposition of the column profile matrix to the centered data as:

$$ \mathbf{F} = \mathbf{D_c^{-1}P\Delta}, \quad \mathbf{Q_c} = \mathbf{D_c^{-1}Q} \quad \text{and} \quad \mathbf{G} = \mathbf{Q_c\Delta} = \mathbf{D_c^{-1}Q\Delta}. \quad (48) $$

Within the framework of generalized PCA, Equations 44, 46, and 48 show, together, that the matrices of the principal factor scores can be obtained as linear combinations of the row (respectively column) profile matrix as (respectively):

$$ \mathbf{F} = \mathbf{R}\mathbf{D_c^{-1}Q} \quad \text{and} \quad \mathbf{G} = \mathbf{C}^{\mathrm{T}}\mathbf{D_r^{-1}P}. \quad (49) $$

In this framework, the matrix $\mathbf{D_c^{-1}Q}$ (respectively $\mathbf{D_r^{-1}P}$) that stores the coefficients of the linear combinations of the columns of $\mathbf{R}$ (respectively $\mathbf{C}^{\mathrm{T}}$) is called the matrix of the *row-weights* (respectively *column-weights*). Note that the weight matrix for one set (e.g., matrix $\mathbf{D_c^{-1}Q}$ for the rows) is the matrix of the standard coordinates for the other set (i.e., $\mathbf{G}^* = \mathbf{D_c^{-1}Q}$, see Equation 33).