<figure><img src="image_1.png" /></figure>

HAL
open science

# A comparison of approximations for perturbation study of covariance matrices with application to principal component analysis

Jacques Bénasséni, Alain Mom

► To cite this version:

Jacques Bénasséni, Alain Mom. A comparison of approximations for perturbation study of covariance matrices with application to principal component analysis. 2022. hal-02083181v2

HAL Id: hal-02083181

https://hal.science/hal-02083181v2

Preprint submitted on 2 Sep 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# A comparison of approximations for perturbation study of covariance matrices with application to principal component analysis

Jacques Bénasséni and Alain Mom

Univ Rennes, CNRS, IRMAR - UMR 6625
F-35000 Rennes, France

## Abstract

Covariance matrices play a central role in a wide range of multivariate statistical methods including for example principal component analysis which is a well known dimensionality reduction technique. Therefore, a large amount of work has been devoted to analyzing the sensitivity of their eigenstructure to influential observations. In order to evaluate the effect of deleting one or a small subset of observations, several approximations to the eigenelements of the perturbed matrix have been proposed. This paper provides a theoretical and numerical comparison of the main approximations. A special emphasis is given to those based on Rayleigh quotients which are seldom used. A general approach, using new inequalities in the study of covariance matrices, is proposed in order to evaluate their accuracy without having to recompute the exact perturbed eigenvalues and eigenvectors. This approach is of specific interest from a computational standpoint. Theoretical developments are illustrated with a numerical study which emphasizes the accuracy of approximations based on Rayleigh quotients.

**Keywords.** approximation; eigenvalue and eigenvector; covariance matrix; perturbation; Rayleigh quotients.

Email addresses : corresponding author jacques.benasseni@univ-rennes2.fr, alain.mom@univ-rennes2.fr
# 1 Introduction

We consider a $n \times p$ data matrix $\mathbf{X}$ in which the $n$ rows are the observation vectors $\mathbf{x}_i^t \in \mathbb{R}^p$, $i = 1, \dots, n$. Letting $\bar{\mathbf{x}} = (1/n) \sum_{i=1}^n \mathbf{x}_i$ denote the mean vector of dimension $p$, the covariance matrix $\mathbf{S} = (1/n) \sum_{i=1}^n (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^t$ is involved in a wide range of statistical methods including principal component analysis or multiple regression for example. Since $\mathbf{S}$ is known to be highly prone to influential observations, sensitivity aspects for principal component analysis have been discussed in several papers including [1], [2], [3], [4], [5], [6] among many others. In order to assess the influence of a small subset $I$ of $r$ observations on the eigenstructure of $\mathbf{S}$, a possible approach consists of studying the effect of removing these $r$ observations on the eigenelements of $\mathbf{S}$. If we note a significant modification of the eigenelements, we know that these observations are strongly influential on the results. In this framework, letting $\tilde{\mathbf{S}}$ denote the covariance matrix obtained without the subset of observations indexed by $I$, several authors have studied the relationship between the eigenelements of $\mathbf{S}$ and those of $\tilde{\mathbf{S}}$. More specifically, providing approximations to the eigenelements of $\tilde{\mathbf{S}}$ allows to detect influential subsets of observations without having to recompute the exact modified eigenvalues and eigenvectors. Hadi & Nyquist [7] and Wang & Nyquist [8] study the effect of deleting a single observation while Wang & Liski [9], Enguix-González et al. [10] and Bénasséni [11] focus on the general case where $I$ comprises several observations. In these works, approximations to the eigenvalues and eigenvectors are obtained by retaining the first terms in power expansions of these parameters. Independently of these works, Bénasséni [12] suggests using approximations based on Rayleigh quotients together with inequalities given in [13]. Finally, it should be noted that approximations for the covariance matrix are still the subject of an ongoing research in a wider framework as emphasized by works published in the past decade including for example [14] which considers the moments of the eigenelements or [15] dealing with the conditional biais of eigenvalues.

The contribution of this work is twofold. First, since approximations based on power ex-
pansions have never been compared with those based on Rayleigh quotients, some elements of comparison are provided. This comparison is based on some simple theoretical relations together with the numerical study of a data table which is intended to provide some guidance on the approximations to choose in practice. Second, inequalities recently given in [16] are introduced in order to evaluate the accuracy of approximations without having to recompute the eigenelements of $\tilde{S}$. It is proved that these inequalities always provide a sharper evaluation than those used in [12]. This is a central result from a computational standpoint since it is of crucial importance to have, as far as possible, the more precise evaluation of the approximations without having to recompute the perturbed eigenelements.

The paper is organized as follows. First, a brief review of approximations based on power expansions is provided in the next section in order to acquaint the reader with the field of research. Section 3 is devoted to approximations based on Rayleigh quotients.

The main contribution of the paper can be found in Section 4 which introduces new inequalities in the study of covariance matrices in order to provide an improved evaluation of approximations. A numerical illustration of the results, based on the soil composition data in [17], can be found in Section 5 in the framework of principal component analysis while Section 6 is devoted to some practical conclusions.

In this work, the eigenvalues $\lambda_1 > \lambda_2 > \dots > \lambda_p \ge 0$ of $S$ are assumed simple and associated to the normalized eigenvectors $\phi_1, \phi_2, \dots, \phi_p$. In the same way, the eigenvalues $\tilde{\lambda}_1 > \tilde{\lambda}_2 > \dots > \tilde{\lambda}_p \ge 0$ of the perturbed matrix $\tilde{S}$ are also assumed simple and associated to the eigenvectors $\tilde{\phi}_1, \tilde{\phi}_2, \dots, \tilde{\phi}_p$.

# 2 Approximations based on power expansions

## 2.1 Theoretical background on matrix perturbations

Referring to [10] or [11] and letting $\bar{x}_I = (1/r) \sum_{i \in I} x_i$, we know that, when a subset of observations indexed by $I$ is deleted, the covariance matrix $S$ is transformed to $\tilde{S}$ which can be expressed as:

$$ \tilde{S} = S + \left(\frac{r}{n-r}\right) \left[ S - \frac{1}{r} \sum_{i \in I} (x_i - \bar{x})(x_i - \bar{x})^t \right] + \left(\frac{r}{n-r}\right)^2 \left[ -(\bar{x}_I - \bar{x})(\bar{x}_I - \bar{x})^t \right] \quad (1) $$

We then have a perturbation of the form $\tilde{S} = S + \epsilon M + \epsilon^2 N$ with $\epsilon = \frac{r}{n-r}$, $M = S$
(1/r) $\sum_{i \in I} (\mathbf{x}_i - \bar{\mathbf{x}}) (\mathbf{x}_i - \bar{\mathbf{x}})^t$ and $\mathbf{N} = -(\bar{\mathbf{x}}_I - \bar{\mathbf{x}}) (\bar{\mathbf{x}}_I - \bar{\mathbf{x}})^t$. Following matrix perturbation theory detailed in [13] for example or referring to [18], we know that, if $\epsilon$ is sufficiently small, for each simple eigenvalue $\lambda$ of $\mathbf{S}$ there is an eigenvalue $\tilde{\lambda}$ of $\tilde{\mathbf{S}}$ given by a convergent power series:

$$ \tilde{\lambda} = \lambda + \gamma_1 \epsilon + \gamma_2 \epsilon^2 + \dots + \gamma_m \epsilon^m + O(\epsilon^{m+1}) \quad (2) $$

with a corresponding eigenvector which can also be expressed under a convergent power series:

$$ \tilde{\phi} = \phi + \psi_1 \epsilon + \psi_2 \epsilon^2 + \dots + \psi_m \epsilon^m + O(\epsilon^{m+1}) \quad (3) $$

The parameters $\gamma_1, \gamma_2, \dots, \gamma_m$ and $\psi_1, \psi_2, \dots, \psi_m$ are derived by equating the coefficients of $\epsilon$, $\epsilon^2, \dots, \epsilon^m$ in the equation $\tilde{\mathbf{S}}\tilde{\phi} = \tilde{\lambda}\tilde{\phi}$. It should also be noted that the perturbation $\tilde{\lambda}_k$ of $\lambda_k$ may not necessarily be the kth largest eigenvalue of $\tilde{\mathbf{S}}$ if the subset of observations indexed by $I$ has initially a strong influence on $\lambda_k$ for example. In this case, we assume simply that the eigenvalues $\tilde{\lambda}_k$ have been reordered in decreasing order and that their corresponding eigenvectors $\tilde{\phi}_k$ have been relabeled.

## 2.2 Formulae for the approximations derived from power expansions

For any integer $m \ge 1$, retaining only terms of order lower or equal to $m$ in $\epsilon$ in (2) and (3) provides the approximations $\tilde{\lambda}^{(m)}$ and $\tilde{\phi}^{(m)}$ of order $m$ for the eigenvalues and eigenvectors of $\tilde{\mathbf{S}}$.

This is the general approach suggested in [9], [10] and [11]. Assuming that $\epsilon = \frac{r}{n-r}$ is sufficiently small to ensure the convergence of the above power series, Enguix-González et al. [10] provide the following approximations for $k = 1, \dots, p$:

$$ \tilde{\lambda}_k^{(1)} = \lambda_k + \left( \frac{r}{n-r} \right) \left( \lambda_k - \frac{1}{r} \sum_{i \in I} \alpha_{ki}^2 \right) \quad (4) $$

$$ \tilde{\lambda}_k^{(2)} = \tilde{\lambda}_k^{(1)} + \left( \frac{r}{n-r} \right)^2 \left[ -\alpha_{kI}^2 - \sum_{j \neq k} \frac{1}{\lambda_j - \lambda_k} \left( \frac{1}{r} \sum_{i \in I} \alpha_{ki} \alpha_{ji} \right)^2 \right] \quad (5) $$

$$ \tilde{\phi}_k^{(1)} = \phi_k + \left( \frac{r}{n-r} \right) \sum_{j \neq k} \left( \frac{1}{r} \sum_{i \in I} \alpha_{ki} \alpha_{ji} \right) \frac{\phi_j}{\lambda_j - \lambda_k} \quad (6) $$

where $\alpha_{ki} = \phi_k^t (\mathbf{x}_i - \bar{\mathbf{x}})$, for $i = 1, \dots, n$ and $\alpha_{kI} = (\sum_{i \in I} \alpha_{ki})/r$.
Details on the derivation of the above expressions are given in [11]. The reader is referred to
[10] for the formula of $\tilde{\phi}_k^{(2)}$ which is fairly long and therefore omitted in this paper.

In the remaining of the paper, we define also the approximations of order zero as $\tilde{\lambda}^{(0)} = \lambda$
and $\tilde{\phi}^{(0)} = \phi$ for notational convenience.

Finally, when studying the influence of a single observation, note that approximations are
simply obtained by taking $r = 1$, $I = \{i\}$ and $\epsilon = \frac{1}{n-1}$ in the previous developments. The
reader more specifically interested by this case will refer to a series of papers including [1],
[7] and [8].

# 3 Approximations based on Rayleigh quotients

## 3.1 Rayleigh quotients as approximations to the perturbed eigenvalues

Assuming that weights are given to the observations, Bénasséni [12] studies the effects of
modifying these weights on the eigenvalues and eigenvectors of the covariance matrix. Delet-
ing a small subset of observations indexed by $I$ is therefore a particular case of his ap-
proach which consists simply of modifying to zero the corresponding weights. As approx-
imations to $\tilde{\lambda}_k$ for $k = 1, \dots, p$, this author suggests using the Rayleigh quotient $q_k^{(0)} =$
$$ \left(\tilde{\phi}_k^{(0)}\right)^t \tilde{S} \tilde{\phi}_k^{(0)} / \left(\tilde{\phi}_k^{(0)}\right)^t \tilde{\phi}_k^{(0)} $$
for $\tilde{S}$ and the initial normalized eigenvector $\tilde{\phi}_k^{(0)}$, and the Rayleigh
quotient $q_k^{(1)} = \left(\tilde{\phi}_k^{(1)}\right)^t \tilde{S} \tilde{\phi}_k^{(1)} / \left(\tilde{\phi}_k^{(1)}\right)^t \tilde{\phi}_k^{(1)}$ for $\tilde{S}$ and the approximation of order one $\tilde{\phi}_k^{(1)}$ to
$\tilde{\phi}_k$.

## 3.2 Error analysis

From a computational standpoint, it is of major interest to evaluate the accuracy of ap-
proximations without having to recompute the exact eigenelements of $\tilde{S}$. In order to do this,
Bénasséni [12] suggests using inequalities provided in [13]. Focusing on $\tilde{\lambda}_k$ and its correspond-
ing eigenvector $\tilde{\phi}_k$ and, from now on, assuming without change of notation that $\tilde{\phi}_k^{(m)}$ has
been normalized, let $\eta_k^{(m)} = \| \tilde{S} \tilde{\phi}_k^{(m)} - q_k^{(m)} \tilde{\phi}_k^{(m)} \|_2$ for $m = 0, 1$ where $\| \cdot \|_2$ stands for the two
norm. Assume that $c_k^{(m)} \in \mathbb{R}^{+*}$ is a nonzero positive constant such that $|\tilde{\lambda}_j - q_k^{(m)}| > c_k^{(m)}$
for $j = 1, \dots, p$ with $j \neq k$. Then the accuracy of $q_k^{(m)}$ as approximation to $\tilde{\lambda}_k$, and of
$\tilde{\phi}_k^{(m)}$ as approximation to $\tilde{\phi}_k$, is analyzed in [12] using the following inequalities given in [13,
pp.172-176] :

$$ \|\tilde{\phi}_k - \tilde{\phi}_k^{(m)}\|_2^2 \le \left(\frac{\eta_k^{(m)}}{c_k^{(m)}}\right)^2 \left[1 + \left(\frac{\eta_k^{(m)}}{c_k^{(m)}}\right)^2\right] \quad (7) $$

and if $\eta_k^{(m)}/c_k^{(m)} < 1$:

$$ |\tilde{\lambda}_k - q_k^{(m)}| \le \left[ \frac{(\eta_k^{(m)})^2}{c_k^{(m)}} \right] / \left[ 1 - \left( \frac{\eta_k^{(m)}}{c_k^{(m)}} \right)^2 \right] \quad (8) $$

Using $\|\tilde{\phi}_k\|_2 = \|\tilde{\phi}_k^{(m)}\|_2 = 1$, note that (7), can be written with the cosine between $\tilde{\phi}_k$ and $\tilde{\phi}_k^{(m)}$ as:

$$ 1 - \left[ \left( \frac{\eta_k^{(m)}}{2} \right)^2 \left( \frac{c_k^{(m)}}{2} \right)^2 \right] \left[ 1 + \left( \frac{\eta_k^{(m)}}{c_k^{(m)}} \right)^2 \right] \le \cos(\tilde{\phi}_k, \tilde{\phi}_k^{(m)}). \quad (9) $$

In practice, it is necessary to give the parameter $c_k^{(m)}$ a value in the previous inequalities. When studying the influence of a single observation, it should be noted that (1) can be expressed as

$$ \tilde{\mathbf{S}} = \frac{n}{n-1} \left[ \mathbf{S} - \frac{1}{n-1} (\mathbf{x}_i - \bar{\mathbf{x}}) (\mathbf{x}_i - \bar{\mathbf{x}})^t \right] \quad (10) $$

so that we have a rank one perturbation. In this case, the parameter $c_k^{(m)}$ is given a value in [12] using bounds, derived from the Courant-Fischer theorem, for the eigenvalues $\tilde{\lambda}_j$ of the symmetric matrix $\tilde{\mathbf{S}}$, $j = 1, \dots, p, j \ne k$. In the case where several observations are deleted this author suggests a fairly lengthy procedure assuming that these observations are removed one after the other, so that we have a series of rank one perturbations.

# 4 New developments

## 4.1 Some relations between the approximations

In his work, Bénasséni [12] only considers the approximations of order one $\tilde{\lambda}_k^{(1)}$ for the eigenvalues of $\tilde{\mathbf{S}}$ and provides no comparison with approximations based on Rayleigh quotients. However, it is easy to derive the following simple relations.

First note that $q_k^{(0)}$ can be written as:
$$q_k^{(0)} = \lambda_k + \left(\frac{r}{n-r}\right) \left(\lambda_k - \frac{1}{r} \sum_{i \in I} \alpha_{ki}^2\right) - \left(\frac{r}{n-r}\right)^2 \alpha_{kI}^2 \quad (11)$$

using (1). Therefore we see that we have always $\tilde{\lambda}_k^{(1)} \ge q_k^{(0)}$. Furthermore, a simple comparison of (11) with (5) shows that:

$$\tilde{\lambda}_k^{(2)} - q_k^{(0)} = - \left(\frac{r}{n-r}\right)^2 \sum_{j \neq k} \frac{1}{\lambda_j - \lambda_k} \left(\frac{1}{r} \sum_{i \in I} \alpha_{ki} \alpha_{ji}\right)^2.$$

In particular, when focusing on the largest eigenvalue which plays for example a central role in PCA, this difference is non negative so that we have $\tilde{\lambda}_1^{(2)} \ge q_1^{(0)}$. In a similar way, when considering the smallest eigenvalue, we get $\tilde{\lambda}_p^{(2)} \le q_p^{(0)}$.

We omit the derivation of $q_k^{(1)}$ which is tedious and leads to a formula too complicated to be interpreted. However, it should be noted that this approximation involves terms up to the order 4 in $\epsilon = \frac{r}{n-r}$.

## 4.2 Improved inequalities in error analysis

In practice, it is of crucial importance to have bounds as close as possible to the true values of $|\tilde{\lambda}_k - q_k^{(m)}|$ and $\cos(\tilde{\phi}_k, \tilde{\phi}_k^{(m)})$ in Inequalities (8) and (9). This will allow for an evaluation as precise as possible of the real accuracy of the approximations without having to recompute the perturbed analysis. For this purpose, we introduce below new inequalities in the study of covariance matrices in order to get an improved error analysis

Indeed, Inequalities (7), (8) and (9) introduced in Subsection 3.2 can be improved using error analysis developed in [16, pp.180-184]. More precisely, it is easily derived from Corollary 4.6.4 in this reference that:

$$|\tilde{\lambda}_k - q_k^{(m)}| \le \frac{(\eta_k^{(m)})^2}{c_k^{(m)}} \quad (12)$$

and

$$\sin(\tilde{\phi}_k, \tilde{\phi}_k^{(m)}) \le \frac{\eta_k^{(m)}}{c_k^{(m)}} \quad (13)$$

under the condition:

$$\eta_k^{(m)} < c_k^{(m)}. \quad (14)$$
It is obvious that (12) is more accurate than (8). A similar remark holds for (13) which improves (9). This last point is easily checked by converting (13) into

$$cos^2(\tilde{\phi}_k, \tilde{\phi}_k^{(m)}) \geq 1 - (\eta_k^{(m)}/c_k^{(m)})^2. \quad (15)$$

Then letting $a = (\eta_k^{(m)}/c_k^{(m)})^2$, $A = 1 - a$ and $B = [1 - (a/2)(1 + a)]^2$, Inequality (9) becomes $cos^2(\tilde{\phi}_k, \tilde{\phi}_k^{(m)}) \geq B$ so that we have simply to prove that $A \geq B$. Developing $B$, we get $B = A + (a^2/4)(a^2 + 2a - 3)$. Thus we have that $A \geq B$ if and only if the polynomial $a^2 + 2a - 3$ is negative. This is the case when $a$ belongs to $[-3, 1]$. Therefore the result follows since $0 \leq a < 1$ from (14).

Furthermore, when dealing with the eigenvector associated to the largest eigenvalue, Chatelin [16, p.204] points out that Inequality (13) can also be refined into the following tangent based inequality:

$$\tan(\tilde{\phi}_1, \tilde{\phi}_1^{(m)}) \leq \frac{\eta_1^{(m)}}{c_1^{(m)}} \quad (16)$$

since this eigenvalue is assumed to be simple. More precisely, letting $\alpha$ denote the angle between the two vectors $\tilde{\phi}_1$ and $\tilde{\phi}_1^{(m)}$, we have $\alpha \leq \arctan(\eta_1^{(m)}/c_1^{(m)}) \leq \arcsin(\eta_1^{(m)}/c_1^{(m)})$ since $0 \leq \arctan x \leq \arcsin x$ for all $x \in [0, 1]$. Thus we obtain a better approximation of $\alpha$ when using the function arctan rather than the function arcsin showing that (16) improves (13).

## 4.3 Improved value for the parameter $c_k^{(m)}$

The sharpness of the bounds in Inequalities (12), (13) and (16) depends on the value of the parameter $c_k^{(m)}$. The larger $c_k^{(m)}$ is, the sharper are these inequalities. In order to get a suitable value for this parameter, some other results in [16, p.180-181] turn out to be also of specific interest since they often improve significantly the value of the parameter $c_k^{(m)}$ obtained through the Courant-Fischer theorem in [12]. More precisely, using this reference we know there is at least one eigenvalue of $\tilde{S}$ in each of the intervals defined for $j = 1, \dots, p$ by $[b_j^{(m)}, B_j^{(m)}] = [q_j^{(m)} - \eta_j^{(m)}, q_j^{(m)} + \eta_j^{(m)}]$ which are often referred to as Krylov-Weinstein intervals. When the interval $[b_k^{(m)}, B_k^{(m)}]$ is isolated from the $p-1$ other
ones, we know that it contains precisely one eigenvalue. Then, assuming that the Rayleigh quotients satisfy $q_1^{(m)} > q_2^{(m)} > \dots > q_p^{(m)}$ (after having been reordered if necessary), a value for $c_k^{(m)}$ can be easily derived as $c_k^{(m)} = \min(b_k^{(m)} - B_{k+1}^{(m)}, b_{k-1}^{(m)} - B_k^{(m)})$ if $k \in \{2, \dots, p-1\}$, $c_1^{(m)} = b_1^{(m)} - B_2^{(m)}$ if $k=1$ and $c_p^{(m)} = b_{p-1}^{(m)} - B_p^{(m)}$ if $k=p$.

It should be noted that for very close eigenvalues, giving a value to this parameter can be a real issue. However, once a value satisfying (14) is obtained, we know from (12) that, for $k=1, \dots, p$, the eigenvalues $\tilde{\lambda}_k$ lie in the intervals $[q_k^{(m)} - (\eta_k^{(m)})^2/c_k^{(m)}, q_k^{(m)} + (\eta_k^{(m)})^2/c_k^{(m)}]$ which are sharper than the Krylov-Weinstein intervals as soon as (14) holds. These new intervals can be used to obtain a larger value for the parameter $c_k^{(m)}$, thus improving (12), (13) and (16). This process could be iterated, but no significative improvement is generally observed.

Finally, it was noted in (10) that we have a rank one perturbation when deleting a single observation. Several inequalities and relations for this specific case are given in [12], [7] and [8]. The reader is also referred to more recent works [19],[20], [21] who suggest new bounds for the perturbed eigenvalues. Although Krylov-Weinstein intervals generally provide a satisfying value for the parameter $c_k^{(m)}$, these recent works can also be interesting in the determination of the largest possible constant $c_k^{(m)}$ in order to make Inequalities (12), (13) and (16) sharper.

# 5 Numerical study

The numerical illustration of the results pertains to principal component analysis. It is based on the soil composition data in [17] which have already been used in several works including, among others, [1], [3], [8], [9], [10], [11] or[15] for sensitivity study of covariance based principal component analysis. The data table consists of 20 observations measured on 4 variables. We have the following four eigenvalues for the corresponding covariance matrix: $\lambda_1 = 82.30827$, $\lambda_2 = 6.73891$, $\lambda_3 = 0.44783$, $\lambda_4 = 0.24552$. In the first subsection, we study the effect of deleting each of the 20 observations on the two largest eigenvalues (which account for more than 99% of the total variation in principal component analysis) and on their corresponding eigenvectors. In the following subsection, we study the effects of deleting subsets of two observations on the largest eigenvalue. These subsets are those considered for the numerical study in [9].
## 5.1 Approximations when deleting one observation

Table 1 provides for each subset $I = \{i\}$, the perturbed eigenvalue $\tilde{\lambda}_1$, its order one and two approximations $\tilde{\lambda}_1^{(1)}$ and $\tilde{\lambda}_1^{(2)}$, the Rayleigh quotients $q_1^{(0)}$ and $q_1^{(1)}$ and the differences between each approximation and the true perturbed eigenvalue $\tilde{\lambda}_1$. In the last two columns we find the bounds to $|\tilde{\lambda}_1 - q_1^{(0)}|$ and $|\tilde{\lambda}_1 - q_1^{(1)}|$ given by Inequality (12).

The first comment regarding the results in this table is that $\tilde{\lambda}_1^{(1)}$ is by far the less accurate approximation in all cases. In contrast $q_1^{(1)}$ always provides extremely sharp approximations since it deviates from $\tilde{\lambda}_1$ by $1.76 \times 10^{-3}$ in the worst case (when deleting observation 4) and that the error is only $2.13 \times 10^{-13}$ when deleting observation number 16. It should be noted that $\tilde{\lambda}_1^{(2)}$ also provides fairly satisfying approximations although clearly less accurate than $q_1^{(1)}$. The Rayleigh quotient $q_1^{(0)}$ is outperformed by $\tilde{\lambda}_1^{(2)}$ but remains significantly sharper than $\tilde{\lambda}_1^{(1)}$. Furthermore, it is worth pointing out that $\tilde{\lambda}_1^{(1)}$ always overestimates the perturbed eigenvalue while the other three estimations slightly underestimate it. Note also that the results agree with inequalities $\tilde{\lambda}_1^{(1)} \ge q_1^{(0)}$ and $\tilde{\lambda}_1^{(2)} \ge q_1^{(0)}$ given in Subsection 4.1.

Second, Inequality (12) provides bounds sufficiently close to $|\tilde{\lambda}_1 - q_1^{(0)}|$ and $|\tilde{\lambda}_1 - q_1^{(1)}|$ to evaluate correctly the accuracy of Rayleigh quotients as approximations to $\tilde{\lambda}_1$ without having to recompute the perturbed analysis.

Third, it is easily seen from (10) that the maximum value for the perturbed eigenvalue is obtained when $x_i = \bar{x}$ with $\tilde{\lambda}_1 = (20\lambda_1)/19 = 86.64028$. We have the highest perturbed eigenvalue when deleting observations number 3, 5, 10, 12, 14, 16, 20 which are fairly close to $\bar{x}$ and in these cases we get the sharper approximations to $\tilde{\lambda}_1$.

Focusing now on the second largest eigenvalue, Table 2 provides results similar to those of Table 1.

It turns out that $\tilde{\lambda}_2^{(1)}$ is the less accurate approximation to $\tilde{\lambda}_2$. Except when deleting observation 13, the Rayleigh quotient $q_2^{(1)}$ again provides the best approximation with a very good accuracy since in the worst case corresponding to this observation we have $\tilde{\lambda}_2 - q_2^{(1)} = 7.69 \times 10^{-4}$. For this observation, $\tilde{\lambda}_2^{(2)}$ is slightly better but less accurate in all the other cases while performing fairly well in general. The Rayleigh quotient $q_2^{(0)}$ performs in a similar way as $q_1^{(0)}$ in Table 1. It should be noted that again $\tilde{\lambda}_2^{(1)}$ always overestimates the perturbed eigenvalue but in contrast to Table 1, the other three approximations can as well slightly underestimate or overestimate $\tilde{\lambda}_2$. Another difference with Table 1 is that bounds provided
<table><caption>Table 1: Approximations and error analysis for the largest eigenvalue with $r = 1$.</caption><thead><tr><th>I</th><th>λ̃<sub>1</sub></th><th>λ̃<sub>1</sub><sup>(1)</sup></th><th>λ̃<sub>1</sub><sup>(2)</sup></th><th>q<sub>1</sub><sup>(0)</sup></th><th>q<sub>1</sub><sup>(1)</sup></th><th>λ̃<sub>1</sub> - λ̃<sub>1</sub><sup>(1)</sup></th><th>λ̃<sub>1</sub> - λ̃<sub>1</sub><sup>(2)</sup></th><th>λ̃<sub>1</sub> - q<sub>1</sub><sup>(0)</sup></th><th>λ̃<sub>1</sub> - q<sub>1</sub><sup>(1)</sup></th><th>|λ̃<sub>1</sub> - q<sub>1</sub><sup>(0)</sup>| ≤ |λ̃<sub>1</sub> - q<sub>1</sub><sup>(1)</sup>| ≤</th></tr></thead><tbody><tr><td>1</td><td>81.59976</td><td>81.8333</td><td>81.59769</td><td>81.58031</td><td>81.59969</td><td>-0.23355</td><td>0.00207</td><td>0.01945</td><td>6.87 × 10<sup>-5</sup></td><td>0.01988</td><td>6.91 × 10<sup>-5</sup></td></tr><tr><td>2</td><td>77.18681</td><td>77.64932</td><td>77.18511</td><td>77.17611</td><td>77.18667</td><td>-0.46251</td><td>0.00170</td><td>0.01070</td><td>1.42 × 10<sup>-4</sup></td><td>0.01115</td><td>1.46 × 10<sup>-4</sup></td></tr><tr><td>3</td><td>86.52309</td><td>86.52844</td><td>86.52306</td><td>86.52255</td><td>86.52309</td><td>-0.00535</td><td>2.53 × 10<sup>-5</sup></td><td>5.38 × 10<sup>-4</sup></td><td>5.20 × 10<sup>-9</sup></td><td>5.40 × 10<sup>-4</sup></td><td>5.22 × 10<sup>-9</sup></td></tr><tr><td>4</td><td>77.34975</td><td>77.64477</td><td>77.32399</td><td>77.17133</td><td>77.34799</td><td>-0.29503</td><td>0.02576</td><td>0.17842</td><td>1.76 × 10<sup>-3</sup></td><td>0.18922</td><td>1.78 × 10<sup>-3</sup></td></tr><tr><td>5</td><td>86.57878</td><td>86.58112</td><td>86.57875</td><td>86.57801</td><td>86.57878</td><td>-0.00234</td><td>2.99 × 10<sup>-5</sup></td><td>7.71 × 10<sup>-4</sup></td><td>1.06 × 10<sup>-7</sup></td><td>7.74 × 10<sup>-4</sup></td><td>1.07 × 10<sup>-7</sup></td></tr><tr><td>6</td><td>76.85039</td><td>77.33877</td><td>76.8502</td><td>76.84922</td><td>76.85037</td><td>-0.48838</td><td>1.86 × 10<sup>-4</sup></td><td>1.17 × 10<sup>-3</sup></td><td>1.53 × 10<sup>-5</sup></td><td>1.28 × 10<sup>-3</sup></td><td>1.66 × 10<sup>-5</sup></td></tr><tr><td>7</td><td>79.95562</td><td>80.27544</td><td>79.95368</td><td>79.94044</td><td>79.95552</td><td>-0.31981</td><td>0.00194</td><td>0.01518</td><td>1.02 × 10<sup>-4</sup></td><td>0.01544</td><td>1.02 × 10<sup>-4</sup></td></tr><tr><td>8</td><td>74.71505</td><td>75.29919</td><td>74.71264</td><td>74.70229</td><td>74.71477</td><td>-0.58414</td><td>0.00241</td><td>0.01276</td><td>2.72 × 10<sup>-4</sup></td><td>0.01326</td><td>2.79 × 10<sup>-4</sup></td></tr><tr><td>9</td><td>74.45165</td><td>75.05167</td><td>74.4498</td><td>74.44174</td><td>74.45144</td><td>-0.60002</td><td>0.00185</td><td>9.91 × 10<sup>-3</sup></td><td>2.04 × 10<sup>-4</sup></td><td>0.01073</td><td>2.19 × 10<sup>-4</sup></td></tr><tr><td>10</td><td>86.02449</td><td>86.05404</td><td>86.02442</td><td>86.02318</td><td>86.02449</td><td>-0.02955</td><td>7.21 × 10<sup>-5</sup></td><td>1.30 × 10<sup>-3</sup></td><td>3.99 × 10<sup>-8</sup></td><td>1.33 × 10<sup>-3</sup></td><td>4.04 × 10<sup>-8</sup></td></tr><tr><td>11</td><td>85.32593</td><td>85.38314</td><td>85.3254</td><td>85.31697</td><td>85.32593</td><td>-0.05721</td><td>5.31 × 10<sup>-4</sup></td><td>0.001782881</td><td>8.62 × 10<sup>-7</sup></td><td>9.06 × 10<sup>-3</sup></td><td>8.63 × 10<sup>-7</sup></td></tr><tr><td>12</td><td>86.58382</td><td>86.58643</td><td>86.58381</td><td>86.58359</td><td>86.58382</td><td>-0.00261</td><td>1.06 × 10<sup>-5</sup></td><td>2.26 × 10<sup>-4</sup></td><td>2.48 × 10<sup>-9</sup></td><td>2.32 × 10<sup>-4</sup></td><td>2.53 × 10<sup>-9</sup></td></tr><tr><td>13</td><td>84.13095</td><td>84.20755</td><td>84.12781</td><td>84.07951</td><td>84.13094</td><td>-0.07660</td><td>0.00314</td><td>0.05143</td><td>6.97 × 10<sup>-6</sup></td><td>0.05293</td><td>6.99 × 10<sup>-6</sup></td></tr><tr><td>14</td><td>86.52446</td><td>86.52855</td><td>86.52439</td><td>86.52267</td><td>86.52446</td><td>-0.00410</td><td>6.56 × 10<sup>-5</sup></td><td>1.78 × 10<sup>-3</sup></td><td>3.46 × 10<sup>-7</sup></td><td>1.82 × 10<sup>-3</sup></td><td>3.50 × 10<sup>-7</sup></td></tr><tr><td>15</td><td>82.72790</td><td>82.91768</td><td>82.72732</td><td>82.72175</td><td>82.72788</td><td>-0.18978</td><td>5.84 × 10<sup>-4</sup></td><td>6.15 × 10<sup>-3</sup></td><td>1.37 × 10<sup>-5</sup></td><td>6.26 × 10<sup>-3</sup></td><td>1.39 × 10<sup>-5</sup></td></tr><tr><td>16</td><td>86.63917</td><td>86.63923</td><td>86.63917</td><td>86.63917</td><td>86.63917</td><td>-5.48 × 10<sup>-5</sup></td><td>3.36 × 10<sup>-8</sup></td><td>6.79 × 10<sup>-7</sup></td><td>2.13 × 10<sup>-13</sup></td><td>7.21 × 10<sup>-7</sup></td><td>2.59 × 10<sup>-13</sup></td></tr><tr><td>17</td><td>83.27902</td><td>83.42004</td><td>83.27668</td><td>83.25055</td><td>83.27899</td><td>-0.14101</td><td>0.00234</td><td>0.02847</td><td>3.27 × 10<sup>-5</sup></td><td>0.02925</td><td>3.29 × 10<sup>-5</sup></td></tr><tr><td>18</td><td>80.80479</td><td>81.08976</td><td>80.80396</td><td>80.79763</td><td>80.80476</td><td>-0.28497</td><td>8.28 × 10<sup>-4</sup></td><td>7.16 × 10<sup>-3</sup></td><td>3.42 × 10<sup>-5</sup></td><td>7.57 × 10<sup>-3</sup></td><td>3.59 × 10<sup>-5</sup></td></tr><tr><td>19</td><td>78.33933</td><td>78.75217</td><td>78.33900</td><td>78.33701</td><td>78.33931</td><td>-0.41283</td><td>3.31 × 10<sup>-4</sup></td><td>2.33 × 10<sup>-3</sup></td><td>2.18 × 10<sup>-5</sup></td><td>2.53 × 10<sup>-3</sup></td><td>2.37 × 10<sup>-5</sup></td></tr><tr><td>20</td><td>86.37150</td><td>86.38477</td><td>86.37149</td><td>86.37133</td><td>86.37150</td><td>-0.01328</td><td>9.05 × 10<sup>-6</sup></td><td>1.73 × 10<sup>-4</sup></td><td>1.09 × 10<sup>-9</sup></td><td>1.86 × 10<sup>-4</sup></td><td>1.17 × 10<sup>-9</sup></td></tr></tbody></table>
<table><caption>Table 2: Approximations and error analysis for the second largest eigenvalue with $r = 1$.</caption><thead><tr><th>I</th><th>λ̃<sub>2</sub></th><th>λ̃<sub>2</sub><sup>(1)</sup></th><th>λ̃<sub>2</sub><sup>(2)</sup></th><th>q<sub>2</sub><sup>(0)</sup></th><th>q<sub>2</sub><sup>(1)</sup></th><th>λ̃<sub>2</sub> - λ̃<sub>2</sub><sup>(1)</sup></th><th>λ̃<sub>2</sub> - λ̃<sub>2</sub><sup>(2)</sup></th><th>λ̃<sub>2</sub> - q<sub>2</sub><sup>(0)</sup></th><th>λ̃<sub>2</sub> - q<sub>2</sub><sup>(1)</sup></th><th>|λ̃<sub>2</sub> - q<sub>2</sub><sup>(0)</sup>| ≤ |λ̃<sub>2</sub> - q<sub>2</sub><sup>(1)</sup>| ≤</th></tr></thead><tbody><tr><td>1</td><td>6.80760</td><td>6.83824</td><td>6.80935</td><td>6.82480</td><td>6.80764</td><td>-0.03065</td><td>-0.00175</td><td>-0.01721</td><td>-4.90 × 10<sup>-5</sup></td><td>0.22557</td><td>7.06 × 10<sup>-4</sup></td></tr><tr><td>2</td><td>7.03262</td><td>7.04231</td><td>7.03372</td><td>7.03961</td><td>7.03271</td><td>-0.00969</td><td>-0.00110</td><td>-0.00699</td><td>-9.02 × 10<sup>-5</sup></td><td>0.08423</td><td>1.02 × 10<sup>-3</sup></td></tr><tr><td>3</td><td>6.73724</td><td>6.75509</td><td>6.73721</td><td>6.73727</td><td>6.73724</td><td>-0.01785</td><td>2.51 × 10<sup>-5</sup></td><td>-3.39 × 10<sup>-5</sup></td><td>1.36 × 10<sup>-6</sup></td><td>7.24 × 10<sup>-3</sup></td><td>1.53 × 10<sup>-6</sup></td></tr><tr><td>4</td><td>5.66955</td><td>5.87923</td><td>5.68474</td><td>5.81532</td><td>5.66891</td><td>-0.20968</td><td>-0.01519</td><td>-0.14577</td><td>6.36 × 10<sup>-4</sup></td><td>2.52388</td><td>0.02060</td></tr><tr><td>5</td><td>6.10058</td><td>6.15035</td><td>6.10049</td><td>6.10071</td><td>6.10057</td><td>-0.04977</td><td>9.24 × 10<sup>-5</sup></td><td>-1.22 × 10<sup>-4</sup></td><td>1.36 × 10<sup>-5</sup></td><td>0.01169</td><td>1.59 × 10<sup>-5</sup></td></tr><tr><td>6</td><td>7.09264</td><td>7.0928</td><td>7.09266</td><td>7.09276</td><td>7.09264</td><td>-1.56 × 10<sup>-4</sup></td><td>-1.90 × 10<sup>-5</sup></td><td>-1.15 × 10<sup>-4</sup></td><td>-1.71 × 10<sup>-6</sup></td><td>1.23 × 10<sup>-3</sup></td><td>1.82 × 10<sup>-5</sup></td></tr><tr><td>7</td><td>6.91733</td><td>6.94011</td><td>6.91920</td><td>6.93203</td><td>6.91743</td><td>-0.02277</td><td>-0.00187</td><td>-0.01470</td><td>-9.66 × 10<sup>-5</sup></td><td>0.17207</td><td>1.12 × 10<sup>-3</sup></td></tr><tr><td>8</td><td>7.03111</td><td>7.04295</td><td>7.03284</td><td>7.04029</td><td>7.03131</td><td>-0.01184</td><td>-0.00173</td><td>-0.00917</td><td>-1.96 × 10<sup>-4</sup></td><td>0.09986</td><td>2.08 × 10<sup>-3</sup></td></tr><tr><td>9</td><td>7.07835</td><td>7.08123</td><td>7.07877</td><td>7.080583</td><td>7.07840</td><td>-0.00288</td><td>-4.22 × 10<sup>-4</sup></td><td>-2.23 × 10<sup>-3</sup></td><td>-4.77 × 10<sup>-5</sup></td><td>0.02615</td><td>5.16 × 10<sup>-4</sup></td></tr><tr><td>10</td><td>6.95776</td><td>6.96488</td><td>6.95776</td><td>6.95811</td><td>6.95776</td><td>-0.00712</td><td>-3.17 × 10<sup>-6</sup></td><td>-3.48 × 10<sup>-4</sup></td><td>3.59 × 10<sup>-7</sup></td><td>0.01375</td><td>3.83 × 10<sup>-7</sup></td></tr><tr><td>11</td><td>6.55595</td><td>6.59092</td><td>6.55642</td><td>6.56447</td><td>6.55595</td><td>-0.03497</td><td>-4.69 × 10<sup>-4</sup></td><td>-0.00851</td><td>2.69 × 10<sup>-6</sup></td><td>0.11689</td><td>1.33 × 10<sup>-5</sup></td></tr><tr><td>12</td><td>6.85444</td><td>6.86361</td><td>6.85422</td><td>6.85150</td><td>6.85444</td><td>-0.00917</td><td>2.21 × 10<sup>-4</sup></td><td>0.00293</td><td>1.79 × 10<sup>-6</sup></td><td>0.00536</td><td>2.30 × 10<sup>-6</sup></td></tr><tr><td>13</td><td>5.50884</td><td>5.62489</td><td>5.50826</td><td>5.54759</td><td>5.50807</td><td>-0.11605</td><td>5.83 × 10<sup>-4</sup></td><td>-0.03875</td><td>7.69 × 10<sup>-4</sup></td><td>0.84790</td><td>8.23 × 10<sup>-4</sup></td></tr><tr><td>14</td><td>6.04911</td><td>6.07404</td><td>6.04375</td><td>6.02038</td><td>6.04857</td><td>-0.02493</td><td>0.00536</td><td>0.02873</td><td>5.42 × 10<sup>-4</sup></td><td>0.05689</td><td>5.82 × 10<sup>-4</sup></td></tr><tr><td>15</td><td>6.98255</td><td>6.99306</td><td>6.98304</td><td>6.98777</td><td>6.98256</td><td>-0.01051</td><td>-4.84 × 10<sup>-4</sup></td><td>-0.00522</td><td>-1.04 × 10<sup>-5</sup></td><td>0.06610</td><td>1.34 × 10<sup>-4</sup></td></tr><tr><td>16</td><td>7.08039</td><td>7.08097</td><td>7.08038</td><td>7.08031</td><td>7.08039</td><td>-5.88 × 10<sup>-4</sup></td><td>3.51 × 10<sup>-6</sup></td><td>7.55 × 10<sup>-5</sup></td><td>1.05 × 10<sup>-9</sup></td><td>7.94 × 10<sup>-5</sup></td><td>1.15 × 10<sup>-9</sup></td></tr><tr><td>17</td><td>6.48050</td><td>6.53032</td><td>6.48147</td><td>6.50067</td><td>6.48043</td><td>-0.04981</td><td>-9.70 × 10<sup>-4</sup></td><td>-0.02017</td><td>7.06 × 10<sup>-5</sup></td><td>0.3604578</td><td>3.20 × 10<sup>-4</sup></td></tr><tr><td>18</td><td>7.04966</td><td>7.05459</td><td>7.04998</td><td>7.05253</td><td>7.04967</td><td>-0.00492</td><td>-3.18 × 10<sup>-4</sup></td><td>-2.87 × 10<sup>-3</sup></td><td>-1.18 × 10<sup>-5</sup></td><td>0.03878</td><td>1.56 × 10<sup>-4</sup></td></tr><tr><td>19</td><td>7.09124</td><td>7.09159</td><td>7.09128</td><td>7.09148</td><td>7.09125</td><td>-3.43 × 10<sup>-4</sup></td><td>-3.48 × 10<sup>-5</sup></td><td>-2.38 × 10<sup>-4</sup></td><td>-2.45 × 10<sup>-6</sup></td><td>2.68 × 10<sup>-3</sup></td><td>2.73 × 10<sup>-5</sup></td></tr><tr><td>20</td><td>7.08664</td><td>7.08696</td><td>7.08664</td><td>7.08662</td><td>7.08664</td><td>-3.23 × 10<sup>-4</sup></td><td>1.23 × 10<sup>-6</sup></td><td>2.54 × 10<sup>-5</sup></td><td>9.25 × 10<sup>-12</sup></td><td>3.36 × 10<sup>-4</sup></td><td>4.31 × 10<sup>-9</sup></td></tr></tbody></table>
to $|\tilde{\lambda}_2 - q_2^{(0)}|$ and $|\tilde{\lambda}_2 - q_2^{(1)}|$ by Inequality (12) are not so close to these quantities as they were previously. For a part, this can be explained by the fact that we have a smaller value for $c_2^{(0)}$ and $c_2^{(1)}$ than for $c_1^{(0)}$ and $c_1^{(1)}$ when considering the largest eigenvalue. Indeed, for $m = 0, 1$ the gap $|\tilde{\lambda}_3 - q_2^{(m)}|$ is smaller than the gap $|\tilde{\lambda}_2 - q_1^{(m)}|$. However, if we except the case observation 4, we know from this bound that $|\tilde{\lambda}_2 - q_2^{(1)}|$ never exceeds $1.02 \times 10^{-3}$ and this is sufficient for practical interpretation.

Results for the eigenvectors corresponding to the two largest eigenvalues are given in Table 3 which provides the sines $\sin(\tilde{\phi}_k, \phi_k)$ and $\sin(\tilde{\phi}_k, \tilde{\phi}_k^{(1)})$ for $k = 1, 2$ and their bounds provided by Inequality (13).

First, we consider the eigenvector associated to the largest eigenvalue. It could be noted that the maximum value of the sine bewteen the unperturbed and perturbed eigenvector is obtained when deleting observation 4. This value corresponds to an angle of $2.85^\circ$. In this case, as in all the other ones, the order one approximation performs fairly well since its sine with the perturbed eigenvector is equal to only 0.00494 which corresponds to an angle of $0.28^\circ$. Furthermore bounds provided by Inequality (13) are always extremely close to the exact value of $\sin(\tilde{\phi}_1, \tilde{\phi}_1^{(1)})$.

Focusing now on the eigenvector corresponding to the second largest eigenvalue, the maximum of $\sin(\tilde{\phi}_2, \phi_2)$ is again obtained when deleting observation 4 with the value of 0.08114. Even in this case, the order one approximation is fairly close to the perturbed eigenvector since $\sin(\tilde{\phi}_2, \tilde{\phi}_2^{(1)}) = 0.01954$. In contrast to the previous eigenvector, it is worth pointing out that bounds provided by Inequality (13) are not always sufficiently close to the true values of the sine to give an exact account of the accuracy for these approximations. Finally, since the angles between the eigenvectors studied in the table are always very close to zero, we do not provide the tangent of these angles which only deviates from the sine by an extremely small amount.

## 5.2 Approximations when deleting subsets of two observations.

Now, we study approximations to the perturbed largest eigenvalue and its corresponding eigenvector when deleting the subsets of two observations considered for the numerical illustration in [9]. Results similar to those of the previous section are provided in Tables 4 and 5.

We note that deleting subsets of two observations can result in larger variations of the
<table><caption>Table 3: Approximations and error analysis for the eigenvectors associated to the two largest eigenvalues with $r = 1$.</caption><thead><tr><th>I</th><th>sin(Φ̃<sub>1</sub>, φ<sub>1</sub>)</th><th>sin(Φ̃<sub>1</sub>, Φ̃<sub>1</sub><sup>(1)</sup>)</th><th>sin(Φ̃<sub>1</sub>, φ<sub>1</sub>) ≤ sin(Φ̃<sub>1</sub>, Φ̃<sub>1</sub><sup>(1)</sup>) ≤ sin(Φ̃<sub>2</sub>, φ<sub>2</sub>)</th><th>sin(Φ̃<sub>2</sub>, Φ̃<sub>2</sub><sup>(1)</sup>)</th><th>sin(Φ̃<sub>2</sub>, φ<sub>2</sub>) ≤ sin(Φ̃<sub>2</sub>, Φ̃<sub>2</sub><sup>(1)</sup>) ≤ sin(Φ̃<sub>2</sub>, Φ̃<sub>2</sub><sup>(1)</sup>)</th></tr></thead><tbody><tr><td>1</td><td>0.01609</td><td>9.56 × 10<sup>-4</sup></td><td>0.01644</td><td>9.62 × 10<sup>-4</sup></td><td>0.01988</td><td>0.00153</td><td>0.19303</td><td>0.01056</td></tr><tr><td>2</td><td>0.01218</td><td>1.40 × 10<sup>-3</sup></td><td>0.01267</td><td>1.44 × 10<sup>-3</sup></td><td>0.01213</td><td>0.00142</td><td>0.11768</td><td>0.01250</td></tr><tr><td>3</td><td>0.00259</td><td>8.07 × 10<sup>-6</sup></td><td>0.00261</td><td>8.09 × 10<sup>-6</sup></td><td>0.00912</td><td>4.62 × 10<sup>-4</sup></td><td>0.03408</td><td>4.94 × 10<sup>-4</sup></td></tr><tr><td>4</td><td>0.04974</td><td>4.94 × 10<sup>-3</sup></td><td>0.05280</td><td>4.99 × 10<sup>-3</sup></td><td>0.08114</td><td>0.01954</td><td>0.72248</td><td>0.06298</td></tr><tr><td>5</td><td>0.00309</td><td>3.64 × 10<sup>-5</sup></td><td>0.00311</td><td>3.64 × 10<sup>-5</sup></td><td>0.01097</td><td>0.00154</td><td>0.04563</td><td>0.00168</td></tr><tr><td>6</td><td>0.00393</td><td>4.50 × 10<sup>-4</sup></td><td>0.00428</td><td>4.88 × 10<sup>-3</sup></td><td>0.01491</td><td>0.00125</td><td>0.16533</td><td>0.01317</td></tr><tr><td>8</td><td>0.01356</td><td>1.98 × 10<sup>-3</sup></td><td>0.01408</td><td>2.03 × 10<sup>-3</sup></td><td>0.01310</td><td>0.00194</td><td>0.12507</td><td>0.01782</td></tr><tr><td>9</td><td>0.01169</td><td>1.68 × 10<sup>-3</sup></td><td>0.01266</td><td>1.80 × 10<sup>-3</sup></td><td>0.00725</td><td>0.00107</td><td>0.06553</td><td>0.00883</td></tr><tr><td>10</td><td>0.00403</td><td>2.23 × 10<sup>-5</sup></td><td>0.00411</td><td>2.26 × 10<sup>-5</sup></td><td>0.01092</td><td>2.32 × 10<sup>-4</sup></td><td>0.04628</td><td>2.43 × 10<sup>-4</sup></td></tr><tr><td>11</td><td>0.01066</td><td>1.05 × 10<sup>-4</sup></td><td>0.01078</td><td>1.05 × 10<sup>-4</sup></td><td>0.01321</td><td>7.59 × 10<sup>-4</sup></td><td>0.13947</td><td>0.00148</td></tr><tr><td>12</td><td>0.00167</td><td>5.52 × 10<sup>-6</sup></td><td>0.00171</td><td>5.63 × 10<sup>-6</sup></td><td>0.02195</td><td>5.32 × 10<sup>-4</sup></td><td>0.02915</td><td>5.96 × 10<sup>-4</sup></td></tr><tr><td>13</td><td>0.02555</td><td>2.98 × 10<sup>-4</sup></td><td>0.02630</td><td>2.98 × 10<sup>-4</sup></td><td>0.05398</td><td>0.01230</td><td>0.42319</td><td>0.01285</td></tr><tr><td>14</td><td>0.00468</td><td>6.52 × 10<sup>-5</sup></td><td>0.00477</td><td>6.60 × 10<sup>-5</sup></td><td>0.07252</td><td>0.00970</td><td>0.10386</td><td>0.01011</td></tr><tr><td>15</td><td>0.00897</td><td>4.24 × 10<sup>-4</sup></td><td>0.00913</td><td>4.28 × 10<sup>-4</sup></td><td>0.01053</td><td>5.52 × 10<sup>-4</sup></td><td>0.10247</td><td>0.00454</td></tr><tr><td>16</td><td>8.97 × 10<sup>-5</sup></td><td>5.16 × 10<sup>-8</sup></td><td>9.52 × 10<sup>-5</sup></td><td>5.70 × 10<sup>-8</sup></td><td>0.00335</td><td>1.25 × 10<sup>-5</sup></td><td>0.00347</td><td>1.32 × 10<sup>-5</sup></td></tr><tr><td>17</td><td>0.01918</td><td>6.50 × 10<sup>-4</sup></td><td>0.01971</td><td>6.55 × 10<sup>-4</sup></td><td>0.03624</td><td>0.00383</td><td>0.25215</td><td>0.00729</td></tr><tr><td>18</td><td>0.00962</td><td>6.66 × 10<sup>-4</sup></td><td>0.01017</td><td>6.97 × 10<sup>-4</sup></td><td>0.00999</td><td>6.86 × 10<sup>-4</sup></td><td>0.07882</td><td>0.00486</td></tr><tr><td>19</td><td>0.00549</td><td>5.31 × 10<sup>-4</sup></td><td>0.00597</td><td>5.76 × 10<sup>-4</sup></td><td>0.00213</td><td>2.16 × 10<sup>-4</sup></td><td>0.02021</td><td>0.00203</td></tr><tr><td>20</td><td>0.00142</td><td>3.58 × 10<sup>-6</sup></td><td>0.00153</td><td>3.84 × 10<sup>-6</sup></td><td>0.00274</td><td>7.34 × 10<sup>-6</sup></td><td>0.00715</td><td>2.55 × 10<sup>-5</sup></td></tr></tbody></table>
<table><caption>Table 4: Approximations and error analysis for the largest eigenvalue with r = 2.</caption><thead><tr><th>I</th><th>λ̃<sub>1</sub></th><th>λ̃<sub>1</sub><sup>(1)</sup></th><th>λ̃<sub>1</sub><sup>(2)</sup></th><th>q<sub>1</sub><sup>(0)</sup></th><th>q<sub>1</sub><sup>(1)</sup></th><th>λ̃<sub>1</sub> - λ̃<sub>1</sub><sup>(1)</sup></th><th>λ̃<sub>1</sub> - λ̃<sub>1</sub><sup>(2)</sup></th><th>λ̃<sub>1</sub> - q<sub>1</sub><sup>(0)</sup></th><th>λ̃<sub>1</sub> - q<sub>1</sub><sup>(1)</sup></th><th>|λ̃<sub>1</sub> - q<sub>1</sub><sup>(0)</sup>| ≤ |λ̃<sub>1</sub> - q<sub>1</sub><sup>(1)</sup>| ≤</th></tr></thead><tbody><tr><td>{8, 9}</td><td>64.59310</td><td>67.25005</td><td>64.58126</td><td>64.56084</td><td>64.59026</td><td>-2.65695</td><td>0.01184</td><td>0.03226</td><td>0.00284</td><td>0.03616</td><td>0.00316</td></tr><tr><td>{4, 9}</td><td>67.51718</td><td>69.72595</td><td>67.4481</td><td>67.32138</td><td>67.50241</td><td>-2.20877</td><td>0.06908</td><td>0.19580</td><td>0.01476</td><td>0.20842</td><td>0.01501</td></tr><tr><td>{4, 8}</td><td>67.98219</td><td>69.98722</td><td>67.85527</td><td>67.61002</td><td>67.95618</td><td>-2.00502</td><td>0.12692</td><td>0.37218</td><td>0.02602</td><td>0.40664</td><td>0.02660</td></tr><tr><td>{6, 9}</td><td>69.40391</td><td>69.40294</td><td>69.40242</td><td>69.39558</td><td>69.40366</td><td>9.72 × 10<sup>-4</sup></td><td>0.00149</td><td>0.00834</td><td>2.52 × 10<sup>-4</sup></td><td>0.00892</td><td>2.66 × 10<sup>-4</sup></td></tr><tr><td>{2, 9}</td><td>69.74526</td><td>69.73074</td><td>69.74115</td><td>69.72109</td><td>69.74455</td><td>0.01451</td><td>0.00410</td><td>0.02417</td><td>7.04 × 10<sup>-4</sup></td><td>0.02512</td><td>7.19 × 10<sup>-4</sup></td></tr><tr><td>{2, 8}</td><td>69.98911</td><td>69.99202</td><td>69.98832</td><td>69.98402</td><td>69.98899</td><td>-0.00291</td><td>7.86 × 10<sup>-4</sup></td><td>0.00509</td><td>1.21 × 10<sup>-4</sup></td><td>0.00565</td><td>1.33 × 10<sup>-4</sup></td></tr><tr><td>{2, 6}</td><td>70.01929</td><td>72.14491</td><td>70.01294</td><td>69.99965</td><td>70.01808</td><td>-2.12561</td><td>0.00636</td><td>0.01964</td><td>0.00121</td><td>0.02072</td><td>0.00126</td></tr><tr><td>{3, 8}</td><td>78.83587</td><td>79.36442</td><td>78.83242</td><td>78.82489</td><td>78.83542</td><td>-0.52854</td><td>0.00345</td><td>0.01099</td><td>4.54 × 10<sup>-4</sup></td><td>0.01144</td><td>4.59 × 10<sup>-4</sup></td></tr><tr><td>{2, 4}</td><td>72.60356</td><td>72.46791</td><td>72.58774</td><td>72.46791</td><td>72.60172</td><td>0.13565</td><td>0.01582</td><td>0.13565</td><td>0.00184</td><td>0.14414</td><td>0.00188</td></tr><tr><td>{4, 7}</td><td>75.53263</td><td>75.23992</td><td>75.50596</td><td>75.22661</td><td>75.53053</td><td>0.29271</td><td>0.02667</td><td>0.30602</td><td>0.00210</td><td>0.33028</td><td>0.00212</td></tr></tbody></table>
<table><caption>Table 5: Approximations and error analysis for the eigenvector associated to the largest eigenvalue with $r = 2$.</caption><thead><tr><th>I</th><th>sin(φ̃<sub>1</sub>, φ<sub>1</sub>)</th><th>sin(φ̃<sub>1</sub>, φ̃<sub>1</sub><sup>(1)</sup>)</th><th>sin(φ̃<sub>1</sub>, φ<sub>1</sub>) ≤ sin(φ̃<sub>1</sub>, φ̃<sub>1</sub><sup>(1)</sup>) ≤</th><th>I</th><th>sin(φ̃<sub>1</sub>, φ<sub>1</sub>)</th><th>sin(φ̃<sub>1</sub>, φ̃<sub>1</sub><sup>(1)</sup>)</th><th>sin(φ̃<sub>1</sub>, φ<sub>1</sub>) ≤ sin(φ̃<sub>1</sub>, φ̃<sub>1</sub><sup>(1)</sup>) ≤</th></tr></thead><tbody><tr><td>{8, 9}</td><td>0.02254</td><td>0.00667</td><td>0.02525</td><td>{2, 8}</td><td>0.00856</td><td>0.00132</td><td>0.00951</td><td>0.00146</td></tr><tr><td>{4, 9}</td><td>0.05637</td><td>0.01548</td><td>0.06010</td><td>{2, 6}</td><td>0.01740</td><td>0.00432</td><td>0.01833</td><td>0.00449</td></tr><tr><td>{4, 8}</td><td>0.07724</td><td>0.02042</td><td>0.08464</td><td>{3, 8}</td><td>0.01220</td><td>0.00251</td><td>0.01269</td><td>0.00253</td></tr><tr><td>{6, 9}</td><td>0.01128</td><td>0.00197</td><td>0.01205</td><td>{2, 4}</td><td>0.04477</td><td>0.00522</td><td>0.04759</td><td>0.00532</td></tr><tr><td>{2, 9}</td><td>0.01950</td><td>0.00333</td><td>0.02026</td><td>{4, 7}</td><td>0.06609</td><td>0.00548</td><td>0.07148</td><td>0.00553</td></tr></tbody></table>
eigenvalue of interest than when deleting single observations. Indeed, the perturbed eigenvalue is lower than 70 in the first six lines of Table 4. Furthermore for all the subsets $I$ studied in this table we have a decrease of the eigenvalue, while we note that this eigenvalue is increased in several cases in Table 1. Despite these significative variations of the eigenvalue, we see that the Rayleigh quotient $q_1^{(1)}$ always provides a very accurate approximation to $\tilde{\lambda}_1$ since the maximum gap $|\tilde{\lambda}_1 - q_1^{(1)}| = 0.02602$ observed for $I = \{4, 8\}$ remains fairly moderate. It should also be noted that $q_1^{(1)}$ always performs better than $\tilde{\lambda}_1^{(2)}$. This point is fairly well illustrated considering again the case of $I = \{4, 8\}$ for which we have $\tilde{\lambda}_1 - \tilde{\lambda}_1^{(2)} = 0.12692$. The Rayleigh quotient $q_1^{(0)}$ provides less accurate approximations than $\tilde{\lambda}_1^{(2)}$ but should generally be preferred to $\tilde{\lambda}_1^{(1)}$ if we except some cases. Finally, it is worth pointing out that bounds to $|\tilde{\lambda}_1 - q_1^{(0)}|$ and $|\tilde{\lambda}_1 - q_1^{(1)}|$ provided by Inequality (12) are always very close to the true values of these two differences thus avoiding to recompute the perturbed analysis.

Turning now to the sine values in Table 5, we note the largest variations of the eigenvector when deleting the subsets $I = \{4, 7\}$, $I = \{4, 8\}$ and $I = \{4, 9\}$. However, the order one approximation $\tilde{\phi}_1^{(1)}$ remains fairly satisfying in all the cases since the maximum value of $\sin(\tilde{\phi}_1, \tilde{\phi}_1^{(1)})$ obtained when deleting the subset $I = \{4, 8\}$ does not exceed 0.02042 which corresponds to an angle of only $1.17^\circ$.

# 6 Concluding remarks

The previous numerical study provides some indications on the sharpness of the various approximations considered in the paper. As a result, it may be useful to provide practitioners with some guidance on the choice of approximations for perturbed covariance matrices.

First, when focusing on eigenvalues, Rayleigh quotients $q_k^{(1)}$ (based on the perturbed matrix $\tilde{S}$ and the approximations of order one $\tilde{\phi}_k^{(1)}$) seem to always provide reliable approximations to $\tilde{\lambda}_k$. They should generally be preferred to the approximation of order two $\tilde{\lambda}_k^{(2)}$. Furthermore, their accuracy can be evaluated in a precise way by Inequality (12) if the eigenvalue of interest is not too close to the other eigenvalues, as emphasized by results in Table 1 when considering the largest eigenvalue. This is a definite advantage over other approximations.

It should also be noted that these Rayleigh quotients seem to perform fairly well even for values of $\epsilon = \frac{r}{n-r}$ which are not necessarily very close to zero. This point is made clear in the first lines of Table 4 where $\epsilon = \frac{2}{18} = 0.11$ and the eigenvalue $\tilde{\lambda}_1$ is significantly decreased
through the perturbation.

Approximations provided by $\tilde{\lambda}_k^{(1)}$ should be avoided as well as the Rayleigh quotients $q_k^{(0)}$ which are not sufficiently accurate.

Second, when considering eigenvectors, we note a satisfying accuracy of approximations of order one $\tilde{\phi}_k^{(1)}$. Again, when the eigenvalue corresponding to the eigenvector of interest is sufficiently distant from the other ones, we have a correct evaluation of this accuracy by Inequality (13).

## References

[1] F. Critchley. Influence in principal component analysis. *Biometrika*, 72:627-636, 1985.

[2] P. Pack, I.T. Jolliffe, and B.J.T Morgan. Influential observations in principal component analysis: a case-study. *Journal of Applied Statistics*, 15:37-50, 1988.

[3] Y. Tanaka. Sensitivity analysis in principal component analysis: Influence on the subspace spanned by principal components. *Communications in Statistics-Theory and Methods*, 17:3157-3175, 1988.

[4] I.T. Jolliffe. *Principal Component Analysis*. Springer-Verlag (2nd ed.), New York, 2002.

[5] L.A. Prendergast. A note on sensitivity of principal component subspaces and the efficient detection of influential observations in high dimensions. *Electronic Journal of Statistics*, 2:454-467, 2008.

[6] L.A. Prendergast and C. Li Wai Suen. A new and practical influence measure for subsets of covariance matrix sample principal components with applications to high dimensional datasets. *Computational Statistics and Data Analysis*, 55:752-764, 2011.

[7] A. Hadi and H. Nyquist. Further theoretical results and a comparison between two methods for approximating eigenvalues of perturbed covariance matrices. *Statistics and Computing*, 3:113-123, 1993.

[8] S.-G. Wang and H. Nyquist. Effects on the eigenstructure of a data matrix when deleting an observation. *Computational Statistics and Data Analysis*, 11:179-188, 1991.

[9] S.-G. Wang and E.P. Liski. Effects of observations on the eigensystem of a sample covariance matrix. *Journal of Statistical Planning and Inference*, 36:215-226, 1993.
[10] A. Enguix-González, J.M. Muñoz Pichardo, J.L. Moreno-Rebollo, and R. Pino-Mejías. Influence analysis in principal component analysis through power-series expansions. *Communications in Statistics-Theory and Methods*, 34:2025–2046, 2005.

[11] J. Bénasséni. A correction of approximations used in sensitivity study of principal component analysis. *Computational Statistics*, 33:1939–1955, 2018.

[12] J. Bénasséni. Perturbation des poids des unités statistiques et approximation en analyse en composantes principales. *R.A.I.R.O. Recherche opérationnelle/Operations Research*, 21:175–198, 1987.

[13] J.H. Wilkinson. *The algebraic eigenvalue problem*. Oxford University Press, 1988.

[14] A. Enguix-González, J.L. Moreno-Rebollo, and J.M Muñoz Pichardo. A better approximation of moments of the eigenvalues and eigenvectors of the sample covariance matrix. *Journal of Multivariate Analysis*, 142:133–143, 2015.

[15] A. Enguix-González, J.M. Muñoz Pichardo, J.L. Moreno-Rebollo, and I. Barranco-Chamorro. Using conditional bias in principal component analysis for the evaluation of joint influence on the eigenvalues of the covariance matrix. *Applied Mathematics and Computation*, 218:8937–8950, 2012.

[16] F. Chatelin. *Eigenvalues of Matrices*. The Society for Industrial and Applied Mathematics, Philadelphia, 2012.

[17] M.G. Kendall. *Multivariate Analysis*. Griffin, London, 1975.

[18] R. Sibson. Studies in robustness of multidimensional scaling: perturbational analysis of classical scaling. *Journal of the Royal Statistical Society Series B*, 41:217–229, 1979.

[19] I.C.F. Ipsen and B. Nadler. Refined perturbation bounds for eigenvalues of hermitian and non-hermitian matrices. *SIAM Journal on Matrix Analysis and Applications*, 31:40–53, 2009.

[20] J. Bénasséni. Lower bounds for the largest eigenvalue of a symmetric matrix under perturbations of rank one. *Linear and Multilinear Algebra*, 59:565–569, 2011.
[21] G. Cheng, Z. Song, J. Yang, and J. Si. The bounds of the eigenvalues for rank-one modification of hermitian matrix. *Numerical Linear Algebra with Applications*, 21:98–107, 2014.