# The Fréchet  Derivative of an Analytic Function of a Bounded Operator with Some Applications

Authors: D. S. Gilliam; T. Hohage; X. Ji; F. Ruymgaart
Journal: International Journal of Mathematics and Mathematical Sciences
 (2009)

---

## Abstract

The main result in this paper is the determination of the Fréchet derivative of an
analytic function of a bounded operator, tangentially to the space of all bounded operators. Some applied problems from statistics and numerical analysis are included as a motivation for this study. The perturbation operator (increment) is not of any special form and is not supposed to commute with the operator at which the derivative is evaluated. This generality is important for the applications. In the Hermitian case, moreover, some results on perturbation of an isolated eigenvalue, its eigenprojection, and its eigenvector if the eigenvalue is simple, are also included. Although these results are known in principle, they are not in general formulated in terms of arbitrary perturbations as required for the applications. Moreover, these results are presented as corollaries to the main theorem, so that this paper also provides a short, essentially self-contained review of these aspects of perturbation theory.


---

## Body

## 1. Introduction

Motivated by certain applications in numerical analysis
and, in particular, statistics, this paper deals with the Fréchet derivative of
an analytic functionφ of a bounded
linear operator T on a separable
Hilbert space ℍ (in the sense
of the usual functional calculus), tangentially to the Banach space ℒ of all bounded
linear operators mapping ℍ into itself.
More precisely, a first order approximation to the difference
(1.1)φ(T˜)−φ(T),T˜=T+Π,Π∈ℒ,is obtained, including the order
of magnitude of the remainder. An example of such a function φ is a
generalized or regularized inverse of the square root(1.2)φ(T˜)=(αI+T˜)−1/2,α>0,where I is the identity
operator. Once the Fréchet derivative has been established (Section 2), it
yields the asymptotic distribution of functions of certain random operators via
an ensuing delta method: a well-known statistical technique (see Section 4).ClearlyT˜ can be regarded
as a perturbed version of T, and it is not surprising that perturbation methods
are employed to obtain the desired result. The authors are aware of the
possibility that the rather straightforward result on the Fréchet derivative
might be hidden somewhere in the rich literature on perturbation theory [1–3]. Yet they have not been successful in
identifying a reference that states the result in its present form, tailored to
the applications they have in mind. Some remarks are particularly in order.(a)
The perturbationsΠ
are typically
of small norm but otherwise arbitrary bounded or Hermitian. In literature, they are often of the form
(1.3)Π=ϵT1+ϵ2T2+⋯for operators T1,T2,…, and a small number ϵ. In statistics, there is no point in representing the
perturbation in such a form.(b)
The
perturbationΠ and the
operator T are not assumed
to commute, because in our applications such an assumption would not in general
be fulfilled. If the operators do commute, however, the Fréchet derivative
would reduce to φ'(T), in the sense of functional calculus with φ' the derivative
of φ. In the case considered here, the actual Fréchet
derivative and φ'(T) may differ
considerably.(c)
A central theme
in perturbation theory concerns the perturbation of an isolated eigenvalue and
corresponding eigenprojection (see, e.g, the references mentioned
before). Some of the results are included, because they can be easily derived
from the main result on the Fréchet derivative by choosing a special functionφ (Section 3). In
this way, the paper presents a concise and essentially self-contained review of some
basic results in this area. They are again presented in terms of a general
(Hermitian) perturbation Π, as being required for statistical application, in the
same vein as, but somewhat more general than, Dauxois et al. [4].As has already been mentioned in the beginning,ℍ will be a
separable Hilbert space and ℒ the Banach
space of all bounded linear operators mapping ℍ into itself.
The inner product on ℍ will be denoted
by 〈⋅,⋅〉 and the norm by ∥⋅∥. The norm on ℒ will be written ∥⋅∥ℒ, and the notation ℒH and 𝒞H will be used to
denote the subspace of all Hermitian and all compact Hermitian operators, respectively.We will exclusively deal with infinite dimensional
Hilbert spaces and will not attempt to include the simpler finite dimensional
case in our formulation. The Fréchet derivative for arbitrary perturbations is
well known in the finite dimensional matrix case. This result and further
references can be found in the recent monograph by Bhatia [5]. In the finite
dimensional case, this derivative is also implicitly present in Theorem 2.1 of
Ruymgaart and Yang [6] to obtain the asymptotic distribution of a function
of a random matrix.


### 2. The Fréchet Derivative

Let us fix an arbitraryT∈ℒ with spectrum σ(T) and a bounded open region Ω⊂ℂ in the complex
plane with smooth boundary Γ=∂Ω, such that(2.1)σ(T)⊂Ω,δΓ=dist(Γ,σ(T))>0.Furthermore, let us consider
functions of type(2.2)φ:D→ℂ,analytic,where D⊃Ω¯ is an open
neighborhood of Ω¯. Let us write(2.3)MΓ=maxz∈Γ|φ(z)|<∞,LΓ=length ofΓ<∞.The resolvent(2.4)R(z)=(zI−T)−1,z∈ρ(T),is analytic on the resolvent set ρ(T)=[σ(T)]c, and the
operator(2.5)φ(T)=12πi∮Γφ(z)R(z)dzis well defined. This relation
establishes an algebra homomorphism [7, Section 17.2] which implies in
particular that(2.6)φ(T)ψ(T)=(φψ)(T),if ψ:D→ℂ is also
analytic. In particular, we have(2.7)T=12πi∮ΓzR(z)dz.The operators(2.8)φT′Π=12πi∮Γφ(z)R(z)ΠR(z)dz,(2.9)Sφ,Π=12πi∮Γφ(z)R(z)(ΠR(z))2(I−ΠR(z))−1dz are well defined for every Π∈ℒ sufficiently
small. Note that according to Dunford and Schwartz [8, Lemma VII.6.11],
there is a constant 0<K<∞, such that(2.10)∥R(z)∥ℒ≤KδΓ,∀z∈Ωc.Theorem 2.1 (Fréchet Derivative).
LetT∈ℒ and suppose
that φ satisfies (2.2). Then φ maps the
neighborhood {T˜=T+Π:Π∈ℒ,∥Π∥ℒ≤cδΓ/K,for some0<c<1} into ℒ, when defined in the usual way of functional
calculus. This mapping is Fréchet differentiable at T, tangentially to ℒ, with bounded derivative φT′:ℒ→ℒ, as defined by (2.8). More specifically, we
have(2.11)φ(T+Π)=φ(T)+φT′Π+Sφ,Π,where Sφ,Π is defined in (2.9) and (2.12)∥φT′Π∥ℒ≤12πMΓLΓ(KδΓ)2∥Π∥ℒ,(2.13)∥Sφ,Π∥ℒ≤12(1−c)πMΓLΓ(KδΓ)3∥Π∥ℒ2.Proof.
Forφ to be well
defined on the neighborhood, let us first show that(2.14)σ(T˜)⊂Ω,∀Π∈ℒwith∥Π∥ℒ≤cδΓK.To verify this, note that by
(2.10) we have ∥ΠR(z)∥ℒ<c for such Π. Consequently, the operator(2.15)R(z)(I−ΠR(z))−1=R(z){(R(z)−1−Π)R(z)}−1=(zI−T−Π)−1=R˜(z)is bounded for each z∈Ωc, which entails (2.14). Hence, (2.16)φ(T˜)=12πi∮Γφ(z)R˜(z)dzis well defined for Π with ∥Π∥ℒ≤cδΓ/K.
Applying a Neumann series expansion [9, Section 5.2] to the inverse on the left in
(2.15), we obtain(2.17)R˜(z)=R(z){I+ΠR(z)+(ΠR(z))2+⋯}=R(z)+R(z)ΠR(z)+R(z)(ΠR(z))2(I−ΠR(z))−1,just as in Watson [10] for
matrices. Term-wise integration yields (2.11).
The upper bounds in (2.12) and
(2.13) are immediate
from (2.8) and (2.9), respectively, by exploiting
(2.3) and (2.15). The boundedness of φT′ as a linear
operator mapping ℒ into itself
follows at once from (2.12).Remark 2.2.
It will be seen in Section4 that for the
applications we have in mind it is important that we do not require that T and Π commute. If
they do, however, it is clear that the Fréchet derivative in (2.8) reduces
to(2.18)φT′Π={12πi∮Γφ(z)R2(z)dz}Π.It has been shown in Dunford
and Schwartz [8, proof of Theorem VII.6.10] that(2.19)12πi∮Γφ(z)R2(z)dz=12πi∮Γφ'(z)R(z)dz.Combination of (2.11) with
(2.18) and (2.19) yields(2.20)φ(T+Π)=φ(T)+φ'(T)Π+𝒪(∥Π∥ℒ2),writing, for any r>0,(2.21)𝒪(∥Π∥ℒr),as∥Π∥ℒ→0,to indicate any quantity
(operator, vector, number) whose norm or absolute value is of the given order.
Note that in (2.20) the operator, φ'(T) is to be understood
in the sense of the usual functional calculus as in (2.5) with φ replaced by its
derivative φ'.
In this situation of commuting operators, Dunford and Schwartz [8] obtain the Taylor expansion(2.22)φ(T+Π)=∑n=0∞φ(n)(T)n!Πn,which implies, of course, (2.20).
Keeping the perturbation as before, we now restrictT to the class 𝒞H of compact
Hermitian operators. The bounded and countable spectrum consists of the number 0, whether an eigenvalue or not, and all the nonzero
eigenvalues λ1,λ2,…∈ℝ. In this work, we avoid technical issues related to λ=0 being an eigenvalue,
and assume that T is one-to-one,
that is, Tf=0 implies that f=0. It is well known [7] that such a T can be
represented as(2.23)T=∑j=1∞λjPj,where the Pj are the
corresponding orthogonal eigenprojections onto the mutually orthogonal finite
dimensional eigenspaces. These projections provide a resolution of the identity
in ℍ, that is,(2.24)Iℍ=∑j=1∞Pj.The resolvent has the
expansion(2.25)R(z)=∑j=1∞1z−λjPj,z∈ρ(T).Corollary 2.3.
Let the conditions of Theorem2.1 be fulfilled for T∈𝒞H with expansion (2.23). In this case the Fréchet derivative φT′:ℒ→ℒ is given
by(2.26)φT′Π=∑jφ′(λj)PjΠPj+∑∑j≠kφ(λk)−φ(λj)λk−λjPjΠPk,Π∈ℒ.Proof.
Let us substitute the expansion (2.25) for R(z) into the
expression for φT′Π in (2.8).
Application of the partial fraction method yields(2.27)φT′Π=∑j∑k{12πi∮Γφ(z)(z−λj)(z−λk)dz}PjΠPk=∑j{12πi∮Γφ(z)(z−λj)2dz}PjΠPj+∑∑j≠k[1λk−λj12πi∮Γ{φ(z)(z−λk)−φ(z)(z−λj)}dz]PjΠPk.The right-hand side of (2.27)
reduces at once to the expression on the right in (2.26) by an application of
Cauchy's integral formula.Example 2.4.
The functionφ(z)=z, z∈ℂ, is analytic on the entire complex plane so that
Corollary 2.3 applies. The Fréchet derivative in (2.26) now reduces
to(2.28)φT′Π=∑j∑kPjΠPk=Π,Π∈ℒ. Of course this result is immediate because in this
simple case φ(T+Π)=T+Π=φ(T)+Π.Example 2.5.
Next let us, forp>0, consider the function(2.29)φα(z)=(α+z)−p,α>δΓ>0,for z≠−α. Note that the choice of α ensures that
the pole at z=−α remains outside
the contour Γ. Clearly there exists an open region Ω of the type
required, such that φ is analytic on
some open neighborhood D of Ω¯. Hence Corollary 2.3 applies again. The operator φα(T)=(αI+T)−p represents a
regularized or generalized inverse of Tikhonov type, according to whether T is injective or
not. The Fréchet derivative in (2.26) now equals(2.30)φα,T′Π=−p∑j1(α+λj)p+1PjΠPj+∑∑j≠k(α+λj)p−(α+λk)p(λk−λj)(α+λj)p(α+λk)pPjΠPk,for Π∈ℒ.Remark 2.6.
ForT∈𝒞H, T and Π commuting the
double sum on the right in (2.26) cancels and we obtain(2.31)φT′Π=∑jφ′(λj)PjΠ,in accordance with (2.20).
Apparently, the double sum is a correction term needed when T and Π do not
commute.


### 3. Perturbation of Eigenvalues and
Eigenvectors

Throughout this section, bothT and Π are assumed to
be Hermitian, so that also T+Π∈ℒH. In addition to this, we assume that(3.1)T∈ℒHhasanisolatedeigenvalueλ1,with one-dimensional eigenspace.
Consequently, the eigenprojection can be written(3.2)P1=p1⊗p1,for somep1∈ℍwith∥p1∥=1,where for a,b∈ℍ the operator a⊗b is defined by (a⊗b)x=〈x,b〉a,x∈ℍ.The regionΩ⊃σ(T) will now be
chosen in such a way that it has a connected component Ω1 with the properties(3.3)Ω1∩σ(T)=λ1,dist(Ω1,Ω∖Ω1)>0.A special analytic function φ1:D→ℂ such
that(3.4)φ1(z)=1,z∈Ω1,φ1(z)=0,z∈Ω0=Ω∖Ω1,will play an important role in
the sequel. Note, for instance, that(3.5)φ1(T)=P1.For the Fréchet derivative ofφ1(T˜) at T, a special
expression can be obtained. Let us write(3.6)T=λ1P1+T0,where T0 is Hermitian
with spectrum σ(T0)⊂Ω0. According to the spectral theorem,  there exists a
resolution of the identity E(λ),  λ∈σ(T0), such that(3.7)T0=∫σ(T0)λdE(λ).It should be noted
that(3.8)P1E(λ)=E(λ)P1=O,∀λ∈σ(T0),where O is the zero
operator, and that(3.9)R(z)=1z−λ1P1+∫σ(T0)1z−λdE(λ).Let us define(3.10)Q1=∫σ(T0)1λ1−λdE(λ).Lemma 3.1.
The Fréchet derivative ofφ1(T˜) at T is given
by(3.11)φ1,T′Π=P1ΠQ1+Q1ΠP1,Π∈ℒH.Proof.
This follows by substitution of (3.9) in the
expression on the right in(3.12)φ1,T′Π=12πi∮Γ1R(z)ΠR(z)dz,Γ1=∂Ω1,for this derivative; see also
(2.8). We thus obtain(3.13)φ1,T′Π=12πi∮Γ11(z−λ1)2P1ΠP1dz+12πi∮Γ11z−λ1P1Π{∫σ(T0)1z−λdE(λ)}dz+12πi∮Γ1{∫σ(T0)1z−λdE(λ)}Π1z−λ1P1dz+12πi∮Γ1{∫σ(T0)1z−λdE(λ)}Π{∫σ(T0)1z−μdE(μ)}.
By Cauchy's integral formula(3.14)12πi∮Γ11(z−λ1)2dz=φ1′(λ1)=0,so that the first term on the
right side in (3.13) is the zero operator. Regarding the second, note
that(3.15)12πi∮Γ11(z−λ1)(z−λ)dz=1λ1−λ(12πi∮Γ11z−λ1dz−12πi∮Γ11z−λdz)=1λ1−λ,because each λ∈σ(T0) lies outside
the contour Γ1. Consequently, the second term equals(3.16)P1Π{12πi∫σ(T0)1λ1−λdE(λ)}=P1ΠQ1.Similarly, the third term equals Q1ΠP1. The last term cancels, because(3.17)12πi∮Γ11(z−λ)(z−μ)dz=0,since both λ and μ lie outside Γ1.Some results
about the perturbation ofλ1 and P1 in a given
direction as in (1.3) that are well known in literature [1, 2] can be partly recovered for perturbations in some neighborhood,
in an essentially self-contained manner, as simple consequences of the results
in Section 2.Corollary 3.2.
Under the assumptions (3.1), (3.2), and for Π∈ℒH sufficiently
small, the operator T˜ has an isolated
eigenvalue λ˜1 with
eigenprojection P˜1=p˜1⊗p˜1 for some unit
vector p˜1∈ℍ, satisfying(3.18)P˜1=P1+P1ΠQ1+Q1ΠP1+𝒪(∥Π∥ℒ2),where Q1 is defined in
(3.10).Proof.
In view of (3.5) and (3.11), application of (2.11)
with φ=φ1 yields φ1(T˜)=P1+P1ΠQ1+Q1ΠP1+𝒪(∥Π∥ℒ2). Clearly φ1(T˜) is Hermitian,
and because (φ1(T˜))2=φ12(T˜)=φ1(T˜) by (2.6), it is
also idempotent so that it is in fact some projection P˜1, for example, it follows that ∥P1−P˜1∥ℒ<1 for all Π sufficiently
small, and hence the range of P˜1 must also have
dimension 1 [11]
so that P˜1=p˜1⊗p˜1 for some p˜1∈ℍ with ∥p˜1∥=1.
Next, letχ(z)=z, z∈ℂ, be the identity function. By (2.6), again, on the
one hand we have (χφ1)(T˜)p˜1=T˜P˜1p˜1=T˜p˜1, and on the other (φ1χ)(T˜)p˜1=P˜1T˜p˜1=(p˜1⊗p˜1)T˜p˜1=〈T˜p˜1,p˜1〉p˜1=λ˜1p˜1. Hence p˜1 is an
eigenvector of T˜ with eigenvalue λ˜1.Corollary 3.3.
Under the assumptions of Corollary3.2, we
have(3.19)p˜1=p1+Q1Πp1+𝒪(∥Π∥ℒ2).Proof.
Let us first observe thatP1ΠQ1p1=0 because of
(3.8). Hence (3.18) yields p˜1−p1=(P˜1−P1)p1+r1+r2=Q1Πp1+r1+r2+𝒪(∥Π∥ℒ2), where(3.20)r1=P1(p˜1−p1),r2=(P˜1−P1)(p˜1−p1).It sufficies to show that ∥rj∥=𝒪(∥Π∥ℒ2) for j=1,2. The idea of the proof can be found in Dauxois et al. [4].
Regardingr1, note that 1−〈p˜1,p1〉2=∥〈p˜1,p1〉p˜1−p1∥2≤∥P˜1−P1∥ℒ2=𝒪(∥Π∥ℒ2), once more using (3.18). Hence 〈p˜1,p1〉→1, as ∥Π∥ℒ→0, and therefore 2≥1+〈p˜1,p1〉≥1 for ∥Π∥ℒ sufficiently
small. This entails(3.21)∥r1∥=∥〈p˜1,p1〉p˜1−p1∥=|1−〈p˜1,p1〉|=|1−〈p˜1,p1〉2||1+〈p˜1,p1〉|=𝒪(∥Π∥ℒ2).
Forr2 we
have(3.22)∥r2∥≤∥P˜1−P1∥ℒ∥p˜1−p1∥=𝒪(∥Π∥ℒ)2(1−〈p˜1,p1〉)=𝒪(∥Π∥ℒ)2∥r1∥=𝒪(∥Π∥ℒ2),as can be seen from (3.21).Corollary 3.4.
Under the assumptions of Corollary3.2, we
have(3.23)λ˜1=λ1+〈Πp1,p1〉+𝒪(∥Π∥ℒ2).Proof.
With the help of (3.19), we see that λ˜1=〈T˜p˜1,p˜1〉=〈(T+Π)(p1+Q1Πp1),p1+Q1Πp1〉+𝒪(∥Π∥ℒ2). The result follows from a routine calculation
combined with the equalities 〈Tp1,p1〉=λ1, 〈Tp1,Q1Πp1〉=λ1〈p1,Q1Πp1〉=λ1〈Q1p1,Πp1〉=0, and 〈TQ1Πp1,p1〉=〈Q1Πp1,Tp1〉=λ1〈Q1Πp1,p1〉=λ1〈Πp1,Q1p1〉=0. For the last two equalities we assume that T and Q1 are Hermitian
and Q1p1=0 by (3.8).Corollary 3.5.
LetT∈𝒞H be given by
(2.23) and satisfy (3.2). Then (3.18) and (3.19) remain true
with(3.24)Q1=∑j=2∞1λ1−λjPj.Proof.
All nonzero eignvalues ofT are isolated,
in particular λ1. It is immediate from (2.23) that T0=∑j=2∞λjPj, and this leads
to the special expression for Q1 in (3.24).Remark 3.6.
The assumption thatΠ be Hermitian is
in fact not necessary. Of course, if we just require Π to be bounded,
the perturbed operator T˜ is not in
general Hermitian anymore. In particular, a suitably modified version of
Corollary 3.3 will now claim the existence of a pair of eigenvectors, p˜1 for T˜ and p˜1* for T˜*, with expansions(3.25)p˜1=p1+Q1Πp1+𝒪(∥Π∥ℒ2),p˜1*=p1+Q1Π*p1+𝒪(∥Π∥ℒ2),as ∥Π∥→0.


### 4. Applications

In this section, we will sketch three applications: two
in statistics and one in numerical analysis.


#### 4.1. Noisy Integral Equations

LetK:L2(0,1)→L2(0,1) be a compact
injective integral operator, with measurable real kernel denoted by the same
symbol without confusion. More specifically, input f∈L2(0,1) and output g∈L2(0,1) are related
according to(4.1)g(s)=∫01K(s,t)f(t)dt.In practice, only finitely many
data regarding the output are available, usually blurred by random measurement
error. If the data are collected according to a random design, we may think of
the data set as of n independent
copies (X1,Y1),…,(Xn,Yn) of a pair (X,Y) of random
variables, where(4.2)Y=g(X)+ϵ=(Kf)(X)+ϵ,the design variable X has a Uniform (0,1) distribution,
the error variable ϵ has finite
variance and zero mean, and where X and ϵ are
stochastically independent.It is the purpose to recoverf from these
data. It is expedient to “precondition” with the adjoint operator K* and recover f from the
equation(4.3)q=K*g=K*Kf=Rf,where R is compact,
Hermitian, and strictly positive. Under suitable conditions, (4.4)q^(t)=1n∑i=1nYiK*(t,Xi)=1n∑i=1nYiK(Xi,t),t∈[0,1],is an unbiased and n-consistent
estimator of q; see, for instance, van Rooij and Ruymgaart
[12]. Since R−1 is unbounded,
an estimator of the input f is obtained by
applying a regularized inverse of R to q^. Here we will use the Tikhonov type
inverse(4.5)(αI+R)−1=φa(R),α>0,where φα(z)=(α+z)−1; see also (2.29). This yields the input
estimator(4.6)f^α=φα(R)q^,α>0.To assess the quality of the
estimator, one considers the mean integrated squared error
(MISE)(4.7)𝔼∥f^α−f∥2.The behavior of the MISE is
well studied in literature.Recently, there is an interest in certain econometric
models where the operatorK (or R) is unknown
but can be estimated from the data. Let R^ denote an
estimator of R and assume that R^ is also
compact, Hermitian, and nonnegative. In this case, the input
estimator(4.8)f^^α=(αI+R^)−1q^=φα(R^)q^,α>0,will be employed. One expects
that estimation of R will increase
the MISE, and naturally the question arises how much bigger the MISE of f^^α will be than
that of f^α.An upper bound for this increase of the MISE can be
easily found from the results in Section2. For large sample size n, R^ will be close
to R, and Π^=R^−R can be
considered as a small random perturbation of R. Writing φα,R′ for the Fréchet
derivative at R, we see from Theorem 2.1 that(4.9)f^^α−f={φα(R^)−φα(R)}q^+φα(R)q^−f=(φα,R′Π^)q^+f^α−f+(∥Π^∥2).Apparently, (φα,R′Π^)q^ is an extra
error term due to the estimation of R.To find an upper bound for its MISE, let us first
observe that (2.30) simplifies for p=1 and
yields(4.10)φα,R′Π^=−∑j∑k1(α+λj)(α+λk)PjΠ^Pk,where now the λj and the Pj are the
spectral characteristics of R. Let us write, for brevity,(4.11)h^=∑k1α+λkPkq^,and note that(4.12)∥h^∥2≤1α2∥q^∥2.We thus arrive
at(4.13)𝔼∥(φα,R′Π^)q^∥2=𝔼∥∑j1α+λjpjΠ^h^∥2≤1α2𝔼∥Π^∥ℒ2∥h^∥2≤1α4𝔼∥Π^∥ℒ2∥q^∥2.Hence, under suitable assumptions, estimation of the
kernel yields an extra term in the MISE of the input estimator which is of
orderα−4. In the Russian literature, sharper bounds can be
found; see in particular Bakushinsky and Kokurin [13,Section 2.2]. For
results of this type in the statistical literature, obtained in a different
manner, see, for instance, Hall and Horowitz [14] and Florens [15].


#### 4.2. Some Asymptotics for Functional Canonical Correlations

LetX be a real
random element in the Hilbert space L2(0,1) and assume that 𝔼∥X∥4<∞. Its mean μ∈L2(0,1) and covariance
operator S:L2(0,1)→L2(0,1) are
well defined by the relations 𝔼〈f,X〉=〈f,μ〉, 𝔼〈f,X−μ〉〈X−μ,g〉=〈f,Sg〉 for all f,g∈L2(0,1). The operator S is known to be
of finite trace and hence Hilbert-Schmidt and compact. It is also nonnegative
Hermitian. Without real loss of generality, we will assume S to be
injective, so that it will be strictly positive.Next suppose that we are given a random sampleX1,…,Xn of independent
copies of X. The usual estimators of μ and S are X¯=(1/n)∑i=1nXi and S^=(1/n)∑i=1n(Xi−X¯)⊗(Xi−X¯), respectively,
where S^ shares all the
properties of S, except that it cannot be injective because it has a
finite dimensional kernel whose range has dimension at most n−1.BecauseS^ cannot be
injective, the finite dimensional definition of sample canonical correlation  has to be modified, and some kind of smoothing
or regularization is recommended in literature [16]. Regularization might even
be useful when the population is considered, although S is injective
[17]. This
regularization yields Tikhonov type inverses in an expression for the canonical
correlation .For a precise definition, letℍ1 and ℍ2 be two closed
subspaces of L2(0,1) and Ij the orthogonal
projection onto ℍj (j=1,2). Let us write Sjk=IjSIk, and note that Ij(αI+S)Ik=Sjk for j≠k. Similar notation will be used for S^. The regularized squared principal canonical
correlation  for the population is now
defined as(4.14)ρ2=supf1,f2≠0〈f1,S12f2〉2〈f1,(αI1+S11)f1〉〈f2,(αI2+S22)f2〉.Its sample analogue ρ^2 is obtained by
replacing the Sjk with S^jk in (4.14). The
supremum is actually a maximum, and pairs of maximizers will be denoted by f1*, f2*, and f1*^, f2*^, respectively.
The corresponding canonical variates then are(4.15)〈X,fj*〉,〈X,fj*^〉,j=1,2.For an alternative description of these canonical
correlations, let us introduce the
operator(4.16)R1=(αI1+S11)−1/2S12(αI2+S22)−1S21(αI1+S11)−1/2.Interchanging the indices 1 and 2 yields R2, and replacing Sjk with S^jk yields R^1 and R^2. It can be seen that all these operators are
Hilbert-Schmidt and strictly positive Hermitian. It will be assumed that Rj has the largest
eigenvalue with one-dimensional eigenspace generated by fj* with ∥fj*∥=1. Under this condition, it has been shown in Cupidon et al. [18] that(4.17)ρ2=largest eigenvalue ofRj=〈fj*,Rjfj*〉,for j=1,2. A similar result holds true for ρ^2.It is well known that the asymptotic distribution of
the eigenvalues and eigenfunctions of a random operator can be derived from the
asymptotic distribution of this random operator itself (see [10] for
Euclidean spaces and [4] for Hilbert spaces). This technique is based on the results of Section 3. In the present situation, this means that we have to show the convergence in
distribution of the suitably standardized R^j. Because all operators are Hilbert-Schmidt, it can be
shown that(4.18)ℛj=n(R^j−Rj)→d𝒢,as  n→∞, inℒ.Result (4.18) follows easily if convergence in
distribution can be established for each of the factors defining R^j, for instance,(4.19)(αI+S^jj)−1/2=φα(S^jj),where this time φα(z)=(α+z)−1/2, compare (2.29).It is known [4] that
(4.20)n{S^jj−Sjj}→d𝒢jj,as n→∞,inℒHS,for some Gaussian random element 𝒢jj, where ℒHS is the Hilbert
space of all Hilbert-Schmidt operators mapping ℍ into itself.
Writing φα,j′ for the Fréchet
derivative evaluated at Sjj (Section 2) and
exploiting the fact that the imbedding of ℒHS is ℒ are continuous,
we obtain via a kind of delta-method  [18, 19](4.21)n{φα(S^jj)−φα(Sjj)}→dφα,j′𝒢jj,asn→∞,inℒthe desired result. A
combination of results like this for each of the factors of R^j yields (4.18).


#### 4.3. Solution of a Nonlinear Operator Equation

In Bakushinsky and  Kokurin [13], the following
problem is considered. Let ℍ1 and ℍ2 be Hilbert
spaces and F:ℍ1→ℍ2 an operator,
not necessarily linear. The (nonlinear) equation(4.22)F(x)=0,x∈ℍ,is studied. Let x* be a solution
of (4.22) and introduce a set Ω={x∈ℍ1:∥x−x*∥1≤r}, for some r>0. It is assumed that F is Fréchet
differentiable on Ω. If Fx′ is the
derivative at x∈Ω it is,
moreover, assumed that(4.23)∥Fx′−Fy′∥ℒ(ℍ1,ℍ2)≤L∥x−y∥1,x,y∈Ω,where 0<L<∞ is a given
number. Given an initial point x0∈Ω and a sequence {αn}, αn>0, of regularization parameters, these authors show
that, under some further conditions, the generalized Gauss-Newton method
generates a sequence of points {xn} such
that(4.24)∥xn−x*∥=𝒪(αnp),for somep≥12.In their proof of this result, the authors need a
crucial upper bound. Under some additional assumptions, we want to derive this
upper bound as an immediate consequence of Theorem2.1. In order to relate the
present problem to the setup of our paper, let us assume that ℍ1=ℍ2=ℍ, and note that(4.25)(Fx*′)*(Fx*′)=T∈ℒH.For xn, let(4.26)(Fxn′)*(Fxn′)=T˜∈ℒH,and set(4.27)T˜=T+T˜−T=T+Π,where obviously Π∈ℒH. It is not hard to see that (4.23)
entails(4.28)∥Π∥ℒ≤a∥x*−xn∥,for some 0<a<∞. Let Γ be the contour
in (2.1) and D the
corresponding domain. As in Bakushinsky and Kokurin [13], a function Θ(z,α), z∈D, is employed in the iteration scheme, which is
analytic on D.Narrowing down the generality in Bakushinsky and
Kokurin [13] somewhat further, so that the current conditions are satisfied,
their proof of the convergence of the iterations requires an upper bound for
the expression (in our notation)(4.29)12πi∮Γ{1−Θ(z,αn)}{R(z)−R˜(z)}dz=1−Θ(T,αn)−{1−Θ(T˜,αn)}=Θ(T˜,αn)−Θ(T,αn).Keeping n fixed, let us
briefly write this last expression as Θ(˜T)−Θ(T). Now Theorem 2.1 applies with φ=Θ, and application yields at once(4.30)∥Θ(T˜)−Θ(T)∥ℒ≤∥ΘT′(T˜−T)∥ℒ+𝒪(∥T˜−T∥ℒ2)≤b∥T˜−T∥ℒ+𝒪(∥T˜−T∥ℒ2)≤ab∥x*−xn∥+𝒪(∥x*−xn∥2),for some 0<b<∞, by (4.28).


## 2. The Fréchet Derivative

Let us fix an arbitraryT∈ℒ with spectrum σ(T) and a bounded open region Ω⊂ℂ in the complex
plane with smooth boundary Γ=∂Ω, such that(2.1)σ(T)⊂Ω,δΓ=dist(Γ,σ(T))>0.Furthermore, let us consider
functions of type(2.2)φ:D→ℂ,analytic,where D⊃Ω¯ is an open
neighborhood of Ω¯. Let us write(2.3)MΓ=maxz∈Γ|φ(z)|<∞,LΓ=length ofΓ<∞.The resolvent(2.4)R(z)=(zI−T)−1,z∈ρ(T),is analytic on the resolvent set ρ(T)=[σ(T)]c, and the
operator(2.5)φ(T)=12πi∮Γφ(z)R(z)dzis well defined. This relation
establishes an algebra homomorphism [7, Section 17.2] which implies in
particular that(2.6)φ(T)ψ(T)=(φψ)(T),if ψ:D→ℂ is also
analytic. In particular, we have(2.7)T=12πi∮ΓzR(z)dz.The operators(2.8)φT′Π=12πi∮Γφ(z)R(z)ΠR(z)dz,(2.9)Sφ,Π=12πi∮Γφ(z)R(z)(ΠR(z))2(I−ΠR(z))−1dz are well defined for every Π∈ℒ sufficiently
small. Note that according to Dunford and Schwartz [8, Lemma VII.6.11],
there is a constant 0<K<∞, such that(2.10)∥R(z)∥ℒ≤KδΓ,∀z∈Ωc.Theorem 2.1 (Fréchet Derivative).
LetT∈ℒ and suppose
that φ satisfies (2.2). Then φ maps the
neighborhood {T˜=T+Π:Π∈ℒ,∥Π∥ℒ≤cδΓ/K,for some0<c<1} into ℒ, when defined in the usual way of functional
calculus. This mapping is Fréchet differentiable at T, tangentially to ℒ, with bounded derivative φT′:ℒ→ℒ, as defined by (2.8). More specifically, we
have(2.11)φ(T+Π)=φ(T)+φT′Π+Sφ,Π,where Sφ,Π is defined in (2.9) and (2.12)∥φT′Π∥ℒ≤12πMΓLΓ(KδΓ)2∥Π∥ℒ,(2.13)∥Sφ,Π∥ℒ≤12(1−c)πMΓLΓ(KδΓ)3∥Π∥ℒ2.Proof.
Forφ to be well
defined on the neighborhood, let us first show that(2.14)σ(T˜)⊂Ω,∀Π∈ℒwith∥Π∥ℒ≤cδΓK.To verify this, note that by
(2.10) we have ∥ΠR(z)∥ℒ<c for such Π. Consequently, the operator(2.15)R(z)(I−ΠR(z))−1=R(z){(R(z)−1−Π)R(z)}−1=(zI−T−Π)−1=R˜(z)is bounded for each z∈Ωc, which entails (2.14). Hence, (2.16)φ(T˜)=12πi∮Γφ(z)R˜(z)dzis well defined for Π with ∥Π∥ℒ≤cδΓ/K.
Applying a Neumann series expansion [9, Section 5.2] to the inverse on the left in
(2.15), we obtain(2.17)R˜(z)=R(z){I+ΠR(z)+(ΠR(z))2+⋯}=R(z)+R(z)ΠR(z)+R(z)(ΠR(z))2(I−ΠR(z))−1,just as in Watson [10] for
matrices. Term-wise integration yields (2.11).
The upper bounds in (2.12) and
(2.13) are immediate
from (2.8) and (2.9), respectively, by exploiting
(2.3) and (2.15). The boundedness of φT′ as a linear
operator mapping ℒ into itself
follows at once from (2.12).Remark 2.2.
It will be seen in Section4 that for the
applications we have in mind it is important that we do not require that T and Π commute. If
they do, however, it is clear that the Fréchet derivative in (2.8) reduces
to(2.18)φT′Π={12πi∮Γφ(z)R2(z)dz}Π.It has been shown in Dunford
and Schwartz [8, proof of Theorem VII.6.10] that(2.19)12πi∮Γφ(z)R2(z)dz=12πi∮Γφ'(z)R(z)dz.Combination of (2.11) with
(2.18) and (2.19) yields(2.20)φ(T+Π)=φ(T)+φ'(T)Π+𝒪(∥Π∥ℒ2),writing, for any r>0,(2.21)𝒪(∥Π∥ℒr),as∥Π∥ℒ→0,to indicate any quantity
(operator, vector, number) whose norm or absolute value is of the given order.
Note that in (2.20) the operator, φ'(T) is to be understood
in the sense of the usual functional calculus as in (2.5) with φ replaced by its
derivative φ'.
In this situation of commuting operators, Dunford and Schwartz [8] obtain the Taylor expansion(2.22)φ(T+Π)=∑n=0∞φ(n)(T)n!Πn,which implies, of course, (2.20).
Keeping the perturbation as before, we now restrictT to the class 𝒞H of compact
Hermitian operators. The bounded and countable spectrum consists of the number 0, whether an eigenvalue or not, and all the nonzero
eigenvalues λ1,λ2,…∈ℝ. In this work, we avoid technical issues related to λ=0 being an eigenvalue,
and assume that T is one-to-one,
that is, Tf=0 implies that f=0. It is well known [7] that such a T can be
represented as(2.23)T=∑j=1∞λjPj,where the Pj are the
corresponding orthogonal eigenprojections onto the mutually orthogonal finite
dimensional eigenspaces. These projections provide a resolution of the identity
in ℍ, that is,(2.24)Iℍ=∑j=1∞Pj.The resolvent has the
expansion(2.25)R(z)=∑j=1∞1z−λjPj,z∈ρ(T).Corollary 2.3.
Let the conditions of Theorem2.1 be fulfilled for T∈𝒞H with expansion (2.23). In this case the Fréchet derivative φT′:ℒ→ℒ is given
by(2.26)φT′Π=∑jφ′(λj)PjΠPj+∑∑j≠kφ(λk)−φ(λj)λk−λjPjΠPk,Π∈ℒ.Proof.
Let us substitute the expansion (2.25) for R(z) into the
expression for φT′Π in (2.8).
Application of the partial fraction method yields(2.27)φT′Π=∑j∑k{12πi∮Γφ(z)(z−λj)(z−λk)dz}PjΠPk=∑j{12πi∮Γφ(z)(z−λj)2dz}PjΠPj+∑∑j≠k[1λk−λj12πi∮Γ{φ(z)(z−λk)−φ(z)(z−λj)}dz]PjΠPk.The right-hand side of (2.27)
reduces at once to the expression on the right in (2.26) by an application of
Cauchy's integral formula.Example 2.4.
The functionφ(z)=z, z∈ℂ, is analytic on the entire complex plane so that
Corollary 2.3 applies. The Fréchet derivative in (2.26) now reduces
to(2.28)φT′Π=∑j∑kPjΠPk=Π,Π∈ℒ. Of course this result is immediate because in this
simple case φ(T+Π)=T+Π=φ(T)+Π.Example 2.5.
Next let us, forp>0, consider the function(2.29)φα(z)=(α+z)−p,α>δΓ>0,for z≠−α. Note that the choice of α ensures that
the pole at z=−α remains outside
the contour Γ. Clearly there exists an open region Ω of the type
required, such that φ is analytic on
some open neighborhood D of Ω¯. Hence Corollary 2.3 applies again. The operator φα(T)=(αI+T)−p represents a
regularized or generalized inverse of Tikhonov type, according to whether T is injective or
not. The Fréchet derivative in (2.26) now equals(2.30)φα,T′Π=−p∑j1(α+λj)p+1PjΠPj+∑∑j≠k(α+λj)p−(α+λk)p(λk−λj)(α+λj)p(α+λk)pPjΠPk,for Π∈ℒ.Remark 2.6.
ForT∈𝒞H, T and Π commuting the
double sum on the right in (2.26) cancels and we obtain(2.31)φT′Π=∑jφ′(λj)PjΠ,in accordance with (2.20).
Apparently, the double sum is a correction term needed when T and Π do not
commute.


## 3. Perturbation of Eigenvalues and
Eigenvectors

Throughout this section, bothT and Π are assumed to
be Hermitian, so that also T+Π∈ℒH. In addition to this, we assume that(3.1)T∈ℒHhasanisolatedeigenvalueλ1,with one-dimensional eigenspace.
Consequently, the eigenprojection can be written(3.2)P1=p1⊗p1,for somep1∈ℍwith∥p1∥=1,where for a,b∈ℍ the operator a⊗b is defined by (a⊗b)x=〈x,b〉a,x∈ℍ.The regionΩ⊃σ(T) will now be
chosen in such a way that it has a connected component Ω1 with the properties(3.3)Ω1∩σ(T)=λ1,dist(Ω1,Ω∖Ω1)>0.A special analytic function φ1:D→ℂ such
that(3.4)φ1(z)=1,z∈Ω1,φ1(z)=0,z∈Ω0=Ω∖Ω1,will play an important role in
the sequel. Note, for instance, that(3.5)φ1(T)=P1.For the Fréchet derivative ofφ1(T˜) at T, a special
expression can be obtained. Let us write(3.6)T=λ1P1+T0,where T0 is Hermitian
with spectrum σ(T0)⊂Ω0. According to the spectral theorem,  there exists a
resolution of the identity E(λ),  λ∈σ(T0), such that(3.7)T0=∫σ(T0)λdE(λ).It should be noted
that(3.8)P1E(λ)=E(λ)P1=O,∀λ∈σ(T0),where O is the zero
operator, and that(3.9)R(z)=1z−λ1P1+∫σ(T0)1z−λdE(λ).Let us define(3.10)Q1=∫σ(T0)1λ1−λdE(λ).Lemma 3.1.
The Fréchet derivative ofφ1(T˜) at T is given
by(3.11)φ1,T′Π=P1ΠQ1+Q1ΠP1,Π∈ℒH.Proof.
This follows by substitution of (3.9) in the
expression on the right in(3.12)φ1,T′Π=12πi∮Γ1R(z)ΠR(z)dz,Γ1=∂Ω1,for this derivative; see also
(2.8). We thus obtain(3.13)φ1,T′Π=12πi∮Γ11(z−λ1)2P1ΠP1dz+12πi∮Γ11z−λ1P1Π{∫σ(T0)1z−λdE(λ)}dz+12πi∮Γ1{∫σ(T0)1z−λdE(λ)}Π1z−λ1P1dz+12πi∮Γ1{∫σ(T0)1z−λdE(λ)}Π{∫σ(T0)1z−μdE(μ)}.
By Cauchy's integral formula(3.14)12πi∮Γ11(z−λ1)2dz=φ1′(λ1)=0,so that the first term on the
right side in (3.13) is the zero operator. Regarding the second, note
that(3.15)12πi∮Γ11(z−λ1)(z−λ)dz=1λ1−λ(12πi∮Γ11z−λ1dz−12πi∮Γ11z−λdz)=1λ1−λ,because each λ∈σ(T0) lies outside
the contour Γ1. Consequently, the second term equals(3.16)P1Π{12πi∫σ(T0)1λ1−λdE(λ)}=P1ΠQ1.Similarly, the third term equals Q1ΠP1. The last term cancels, because(3.17)12πi∮Γ11(z−λ)(z−μ)dz=0,since both λ and μ lie outside Γ1.Some results
about the perturbation ofλ1 and P1 in a given
direction as in (1.3) that are well known in literature [1, 2] can be partly recovered for perturbations in some neighborhood,
in an essentially self-contained manner, as simple consequences of the results
in Section 2.Corollary 3.2.
Under the assumptions (3.1), (3.2), and for Π∈ℒH sufficiently
small, the operator T˜ has an isolated
eigenvalue λ˜1 with
eigenprojection P˜1=p˜1⊗p˜1 for some unit
vector p˜1∈ℍ, satisfying(3.18)P˜1=P1+P1ΠQ1+Q1ΠP1+𝒪(∥Π∥ℒ2),where Q1 is defined in
(3.10).Proof.
In view of (3.5) and (3.11), application of (2.11)
with φ=φ1 yields φ1(T˜)=P1+P1ΠQ1+Q1ΠP1+𝒪(∥Π∥ℒ2). Clearly φ1(T˜) is Hermitian,
and because (φ1(T˜))2=φ12(T˜)=φ1(T˜) by (2.6), it is
also idempotent so that it is in fact some projection P˜1, for example, it follows that ∥P1−P˜1∥ℒ<1 for all Π sufficiently
small, and hence the range of P˜1 must also have
dimension 1 [11]
so that P˜1=p˜1⊗p˜1 for some p˜1∈ℍ with ∥p˜1∥=1.
Next, letχ(z)=z, z∈ℂ, be the identity function. By (2.6), again, on the
one hand we have (χφ1)(T˜)p˜1=T˜P˜1p˜1=T˜p˜1, and on the other (φ1χ)(T˜)p˜1=P˜1T˜p˜1=(p˜1⊗p˜1)T˜p˜1=〈T˜p˜1,p˜1〉p˜1=λ˜1p˜1. Hence p˜1 is an
eigenvector of T˜ with eigenvalue λ˜1.Corollary 3.3.
Under the assumptions of Corollary3.2, we
have(3.19)p˜1=p1+Q1Πp1+𝒪(∥Π∥ℒ2).Proof.
Let us first observe thatP1ΠQ1p1=0 because of
(3.8). Hence (3.18) yields p˜1−p1=(P˜1−P1)p1+r1+r2=Q1Πp1+r1+r2+𝒪(∥Π∥ℒ2), where(3.20)r1=P1(p˜1−p1),r2=(P˜1−P1)(p˜1−p1).It sufficies to show that ∥rj∥=𝒪(∥Π∥ℒ2) for j=1,2. The idea of the proof can be found in Dauxois et al. [4].
Regardingr1, note that 1−〈p˜1,p1〉2=∥〈p˜1,p1〉p˜1−p1∥2≤∥P˜1−P1∥ℒ2=𝒪(∥Π∥ℒ2), once more using (3.18). Hence 〈p˜1,p1〉→1, as ∥Π∥ℒ→0, and therefore 2≥1+〈p˜1,p1〉≥1 for ∥Π∥ℒ sufficiently
small. This entails(3.21)∥r1∥=∥〈p˜1,p1〉p˜1−p1∥=|1−〈p˜1,p1〉|=|1−〈p˜1,p1〉2||1+〈p˜1,p1〉|=𝒪(∥Π∥ℒ2).
Forr2 we
have(3.22)∥r2∥≤∥P˜1−P1∥ℒ∥p˜1−p1∥=𝒪(∥Π∥ℒ)2(1−〈p˜1,p1〉)=𝒪(∥Π∥ℒ)2∥r1∥=𝒪(∥Π∥ℒ2),as can be seen from (3.21).Corollary 3.4.
Under the assumptions of Corollary3.2, we
have(3.23)λ˜1=λ1+〈Πp1,p1〉+𝒪(∥Π∥ℒ2).Proof.
With the help of (3.19), we see that λ˜1=〈T˜p˜1,p˜1〉=〈(T+Π)(p1+Q1Πp1),p1+Q1Πp1〉+𝒪(∥Π∥ℒ2). The result follows from a routine calculation
combined with the equalities 〈Tp1,p1〉=λ1, 〈Tp1,Q1Πp1〉=λ1〈p1,Q1Πp1〉=λ1〈Q1p1,Πp1〉=0, and 〈TQ1Πp1,p1〉=〈Q1Πp1,Tp1〉=λ1〈Q1Πp1,p1〉=λ1〈Πp1,Q1p1〉=0. For the last two equalities we assume that T and Q1 are Hermitian
and Q1p1=0 by (3.8).Corollary 3.5.
LetT∈𝒞H be given by
(2.23) and satisfy (3.2). Then (3.18) and (3.19) remain true
with(3.24)Q1=∑j=2∞1λ1−λjPj.Proof.
All nonzero eignvalues ofT are isolated,
in particular λ1. It is immediate from (2.23) that T0=∑j=2∞λjPj, and this leads
to the special expression for Q1 in (3.24).Remark 3.6.
The assumption thatΠ be Hermitian is
in fact not necessary. Of course, if we just require Π to be bounded,
the perturbed operator T˜ is not in
general Hermitian anymore. In particular, a suitably modified version of
Corollary 3.3 will now claim the existence of a pair of eigenvectors, p˜1 for T˜ and p˜1* for T˜*, with expansions(3.25)p˜1=p1+Q1Πp1+𝒪(∥Π∥ℒ2),p˜1*=p1+Q1Π*p1+𝒪(∥Π∥ℒ2),as ∥Π∥→0.


## 4. Applications

In this section, we will sketch three applications: two
in statistics and one in numerical analysis.


### 4.1. Noisy Integral Equations

LetK:L2(0,1)→L2(0,1) be a compact
injective integral operator, with measurable real kernel denoted by the same
symbol without confusion. More specifically, input f∈L2(0,1) and output g∈L2(0,1) are related
according to(4.1)g(s)=∫01K(s,t)f(t)dt.In practice, only finitely many
data regarding the output are available, usually blurred by random measurement
error. If the data are collected according to a random design, we may think of
the data set as of n independent
copies (X1,Y1),…,(Xn,Yn) of a pair (X,Y) of random
variables, where(4.2)Y=g(X)+ϵ=(Kf)(X)+ϵ,the design variable X has a Uniform (0,1) distribution,
the error variable ϵ has finite
variance and zero mean, and where X and ϵ are
stochastically independent.It is the purpose to recoverf from these
data. It is expedient to “precondition” with the adjoint operator K* and recover f from the
equation(4.3)q=K*g=K*Kf=Rf,where R is compact,
Hermitian, and strictly positive. Under suitable conditions, (4.4)q^(t)=1n∑i=1nYiK*(t,Xi)=1n∑i=1nYiK(Xi,t),t∈[0,1],is an unbiased and n-consistent
estimator of q; see, for instance, van Rooij and Ruymgaart
[12]. Since R−1 is unbounded,
an estimator of the input f is obtained by
applying a regularized inverse of R to q^. Here we will use the Tikhonov type
inverse(4.5)(αI+R)−1=φa(R),α>0,where φα(z)=(α+z)−1; see also (2.29). This yields the input
estimator(4.6)f^α=φα(R)q^,α>0.To assess the quality of the
estimator, one considers the mean integrated squared error
(MISE)(4.7)𝔼∥f^α−f∥2.The behavior of the MISE is
well studied in literature.Recently, there is an interest in certain econometric
models where the operatorK (or R) is unknown
but can be estimated from the data. Let R^ denote an
estimator of R and assume that R^ is also
compact, Hermitian, and nonnegative. In this case, the input
estimator(4.8)f^^α=(αI+R^)−1q^=φα(R^)q^,α>0,will be employed. One expects
that estimation of R will increase
the MISE, and naturally the question arises how much bigger the MISE of f^^α will be than
that of f^α.An upper bound for this increase of the MISE can be
easily found from the results in Section2. For large sample size n, R^ will be close
to R, and Π^=R^−R can be
considered as a small random perturbation of R. Writing φα,R′ for the Fréchet
derivative at R, we see from Theorem 2.1 that(4.9)f^^α−f={φα(R^)−φα(R)}q^+φα(R)q^−f=(φα,R′Π^)q^+f^α−f+(∥Π^∥2).Apparently, (φα,R′Π^)q^ is an extra
error term due to the estimation of R.To find an upper bound for its MISE, let us first
observe that (2.30) simplifies for p=1 and
yields(4.10)φα,R′Π^=−∑j∑k1(α+λj)(α+λk)PjΠ^Pk,where now the λj and the Pj are the
spectral characteristics of R. Let us write, for brevity,(4.11)h^=∑k1α+λkPkq^,and note that(4.12)∥h^∥2≤1α2∥q^∥2.We thus arrive
at(4.13)𝔼∥(φα,R′Π^)q^∥2=𝔼∥∑j1α+λjpjΠ^h^∥2≤1α2𝔼∥Π^∥ℒ2∥h^∥2≤1α4𝔼∥Π^∥ℒ2∥q^∥2.Hence, under suitable assumptions, estimation of the
kernel yields an extra term in the MISE of the input estimator which is of
orderα−4. In the Russian literature, sharper bounds can be
found; see in particular Bakushinsky and Kokurin [13,Section 2.2]. For
results of this type in the statistical literature, obtained in a different
manner, see, for instance, Hall and Horowitz [14] and Florens [15].


### 4.2. Some Asymptotics for Functional Canonical Correlations

LetX be a real
random element in the Hilbert space L2(0,1) and assume that 𝔼∥X∥4<∞. Its mean μ∈L2(0,1) and covariance
operator S:L2(0,1)→L2(0,1) are
well defined by the relations 𝔼〈f,X〉=〈f,μ〉, 𝔼〈f,X−μ〉〈X−μ,g〉=〈f,Sg〉 for all f,g∈L2(0,1). The operator S is known to be
of finite trace and hence Hilbert-Schmidt and compact. It is also nonnegative
Hermitian. Without real loss of generality, we will assume S to be
injective, so that it will be strictly positive.Next suppose that we are given a random sampleX1,…,Xn of independent
copies of X. The usual estimators of μ and S are X¯=(1/n)∑i=1nXi and S^=(1/n)∑i=1n(Xi−X¯)⊗(Xi−X¯), respectively,
where S^ shares all the
properties of S, except that it cannot be injective because it has a
finite dimensional kernel whose range has dimension at most n−1.BecauseS^ cannot be
injective, the finite dimensional definition of sample canonical correlation  has to be modified, and some kind of smoothing
or regularization is recommended in literature [16]. Regularization might even
be useful when the population is considered, although S is injective
[17]. This
regularization yields Tikhonov type inverses in an expression for the canonical
correlation .For a precise definition, letℍ1 and ℍ2 be two closed
subspaces of L2(0,1) and Ij the orthogonal
projection onto ℍj (j=1,2). Let us write Sjk=IjSIk, and note that Ij(αI+S)Ik=Sjk for j≠k. Similar notation will be used for S^. The regularized squared principal canonical
correlation  for the population is now
defined as(4.14)ρ2=supf1,f2≠0〈f1,S12f2〉2〈f1,(αI1+S11)f1〉〈f2,(αI2+S22)f2〉.Its sample analogue ρ^2 is obtained by
replacing the Sjk with S^jk in (4.14). The
supremum is actually a maximum, and pairs of maximizers will be denoted by f1*, f2*, and f1*^, f2*^, respectively.
The corresponding canonical variates then are(4.15)〈X,fj*〉,〈X,fj*^〉,j=1,2.For an alternative description of these canonical
correlations, let us introduce the
operator(4.16)R1=(αI1+S11)−1/2S12(αI2+S22)−1S21(αI1+S11)−1/2.Interchanging the indices 1 and 2 yields R2, and replacing Sjk with S^jk yields R^1 and R^2. It can be seen that all these operators are
Hilbert-Schmidt and strictly positive Hermitian. It will be assumed that Rj has the largest
eigenvalue with one-dimensional eigenspace generated by fj* with ∥fj*∥=1. Under this condition, it has been shown in Cupidon et al. [18] that(4.17)ρ2=largest eigenvalue ofRj=〈fj*,Rjfj*〉,for j=1,2. A similar result holds true for ρ^2.It is well known that the asymptotic distribution of
the eigenvalues and eigenfunctions of a random operator can be derived from the
asymptotic distribution of this random operator itself (see [10] for
Euclidean spaces and [4] for Hilbert spaces). This technique is based on the results of Section 3. In the present situation, this means that we have to show the convergence in
distribution of the suitably standardized R^j. Because all operators are Hilbert-Schmidt, it can be
shown that(4.18)ℛj=n(R^j−Rj)→d𝒢,as  n→∞, inℒ.Result (4.18) follows easily if convergence in
distribution can be established for each of the factors defining R^j, for instance,(4.19)(αI+S^jj)−1/2=φα(S^jj),where this time φα(z)=(α+z)−1/2, compare (2.29).It is known [4] that
(4.20)n{S^jj−Sjj}→d𝒢jj,as n→∞,inℒHS,for some Gaussian random element 𝒢jj, where ℒHS is the Hilbert
space of all Hilbert-Schmidt operators mapping ℍ into itself.
Writing φα,j′ for the Fréchet
derivative evaluated at Sjj (Section 2) and
exploiting the fact that the imbedding of ℒHS is ℒ are continuous,
we obtain via a kind of delta-method  [18, 19](4.21)n{φα(S^jj)−φα(Sjj)}→dφα,j′𝒢jj,asn→∞,inℒthe desired result. A
combination of results like this for each of the factors of R^j yields (4.18).


### 4.3. Solution of a Nonlinear Operator Equation

In Bakushinsky and  Kokurin [13], the following
problem is considered. Let ℍ1 and ℍ2 be Hilbert
spaces and F:ℍ1→ℍ2 an operator,
not necessarily linear. The (nonlinear) equation(4.22)F(x)=0,x∈ℍ,is studied. Let x* be a solution
of (4.22) and introduce a set Ω={x∈ℍ1:∥x−x*∥1≤r}, for some r>0. It is assumed that F is Fréchet
differentiable on Ω. If Fx′ is the
derivative at x∈Ω it is,
moreover, assumed that(4.23)∥Fx′−Fy′∥ℒ(ℍ1,ℍ2)≤L∥x−y∥1,x,y∈Ω,where 0<L<∞ is a given
number. Given an initial point x0∈Ω and a sequence {αn}, αn>0, of regularization parameters, these authors show
that, under some further conditions, the generalized Gauss-Newton method
generates a sequence of points {xn} such
that(4.24)∥xn−x*∥=𝒪(αnp),for somep≥12.In their proof of this result, the authors need a
crucial upper bound. Under some additional assumptions, we want to derive this
upper bound as an immediate consequence of Theorem2.1. In order to relate the
present problem to the setup of our paper, let us assume that ℍ1=ℍ2=ℍ, and note that(4.25)(Fx*′)*(Fx*′)=T∈ℒH.For xn, let(4.26)(Fxn′)*(Fxn′)=T˜∈ℒH,and set(4.27)T˜=T+T˜−T=T+Π,where obviously Π∈ℒH. It is not hard to see that (4.23)
entails(4.28)∥Π∥ℒ≤a∥x*−xn∥,for some 0<a<∞. Let Γ be the contour
in (2.1) and D the
corresponding domain. As in Bakushinsky and Kokurin [13], a function Θ(z,α), z∈D, is employed in the iteration scheme, which is
analytic on D.Narrowing down the generality in Bakushinsky and
Kokurin [13] somewhat further, so that the current conditions are satisfied,
their proof of the convergence of the iterations requires an upper bound for
the expression (in our notation)(4.29)12πi∮Γ{1−Θ(z,αn)}{R(z)−R˜(z)}dz=1−Θ(T,αn)−{1−Θ(T˜,αn)}=Θ(T˜,αn)−Θ(T,αn).Keeping n fixed, let us
briefly write this last expression as Θ(˜T)−Θ(T). Now Theorem 2.1 applies with φ=Θ, and application yields at once(4.30)∥Θ(T˜)−Θ(T)∥ℒ≤∥ΘT′(T˜−T)∥ℒ+𝒪(∥T˜−T∥ℒ2)≤b∥T˜−T∥ℒ+𝒪(∥T˜−T∥ℒ2)≤ab∥x*−xn∥+𝒪(∥x*−xn∥2),for some 0<b<∞, by (4.28).


## 4.1. Noisy Integral Equations

LetK:L2(0,1)→L2(0,1) be a compact
injective integral operator, with measurable real kernel denoted by the same
symbol without confusion. More specifically, input f∈L2(0,1) and output g∈L2(0,1) are related
according to(4.1)g(s)=∫01K(s,t)f(t)dt.In practice, only finitely many
data regarding the output are available, usually blurred by random measurement
error. If the data are collected according to a random design, we may think of
the data set as of n independent
copies (X1,Y1),…,(Xn,Yn) of a pair (X,Y) of random
variables, where(4.2)Y=g(X)+ϵ=(Kf)(X)+ϵ,the design variable X has a Uniform (0,1) distribution,
the error variable ϵ has finite
variance and zero mean, and where X and ϵ are
stochastically independent.It is the purpose to recoverf from these
data. It is expedient to “precondition” with the adjoint operator K* and recover f from the
equation(4.3)q=K*g=K*Kf=Rf,where R is compact,
Hermitian, and strictly positive. Under suitable conditions, (4.4)q^(t)=1n∑i=1nYiK*(t,Xi)=1n∑i=1nYiK(Xi,t),t∈[0,1],is an unbiased and n-consistent
estimator of q; see, for instance, van Rooij and Ruymgaart
[12]. Since R−1 is unbounded,
an estimator of the input f is obtained by
applying a regularized inverse of R to q^. Here we will use the Tikhonov type
inverse(4.5)(αI+R)−1=φa(R),α>0,where φα(z)=(α+z)−1; see also (2.29). This yields the input
estimator(4.6)f^α=φα(R)q^,α>0.To assess the quality of the
estimator, one considers the mean integrated squared error
(MISE)(4.7)𝔼∥f^α−f∥2.The behavior of the MISE is
well studied in literature.Recently, there is an interest in certain econometric
models where the operatorK (or R) is unknown
but can be estimated from the data. Let R^ denote an
estimator of R and assume that R^ is also
compact, Hermitian, and nonnegative. In this case, the input
estimator(4.8)f^^α=(αI+R^)−1q^=φα(R^)q^,α>0,will be employed. One expects
that estimation of R will increase
the MISE, and naturally the question arises how much bigger the MISE of f^^α will be than
that of f^α.An upper bound for this increase of the MISE can be
easily found from the results in Section2. For large sample size n, R^ will be close
to R, and Π^=R^−R can be
considered as a small random perturbation of R. Writing φα,R′ for the Fréchet
derivative at R, we see from Theorem 2.1 that(4.9)f^^α−f={φα(R^)−φα(R)}q^+φα(R)q^−f=(φα,R′Π^)q^+f^α−f+(∥Π^∥2).Apparently, (φα,R′Π^)q^ is an extra
error term due to the estimation of R.To find an upper bound for its MISE, let us first
observe that (2.30) simplifies for p=1 and
yields(4.10)φα,R′Π^=−∑j∑k1(α+λj)(α+λk)PjΠ^Pk,where now the λj and the Pj are the
spectral characteristics of R. Let us write, for brevity,(4.11)h^=∑k1α+λkPkq^,and note that(4.12)∥h^∥2≤1α2∥q^∥2.We thus arrive
at(4.13)𝔼∥(φα,R′Π^)q^∥2=𝔼∥∑j1α+λjpjΠ^h^∥2≤1α2𝔼∥Π^∥ℒ2∥h^∥2≤1α4𝔼∥Π^∥ℒ2∥q^∥2.Hence, under suitable assumptions, estimation of the
kernel yields an extra term in the MISE of the input estimator which is of
orderα−4. In the Russian literature, sharper bounds can be
found; see in particular Bakushinsky and Kokurin [13,Section 2.2]. For
results of this type in the statistical literature, obtained in a different
manner, see, for instance, Hall and Horowitz [14] and Florens [15].


## 4.2. Some Asymptotics for Functional Canonical Correlations

LetX be a real
random element in the Hilbert space L2(0,1) and assume that 𝔼∥X∥4<∞. Its mean μ∈L2(0,1) and covariance
operator S:L2(0,1)→L2(0,1) are
well defined by the relations 𝔼〈f,X〉=〈f,μ〉, 𝔼〈f,X−μ〉〈X−μ,g〉=〈f,Sg〉 for all f,g∈L2(0,1). The operator S is known to be
of finite trace and hence Hilbert-Schmidt and compact. It is also nonnegative
Hermitian. Without real loss of generality, we will assume S to be
injective, so that it will be strictly positive.Next suppose that we are given a random sampleX1,…,Xn of independent
copies of X. The usual estimators of μ and S are X¯=(1/n)∑i=1nXi and S^=(1/n)∑i=1n(Xi−X¯)⊗(Xi−X¯), respectively,
where S^ shares all the
properties of S, except that it cannot be injective because it has a
finite dimensional kernel whose range has dimension at most n−1.BecauseS^ cannot be
injective, the finite dimensional definition of sample canonical correlation  has to be modified, and some kind of smoothing
or regularization is recommended in literature [16]. Regularization might even
be useful when the population is considered, although S is injective
[17]. This
regularization yields Tikhonov type inverses in an expression for the canonical
correlation .For a precise definition, letℍ1 and ℍ2 be two closed
subspaces of L2(0,1) and Ij the orthogonal
projection onto ℍj (j=1,2). Let us write Sjk=IjSIk, and note that Ij(αI+S)Ik=Sjk for j≠k. Similar notation will be used for S^. The regularized squared principal canonical
correlation  for the population is now
defined as(4.14)ρ2=supf1,f2≠0〈f1,S12f2〉2〈f1,(αI1+S11)f1〉〈f2,(αI2+S22)f2〉.Its sample analogue ρ^2 is obtained by
replacing the Sjk with S^jk in (4.14). The
supremum is actually a maximum, and pairs of maximizers will be denoted by f1*, f2*, and f1*^, f2*^, respectively.
The corresponding canonical variates then are(4.15)〈X,fj*〉,〈X,fj*^〉,j=1,2.For an alternative description of these canonical
correlations, let us introduce the
operator(4.16)R1=(αI1+S11)−1/2S12(αI2+S22)−1S21(αI1+S11)−1/2.Interchanging the indices 1 and 2 yields R2, and replacing Sjk with S^jk yields R^1 and R^2. It can be seen that all these operators are
Hilbert-Schmidt and strictly positive Hermitian. It will be assumed that Rj has the largest
eigenvalue with one-dimensional eigenspace generated by fj* with ∥fj*∥=1. Under this condition, it has been shown in Cupidon et al. [18] that(4.17)ρ2=largest eigenvalue ofRj=〈fj*,Rjfj*〉,for j=1,2. A similar result holds true for ρ^2.It is well known that the asymptotic distribution of
the eigenvalues and eigenfunctions of a random operator can be derived from the
asymptotic distribution of this random operator itself (see [10] for
Euclidean spaces and [4] for Hilbert spaces). This technique is based on the results of Section 3. In the present situation, this means that we have to show the convergence in
distribution of the suitably standardized R^j. Because all operators are Hilbert-Schmidt, it can be
shown that(4.18)ℛj=n(R^j−Rj)→d𝒢,as  n→∞, inℒ.Result (4.18) follows easily if convergence in
distribution can be established for each of the factors defining R^j, for instance,(4.19)(αI+S^jj)−1/2=φα(S^jj),where this time φα(z)=(α+z)−1/2, compare (2.29).It is known [4] that
(4.20)n{S^jj−Sjj}→d𝒢jj,as n→∞,inℒHS,for some Gaussian random element 𝒢jj, where ℒHS is the Hilbert
space of all Hilbert-Schmidt operators mapping ℍ into itself.
Writing φα,j′ for the Fréchet
derivative evaluated at Sjj (Section 2) and
exploiting the fact that the imbedding of ℒHS is ℒ are continuous,
we obtain via a kind of delta-method  [18, 19](4.21)n{φα(S^jj)−φα(Sjj)}→dφα,j′𝒢jj,asn→∞,inℒthe desired result. A
combination of results like this for each of the factors of R^j yields (4.18).


## 4.3. Solution of a Nonlinear Operator Equation

In Bakushinsky and  Kokurin [13], the following
problem is considered. Let ℍ1 and ℍ2 be Hilbert
spaces and F:ℍ1→ℍ2 an operator,
not necessarily linear. The (nonlinear) equation(4.22)F(x)=0,x∈ℍ,is studied. Let x* be a solution
of (4.22) and introduce a set Ω={x∈ℍ1:∥x−x*∥1≤r}, for some r>0. It is assumed that F is Fréchet
differentiable on Ω. If Fx′ is the
derivative at x∈Ω it is,
moreover, assumed that(4.23)∥Fx′−Fy′∥ℒ(ℍ1,ℍ2)≤L∥x−y∥1,x,y∈Ω,where 0<L<∞ is a given
number. Given an initial point x0∈Ω and a sequence {αn}, αn>0, of regularization parameters, these authors show
that, under some further conditions, the generalized Gauss-Newton method
generates a sequence of points {xn} such
that(4.24)∥xn−x*∥=𝒪(αnp),for somep≥12.In their proof of this result, the authors need a
crucial upper bound. Under some additional assumptions, we want to derive this
upper bound as an immediate consequence of Theorem2.1. In order to relate the
present problem to the setup of our paper, let us assume that ℍ1=ℍ2=ℍ, and note that(4.25)(Fx*′)*(Fx*′)=T∈ℒH.For xn, let(4.26)(Fxn′)*(Fxn′)=T˜∈ℒH,and set(4.27)T˜=T+T˜−T=T+Π,where obviously Π∈ℒH. It is not hard to see that (4.23)
entails(4.28)∥Π∥ℒ≤a∥x*−xn∥,for some 0<a<∞. Let Γ be the contour
in (2.1) and D the
corresponding domain. As in Bakushinsky and Kokurin [13], a function Θ(z,α), z∈D, is employed in the iteration scheme, which is
analytic on D.Narrowing down the generality in Bakushinsky and
Kokurin [13] somewhat further, so that the current conditions are satisfied,
their proof of the convergence of the iterations requires an upper bound for
the expression (in our notation)(4.29)12πi∮Γ{1−Θ(z,αn)}{R(z)−R˜(z)}dz=1−Θ(T,αn)−{1−Θ(T˜,αn)}=Θ(T˜,αn)−Θ(T,αn).Keeping n fixed, let us
briefly write this last expression as Θ(˜T)−Θ(T). Now Theorem 2.1 applies with φ=Θ, and application yields at once(4.30)∥Θ(T˜)−Θ(T)∥ℒ≤∥ΘT′(T˜−T)∥ℒ+𝒪(∥T˜−T∥ℒ2)≤b∥T˜−T∥ℒ+𝒪(∥T˜−T∥ℒ2)≤ab∥x*−xn∥+𝒪(∥x*−xn∥2),for some 0<b<∞, by (4.28).



---

*Source: 239025-2009-04-14.xml*