# J-Self-Adjoint Extensions for a Class of  Discrete Linear Hamiltonian Systems

Authors: Guojing Ren; Huaqing Sun
Journal: Abstract and Applied Analysis
 (2013)

---

## Abstract

This paper is concerned with formallyJ-self-adjoint discrete linear Hamiltonian systems on finite or infinite intervals. The minimal and maximal subspaces are characterized, and the defect indices of the minimal subspaces are discussed. All the J-self-adjoint subspace extensions of the minimal subspace are completely characterized in terms of the square summable solutions and boundary conditions. As a consequence, characterizations of all the J-self-adjoint subspace extensions are given in the limit point and limit circle cases.


---

## Body

## 1. Introduction

Consider the following discrete linear Hamiltonian system:(1λ)𝒥Δy(t)=(P(t)+λW(t))R(y)(t),t∈ℐ,
where ℐ:={t}t=ab, a is a finite integer or a=-∞, b is a finite integer or b=+∞, and b-a≥1; Δ is the forward difference operator, that is, Δy(t)=y(t+1)-y(t); 𝒥 is the canonical symplectic matrix, that is,
(1)𝒥=(0-InIn0),
where In is the n×n unit matrix; the weighted function W(t) is a 2n×2n real symmetric matrix with W(t)≥0 for t∈ℐ, and it is of the block diagonal form,
(2)W(t)=diag{W1(t),W2(t)},
where P(t) is a 2n×2n complex symmetric matrix, that is, PT(t)=P(t). The partial right shift operator R(y)(t)=(uT(t+1),vT(t))T with y(t)=(uT(t),vT(t))T and u(t),v(t)∈ℂn; λ is a complex spectral parameter.For briefness, denoteℐ=[a,b] in the case where a and b are finite integers; ℐ=[a,+∞) in the case where a is finite and b=+∞; ℐ=(-∞,b] in the case where a=-∞ and b is finite; ℐ=(-∞,+∞) in the case where a=-∞ and b=+∞.SinceP(t) is symmetric, it can be blocked as
(3)P(t)=(-C(t)AT(t)A(t)B(t)),
where A, B, and C are n×n complex-valued matrices with CT=C and BT=B. Then, (1λ) can be rewritten as
(2λ)(In-A(t))u(t+1)=u(t)+(B(t)+λW2(t))v(t),v(t+1)=(C(t)-λW1(t))u(t+1)+(In-AT(t))v(t),t∈ℐ.To ensure the existence and uniqueness of the solution of any initial value problem for(1λ), we always assume in the present paper that  (A1)
In-A(t) is invertible in ℐ.It can be easily verified that(1λ) contains the following complex coefficients vector difference equation of order 2m:
(3λ)∑j=0m‍(-1)jΔj[pj(t)Δjz(t-j)]=λw(t)z(t),t∈ℐ,
where pj(t) are l×l complex-valued matrices with pjT(t)=pj(t), 0≤j≤m; pm(t) is invertible in ℐ; w(t) is an l×l real-valued with w(t)≥0. In fact, by letting y=(uT,vT)T with u=(u1T,u2T,…,umT)T, v=(v1T,v2T,…,vmT)T, and
(4)uj(t)=Δj-1z(t-j),vj(t)=∑k=jm‍(-1)k-jΔk-j(pk(t)Δkz(t-k))
for 1≤j≤m, (3λ) can be converted into (1λ), as well as (2λ), with
(5)-C(t)=diag{p0(t),p1(t),…,pm-1(t)},A(t)=(0Il(m-1)00),B(t)=diag{0,…,0,pm-1(t)},W(t)=diag{w(t),0,…,0}.
It is obvious that (A1) is satisfied for (3λ).The spectral theory of self-adjoint operators and self-adjoint extensions of symmetric operators (i.e., densely defined Hermitian operators) in Hilbert spaces has been well developed (cf. [1–4]). In general, under certain definiteness conditions, a formally self-adjoint differential expression can generate a minimal operator which is symmetric, and the defect index of the minimal operator is equal to the number of linearly independent square integrable solutions. All the characterizations of self-adjoint extensions of differential equation are obtained [5–8].However, for difference equations, it was found in [9] that the minimal operator defined in [10] may be neither densely defined nor single-valued even if the definiteness condition is satisfied. This is an important difference between the differential and difference equations. In order to study the self-adjoint extensions of nondensely defined or multivalued Hermitian operators, some scholars tried to extend the concepts and theory for densely defined Hermitian operators to Hermitian subspaces [11–15]. Recently, Shi extended the Glazman-Krein-Naimark (GKN) theory for symmetric operators to Hermitian subspaces [9]. Applying this GKN theory, the first author, with Shi and Sun, gave complete characterizations of self-adjoint extensions for second-order formally self-adjoint difference equations and general linear discrete Hamiltonian systems, separately [16, 17].We note that when the coefficientP(t) in (1λ) is not a Hermitian matrix, that is, P*(t)≠P(t), system (1λ) is not formally self-adjoint, and the minimal subspace generated by (1λ) is not Hermitian. Hence the spectral theory of self-adjoint operators or self-adjoint subspaces is not applicable. To solve this problem, Glazman introduced a concept of J-symmetric operators in [3, 18] where J is an operator. The minimal operators generated by certain differential expressions are J-symmetric operators in the related Hilbert spaces [19, 20]. Monaquel and Schmidt [21] discussed the M-functions of the following discrete Hamiltonian system:
(4λ)𝒥(Δz1(t)∇z2(t))=(Q(t)+λH(t))(z1(t)z2(t)),t∈[0,+∞),
where ∇ is the backward difference operator, that is, ∇z(t)=z(t)-z(t-1), and weighted function H(t)=diag{H1(t),H2(t)}. By letting z1(t)=v(t), z2(t)=u(t+1), (1λ) can be converted into (4-λ) with
(6)Q(t)=(-B(t)-A(t)-AT(t)C(t)),H(t)=(W2(t)00W1(t)).In [22], the result that every J-Hermitian subspace has a J-self-adjoint subspace extension has been given. Furthermore, a result about J-self-adjoint subspace extension was obtained [22], which can be regarded as a GKN theorem for J-Hermitian subspaces.In the present paper, enlightened by the methods used in the study of self-adjoint subspace extensions of Hermitian subspaces, we will study theJ-self-adjoint subspace extensions of the minimal operator corresponding to system (1λ). A complete characterization of them in terms of boundary conditions is given by employing the GKN theorem for J-Hermitian subspaces. The rest of this paper is organized as follows. In Section 2, some basic concepts and useful results about subspaces are briefly recalled. In Section 3, a conjugation operator J is defined in the corresponding Hilbert space, and the maximal and minimal subspaces are discussed. In Section 4, the description of the minimal subspaces is given by the properties of their elements at the endpoints of the discussed intervals, the defect indices of minimal subspaces are discussed, and characterizations of the maximal subspaces are established. Section 5 pays attention to two characterizations of all the self-adjoint subspace extensions of the minimal subspace in terms of boundary conditions via linearly independent square summable solutions of (1λ). As a consequence, characterizations of all the self-adjoint subspace extensions are given in two special cases: the limit point and limit circle cases.


## 2. Fundamental Results on Subspaces

In this section, we recall some basic concepts and useful results about subspaces. For more results about nondensely definedJ-Hermitian operators or J-Hermitian subspaces, we refer to [17–19, 22] and some references cited therein. In addition, some properties of solutions of (1λ) and a result about matrices are given at the end of this section.Byℝ and ℂ we denote the sets of the real and the complex numbers, respectively. Let X be a complex Hilbert space equipped with inner product 〈·,·〉, T and S two linear subspaces (briefly, subspace) in X2:=X×X, and λ∈ℂ. Denote
(7)DomT≔{x∈X:(x,f)∈Tforsomef∈X},RanT:={f∈X:(x,f)∈Tforsomex∈X},KerT:={x∈X:(x,0)∈T},T*:={(x,f)∈X2:〈x,g〉=〈f,y〉∀(y,g)∈T},T-λI:={(x,f-λx):(x,f)∈T}.
If T∩S={0}, we write
(8)T∔S≔{(x+y,f+g):(x,f)∈T,(y,g)∈S},
which is denoted by T⊕S in the case that T and S are orthogonal.Denote(9)T(x):={f∈X:(x,f)∈T}.
It can be easily verified that T(0)={0} if and only if T can determine a unique linear operator from DomT into X whose graph is just T. For convenience, we will identify a linear operator in X with a subspace in X2 via its graph.Definition 1 (see [11]).
LetT be a subspace in X2. (1)
T is said to be a Hermitian subspace if T⊂T*. Furthermore, T is said to be a Hermitian operator if it is an operator, that is, T(0)={0}.(2)
T is said to be a self-adjoint subspace if T=T*. Furthermore, T is said to be a self-adjoint operator if it is an operator, that is, T(0)={0}.(3)
LetT be a Hermitian subspace. T1 is said to be a self-adjoint subspace extension (briefly, SSE) of  T if T⊂T1 and T1 is a self-adjoint subspace.(4)
LetT be a Hermitian operator. T1 is said to be a self-adjoint operator extension (briefly, SOE) of T if T⊂T1 and T1 is a self-adjoint operator.Lemma 2 (see [11]).
LetT be a subspace in X2. Then  (1)
T* is a closed subspace in X2;(2)
T*=(T¯)* and T**=T¯, where T¯ is the closure of T;(3)
KerT*=(RanT)⊥=(RanT¯)⊥.In [19], an operator J defined in X is said to be a conjugation operator if for all x,y∈X,
(10)〈Jx,Jy〉=〈y,x〉,J2x=x.Definition 3.
LetT be a subspace in X2 and J be a conjugation operator.  (1)
TheJ-adjoint of T is defined by
(11)TJ*:={(y,g)∈X2:〈f,Jy〉=〈x,Jg〉∀(x,f)∈T}.(2)
T is said to be a J-Hermitian subspace if T⊂TJ*. Furthermore, T is said to be a J-Hermitian operator if it is an operator, that is, T(0)={0}.(3)
T is said to be a J-self-adjoint subspace if T=TJ*. Furthermore, T is said to be a J-self-adjoint operator if it is an operator, that is, T(0)={0}.(4)
LetT be a J-Hermitian subspace. T1 is said to be a J-self-adjoint subspace extension (briefly, J-SSE) of T if T⊂T1 and T1 is a J-self-adjoint subspace.(5)
LetT be a J-Hermitian operator. T1 is said to be a J-self-adjoint operator extension (briefly, J-SOE) of T if T⊂T1 and T1 is a J-self-adjoint operator.Remark 4.
(1) It can be easily verified thatTJ* is a closed subspace. Consequently, a J-self-adjoint subspace T is a closed subspace since T=TJ*. In addition, SJ*⊂TJ* if T⊂S.
(2) From the definition, we have that〈f,Jy〉=〈x,Jg〉 holds for all (x,f)∈T and (y,g)∈TJ*, and that T is a J-Hermitian subspace if and only if 〈f,Jy〉=〈x,Jg〉 for all (x,f),(y,g)∈T.Lemma 5 (see [22]).
LetT be a subspace in X2. Then  (1)
T*:={(Jy,Jg):(y,g)∈TJ*};(2)
TJ*:={(Jy,Jg):(y,g)∈T*}.It follows from Lemmas2 and 5 that TJ*=(T¯)J*, and T¯ is J-Hermitian if T is J-Hermitian.Lemma 6 (see [22]).
Every J-Hermitian subspace has aJ-SSE.Definition 7.
Let T be aJ-Hermitian subspace. Then d(T)=(1/2)dimTJ*/T¯ is called to be the defect index of T.Next, we introduce a form onX2×X2 by
(12)[(x,f):(y,g)]≔〈f,Jy〉-〈x,Jg〉,(x,f),(y,g)∈X2.Lemma 8 (see [22]).
LetT be a J-Hermitian subspace. Then
(13)T¯={(x,f)∈TJ*:[(x,f):(y,g)]=0,∀(y,g)∈TJ*}.Lemma 9 (see [22]).
LetT be a closed J-Hermitian subspace in X2 and satisfy d=d(T)<+∞. Then a subspace T1 is a J-SSE of T if and only if T⊂T1⊂TJ* and there exists {(xj,fj)}j=1d⊂TJ* such that  (1)
(x1,f1),(x2,f2),…,(xd,fd) are linearly independent in TJ* (modulo T);(2)
[(xj,fj):(xk,fk)]=0, 1≤j, k≤d;(3)
T1={(y,g)∈TJ*:[(y,g):(xj,fj)]=0,1≤j≤d}.Lemma9 can be regarded as a GKN theorem for J-Hermitian subspaces. A set of {(xj,fj)}j=1d⊂TJ* which is satisfying (1) and (2) in Lemma 9 is called a GKN set of T.Definition 10.
LetT be a subspace in X2.(1)
The set(14)ρ(T):={λ∈ℂ:(T-λ)-1isaboundedlinearoperatordefinedinX(T-λ)-1}
is called the resolvent set of T.(2)
The setσ(T):=ℂ∖ρ(T) is called the spectrum of T.(3)
The set(15)Γ(T):={∥f-λx∥λ∈ℂ:∃c(λ)>0s.t.∥f-λx∥∥f-λx∥≥c(λ)∥x∥,∀(x,f)∈T}
is called to be the regularity field of T.
It is evident thatρ(T)⊂Γ(T) for any subspace T in X2.Lemma 11 (see [22]).
LetT be a J-Hermitian subspace in X2 with Γ(T)≠∅, and λ∈Γ(T). Then
(16)TJ*=T-∔{(y,g)∈TJ*:g-λy∈Ker(T*-λ-)},d(T)=dimRan(T-λ)⊥=dimKer(T*-λ-).The following is a well-known result on the rank of matrices.Lemma 12.
LetA be an m×l matrix and B an l×n matrix. Then
(17)rankA+rankB-l≤rank(AB)≤min{rankA,rankB}.
In particular, if AB=0, then
(18)rankA+rankB≤l.


## 3. Relationship between the Maximal and Minimal Subspaces

This section is divided into three subsections. In the first subsection, we define a conjugation operator in a Hilbert space. In the second subsection, we define maximal and minimal subspaces generated by(1λ) and discuss relationship between them. In the last subsection, we discuss the definiteness condition corresponding to (1λ).


### 3.1. Conjugation Operator

In this subsection, we define a conjugation operator in a Hilbert space and then discuss its properties.Sinceb and a may be finite or infinite, we introduce the following conventions for briefness: b+1 means +∞ in the case of b=+∞ and a-1 means -∞ in the case of a=-∞. Denote
(19)ℐ*:={t}t=ab+1,l(ℐ):={y:y={y(t)}t∈ℐ*⊂ℂ2n}.
For any 2n×2n Hermitian matrix W(t)≥0 defined in ℐ, we define
(20)ℒW2(ℐ):={y∈l(ℐ):∑t∈ℐR(y)*(t)W(t)R(y)(t)<+∞}
with the semiscalar product
(21)〈y,z〉:=∑t∈ℐR*(z)(t)W(t)R(y)(t).
Furthermore, denote ∥y∥:=(〈y,y〉)1/2 for y∈ℒW2(ℐ). Since the weighted function W(t) may be singular in ℐ, ∥·∥ is a seminorm. Introduce the quotient space
(22)LW2(ℐ):=ℒW2(ℐ){y∈ℒW2(ℐ):∥y∥=0}.
Then LW2(ℐ) is a Hilbert space with the inner product 〈·,·〉.For a functiony∈ℒW2(ℐ), denote by yπ the corresponding class in LW2(ℐ). And for any yπ∈LW2(ℐ), denote by y∈ℒW2(ℐ) a representative of yπ. It is evident that 〈yπ,zπ〉=〈y,z〉 for any yπ,zπ∈LW2(ℐ).For anyy∈ℒW2(ℐ), denote by J0y the conjugation of y; that is,
(23)J0y:={y-(t)}t∈ℐ*,y∈ℒW2(ℐ).
It can be easily verified that y∈ℒW2(ℐ) if and only if J0y∈ℒW-2(ℐ). Here W- is the conjugation of matrix W. Since each yπ∈LW2(ℐ) is an equivalent class, we define a operator J defined on LW2(ℐ) by
(24)J(yπ):=(J0y)π,yπ∈LW2(ℐ).
The following result is obtained.Lemma 13.
J defined by (24) is a conjugation operator defined on LW2(ℐ) if and only if W(t) is real and symmetric in ℐ.Proof.
The sufficiency is evident. Next, we consider the necessity. Assume thatJ defined by (24) is a conjugation operator in LW2(ℐ). Then for any xπ,yπ∈LW2(ℐ), it follows from 〈Jxπ,Jyπ〉=〈yπ,xπ〉 that
(25)∑t∈ℐR(x)*(t)WT(t)R(y)(t)=∑t∈ℐR(x)*(t)W(t)R(y)(t).
By the arbitrariness of xπ,yπ, one has that W(t)=WT(t). This, together with W(t)=W*(t), yields that W(t) is real. The proof is complete.For anyx,y∈l(ℐ), we denote
(26)(x,y)(t)=yT(t)𝒥x(t),t∈ℐ*,
where 𝒥 is the canonical symplectic matrix given in Section 1. In the case of b=+∞, if limt→b(x,y)(t) exists and is finite, then its limit is denoted by (x,y)(+∞). In the case of a=-∞, if limt→a(x,y)(t) exists and is finite, then its limit is denoted by (x,y)(-∞).Denote(27)τ(y)(t):=𝒥Δy(t)-P(t)R(y)(t),δ(y)(t):=𝒥Δy(t)-P¯(t)R(y)(t),
where τ and δ are called the natural difference operators corresponding to system (1λ). The following result can be easily verified, and so we omit the proof.Lemma 14.
Assume that(A1) holds. Let x,y∈l(ℐ).  (1)
δ(J0y)=J0τ(y),τ(J0y)=J0δ(y).(2)
For anys,k∈ℐ,
(28)∑t=sk[R(J0y)*(t)τ(x)(t)∑-(J0τ(y))*(t)R(x)(t)]=(x,y)(t)|sk+1.(3)
For anyλ∈ℂ, c0∈ℐ, and any two solutions x(t) and y(t) of (1λ), it follows that
(29)(x,y)(t)=(x,y)(c0),t∈ℐ*.
Moreover, letY(t,λ) be a fundamental solution of (1λ), then
(30)YT(t,λ)𝒥Y(t,λ)=YT(c0,λ)𝒥Y(c0,λ),t∈ℐ*.


### 3.2. Relationship between the Maximal and Minimal Subspaces

In this subsection, we first introduce the maximal and minimal subspaces corresponding to(1λ) and then show that the minimal subspace is J-Hermitian, and its J-adjoint subspace is just the maximal subspace.Denote(31)ℒW,02(ℐ):={y∈ℒW2(ℐ):thereexisttwointegerss,k∈ℐwiths≤ksuchthaty(t)=0fort≤sandt≥k+1ℒW2(ℐ)},
and define
(32)H(τ):={(yπ,gπ)∈(LW2(ℐ))2:∃y∈yπs.t.(y)(t)=W(t)R(g)(t)inℐ(LW2(ℐ))2},H00(τ):={(yπ,gπ)∈H(τ):∃y∈yπs.t.y∈ℒW,02(ℐ)andτ(y)(t)=W(t)R(g)(t)inℐ(yπ,gπ)(yπ,gπ)}.
It can be easily verified that H(τ) and H00(τ) are both linear subspaces in (LW2(ℐ))2. Here, H(τ) and H00(τ) are called the maximal and preminimal subspaces corresponding to τ or (1λ) in (LW2(ℐ))2, and H0(τ):=H¯00(τ) is called the minimal subspace corresponding to τ or (1λ) in (LW2(ℐ))2.Since the end pointsa and b may be finite or infinite, we need to divide ℐ into two subintervals in order to characterize the maximal and minimal subspaces in a unified form. Choose a<c0<b and fix it. Denote
(33)ℐa:={t}t=ac0-1,ℐb:={t}t=c0b,
and denote by 〈·,·〉a, 〈·,·〉b and ∥·∥a, ∥·∥b the inner products and norms of ℒW2(ℐa), ℒW2(ℐb), respectively. Let ℒW,02(ℐa) and ℒW,02(ℐb) be defined by (31) with ℐ replaced by ℐa and ℐb, respectively. Furthermore, let Ha(τ) and Ha,00(τ) be the left maximal and preminimal subspaces defined by (32) with ℐ replaced by ℐa, respectively, and Hb(τ) and Hb,00(τ) the right maximal and preminimal subspaces defined by (32) with ℐ replaced by ℐb, respectively. The subspaces Ha,0(τ):=H¯a,00(τ) and Hb,0(τ):=H¯b,00(τ) are called the left and right minimal subspaces corresponding to system (1λ) in ℐa and ℐb, respectively. Similarly, we can define H(δ), H00(δ), and H0(δ); Hb(δ), Hb,00(δ), and Hb,0(δ); Ha(δ), Ha,00(δ) and Ha,0(δ).The following result is directly derived from (1) of Lemma14.Lemma 15.
Assume that(A1) holds. Then (yπ,gπ)∈H(τ) if and only if (Jyπ,Jgπ)∈H(δ).In order to study properties of the above subspaces, we first make some preparation.LetY(t) be the fundamental solution matrix of (10) with Y(c0)=I2n. For any finite subinterval ℐ′ with c0∈ℐ′⊂ℐ, denote
(34)Φℐ′:=∑t∈ℐ′R(Y)*(t)W(t)R(Y)(t).
It is evident that Φℐ' is a 2n×2n positive semidefinite matrix and dependent on ℐ′. By the same method used in [23, Lemma 3.2], it follows that there exists a finite subinterval ℐ0 with c0∈ℐ0⊂ℐ such that
(35)rankΦℐ0=rankΦℐ′,RanΦℐ0=RanΦℐ′,
for any finite subinterval ℐ′ with ℐ0⊂ℐ′⊂ℐ. In the present paper, we will always denote ℐ0:=[s0,t0] and define
(36)rankΦℐ:=rankΦℐ0,RanΦℐ:=RanΦℐ0,
whenever ℐ is finite or infinite. In the case that ℐ is finite, ℐ0 can be taken as ℐ.In the case thatℐ is finite, we define
(37)ϕℐ:LW2(ℐ)⟶ℂ2n,gπ⟼∑s∈ℐR(J0Y)*(s)W(s)R(g)(s).
It is evident that ϕℐ is a bounded linear map and its range is a closed subset in ℂ2n.In the case thatℐ is infinite, that is, ℐ=[a,+∞) or ℐ=(-∞,b] or ℐ=(-∞,+∞), where a, b are finite integers, we introduce the following subspaces of LW2(ℐ), respectively:
(38)LW,12(ℐ):={yπ∈LW2(ℐ):∃k∈ℐs.t.W(t)R(y)(t)=0fort≥kLW2(ℐ)},(39)LW,12(ℐ):={yπ∈LW2(ℐ):∃s∈ℐs.t.W(t)R(y)(t)=0fort≤sLW2},(40)LW,12(ℐ):={yπ∈LW2(ℐ):∃s≤k∈ℐs.t.W(t)R(y)(t)=0fort≤sandt≥kLW2}.
It can be easily shown that LW,12(ℐ) is dense in LW2(ℐ). In this case, we define
(41)ϕℐ:LW,12(ℐ)⟶ℂ2n,gπ⟼∑s∈ℐ‍R(J0Y)*(s)W(s)R(g)(s).
By the method used in [23, Lemma 3.3], one has the following properties of ϕℐ.Lemma 16.
Assume that(A1) holds.(1)
Ranϕℐ=RanΦℐ.(2)
In the case thatℐ is finite,
(42)LW2(ℐ)=Kerϕℐ⊕{(J0(Yξ))π:ξ∈RanΦℐ};in the case thatℐ is infinite, let l=rankΦℐ. Then there exist linearly independent elements hjπ∈LW,12(ℐ), 1≤j≤l, such that
(43)LW,12(ℐ)=Kerϕℐ∔span{h1π,h2π,…,hlπ}.(3)
Kerϕℐ⊂Ran(H00(τ)).The following is the main result of this section.Theorem 17.
Assume that(A1) holds. Then H(τ)=(H00(τ))J*=(H0(τ))J*, Hb(τ)=(Hb,00(τ))J*=(Hb,0(τ))J*, and Ha(τ)=(Ha,00(τ))J*=(Ha,0(τ))J*.Proof.
Since the method of the proofs is similar, we only show the first assertion. By(H0(τ))J*=(H00(τ))J*, it suffices to show H(τ)=(H00(τ))J*.
We first show thatH(τ)⊂(H00(τ))J*. Let (yπ,gπ)∈H(τ). Then for any (xπ,fπ)∈H00(τ), there exists x∈xπ with x∈ℒW2(ℐ) such that τ(x)(t)=W(t)R(f)(t) in ℐ. So, it follows from (2) of Lemma 14 that
(44)〈fπ,Jyπ〉-〈xπ,Jgπ〉=∑t∈ℐ[R(J0y)*(t)W(t)R(f)(t)-R(J0g)*(t)W(t)R(x)(t)]=∑t∈ℐ[R(J0y)*(t)τ(x)(t)-(J0τ(y))*(t)R(x)(t)]=(x,y)(t)|ab+1=0.
This implies that H(τ)⊂(H00(τ))J*.
Next, we show(H00(τ))J*⊂H(τ). Fix any (yπ,gπ)∈(H00(τ))J*. It suffices to show that there exists y0∈yπ such that τ(y0)(t)=W(t)R(g)(t) in ℐ. Let z be a solution of τ(z)(t)=W(t)R(g)(t) on ℐ. For any (xπ,fπ)∈H00(τ), there exits x∈xπ with x∈ℒW,02(ℐ) such that τ(x)(t)=W(t)R(f)(t) in ℐ. Thus, it follows from (2) of Lemma 14 that
(45)=∑t∈ℐ[R(J0z)*(t)W(t)R(f)(t)-R(J0g)*(t)W(t)R(x)(t)]=∑t∈ℐ[R(J0z)*(t)τ(x)(t)-(J0τ(z))*(t))R(x)(t)]=(x,z)(t)|ab+1=0.
In addition, it is clear that
(46)〈fπ,Jyπ〉-〈xπ,Jgπ〉=∑t∈ℐ[R(J0y)*(t)W(t)R(f)(t)-R(J0g)*(t)W(t)R(x)(t)]=0.
Combining (45) and (46), one has that for all (xπ,fπ)∈H00(τ),
(47)∑t∈ℐR(J0(y-z))*(t)W(t)R(f)(t)=0.
By (2) and (3) of Lemma 16, we get that for any hπ∈Kerϕℐ and any ξ∈ℂ2n(48)∑t∈ℐR(J0(y-z-Yξ))*(t)W(t)R(h)(t)=0.
The following discussion is divided into two parts.
Case
1. ℐ is finite. It is evident that yπ,zπ∈LW2(ℐ). Then, from (2) of Lemma 16, there exists ξ0∈RanΦℐ such that (J0(y-z-Yξ0))π∈Kerϕℐ. This, together with (48), implies that (J0(y-z-Yξ0))π=0. This is equivalent to (y-z-Yξ0)π=0. Let y0(t):=z(t)+Y(t)ξ0. Then y0∈yπ and satisfies
(49)τ(y0)(t)=τ(z)(t)+τ(Yξ)(t)=W(t)R(g)(t),t∈ℐ.
Hence, (yπ,gπ)∈H(τ). Since (yπ,gπ)∈H00*(τ) is arbitrary, we have (H00(τ))J*⊂H(τ).
Case
2. ℐ is infinite. We only consider the case that ℐ=[a,+∞). For the other two cases, it can be proved with similar arguments.
LetrankΦℐ=l. With a similar argument as Case 2 of the proof of [23, Theorem 3.1], it can be shown that there exist linearly independent elements hjπ∈LW,12,1≤j≤l, and ξ0∈RanΦℐ such that (y-z-Yξ0)π∈Kerϕℐ,
(50)LW,12(ℐ)=Kerϕℐ∔span{h1π,h2π,…,hlπ},∑t∈ℐR(J0(y-z-Yξ0))*(t)W(t)R(hj)(t)=0,1≤j≤l.
Combining (48)–(50), one has that for any hπ∈LW,12(ℐ)(51)∑t∈ℐR(J0(y-z-Yξ0))*(t)W(t)R(h)(t)=0.
This implies that (y-z-Yξ0)π=0 and consequently y0:=z+Yξ0 is a representative of yπ such that τ(y0)(t)=W(t)R(g)(t). So (yπ,gπ)∈H(τ). By the arbitrariness of (yπ,gπ) one has (H00(τ))J*⊂H(τ).
The entire proof is complete.The following result is directly derived from Lemmas5 and 15, and Theorem 17.Theorem 18.
Assume that(A1) holds. Then H(δ)=(H00(τ))*, Hb(δ)=(Hb,00(τ))*, and Ha(δ)=(Ha,00(τ))*.


### 3.3. Definiteness Condition

In this subsection, we introduce the definiteness condition for(1λ), and give some important results on it. Since the proofs are similar to those given in [23], we omit the proofs.The definiteness condition for(1λ) or H(τ) is given by the following.  (A2)
There exists a finite subintervalℐ1⊂ℐ such that for any λ∈ℂ and for any nontrivial solution y(t) of (1λ), the following always holds:
(52)∑t∈ℐ1R(y)*(t)W(t)R(y)(t)>0.In particular, the definiteness condition for (3λ) can be described as there exists a finite subinterval ℐ1⊂ℐ such that for any λ∈ℂ and for any nontrivial solution z(t) of (3λ), the following always holds:
(53)∑t∈ℐ1z*(t)w(t)z(t)>0.Lemma 19.
Assume that(A1) holds. Then (A2) holds if and only if there exists a finite subinterval ℐ1⊂ℐ such that one of the following holds:  (1)
rankΦℐ1=2n;(2)
for someλ∈ℂ, every nontrivial solution y(t) of (1λ) satisfies
(54)∑t∈ℐ1R(y)*(t)W(t)R(y)(t)>0.By Lemma19, if (52) (or (53)) holds for some λ∈ℂ, then it holds for every λ∈ℂ. In addition, if (A2) holds on some finite interval ℐ1, then it holds on ℐ0=[s0,t0].The following is another sufficient and necessary condition for the definiteness condition.Lemma 20.
Assume that(A1) holds. Then (A2) holds if and only if for any (yπ,gπ)∈H(τ), there exists a unique y∈yπ such that τ(y)(t)=W(t)R(g)(t) for t∈ℐ.Remark 21.
(1) It can be easily verified that the definiteness condition forH(τ) holds if and only if that for H(δ) holds.
(2) In the following of the present paper, we always assume that(A2) holds. In this case, we can write (y,gπ)∈H(τ) instead of (yπ,gπ)∈H(τ) in the rest of the present paper.
(3) Denote by(Aa,2) and (Ab,2), the definiteness conditions for (1λ) in ℐa and ℐb, and
(55)ℐa,0:=[sa0,ta0],ℐb,0:=[sb0,tb0].
By the corresponding intervals, respectively. It is evident that one of (Aa,2) and (Ab,2) implies (A2).
But(A2) cannot imply that there exists c0∈ℐ such that both (Aa,2) and (Ab,2) hold.
(4) Several sufficient conditions for the definiteness condition can be given. The reader is referred to [23, Section 4].
For convenience, denote(56)ℳλ:={y∈ℒW2(ℐ):τ(y)(t)=λW(t)R(y)(t)fort∈ℐ},Mλ:={yπ∈LW2(ℐ):(yπ,λyπ)∈H(τ)}.Lemma 22.
Assume that(A1) holds. For any λ∈ℂ, dimℳλ=dimMλ if and only if (A2) holds.


## 3.1. Conjugation Operator

In this subsection, we define a conjugation operator in a Hilbert space and then discuss its properties.Sinceb and a may be finite or infinite, we introduce the following conventions for briefness: b+1 means +∞ in the case of b=+∞ and a-1 means -∞ in the case of a=-∞. Denote
(19)ℐ*:={t}t=ab+1,l(ℐ):={y:y={y(t)}t∈ℐ*⊂ℂ2n}.
For any 2n×2n Hermitian matrix W(t)≥0 defined in ℐ, we define
(20)ℒW2(ℐ):={y∈l(ℐ):∑t∈ℐR(y)*(t)W(t)R(y)(t)<+∞}
with the semiscalar product
(21)〈y,z〉:=∑t∈ℐR*(z)(t)W(t)R(y)(t).
Furthermore, denote ∥y∥:=(〈y,y〉)1/2 for y∈ℒW2(ℐ). Since the weighted function W(t) may be singular in ℐ, ∥·∥ is a seminorm. Introduce the quotient space
(22)LW2(ℐ):=ℒW2(ℐ){y∈ℒW2(ℐ):∥y∥=0}.
Then LW2(ℐ) is a Hilbert space with the inner product 〈·,·〉.For a functiony∈ℒW2(ℐ), denote by yπ the corresponding class in LW2(ℐ). And for any yπ∈LW2(ℐ), denote by y∈ℒW2(ℐ) a representative of yπ. It is evident that 〈yπ,zπ〉=〈y,z〉 for any yπ,zπ∈LW2(ℐ).For anyy∈ℒW2(ℐ), denote by J0y the conjugation of y; that is,
(23)J0y:={y-(t)}t∈ℐ*,y∈ℒW2(ℐ).
It can be easily verified that y∈ℒW2(ℐ) if and only if J0y∈ℒW-2(ℐ). Here W- is the conjugation of matrix W. Since each yπ∈LW2(ℐ) is an equivalent class, we define a operator J defined on LW2(ℐ) by
(24)J(yπ):=(J0y)π,yπ∈LW2(ℐ).
The following result is obtained.Lemma 13.
J defined by (24) is a conjugation operator defined on LW2(ℐ) if and only if W(t) is real and symmetric in ℐ.Proof.
The sufficiency is evident. Next, we consider the necessity. Assume thatJ defined by (24) is a conjugation operator in LW2(ℐ). Then for any xπ,yπ∈LW2(ℐ), it follows from 〈Jxπ,Jyπ〉=〈yπ,xπ〉 that
(25)∑t∈ℐR(x)*(t)WT(t)R(y)(t)=∑t∈ℐR(x)*(t)W(t)R(y)(t).
By the arbitrariness of xπ,yπ, one has that W(t)=WT(t). This, together with W(t)=W*(t), yields that W(t) is real. The proof is complete.For anyx,y∈l(ℐ), we denote
(26)(x,y)(t)=yT(t)𝒥x(t),t∈ℐ*,
where 𝒥 is the canonical symplectic matrix given in Section 1. In the case of b=+∞, if limt→b(x,y)(t) exists and is finite, then its limit is denoted by (x,y)(+∞). In the case of a=-∞, if limt→a(x,y)(t) exists and is finite, then its limit is denoted by (x,y)(-∞).Denote(27)τ(y)(t):=𝒥Δy(t)-P(t)R(y)(t),δ(y)(t):=𝒥Δy(t)-P¯(t)R(y)(t),
where τ and δ are called the natural difference operators corresponding to system (1λ). The following result can be easily verified, and so we omit the proof.Lemma 14.
Assume that(A1) holds. Let x,y∈l(ℐ).  (1)
δ(J0y)=J0τ(y),τ(J0y)=J0δ(y).(2)
For anys,k∈ℐ,
(28)∑t=sk[R(J0y)*(t)τ(x)(t)∑-(J0τ(y))*(t)R(x)(t)]=(x,y)(t)|sk+1.(3)
For anyλ∈ℂ, c0∈ℐ, and any two solutions x(t) and y(t) of (1λ), it follows that
(29)(x,y)(t)=(x,y)(c0),t∈ℐ*.
Moreover, letY(t,λ) be a fundamental solution of (1λ), then
(30)YT(t,λ)𝒥Y(t,λ)=YT(c0,λ)𝒥Y(c0,λ),t∈ℐ*.


## 3.2. Relationship between the Maximal and Minimal Subspaces

In this subsection, we first introduce the maximal and minimal subspaces corresponding to(1λ) and then show that the minimal subspace is J-Hermitian, and its J-adjoint subspace is just the maximal subspace.Denote(31)ℒW,02(ℐ):={y∈ℒW2(ℐ):thereexisttwointegerss,k∈ℐwiths≤ksuchthaty(t)=0fort≤sandt≥k+1ℒW2(ℐ)},
and define
(32)H(τ):={(yπ,gπ)∈(LW2(ℐ))2:∃y∈yπs.t.(y)(t)=W(t)R(g)(t)inℐ(LW2(ℐ))2},H00(τ):={(yπ,gπ)∈H(τ):∃y∈yπs.t.y∈ℒW,02(ℐ)andτ(y)(t)=W(t)R(g)(t)inℐ(yπ,gπ)(yπ,gπ)}.
It can be easily verified that H(τ) and H00(τ) are both linear subspaces in (LW2(ℐ))2. Here, H(τ) and H00(τ) are called the maximal and preminimal subspaces corresponding to τ or (1λ) in (LW2(ℐ))2, and H0(τ):=H¯00(τ) is called the minimal subspace corresponding to τ or (1λ) in (LW2(ℐ))2.Since the end pointsa and b may be finite or infinite, we need to divide ℐ into two subintervals in order to characterize the maximal and minimal subspaces in a unified form. Choose a<c0<b and fix it. Denote
(33)ℐa:={t}t=ac0-1,ℐb:={t}t=c0b,
and denote by 〈·,·〉a, 〈·,·〉b and ∥·∥a, ∥·∥b the inner products and norms of ℒW2(ℐa), ℒW2(ℐb), respectively. Let ℒW,02(ℐa) and ℒW,02(ℐb) be defined by (31) with ℐ replaced by ℐa and ℐb, respectively. Furthermore, let Ha(τ) and Ha,00(τ) be the left maximal and preminimal subspaces defined by (32) with ℐ replaced by ℐa, respectively, and Hb(τ) and Hb,00(τ) the right maximal and preminimal subspaces defined by (32) with ℐ replaced by ℐb, respectively. The subspaces Ha,0(τ):=H¯a,00(τ) and Hb,0(τ):=H¯b,00(τ) are called the left and right minimal subspaces corresponding to system (1λ) in ℐa and ℐb, respectively. Similarly, we can define H(δ), H00(δ), and H0(δ); Hb(δ), Hb,00(δ), and Hb,0(δ); Ha(δ), Ha,00(δ) and Ha,0(δ).The following result is directly derived from (1) of Lemma14.Lemma 15.
Assume that(A1) holds. Then (yπ,gπ)∈H(τ) if and only if (Jyπ,Jgπ)∈H(δ).In order to study properties of the above subspaces, we first make some preparation.LetY(t) be the fundamental solution matrix of (10) with Y(c0)=I2n. For any finite subinterval ℐ′ with c0∈ℐ′⊂ℐ, denote
(34)Φℐ′:=∑t∈ℐ′R(Y)*(t)W(t)R(Y)(t).
It is evident that Φℐ' is a 2n×2n positive semidefinite matrix and dependent on ℐ′. By the same method used in [23, Lemma 3.2], it follows that there exists a finite subinterval ℐ0 with c0∈ℐ0⊂ℐ such that
(35)rankΦℐ0=rankΦℐ′,RanΦℐ0=RanΦℐ′,
for any finite subinterval ℐ′ with ℐ0⊂ℐ′⊂ℐ. In the present paper, we will always denote ℐ0:=[s0,t0] and define
(36)rankΦℐ:=rankΦℐ0,RanΦℐ:=RanΦℐ0,
whenever ℐ is finite or infinite. In the case that ℐ is finite, ℐ0 can be taken as ℐ.In the case thatℐ is finite, we define
(37)ϕℐ:LW2(ℐ)⟶ℂ2n,gπ⟼∑s∈ℐR(J0Y)*(s)W(s)R(g)(s).
It is evident that ϕℐ is a bounded linear map and its range is a closed subset in ℂ2n.In the case thatℐ is infinite, that is, ℐ=[a,+∞) or ℐ=(-∞,b] or ℐ=(-∞,+∞), where a, b are finite integers, we introduce the following subspaces of LW2(ℐ), respectively:
(38)LW,12(ℐ):={yπ∈LW2(ℐ):∃k∈ℐs.t.W(t)R(y)(t)=0fort≥kLW2(ℐ)},(39)LW,12(ℐ):={yπ∈LW2(ℐ):∃s∈ℐs.t.W(t)R(y)(t)=0fort≤sLW2},(40)LW,12(ℐ):={yπ∈LW2(ℐ):∃s≤k∈ℐs.t.W(t)R(y)(t)=0fort≤sandt≥kLW2}.
It can be easily shown that LW,12(ℐ) is dense in LW2(ℐ). In this case, we define
(41)ϕℐ:LW,12(ℐ)⟶ℂ2n,gπ⟼∑s∈ℐ‍R(J0Y)*(s)W(s)R(g)(s).
By the method used in [23, Lemma 3.3], one has the following properties of ϕℐ.Lemma 16.
Assume that(A1) holds.(1)
Ranϕℐ=RanΦℐ.(2)
In the case thatℐ is finite,
(42)LW2(ℐ)=Kerϕℐ⊕{(J0(Yξ))π:ξ∈RanΦℐ};in the case thatℐ is infinite, let l=rankΦℐ. Then there exist linearly independent elements hjπ∈LW,12(ℐ), 1≤j≤l, such that
(43)LW,12(ℐ)=Kerϕℐ∔span{h1π,h2π,…,hlπ}.(3)
Kerϕℐ⊂Ran(H00(τ)).The following is the main result of this section.Theorem 17.
Assume that(A1) holds. Then H(τ)=(H00(τ))J*=(H0(τ))J*, Hb(τ)=(Hb,00(τ))J*=(Hb,0(τ))J*, and Ha(τ)=(Ha,00(τ))J*=(Ha,0(τ))J*.Proof.
Since the method of the proofs is similar, we only show the first assertion. By(H0(τ))J*=(H00(τ))J*, it suffices to show H(τ)=(H00(τ))J*.
We first show thatH(τ)⊂(H00(τ))J*. Let (yπ,gπ)∈H(τ). Then for any (xπ,fπ)∈H00(τ), there exists x∈xπ with x∈ℒW2(ℐ) such that τ(x)(t)=W(t)R(f)(t) in ℐ. So, it follows from (2) of Lemma 14 that
(44)〈fπ,Jyπ〉-〈xπ,Jgπ〉=∑t∈ℐ[R(J0y)*(t)W(t)R(f)(t)-R(J0g)*(t)W(t)R(x)(t)]=∑t∈ℐ[R(J0y)*(t)τ(x)(t)-(J0τ(y))*(t)R(x)(t)]=(x,y)(t)|ab+1=0.
This implies that H(τ)⊂(H00(τ))J*.
Next, we show(H00(τ))J*⊂H(τ). Fix any (yπ,gπ)∈(H00(τ))J*. It suffices to show that there exists y0∈yπ such that τ(y0)(t)=W(t)R(g)(t) in ℐ. Let z be a solution of τ(z)(t)=W(t)R(g)(t) on ℐ. For any (xπ,fπ)∈H00(τ), there exits x∈xπ with x∈ℒW,02(ℐ) such that τ(x)(t)=W(t)R(f)(t) in ℐ. Thus, it follows from (2) of Lemma 14 that
(45)=∑t∈ℐ[R(J0z)*(t)W(t)R(f)(t)-R(J0g)*(t)W(t)R(x)(t)]=∑t∈ℐ[R(J0z)*(t)τ(x)(t)-(J0τ(z))*(t))R(x)(t)]=(x,z)(t)|ab+1=0.
In addition, it is clear that
(46)〈fπ,Jyπ〉-〈xπ,Jgπ〉=∑t∈ℐ[R(J0y)*(t)W(t)R(f)(t)-R(J0g)*(t)W(t)R(x)(t)]=0.
Combining (45) and (46), one has that for all (xπ,fπ)∈H00(τ),
(47)∑t∈ℐR(J0(y-z))*(t)W(t)R(f)(t)=0.
By (2) and (3) of Lemma 16, we get that for any hπ∈Kerϕℐ and any ξ∈ℂ2n(48)∑t∈ℐR(J0(y-z-Yξ))*(t)W(t)R(h)(t)=0.
The following discussion is divided into two parts.
Case
1. ℐ is finite. It is evident that yπ,zπ∈LW2(ℐ). Then, from (2) of Lemma 16, there exists ξ0∈RanΦℐ such that (J0(y-z-Yξ0))π∈Kerϕℐ. This, together with (48), implies that (J0(y-z-Yξ0))π=0. This is equivalent to (y-z-Yξ0)π=0. Let y0(t):=z(t)+Y(t)ξ0. Then y0∈yπ and satisfies
(49)τ(y0)(t)=τ(z)(t)+τ(Yξ)(t)=W(t)R(g)(t),t∈ℐ.
Hence, (yπ,gπ)∈H(τ). Since (yπ,gπ)∈H00*(τ) is arbitrary, we have (H00(τ))J*⊂H(τ).
Case
2. ℐ is infinite. We only consider the case that ℐ=[a,+∞). For the other two cases, it can be proved with similar arguments.
LetrankΦℐ=l. With a similar argument as Case 2 of the proof of [23, Theorem 3.1], it can be shown that there exist linearly independent elements hjπ∈LW,12,1≤j≤l, and ξ0∈RanΦℐ such that (y-z-Yξ0)π∈Kerϕℐ,
(50)LW,12(ℐ)=Kerϕℐ∔span{h1π,h2π,…,hlπ},∑t∈ℐR(J0(y-z-Yξ0))*(t)W(t)R(hj)(t)=0,1≤j≤l.
Combining (48)–(50), one has that for any hπ∈LW,12(ℐ)(51)∑t∈ℐR(J0(y-z-Yξ0))*(t)W(t)R(h)(t)=0.
This implies that (y-z-Yξ0)π=0 and consequently y0:=z+Yξ0 is a representative of yπ such that τ(y0)(t)=W(t)R(g)(t). So (yπ,gπ)∈H(τ). By the arbitrariness of (yπ,gπ) one has (H00(τ))J*⊂H(τ).
The entire proof is complete.The following result is directly derived from Lemmas5 and 15, and Theorem 17.Theorem 18.
Assume that(A1) holds. Then H(δ)=(H00(τ))*, Hb(δ)=(Hb,00(τ))*, and Ha(δ)=(Ha,00(τ))*.


## 3.3. Definiteness Condition

In this subsection, we introduce the definiteness condition for(1λ), and give some important results on it. Since the proofs are similar to those given in [23], we omit the proofs.The definiteness condition for(1λ) or H(τ) is given by the following.  (A2)
There exists a finite subintervalℐ1⊂ℐ such that for any λ∈ℂ and for any nontrivial solution y(t) of (1λ), the following always holds:
(52)∑t∈ℐ1R(y)*(t)W(t)R(y)(t)>0.In particular, the definiteness condition for (3λ) can be described as there exists a finite subinterval ℐ1⊂ℐ such that for any λ∈ℂ and for any nontrivial solution z(t) of (3λ), the following always holds:
(53)∑t∈ℐ1z*(t)w(t)z(t)>0.Lemma 19.
Assume that(A1) holds. Then (A2) holds if and only if there exists a finite subinterval ℐ1⊂ℐ such that one of the following holds:  (1)
rankΦℐ1=2n;(2)
for someλ∈ℂ, every nontrivial solution y(t) of (1λ) satisfies
(54)∑t∈ℐ1R(y)*(t)W(t)R(y)(t)>0.By Lemma19, if (52) (or (53)) holds for some λ∈ℂ, then it holds for every λ∈ℂ. In addition, if (A2) holds on some finite interval ℐ1, then it holds on ℐ0=[s0,t0].The following is another sufficient and necessary condition for the definiteness condition.Lemma 20.
Assume that(A1) holds. Then (A2) holds if and only if for any (yπ,gπ)∈H(τ), there exists a unique y∈yπ such that τ(y)(t)=W(t)R(g)(t) for t∈ℐ.Remark 21.
(1) It can be easily verified that the definiteness condition forH(τ) holds if and only if that for H(δ) holds.
(2) In the following of the present paper, we always assume that(A2) holds. In this case, we can write (y,gπ)∈H(τ) instead of (yπ,gπ)∈H(τ) in the rest of the present paper.
(3) Denote by(Aa,2) and (Ab,2), the definiteness conditions for (1λ) in ℐa and ℐb, and
(55)ℐa,0:=[sa0,ta0],ℐb,0:=[sb0,tb0].
By the corresponding intervals, respectively. It is evident that one of (Aa,2) and (Ab,2) implies (A2).
But(A2) cannot imply that there exists c0∈ℐ such that both (Aa,2) and (Ab,2) hold.
(4) Several sufficient conditions for the definiteness condition can be given. The reader is referred to [23, Section 4].
For convenience, denote(56)ℳλ:={y∈ℒW2(ℐ):τ(y)(t)=λW(t)R(y)(t)fort∈ℐ},Mλ:={yπ∈LW2(ℐ):(yπ,λyπ)∈H(τ)}.Lemma 22.
Assume that(A1) holds. For any λ∈ℂ, dimℳλ=dimMλ if and only if (A2) holds.


## 4. Characterizations of Minimal and Maximal Subspaces and Defect Indices of Minimal Subspaces

This section is divided into three subsections. In the first subsection, we give all the characterizations of the minimal subspaces generated by(1λ) in ℐ, ℐa, and ℐb. In the second subsection, we study the defect indices of the minimal subspaces. In the third subsection, characterizations of the maximal subspaces are established.


### 4.1. Characterizations of the Minimal Subspaces

In this subsection, we study characterizations of the minimal subspaces generated by(1λ) in ℐ, ℐa, and ℐb.The following result is a direct consequence of Theorem17.Theorem 23.
Assume that(A1) holds. Then H0(τ), Hb,0(τ), and Ha,0(τ) are closed J-Hermitian subspace in (LW2(ℐ))2, (LW2(ℐb))2, and (LW2(ℐa))2, respectively.Now, we introduce boundary forms on(LW2(ℐ))2, (LW2(ℐa))2, and (LW2(ℐb))2 by
(57)[:]:(LW2(ℐ))2×(LW2(ℐ))2⟶ℂ,[(xπ,fπ):(yπ,gπ)]=〈fπ,Jyπ〉-〈xπ,Jgπ〉,[:]a:(LW2(ℐa))2×(LW2(ℐa))2⟶ℂ,[(xπ,fπ):(yπ,gπ)]a=〈fπ,Jyπ〉a-〈xπ,Jgπ〉a,[:]a:(LW2(ℐb))2×(LW2(ℐb))2⟶ℂ,[(xπ,fπ):(yπ,gπ)]b=〈fπ,Jyπ〉b-〈xπ,Jgπ〉b.Lemma 24.
Assume that(A1) holds.  (1)
If(A2) holds, then for any (x,fπ),(y,gπ)∈H(τ),
(58)[(x,fπ):(y,gπ)]=(x,y)(b+1)-(x,y)(a).(2)
If(Aa,2) holds, then for any (x,fπ),(y,gπ)∈Ha(τ),
(59)[(x,fπ):(y,gπ)]a=(x,y)(c0)-(x,y)(a).(3)
If(Ab,2) holds, then for any (x,fπ),(y,gπ)∈Hb(τ),
(60)[(x,fπ):(y,gπ)]b=(x,y)(b+1)-(x,y)(c0).Proof.
Since the proofs of (1)–(3) are similar, we only show that assertion (1) holds.
For any(x,fπ),(y,gπ)∈H(τ), we have from (2) of Lemma 14 that
(61)〈fπ,Jyπ〉-〈xπ,Jgπ〉=∑t=sk[R(J0y)*(t)W(t)R(f)(t)-R(J0g)*(t)W(t)R(x)(t)]=∑t=sk[R(J0y)*(t)τ(x)(t)-(J0τ(y))*(t)R(x)(t)]=(x,y)(t)|sk+1
for any s<k∈ℐ. This yields that limt→b(x,y)(t) exists and is finite for any (x,fπ),(y,gπ)∈H(τ). Similarly, it can be shown that limt→a(x,y)(a) exists and is finite for any (x,fπ),(y,gπ)∈H(τ). Hence, assertion (1) holds. The proof is complete.Lemma 25.
Assume that(A1) and (A2) hold. Then for any given finite subset ℐ1=[s,k] with ℐ0⊂ℐ1⊂ℐ and for any given α,β∈ℂ2n, there exists f={f(t)}t=sk+1⊂ℂ2n such that the following boundary value problem:
(62)τ(x)(t)=W(t)R(f)(t),t∈ℐ1,x(s)=α,x(k+1)=β
has a solution x={x(t)}t=sk+1⊂ℂ2n.Proof.
Set(63)〈x,y〉′:=∑t=skR(y)*(t)W(t)R(x)(t),∀x,y∈l([s,k]).
Let ϕj,1≤j≤2n, be the linearly independent solutions of system (10). Then we have
(64)rank(〈ϕi,ϕj〉′)1≤i,j≤2n=2n.
In fact, the linear algebraic system
(65)(〈ϕi,ϕj〉′)1≤i,j≤2nC=0,
where C=(c1,c2,…,c2n)T∈ℂ2n, can be written as
(66)〈ϕi,∑j=12nc-jϕj〉′=0,1≤i≤2n,
which yields
(67)〈∑j=12nc-jϕj,∑j=12nc-jϕj〉′=0.
Since ∑j=12nc-jϕj is a solution of system (10), it follows from (A2) that ∑j=12nc-jϕj=0. Then C=0; that is, (65) has only a zero solution. Consequently, (64) holds.
Letα, β be any given vectors in ℂ2n. By (64), the linear algebraic system
(68)(〈ϕi,ϕj〉′)1≤i,j≤2nTC=(ϕ1(k+1),ϕ2(k+1),…,ϕ2n(k+1))*𝒥β
has a unique solution C1∈ℂ2n. Set ψ1=(ϕ1,ϕ2,…,ϕ2n)C1. It follows from (68) that
(69)〈ψ1,ϕi〉′=ϕi*(k+1)𝒥β,1≤i≤2n.
Let u(t) be a solution of the following initial value problem:
(70)τ(x)(t)=W(t)R(ψ1)(t),t∈[s,k],x(s)=0.
Since τ(ϕi)(t)=0 for t∈ℐ and 1≤i≤2n, we get by (70) and (2) of Lemma 14 that
(71)〈ψ1,ϕi〉′=∑t=sk[R(ϕi)*(t)τ(u)(t)-τ(ϕi)*(t)R(u)(t)]=(u,J0ϕi)(t)|sk+1=ϕi*(k+1)𝒥u(k+1),1≤i≤2n.
Since ϕ1,ϕ2,…,ϕ2n are linearly independent in l(ℐ), we get from (69) and (71) that u(k+1)=β. So, u(t) is a solution of the following boundary value problem:
(72)τ(x)(t)=W(t)R(ψ1)(t),t∈[s,k],x(s)=0,x(k+1)=β.
On the other hand, the linear algebraic system(73)(〈ϕi,ϕj〉′)1≤i,j≤2nTC=(ϕ1(s),ϕ2(s),…,ϕ2n(s))*Jα
has a unique solution C2∈ℂ2n by (64). Set ψ2=(ϕ1,ϕ2,…,ϕ2n)C2. Then, by (73)
(74)〈ψ2,ϕi〉′=ϕi*(s)Jα,1≤i≤2n.
Let v(t) be a solution of the following initial value problem:
(75)τ(x)(t)=-W(t)R(ψ2)(t),x(k+1)=0,t∈[s,k].
Since τ(ϕi)(t)=0 for t∈ℐ and 1≤i≤2n, we get by (2) of Lemma 14 and (75) that
(76)〈ψ2,ϕi〉′=ϕi*(s)Jv(s),1≤i≤2n,
which, together with (74), implies that v(s)=α. So, v(t) is a solution of the following boundary value problem:
(77)τ(x)(t)=-W(t)R(ψ2)(t),t∈[s,k],x(s)=α,x(k+1)=0.
Set ψ=ψ1-ψ2 and ϕ=u+v. Then ϕ is a solution of the boundary value problem (62). The proof is complete.Remark 26.
Lemma25 is called a patch lemma. Based on Lemma 25, any two elements of H(τ) (Hb(τ), Ha(τ), resp.) can be patched up to construct another new element of H(τ) (Hb(τ), Ha(τ), resp.). In particular,  (1)
if(Ab,2) holds, we can take ℐ1=[c0,tb0], α=ei:=(0,…,0︸i-1,1,0,…,0)T, and β=0, 1≤i≤2n. Then there exist (zbi,(hπ)bi)∈Hb(τ) satisfying
(78)zbi(c0)=ei;zbi(t)=0,t≥tb0+1,1≤i≤2n;(2)
if(Aa,2) holds, we take ℐ1=[sa0,c0-1], α=0, and β=ei. Then there exist (zai,(hπ)ai)∈Ha(τ) satisfying
(79)zai(c0)=ei;zai(t)=0,t≤sa0,1≤i≤2n;(3)
if both(Ab,2) and (Aa,2) hold, then there exist (zi,hiπ)∈H(τ) satisfying
(80)zi(c0)=ei;zi(t)=0fort≤sa0,t≥tb0+1,1≤i≤2n.The above auxiliary elements (zbi,(hπ)bi), (zai,(hπ)ai), and (zi,hiπ)(1≤i≤2n) will be very useful in the sequent discussions.Theorem 27.
Assume that(A1) holds.(1)
If(A2) holds, then
(81)H0(τ)={(x,fπ)∈H(τ):(x,y)(b+1)=(x,y)(a)=0∀y∈DomH(τ)}.
In particular, if ℐ=[a,b], then
(82)H00(τ)={(x,fπ)∈H(τ):x(a)=x(b+1)=0}.(2)
If(Ab,2) holds, then
(83)Hb,0(τ)={(x,fπ)∈Hb(τ):x(c0)=0,(x,y)(b+1)=0∀y∈DomHb(τ)}.(3)
If(Aa,2) holds, then
(84)Ha,0(τ)={(x,fπ)∈Ha(τ):x(c0)=0,(x,y)(a)=0∀y∈DomHa(τ)}.Proof.
We first show that assertion (1) holds. By Lemmas8 and 24, and Theorem 17, one has
(85)H0(τ)={(x,fπ)∈H(τ):(x,y)(b+1)=(x,y)(a)∀y∈DomH(τ)}.
For convenience, denote
(86)H0(τ)={(x,fπ)∈H(τ):(x,y)(b+1)=(x,y)(a)=0∀y∈DomH(τ)}.
Clearly, H0(τ)⊂H0(τ). We now show that H0(τ)⊂H0(τ). Fix any (x,fπ)∈H0(τ). It follows from (85) that for all (y,gπ)∈H(τ),
(87)(x,y)(b+1)=(x,y)(a).
For any given (y,gπ)∈H(τ), by Remark 26 there exists (z,hπ)∈H(τ) such that
(88)z(t)=0,t≤s0;z(t)=y(t),t≥t0+1.
Thus, it follows from (87) that (x,y)(b+1)=(x,z)(b+1)=(x,z)(a)=0, and consequently (x,y)(a)=0 for all (y,gπ)∈H(τ).
In the case thatℐ=[a,b], it is clear that
(89)H00(τ)={(x,fπ)∈H(τ):x(a)=x(b+1)=0}.
So it remains to show that H0(τ)=H00(τ). It suffices to show that x(a)=x(b+1)=0 for any (x,fπ)∈H0(τ). Fix any (x,fπ)∈H0(τ), and let ℐ0=ℐ, α=ei, 1≤i≤2n, and β=0. Then by Lemma 25, there exist (yi,giπ)∈H(τ) with yi(a)=ei and yi(b+1)=0. Inserting these yi into (87) one has that x(a)=0. Similarly, one can show that x(b+1)=0. Thus H0(τ)=H00(τ). Therefore, assertion (1) has been shown.
With similar arguments, one can show that assertion (2) and (3) hold by using (78) and (79), separately. This completes the proof.


### 4.2. Defect Indices of Minimal Subspaces

In this subsection, we first give a valued range of the defect indices ofHb,0(τ) and Ha,0(τ) and then discuss the relationship among the defect indices of Hb,0(τ), Ha,0(τ), and H0(τ).For briefness, denote(90)d:=d(H0(τ)),db:=d(Hb,0(τ)),da:=d(Ha,0(τ)).
For any λ∈ℂ, let ℳb,λ, Mb,λ, ℳa,λ, and Ma,λ be defined as (56) with ℐ replaced by ℐb and ℐa, respectively.The following results are obtained.Theorem 28.
Assume that(A1) holds.(1)
If(Ab,2) holds and Γ(Hb,0(τ))≠∅, then db=dimℳb,λ for any λ∈Γ(Hb,0(τ)), and n≤db≤2n.(2)
If(Aa,2) holds and Γ(Ha,0(τ))≠∅, then da=dimℳa,λ for any λ∈Γ(Ha,0(τ)), and n≤da≤2n.Proof.
Since the method of the proofs is the same, we only give the proof of assertion (1).
For anyλ∈Γ(Hb,0(τ)), it follows from Lemma 11 and Theorem 18 that
(91)db=dimKer(Hb(δ)-λ-).
On the other hand, by using Lemma 5 and Theorems 17 and 18, one has that
(92)dimKer(Hb(τ)-λ)=dimKer(Hb(δ)-λ-).
It is clear that
(93)Mb,λ=Ker(Hb(τ)-λ).
Combining (91)–(93), one has db=dimMb,λ. This, together with Lemma 22, implies that db=dimℳb,λ. In addition, it has been shown in [21] that n≤dimℳb,λ≤2n for any λ∈ℂ. So assertion (1) is true. The proof is complete.Next, we discuss relationship among defect indices ofH0(τ), Ha,0(τ), and Hb,0(τ). For convenience, denote
(94)ya:={y(t)}t=ac0,yb:={y(t)}t=c0b+1
for any y∈ℒW2(ℐ). It is evident that ya∈ℒW2(ℐa), yb∈ℒW2(ℐb), and
(95)∥y∥2=∥ya∥a2+∥yb∥b2.
On the other hand, for any ya=(uaT,vaT)T∈ℒW2(ℐa) and yb=(ubT,vbT)T∈ℒW2(ℐb), we define y by
(96)y(t)={ya(t),t≤c0-1,(uaT(c0),vbT(c0))T,t=c0,yb(t),t≥c0+1.
Then (95) still holds, and consequently y∈ℒW2(ℐ). Furthermore, it is clear that
(97)R(y)(t)=R(ya)(t),t≤c0-1;R(y)(t)=R(yb)(t),t≥c0.The following result can be easily verified. So we omit its proof.Lemma 29.
Assume that(A1), (Aa,2), and (Ab,2) hold.  (1)
If(y,gπ)∈H(τ), then (yb,gbπ)∈Hb(τ) and (ya,gaπ)∈Ha(τ).(2)
If(ya,gaπ)∈Ha(τ) and (yb,gbπ)∈Hb(τ) with ya(c0)=yb(c0), then (y,gπ)∈H(τ).LetH^0(τ) be the restriction of subspace H0(τ), defined by
(98)H^0(τ)={(y,gπ)∈H0(τ):y(c0)=0}.Lemma 30.
Assume that(A1), (Aa,2), and (Ab,2) hold. Then  (1)
(y,gπ)∈H^0(τ) if and only if (ya,gaπ)∈Ha,0(τ) and (yb,gbπ)∈Hb,0(τ);(2)
(y,gπ)∈(H^0(τ))J* if and only if (ya,gaπ)∈Ha(τ) and (yb,gbπ)∈Hb(τ).Proof.
(1) We first consider the necessity. Fix any(y,gπ)∈H^0(τ). Let ya, yb, ga, and gb be defined as (94). Then ya(c0)=yb(c0)=0. By (1) of Lemma 29 one has that (ya,gaπ)∈Ha(τ),(yb,gbπ)∈Hb(τ). In addition, for any (xa,faπ)∈Ha(τ), it follows from Remark 26 that there exits (z,hπ)∈H(τ) with
(99)z(t)=xa(t),t≤c0,z(t)=0,t≥tb0+1.
So one has
(100)(ya,xa)(a)=(y,z)(a)=0,
which implies that (ya,gaπ)∈Ha,0(τ) by the arbitrariness of (xa,faπ)∈Ha(τ) and (3) of Theorem 27. With a similar argument, one can show (yb,gbπ)∈Hb,0(τ).
Next, we consider the sufficiency. Fix any(ya,gaπ)∈Ha,0(τ) and (yb,gbπ)∈Hb,0(τ). Let y and g be defined by (96). By (2) and (3) of Theorem 27 one has that ya(c0)=yb(c0)=0. So y(c0)=0. It follows from (2) of Lemma 29 that (y,gπ)∈H(τ). For any (x,fπ)∈H(τ), it follows from (1) of Lemma 29 that (xa,faπ)∈Ha(τ) and (xb,fbπ)∈Hb(τ). Thus one has by (2) and (3) of Theorem 27 that
(101)(y,x)(a)=(ya,xa)(a)=0,(y,x)(b+1)=(yb,xb)(b+1)=0.
This implies that (y,gπ)∈H0(τ) by (1) of Theorem 27, and consequently, (y,gπ)∈H^0(τ).
(2) We first consider the necessity. Fix any(y,gπ)∈(H^0(τ))J*. Let ya, yb, ga, and gb be defined as (94). Then (ya,gaπ)∈(LW2(ℐa))2 and (yb,gbπ)∈(LW2(ℐb))2.
Setxb(t)=fb(t)=0 for t∈ℐb*, where ℐb* is defined by (19) with ℐ replaced by ℐb. It is clear that (xb,fbπ)∈Hb,0(τ). For any (xa,faπ)∈Ha,0(τ), let x and f be defined by (96). Then by the above result (1) one has that (x,fπ)∈H^0(τ). It follows that
(102)0=〈f,Jy〉-〈x,Jg〉=〈ga,Jya〉a-〈xa,Jga〉a.
By the arbitrariness of (xa,faπ)∈Ha,0(τ), one has that (ya,gaπ)∈(Ha,0(τ))J*=Ha(τ) by Theorem 17. With a similar argument one can show (yb,gbπ)∈Hb(τ).
Next, we consider the sufficiency. Fix any(ya,gaπ)∈Ha(τ) and (yb,gbπ)∈Hb(τ). Let y and g be defined by (96). For any (x,fπ)∈H^0(τ), it follows from the above result (1) that (xa,faπ)∈Ha,0(τ) and (xb,fbπ)∈Hb,0(τ). So one has by Theorem 17 that
(103)〈f,J0y〉-〈x,J0g〉=〈ga,J0ya〉a-〈xa,J0ga〉a+〈gb,J0yb〉b-〈xb,J0gb〉b=0.
This yields that (y,gπ)∈(H^0(τ))J*. The whole proof is complete.Lemma 31.
Assume that(A1), (Aa,2), and (Ab,2) hold. Then Γ(H0(τ))⊂Γ(H^0(τ)) and Γ(Hb,0(τ))∩Γ(Ha,0(τ))=Γ(H^0(τ)).Proof.
The first assertion holds becauseH^0(τ)⊂H0(τ), and the second assertion can be proved by (1) of Lemma 30 and (95). The proof is complete.In the following of the present paper, we assume that(A3)
Γ(H0(τ))≠∅.By Definition10, one has that if Γ(H0(τ))=∅, then ρ(H0(τ))=∅, and consequently σ(H0(τ))=ℂ. We do not consider this case in the present paper.Theorem 32.
Assume that(A1), (Aa,2), (Ab,2), and (A3) hold. Then
(104)d=da+db-2n.Proof.
The proof is divided into two steps.
Step  1. We show that
(105)d(H^0(τ))=da+db.
It follows fromΓ(H0(τ))≠∅ and Lemma 31 that Γ(Hb,0(τ))∩Γ(Ha,0(τ))≠∅. This, together with (Aa,2), (Ab,2), and Theorem 28, implies that for any λ∈Γ(Hb,0(τ))∩Γ(Ha,0(τ)), (1λ) has just db linearly independent solutions ybj∈ℒW2(ℐb), 1≤j≤db, and (1λ) has just da linearly independent solutions xaj∈ℒW2(ℐa), 1≤j≤da; that is, (ybj,λ(ybj)π)∈Hb(τ) for 1≤j≤db and (xaj,λ(xaj)π)∈Ha(τ) for 1≤j≤da.
Setxbj(t)=0 for t∈ℐb*. It is clear that (xbj,λ(xbj)π)∈Hb(τ) for 1≤j≤da. Let xj, 1≤j≤da, be defined by (96). Then it follows from (2) of Lemma 30 that (xj,λxjπ)∈(H^0(τ))J*. Similarly, set yaj(t)=0 for t∈ℐa*. Then one has (yj,λyjπ)∈(H^0(τ))J* for 1≤j≤db. It is evident that x1,…,xda,y1,…,ydb∈Dom(H^0(τ))J* are linearly independent.
On the other hand, for any(y,λyπ)∈(H^0(τ))J*, it follows from (2) of Lemma 31 that (ya,λyaπ)∈Ha(τ) and (yb,λybπ)∈Hb(τ); that is, ya∈ℒW2(ℐa) is a solution of (1λ) in ℐa, and yb∈ℒW2(ℐb) is a solution of (1λ) in ℐb. Therefore, there exist unique cj∈ℂ(1≤j≤da) and dj∈ℂ(1≤j≤db) such that
(106)ya=∑j=1dacjxaj,yb=∑j=1dbdjybj.
Noting the constructions of xj and yj by (96), one has that
(107)y=∑j=1dacjxj+∑j=1dbdjyj.
This, together with Lemma 11, implies that (105) holds.
Step  2. We show that
(108)d=d(H^0(τ))-2n.
It is evident that(zi,hiπ)∈H0(τ), 1≤i≤2n, where (zi,hiπ) are defined by (80). We claim that
(109)H0(τ)=H^0(τ)∔span{(z1,h1π),(z2,h2π),…,(z2n,h2nπ)}.
In fact, for each (y,gπ)∈H0(τ), set C0=(c1,c2,…,c2n)T=y(c0). Let
(110)y0=y-∑i=12ncizi,g0=g-∑i=12ncihi.
Then
(111)(y,gπ)=(y0,g0π)+∑i=12nci(zi,hiπ),(y0,g0π)∈H^0(τ).
It can be easily verified that this decomposition is unique. Hence, (109) holds.
For anyλ∈Γ(H0(τ)), we claim that
(112)Ran(H0(τ)-λI)=Ran(H^0(τ)-λI)∔span{h1π-λz1π,h2π-λz2π,…,h2nπ-λz2nπ}.
Since (109) holds, it suffices to show that
(113)Ran(H^0(τ)-λI)∩span{h1π-λz1π,h2π-λz2π,…,h2nπ-λz2nπ}={0}.
Suppose that (y,gπ)∈H^0(τ) and c1,c2,…,c2n∈ℂ satisfy
(114)gπ-λyπ+∑j=12ncj(hjπ-λzjπ)=0,
that is,
(115)gπ+∑j=12ncjhjπ=λ(yπ+∑j=12ncjzjπ).
Since (y,gπ)∈H^0(τ) and (zi,hiπ)∈H0(τ), it follows that (y+∑j=12n‍cjzj,λ(yπ+∑j=12n‍cjzjπ))∈H0(τ). Since λ∈Γ(H0(τ)), it yields
(116)y+∑j=12ncjzj=0,gπ+∑j=12ncjhjπ=0,
which, together with (109), implies that cj=0 for 1≤j≤2n, and consequently gπ=yπ=0. This yields that (113) holds.
SinceH0(τ) and H^0(τ) are closed J-Hermitian subspaces, it follows that Ran(H0(τ)-λ) and Ran(H^0(τ)-λ) are closed subspaces in LW2(ℐ), respectively. Hence, there exists a closed subspace Q in LW2(ℐ) such that
(117)Ran(H0(τ)-λI)=Ran(H^0(τ)-λI)⊕Q.
In addition, again by the fact that λ∈Γ(Hb,0(τ))∩Γ(Ha,0(τ)), it follows that {hjπ-λzjπ}j=12n are linearly independent in LW2(ℐ). It follows from (113) that dimQ=2n. Consequently,
(118)dim(Ran(H0(τ)-λI)⊥)=dim(Ran(H^0(τ)-λI)⊥)-2n.
This yields that (108) holds. It follows from (105) and (108) that (104) holds. The proof is complete.Remark 33.
Theorem32 (formula (104)) generalizes the classical result for 2nth order ordinary differential equations that go back to the classical work by Akhiezer and Glazman [1, Theorem 3 in Appendix 2]. To the case of symmetric Hamiltonian systems, formula (104) was extended in [15].
So it follows from Theorems28 and 32 that d=0 if and only if da=db=n, and d=2n if and only if da=db=2n. The following definition is obtained.Definition 34.
Assume that(A1), (A2), and (Ab,2) hold. Then (1λ) is said to be in the limit db case at t=b. In the special case that db=n, (1λ) is said to be in the limit point case (l.p.c.) at t=b, and in the other special case that db=2n, (1λ) is said to be in the limit circle case (l.c.c.) at t=b.
The same definition can be given att=a provided that (Aa,2) holds.


### 4.3. Characterizations ofHa(τ) and Hb(τ)

In this subsection, we characterize the maximal subspacesHa(τ) and Hb(τ). We first consider characterization of Hb(τ). Assume that (Ab,2) holds and Γ(Hb,0(τ))≠∅. Let λ∈Γ(Hb,0(τ)). It follows from the proof of Lemma 11 that
(119)Hb(τ)=Hb,0(τ)∔U1∔U2,
where U1={(xbj,λ(xbj)π)∈Hb(τ):1≤j≤db} and U2={(ybj,(gbj)π)∈Hb(τ):1≤j≤db}, and {(ybj,(gbj)π)}j=1db are linearly independent (mod U1). For convenience, denote
(120)χbi(t):=xbi(t),χbdb+j(t):=ybj(t),1≤i≤db,1≤j≤db.
Clearly, χbi∈DomHb(τ), 1≤i≤2db. So (χbi,χbj)(b+1) is finite for all 1≤i, j≤2db. Then the following result can be directly derived from (119) and (120).Lemma 35.
Assume that(A1) and (Ab,2) hold, and Γ(Hb,0(τ))≠∅. Then every y∈DomHb(τ) can be expressed as
(121)y=yb0+∑i=12dbaiχbi,
where yb0∈DomHb,0(τ) and ai∈ℂ.By Lemma35, zbi, defined by (78), can be uniquely expressed as
(122)zbi=ybi,0+∑j=12dbaijχbj,1≤i≤2n,
where ybi,0∈DomHb,0(τ) and aij∈ℂ. Denote
(123)E:=(aij)2n×2db,F1:=((χbi,χbj)(b+1))1≤i,j≤2db.Lemma 36.
Assume that(A1) and (Ab,2) hold, and Γ(Hb,0(τ))≠∅. Then rankE=2n and rankF1=2db-2n. Furthermore, we can rearrange the order of χb1,χb2,…,χbdb such that
(124)rank((χbi,χbj)(b+1))1≤i≤2db-2n,1≤j≤db=2db-2n.Proof.
It follows from (122) that
(125)(zb1,zb2,…,zb2n)=(yb1,0,yb2,0,…,yb2n,0)+(χb1,χb2,…,χb2db)ET,
which, together with (78), implies that
(126)I2n=(χb1(c0),χb2(c0),…,χb2db(c0))ET.
By Lemma 12, one has rankE=2n.
On the other hand, it follows from (122) that
(127)(zbi,χbs)(b+1)=(ybi,0,χbs)(b+1)+∑j=12dbaij(χjb,χbs)(b+1),
for 1≤i≤2n, 1≤s≤2db, which, together with (78) and (2) of Theorem 27, implies that
(128)EF1=0.
Noting that rankE=2n, one has
(129)rankF1≤2db-2n.
We now want to show rankF1≥2db-2n. By (3) of Lemma 14 one has
(130)F2:=((χbi,χbj)(b+1))1≤i≤db,1≤j≤db=(X0T𝒥X0)T,
where X0:=(xb1(c0),xb2(c0),…,xbdb(c0)). Since rankX0=db, we have that
(131)rankF1≥rankF2≥2db-2n,
which, together with (129), yields that
(132)rankF1=rankF2=2db-2n.
Because rankF2=2db-2n≤db, one can rearrange the order of χb1,χb2,…,χbdb such that the first 2db-2n rows of F2 are linearly independent; that is, (124) holds. The proof is complete.
Without loss of generality, we assume that (124) holds in the rest of this paper. Now, we can give a characterization of Hb(τ).Theorem 37.
Assume that(A1) and (Ab,2) hold, λ∈Γ(Hb,0(τ))≠∅, and χb1,χb2,…,χb2db-2n are linearly independent solutions of (1λ) in ℒw2(ℐb) such that (124) holds. Then
(133)Gb:=((χbi,χbj)(b+1))1≤i,j≤2db-2n
is invertible, and any y∈DomHb(τ) can be uniquely expressed as
(134)y=yb0+∑i=12ncizbi+∑j=12db-2ndjχbj,
where yb0∈DomHb,0(τ), zbi are defined by (78), and ci,dj∈ℂ.Proof.
LetE=(E1,E2), where E1 and E2 are 2n×(2db-2n) and 2n×2n matrices, respectively. It follows from (124) that there exists an invertible matrix L such that
(135)F1L=(I2db-2n0F3F4).
So, it follows from (128) that E1+E2F3=0, which is equivalent to E1=-E2F3. Since rankE=2n, it follows that E2 is invertible. Multiplying (125) by (E2-1)T from the right-hand side, we get
(136)(zb1,zb2,…,zb2n)(E2-1)T=(yb1,0,yb2,0,…,yb2n,0)(E2-1)T+(χb1,χb2,…,χb2db-2n)(E2-1E1)T+(χb2db-2n+1,…,χb2db).
This implies that each of χb2db-2n+1,…,χb2db can be uniquely expressed as a linear combination of χb1,χb2,…,χb2db-2n, yb1,0,yb2,0,…,yb2n,0, and zb1,zb2,…,zb2n. Therefore, (134) follows from Lemma 35.
Sinceχbj∈DomHb(τ) for 1≤j≤db, χbj can be uniquely expressed as
(137)χbj=ybj,0+∑l=12nbjlzbl+∑s=12db-2ncjsχbs,1≤j≤db,
where ybj,0∈DomHb,0(τ), and bjl,cjs∈ℂ. This, together with (78) and (2) of Theorem 27, implies that for 1≤i≤2db-2n,1≤j≤db(138)(χbi,χbj)(b+1)=∑s=12db-2ncjs(χbi,χsb)(b+1),
that is,
(139)(χbi,χbj)(b+1)1≤i≤2db-2n,1≤j≤db=Gb(cij)1≤i≤db,1≤j≤2db-2nT.
Therefore, Gb is invertible from (124). This completes the proof.With a similar argument, one can obtain the following characterization ofHa(τ).Theorem 38.
Assume that(A1) and (Aa,2) hold, Γ(Ha,0(τ))≠∅. Then, for some λ∈Γ(Ha,0(τ)), system (1λ) has 2da-2n linearly independent solutions χa1,…,χa2da-2n in ℒW2(ℐa) such that Ga is invertible, where
(140)Ga:=((χai,χaj)(a))1≤i,j≤2da-2n,
and each y∈DomHa(τ) can be uniquely expressed as
(141)y=ya0+∑i=12ncizai+∑j=12da-2ndjχaj,
where ya0∈DomHa,0(τ), ci,dj∈ℂ, and zai are defined as (79).


## 4.1. Characterizations of the Minimal Subspaces

In this subsection, we study characterizations of the minimal subspaces generated by(1λ) in ℐ, ℐa, and ℐb.The following result is a direct consequence of Theorem17.Theorem 23.
Assume that(A1) holds. Then H0(τ), Hb,0(τ), and Ha,0(τ) are closed J-Hermitian subspace in (LW2(ℐ))2, (LW2(ℐb))2, and (LW2(ℐa))2, respectively.Now, we introduce boundary forms on(LW2(ℐ))2, (LW2(ℐa))2, and (LW2(ℐb))2 by
(57)[:]:(LW2(ℐ))2×(LW2(ℐ))2⟶ℂ,[(xπ,fπ):(yπ,gπ)]=〈fπ,Jyπ〉-〈xπ,Jgπ〉,[:]a:(LW2(ℐa))2×(LW2(ℐa))2⟶ℂ,[(xπ,fπ):(yπ,gπ)]a=〈fπ,Jyπ〉a-〈xπ,Jgπ〉a,[:]a:(LW2(ℐb))2×(LW2(ℐb))2⟶ℂ,[(xπ,fπ):(yπ,gπ)]b=〈fπ,Jyπ〉b-〈xπ,Jgπ〉b.Lemma 24.
Assume that(A1) holds.  (1)
If(A2) holds, then for any (x,fπ),(y,gπ)∈H(τ),
(58)[(x,fπ):(y,gπ)]=(x,y)(b+1)-(x,y)(a).(2)
If(Aa,2) holds, then for any (x,fπ),(y,gπ)∈Ha(τ),
(59)[(x,fπ):(y,gπ)]a=(x,y)(c0)-(x,y)(a).(3)
If(Ab,2) holds, then for any (x,fπ),(y,gπ)∈Hb(τ),
(60)[(x,fπ):(y,gπ)]b=(x,y)(b+1)-(x,y)(c0).Proof.
Since the proofs of (1)–(3) are similar, we only show that assertion (1) holds.
For any(x,fπ),(y,gπ)∈H(τ), we have from (2) of Lemma 14 that
(61)〈fπ,Jyπ〉-〈xπ,Jgπ〉=∑t=sk[R(J0y)*(t)W(t)R(f)(t)-R(J0g)*(t)W(t)R(x)(t)]=∑t=sk[R(J0y)*(t)τ(x)(t)-(J0τ(y))*(t)R(x)(t)]=(x,y)(t)|sk+1
for any s<k∈ℐ. This yields that limt→b(x,y)(t) exists and is finite for any (x,fπ),(y,gπ)∈H(τ). Similarly, it can be shown that limt→a(x,y)(a) exists and is finite for any (x,fπ),(y,gπ)∈H(τ). Hence, assertion (1) holds. The proof is complete.Lemma 25.
Assume that(A1) and (A2) hold. Then for any given finite subset ℐ1=[s,k] with ℐ0⊂ℐ1⊂ℐ and for any given α,β∈ℂ2n, there exists f={f(t)}t=sk+1⊂ℂ2n such that the following boundary value problem:
(62)τ(x)(t)=W(t)R(f)(t),t∈ℐ1,x(s)=α,x(k+1)=β
has a solution x={x(t)}t=sk+1⊂ℂ2n.Proof.
Set(63)〈x,y〉′:=∑t=skR(y)*(t)W(t)R(x)(t),∀x,y∈l([s,k]).
Let ϕj,1≤j≤2n, be the linearly independent solutions of system (10). Then we have
(64)rank(〈ϕi,ϕj〉′)1≤i,j≤2n=2n.
In fact, the linear algebraic system
(65)(〈ϕi,ϕj〉′)1≤i,j≤2nC=0,
where C=(c1,c2,…,c2n)T∈ℂ2n, can be written as
(66)〈ϕi,∑j=12nc-jϕj〉′=0,1≤i≤2n,
which yields
(67)〈∑j=12nc-jϕj,∑j=12nc-jϕj〉′=0.
Since ∑j=12nc-jϕj is a solution of system (10), it follows from (A2) that ∑j=12nc-jϕj=0. Then C=0; that is, (65) has only a zero solution. Consequently, (64) holds.
Letα, β be any given vectors in ℂ2n. By (64), the linear algebraic system
(68)(〈ϕi,ϕj〉′)1≤i,j≤2nTC=(ϕ1(k+1),ϕ2(k+1),…,ϕ2n(k+1))*𝒥β
has a unique solution C1∈ℂ2n. Set ψ1=(ϕ1,ϕ2,…,ϕ2n)C1. It follows from (68) that
(69)〈ψ1,ϕi〉′=ϕi*(k+1)𝒥β,1≤i≤2n.
Let u(t) be a solution of the following initial value problem:
(70)τ(x)(t)=W(t)R(ψ1)(t),t∈[s,k],x(s)=0.
Since τ(ϕi)(t)=0 for t∈ℐ and 1≤i≤2n, we get by (70) and (2) of Lemma 14 that
(71)〈ψ1,ϕi〉′=∑t=sk[R(ϕi)*(t)τ(u)(t)-τ(ϕi)*(t)R(u)(t)]=(u,J0ϕi)(t)|sk+1=ϕi*(k+1)𝒥u(k+1),1≤i≤2n.
Since ϕ1,ϕ2,…,ϕ2n are linearly independent in l(ℐ), we get from (69) and (71) that u(k+1)=β. So, u(t) is a solution of the following boundary value problem:
(72)τ(x)(t)=W(t)R(ψ1)(t),t∈[s,k],x(s)=0,x(k+1)=β.
On the other hand, the linear algebraic system(73)(〈ϕi,ϕj〉′)1≤i,j≤2nTC=(ϕ1(s),ϕ2(s),…,ϕ2n(s))*Jα
has a unique solution C2∈ℂ2n by (64). Set ψ2=(ϕ1,ϕ2,…,ϕ2n)C2. Then, by (73)
(74)〈ψ2,ϕi〉′=ϕi*(s)Jα,1≤i≤2n.
Let v(t) be a solution of the following initial value problem:
(75)τ(x)(t)=-W(t)R(ψ2)(t),x(k+1)=0,t∈[s,k].
Since τ(ϕi)(t)=0 for t∈ℐ and 1≤i≤2n, we get by (2) of Lemma 14 and (75) that
(76)〈ψ2,ϕi〉′=ϕi*(s)Jv(s),1≤i≤2n,
which, together with (74), implies that v(s)=α. So, v(t) is a solution of the following boundary value problem:
(77)τ(x)(t)=-W(t)R(ψ2)(t),t∈[s,k],x(s)=α,x(k+1)=0.
Set ψ=ψ1-ψ2 and ϕ=u+v. Then ϕ is a solution of the boundary value problem (62). The proof is complete.Remark 26.
Lemma25 is called a patch lemma. Based on Lemma 25, any two elements of H(τ) (Hb(τ), Ha(τ), resp.) can be patched up to construct another new element of H(τ) (Hb(τ), Ha(τ), resp.). In particular,  (1)
if(Ab,2) holds, we can take ℐ1=[c0,tb0], α=ei:=(0,…,0︸i-1,1,0,…,0)T, and β=0, 1≤i≤2n. Then there exist (zbi,(hπ)bi)∈Hb(τ) satisfying
(78)zbi(c0)=ei;zbi(t)=0,t≥tb0+1,1≤i≤2n;(2)
if(Aa,2) holds, we take ℐ1=[sa0,c0-1], α=0, and β=ei. Then there exist (zai,(hπ)ai)∈Ha(τ) satisfying
(79)zai(c0)=ei;zai(t)=0,t≤sa0,1≤i≤2n;(3)
if both(Ab,2) and (Aa,2) hold, then there exist (zi,hiπ)∈H(τ) satisfying
(80)zi(c0)=ei;zi(t)=0fort≤sa0,t≥tb0+1,1≤i≤2n.The above auxiliary elements (zbi,(hπ)bi), (zai,(hπ)ai), and (zi,hiπ)(1≤i≤2n) will be very useful in the sequent discussions.Theorem 27.
Assume that(A1) holds.(1)
If(A2) holds, then
(81)H0(τ)={(x,fπ)∈H(τ):(x,y)(b+1)=(x,y)(a)=0∀y∈DomH(τ)}.
In particular, if ℐ=[a,b], then
(82)H00(τ)={(x,fπ)∈H(τ):x(a)=x(b+1)=0}.(2)
If(Ab,2) holds, then
(83)Hb,0(τ)={(x,fπ)∈Hb(τ):x(c0)=0,(x,y)(b+1)=0∀y∈DomHb(τ)}.(3)
If(Aa,2) holds, then
(84)Ha,0(τ)={(x,fπ)∈Ha(τ):x(c0)=0,(x,y)(a)=0∀y∈DomHa(τ)}.Proof.
We first show that assertion (1) holds. By Lemmas8 and 24, and Theorem 17, one has
(85)H0(τ)={(x,fπ)∈H(τ):(x,y)(b+1)=(x,y)(a)∀y∈DomH(τ)}.
For convenience, denote
(86)H0(τ)={(x,fπ)∈H(τ):(x,y)(b+1)=(x,y)(a)=0∀y∈DomH(τ)}.
Clearly, H0(τ)⊂H0(τ). We now show that H0(τ)⊂H0(τ). Fix any (x,fπ)∈H0(τ). It follows from (85) that for all (y,gπ)∈H(τ),
(87)(x,y)(b+1)=(x,y)(a).
For any given (y,gπ)∈H(τ), by Remark 26 there exists (z,hπ)∈H(τ) such that
(88)z(t)=0,t≤s0;z(t)=y(t),t≥t0+1.
Thus, it follows from (87) that (x,y)(b+1)=(x,z)(b+1)=(x,z)(a)=0, and consequently (x,y)(a)=0 for all (y,gπ)∈H(τ).
In the case thatℐ=[a,b], it is clear that
(89)H00(τ)={(x,fπ)∈H(τ):x(a)=x(b+1)=0}.
So it remains to show that H0(τ)=H00(τ). It suffices to show that x(a)=x(b+1)=0 for any (x,fπ)∈H0(τ). Fix any (x,fπ)∈H0(τ), and let ℐ0=ℐ, α=ei, 1≤i≤2n, and β=0. Then by Lemma 25, there exist (yi,giπ)∈H(τ) with yi(a)=ei and yi(b+1)=0. Inserting these yi into (87) one has that x(a)=0. Similarly, one can show that x(b+1)=0. Thus H0(τ)=H00(τ). Therefore, assertion (1) has been shown.
With similar arguments, one can show that assertion (2) and (3) hold by using (78) and (79), separately. This completes the proof.


## 4.2. Defect Indices of Minimal Subspaces

In this subsection, we first give a valued range of the defect indices ofHb,0(τ) and Ha,0(τ) and then discuss the relationship among the defect indices of Hb,0(τ), Ha,0(τ), and H0(τ).For briefness, denote(90)d:=d(H0(τ)),db:=d(Hb,0(τ)),da:=d(Ha,0(τ)).
For any λ∈ℂ, let ℳb,λ, Mb,λ, ℳa,λ, and Ma,λ be defined as (56) with ℐ replaced by ℐb and ℐa, respectively.The following results are obtained.Theorem 28.
Assume that(A1) holds.(1)
If(Ab,2) holds and Γ(Hb,0(τ))≠∅, then db=dimℳb,λ for any λ∈Γ(Hb,0(τ)), and n≤db≤2n.(2)
If(Aa,2) holds and Γ(Ha,0(τ))≠∅, then da=dimℳa,λ for any λ∈Γ(Ha,0(τ)), and n≤da≤2n.Proof.
Since the method of the proofs is the same, we only give the proof of assertion (1).
For anyλ∈Γ(Hb,0(τ)), it follows from Lemma 11 and Theorem 18 that
(91)db=dimKer(Hb(δ)-λ-).
On the other hand, by using Lemma 5 and Theorems 17 and 18, one has that
(92)dimKer(Hb(τ)-λ)=dimKer(Hb(δ)-λ-).
It is clear that
(93)Mb,λ=Ker(Hb(τ)-λ).
Combining (91)–(93), one has db=dimMb,λ. This, together with Lemma 22, implies that db=dimℳb,λ. In addition, it has been shown in [21] that n≤dimℳb,λ≤2n for any λ∈ℂ. So assertion (1) is true. The proof is complete.Next, we discuss relationship among defect indices ofH0(τ), Ha,0(τ), and Hb,0(τ). For convenience, denote
(94)ya:={y(t)}t=ac0,yb:={y(t)}t=c0b+1
for any y∈ℒW2(ℐ). It is evident that ya∈ℒW2(ℐa), yb∈ℒW2(ℐb), and
(95)∥y∥2=∥ya∥a2+∥yb∥b2.
On the other hand, for any ya=(uaT,vaT)T∈ℒW2(ℐa) and yb=(ubT,vbT)T∈ℒW2(ℐb), we define y by
(96)y(t)={ya(t),t≤c0-1,(uaT(c0),vbT(c0))T,t=c0,yb(t),t≥c0+1.
Then (95) still holds, and consequently y∈ℒW2(ℐ). Furthermore, it is clear that
(97)R(y)(t)=R(ya)(t),t≤c0-1;R(y)(t)=R(yb)(t),t≥c0.The following result can be easily verified. So we omit its proof.Lemma 29.
Assume that(A1), (Aa,2), and (Ab,2) hold.  (1)
If(y,gπ)∈H(τ), then (yb,gbπ)∈Hb(τ) and (ya,gaπ)∈Ha(τ).(2)
If(ya,gaπ)∈Ha(τ) and (yb,gbπ)∈Hb(τ) with ya(c0)=yb(c0), then (y,gπ)∈H(τ).LetH^0(τ) be the restriction of subspace H0(τ), defined by
(98)H^0(τ)={(y,gπ)∈H0(τ):y(c0)=0}.Lemma 30.
Assume that(A1), (Aa,2), and (Ab,2) hold. Then  (1)
(y,gπ)∈H^0(τ) if and only if (ya,gaπ)∈Ha,0(τ) and (yb,gbπ)∈Hb,0(τ);(2)
(y,gπ)∈(H^0(τ))J* if and only if (ya,gaπ)∈Ha(τ) and (yb,gbπ)∈Hb(τ).Proof.
(1) We first consider the necessity. Fix any(y,gπ)∈H^0(τ). Let ya, yb, ga, and gb be defined as (94). Then ya(c0)=yb(c0)=0. By (1) of Lemma 29 one has that (ya,gaπ)∈Ha(τ),(yb,gbπ)∈Hb(τ). In addition, for any (xa,faπ)∈Ha(τ), it follows from Remark 26 that there exits (z,hπ)∈H(τ) with
(99)z(t)=xa(t),t≤c0,z(t)=0,t≥tb0+1.
So one has
(100)(ya,xa)(a)=(y,z)(a)=0,
which implies that (ya,gaπ)∈Ha,0(τ) by the arbitrariness of (xa,faπ)∈Ha(τ) and (3) of Theorem 27. With a similar argument, one can show (yb,gbπ)∈Hb,0(τ).
Next, we consider the sufficiency. Fix any(ya,gaπ)∈Ha,0(τ) and (yb,gbπ)∈Hb,0(τ). Let y and g be defined by (96). By (2) and (3) of Theorem 27 one has that ya(c0)=yb(c0)=0. So y(c0)=0. It follows from (2) of Lemma 29 that (y,gπ)∈H(τ). For any (x,fπ)∈H(τ), it follows from (1) of Lemma 29 that (xa,faπ)∈Ha(τ) and (xb,fbπ)∈Hb(τ). Thus one has by (2) and (3) of Theorem 27 that
(101)(y,x)(a)=(ya,xa)(a)=0,(y,x)(b+1)=(yb,xb)(b+1)=0.
This implies that (y,gπ)∈H0(τ) by (1) of Theorem 27, and consequently, (y,gπ)∈H^0(τ).
(2) We first consider the necessity. Fix any(y,gπ)∈(H^0(τ))J*. Let ya, yb, ga, and gb be defined as (94). Then (ya,gaπ)∈(LW2(ℐa))2 and (yb,gbπ)∈(LW2(ℐb))2.
Setxb(t)=fb(t)=0 for t∈ℐb*, where ℐb* is defined by (19) with ℐ replaced by ℐb. It is clear that (xb,fbπ)∈Hb,0(τ). For any (xa,faπ)∈Ha,0(τ), let x and f be defined by (96). Then by the above result (1) one has that (x,fπ)∈H^0(τ). It follows that
(102)0=〈f,Jy〉-〈x,Jg〉=〈ga,Jya〉a-〈xa,Jga〉a.
By the arbitrariness of (xa,faπ)∈Ha,0(τ), one has that (ya,gaπ)∈(Ha,0(τ))J*=Ha(τ) by Theorem 17. With a similar argument one can show (yb,gbπ)∈Hb(τ).
Next, we consider the sufficiency. Fix any(ya,gaπ)∈Ha(τ) and (yb,gbπ)∈Hb(τ). Let y and g be defined by (96). For any (x,fπ)∈H^0(τ), it follows from the above result (1) that (xa,faπ)∈Ha,0(τ) and (xb,fbπ)∈Hb,0(τ). So one has by Theorem 17 that
(103)〈f,J0y〉-〈x,J0g〉=〈ga,J0ya〉a-〈xa,J0ga〉a+〈gb,J0yb〉b-〈xb,J0gb〉b=0.
This yields that (y,gπ)∈(H^0(τ))J*. The whole proof is complete.Lemma 31.
Assume that(A1), (Aa,2), and (Ab,2) hold. Then Γ(H0(τ))⊂Γ(H^0(τ)) and Γ(Hb,0(τ))∩Γ(Ha,0(τ))=Γ(H^0(τ)).Proof.
The first assertion holds becauseH^0(τ)⊂H0(τ), and the second assertion can be proved by (1) of Lemma 30 and (95). The proof is complete.In the following of the present paper, we assume that(A3)
Γ(H0(τ))≠∅.By Definition10, one has that if Γ(H0(τ))=∅, then ρ(H0(τ))=∅, and consequently σ(H0(τ))=ℂ. We do not consider this case in the present paper.Theorem 32.
Assume that(A1), (Aa,2), (Ab,2), and (A3) hold. Then
(104)d=da+db-2n.Proof.
The proof is divided into two steps.
Step  1. We show that
(105)d(H^0(τ))=da+db.
It follows fromΓ(H0(τ))≠∅ and Lemma 31 that Γ(Hb,0(τ))∩Γ(Ha,0(τ))≠∅. This, together with (Aa,2), (Ab,2), and Theorem 28, implies that for any λ∈Γ(Hb,0(τ))∩Γ(Ha,0(τ)), (1λ) has just db linearly independent solutions ybj∈ℒW2(ℐb), 1≤j≤db, and (1λ) has just da linearly independent solutions xaj∈ℒW2(ℐa), 1≤j≤da; that is, (ybj,λ(ybj)π)∈Hb(τ) for 1≤j≤db and (xaj,λ(xaj)π)∈Ha(τ) for 1≤j≤da.
Setxbj(t)=0 for t∈ℐb*. It is clear that (xbj,λ(xbj)π)∈Hb(τ) for 1≤j≤da. Let xj, 1≤j≤da, be defined by (96). Then it follows from (2) of Lemma 30 that (xj,λxjπ)∈(H^0(τ))J*. Similarly, set yaj(t)=0 for t∈ℐa*. Then one has (yj,λyjπ)∈(H^0(τ))J* for 1≤j≤db. It is evident that x1,…,xda,y1,…,ydb∈Dom(H^0(τ))J* are linearly independent.
On the other hand, for any(y,λyπ)∈(H^0(τ))J*, it follows from (2) of Lemma 31 that (ya,λyaπ)∈Ha(τ) and (yb,λybπ)∈Hb(τ); that is, ya∈ℒW2(ℐa) is a solution of (1λ) in ℐa, and yb∈ℒW2(ℐb) is a solution of (1λ) in ℐb. Therefore, there exist unique cj∈ℂ(1≤j≤da) and dj∈ℂ(1≤j≤db) such that
(106)ya=∑j=1dacjxaj,yb=∑j=1dbdjybj.
Noting the constructions of xj and yj by (96), one has that
(107)y=∑j=1dacjxj+∑j=1dbdjyj.
This, together with Lemma 11, implies that (105) holds.
Step  2. We show that
(108)d=d(H^0(τ))-2n.
It is evident that(zi,hiπ)∈H0(τ), 1≤i≤2n, where (zi,hiπ) are defined by (80). We claim that
(109)H0(τ)=H^0(τ)∔span{(z1,h1π),(z2,h2π),…,(z2n,h2nπ)}.
In fact, for each (y,gπ)∈H0(τ), set C0=(c1,c2,…,c2n)T=y(c0). Let
(110)y0=y-∑i=12ncizi,g0=g-∑i=12ncihi.
Then
(111)(y,gπ)=(y0,g0π)+∑i=12nci(zi,hiπ),(y0,g0π)∈H^0(τ).
It can be easily verified that this decomposition is unique. Hence, (109) holds.
For anyλ∈Γ(H0(τ)), we claim that
(112)Ran(H0(τ)-λI)=Ran(H^0(τ)-λI)∔span{h1π-λz1π,h2π-λz2π,…,h2nπ-λz2nπ}.
Since (109) holds, it suffices to show that
(113)Ran(H^0(τ)-λI)∩span{h1π-λz1π,h2π-λz2π,…,h2nπ-λz2nπ}={0}.
Suppose that (y,gπ)∈H^0(τ) and c1,c2,…,c2n∈ℂ satisfy
(114)gπ-λyπ+∑j=12ncj(hjπ-λzjπ)=0,
that is,
(115)gπ+∑j=12ncjhjπ=λ(yπ+∑j=12ncjzjπ).
Since (y,gπ)∈H^0(τ) and (zi,hiπ)∈H0(τ), it follows that (y+∑j=12n‍cjzj,λ(yπ+∑j=12n‍cjzjπ))∈H0(τ). Since λ∈Γ(H0(τ)), it yields
(116)y+∑j=12ncjzj=0,gπ+∑j=12ncjhjπ=0,
which, together with (109), implies that cj=0 for 1≤j≤2n, and consequently gπ=yπ=0. This yields that (113) holds.
SinceH0(τ) and H^0(τ) are closed J-Hermitian subspaces, it follows that Ran(H0(τ)-λ) and Ran(H^0(τ)-λ) are closed subspaces in LW2(ℐ), respectively. Hence, there exists a closed subspace Q in LW2(ℐ) such that
(117)Ran(H0(τ)-λI)=Ran(H^0(τ)-λI)⊕Q.
In addition, again by the fact that λ∈Γ(Hb,0(τ))∩Γ(Ha,0(τ)), it follows that {hjπ-λzjπ}j=12n are linearly independent in LW2(ℐ). It follows from (113) that dimQ=2n. Consequently,
(118)dim(Ran(H0(τ)-λI)⊥)=dim(Ran(H^0(τ)-λI)⊥)-2n.
This yields that (108) holds. It follows from (105) and (108) that (104) holds. The proof is complete.Remark 33.
Theorem32 (formula (104)) generalizes the classical result for 2nth order ordinary differential equations that go back to the classical work by Akhiezer and Glazman [1, Theorem 3 in Appendix 2]. To the case of symmetric Hamiltonian systems, formula (104) was extended in [15].
So it follows from Theorems28 and 32 that d=0 if and only if da=db=n, and d=2n if and only if da=db=2n. The following definition is obtained.Definition 34.
Assume that(A1), (A2), and (Ab,2) hold. Then (1λ) is said to be in the limit db case at t=b. In the special case that db=n, (1λ) is said to be in the limit point case (l.p.c.) at t=b, and in the other special case that db=2n, (1λ) is said to be in the limit circle case (l.c.c.) at t=b.
The same definition can be given att=a provided that (Aa,2) holds.


## 4.3. Characterizations ofHa(τ) and Hb(τ)

In this subsection, we characterize the maximal subspacesHa(τ) and Hb(τ). We first consider characterization of Hb(τ). Assume that (Ab,2) holds and Γ(Hb,0(τ))≠∅. Let λ∈Γ(Hb,0(τ)). It follows from the proof of Lemma 11 that
(119)Hb(τ)=Hb,0(τ)∔U1∔U2,
where U1={(xbj,λ(xbj)π)∈Hb(τ):1≤j≤db} and U2={(ybj,(gbj)π)∈Hb(τ):1≤j≤db}, and {(ybj,(gbj)π)}j=1db are linearly independent (mod U1). For convenience, denote
(120)χbi(t):=xbi(t),χbdb+j(t):=ybj(t),1≤i≤db,1≤j≤db.
Clearly, χbi∈DomHb(τ), 1≤i≤2db. So (χbi,χbj)(b+1) is finite for all 1≤i, j≤2db. Then the following result can be directly derived from (119) and (120).Lemma 35.
Assume that(A1) and (Ab,2) hold, and Γ(Hb,0(τ))≠∅. Then every y∈DomHb(τ) can be expressed as
(121)y=yb0+∑i=12dbaiχbi,
where yb0∈DomHb,0(τ) and ai∈ℂ.By Lemma35, zbi, defined by (78), can be uniquely expressed as
(122)zbi=ybi,0+∑j=12dbaijχbj,1≤i≤2n,
where ybi,0∈DomHb,0(τ) and aij∈ℂ. Denote
(123)E:=(aij)2n×2db,F1:=((χbi,χbj)(b+1))1≤i,j≤2db.Lemma 36.
Assume that(A1) and (Ab,2) hold, and Γ(Hb,0(τ))≠∅. Then rankE=2n and rankF1=2db-2n. Furthermore, we can rearrange the order of χb1,χb2,…,χbdb such that
(124)rank((χbi,χbj)(b+1))1≤i≤2db-2n,1≤j≤db=2db-2n.Proof.
It follows from (122) that
(125)(zb1,zb2,…,zb2n)=(yb1,0,yb2,0,…,yb2n,0)+(χb1,χb2,…,χb2db)ET,
which, together with (78), implies that
(126)I2n=(χb1(c0),χb2(c0),…,χb2db(c0))ET.
By Lemma 12, one has rankE=2n.
On the other hand, it follows from (122) that
(127)(zbi,χbs)(b+1)=(ybi,0,χbs)(b+1)+∑j=12dbaij(χjb,χbs)(b+1),
for 1≤i≤2n, 1≤s≤2db, which, together with (78) and (2) of Theorem 27, implies that
(128)EF1=0.
Noting that rankE=2n, one has
(129)rankF1≤2db-2n.
We now want to show rankF1≥2db-2n. By (3) of Lemma 14 one has
(130)F2:=((χbi,χbj)(b+1))1≤i≤db,1≤j≤db=(X0T𝒥X0)T,
where X0:=(xb1(c0),xb2(c0),…,xbdb(c0)). Since rankX0=db, we have that
(131)rankF1≥rankF2≥2db-2n,
which, together with (129), yields that
(132)rankF1=rankF2=2db-2n.
Because rankF2=2db-2n≤db, one can rearrange the order of χb1,χb2,…,χbdb such that the first 2db-2n rows of F2 are linearly independent; that is, (124) holds. The proof is complete.
Without loss of generality, we assume that (124) holds in the rest of this paper. Now, we can give a characterization of Hb(τ).Theorem 37.
Assume that(A1) and (Ab,2) hold, λ∈Γ(Hb,0(τ))≠∅, and χb1,χb2,…,χb2db-2n are linearly independent solutions of (1λ) in ℒw2(ℐb) such that (124) holds. Then
(133)Gb:=((χbi,χbj)(b+1))1≤i,j≤2db-2n
is invertible, and any y∈DomHb(τ) can be uniquely expressed as
(134)y=yb0+∑i=12ncizbi+∑j=12db-2ndjχbj,
where yb0∈DomHb,0(τ), zbi are defined by (78), and ci,dj∈ℂ.Proof.
LetE=(E1,E2), where E1 and E2 are 2n×(2db-2n) and 2n×2n matrices, respectively. It follows from (124) that there exists an invertible matrix L such that
(135)F1L=(I2db-2n0F3F4).
So, it follows from (128) that E1+E2F3=0, which is equivalent to E1=-E2F3. Since rankE=2n, it follows that E2 is invertible. Multiplying (125) by (E2-1)T from the right-hand side, we get
(136)(zb1,zb2,…,zb2n)(E2-1)T=(yb1,0,yb2,0,…,yb2n,0)(E2-1)T+(χb1,χb2,…,χb2db-2n)(E2-1E1)T+(χb2db-2n+1,…,χb2db).
This implies that each of χb2db-2n+1,…,χb2db can be uniquely expressed as a linear combination of χb1,χb2,…,χb2db-2n, yb1,0,yb2,0,…,yb2n,0, and zb1,zb2,…,zb2n. Therefore, (134) follows from Lemma 35.
Sinceχbj∈DomHb(τ) for 1≤j≤db, χbj can be uniquely expressed as
(137)χbj=ybj,0+∑l=12nbjlzbl+∑s=12db-2ncjsχbs,1≤j≤db,
where ybj,0∈DomHb,0(τ), and bjl,cjs∈ℂ. This, together with (78) and (2) of Theorem 27, implies that for 1≤i≤2db-2n,1≤j≤db(138)(χbi,χbj)(b+1)=∑s=12db-2ncjs(χbi,χsb)(b+1),
that is,
(139)(χbi,χbj)(b+1)1≤i≤2db-2n,1≤j≤db=Gb(cij)1≤i≤db,1≤j≤2db-2nT.
Therefore, Gb is invertible from (124). This completes the proof.With a similar argument, one can obtain the following characterization ofHa(τ).Theorem 38.
Assume that(A1) and (Aa,2) hold, Γ(Ha,0(τ))≠∅. Then, for some λ∈Γ(Ha,0(τ)), system (1λ) has 2da-2n linearly independent solutions χa1,…,χa2da-2n in ℒW2(ℐa) such that Ga is invertible, where
(140)Ga:=((χai,χaj)(a))1≤i,j≤2da-2n,
and each y∈DomHa(τ) can be uniquely expressed as
(141)y=ya0+∑i=12ncizai+∑j=12da-2ndjχaj,
where ya0∈DomHa,0(τ), ci,dj∈ℂ, and zai are defined as (79).


## 5. Characterizations ofJ-SSEs of H0(τ)

In this section, we give a complete characterization of all theJ-SSEs of minimal subspace H0(τ) in terms of the square summable solutions of system (1λ). As a consequence, characterizations of all the J-self-adjoint subspace extensions are obtained in the two special cases: the limit point and limit circle cases. The following discussion is divided into two parts based on the form of ℐ.


### 5.1. Both the Endpoints Are Infinite

Letℐ=(-∞,+∞), and assume that (A1), (Aa,2), and (Ab,2) hold, and Γ(H0(τ))≠∅ in this subsection. It follows from Theorem 32 that
(142)da+db-2n=d.
In addition, for some λ∈Γ(H0(τ), let χb1,…,χb2db-2n given in Theorems 37 and χa1,…,χa2da-2n be given in Theorems 38.Theorem 39.
Assume that(A1), (Aa,2), (Ab,2), and (A3) hold. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two matrices Md×(2da-2n) and Nd×(2db-2n) such that  (1)
rank(M,N)=d,(2)
NGbNT-MGaMT=0, and
(143)T={(y,gπ)∈H(τ):M((y,χa1)(-∞)⋮(y,χa2da-2n)(-∞))-N((y,χb1)(+∞)⋮(y,χb2db-2n)(+∞))=0},where Gb and Ga are the same as those in Theorems 37 and 38.Proof.
We first show the sufficiency.
Suppose that there exist two matricesMd×(2da-2n) and Nd×(2db-2n) such that conditions (1) and (2) hold and T is defined by (143). We now prove that T is a J-self-adjoint subspace extension of H0(τ) by Lemma 9.
Denote(144)M=(mij)d×(2da-2n),N=(nij)d×(2db-2n),
and set
(145)wai:=∑j=12da-2nmijχaj,wbi:=∑j=12db-2nnijχbj,1≤i≤d.
Clearly, wai∈DomHa(τ) and wbi∈DomHb(τ) for 1≤i≤d. By Remark 26, there exist βi:=(ωi,ρiπ)∈H(τ)(1≤i≤d) such that
(146)ωi(t)=wai(t),t≤sa0,ωi(t)=wbi(t),t≥tb0+1,
where sa0 and tb0 are specified by (Aa,2) and (Ab,2), respectively.
SinceH(τ) and H0(τ) are liner subspaces, β1,β2,…,βd are linearly independent in H(τ) (modulo H0(τ)) if and only if ω1,ω2,…,ωd are linearly independent in DomH(τ) (modulo DomH0(τ)). So it suffices to show that ω1,ω2,…,ωd are linearly independent in DomH(τ) (modulo DomH0(τ)). Suppose that there exists C=(c1,c2,…,cd)∈ℂd such that
(147)ω=∑j=1dcjωj∈DomH0(τ).
It follows from (145), (146), and (2) and (3) of Theorem 27 that
(148)0=((ω,χa1)(-∞),…,(ω,χa2da-2n)(-∞))=CMGa,0=((ω,χb1)(+∞),…,(ω,χb2d-2n)(+∞))=CNGb.
Since Gb and Ga are invertible, we get from (148) that CM=CN=0. Then C=0 by condition (1). So, ω1,ω2,…,ωd are linearly independent in DomH(τ) (modulo DomH0(τ)), and consequently β1,β2,…,βd are linearly independent in H(τ) (modulo H0(τ)).
Next, we show that[βi:βj]=0 for 1≤i,j≤d. It follows from (145) and (146) that
(149)((ωi,ωj)(-∞))1≤i,j≤d=MGaMT,((ωi,ωj)(+∞))1≤i,j≤d=NGbNT,
which, together with Lemma 24 and condition (2), implies that
(150)([βi:βj])1≤i,j≤d=NGbNT-MGaMT=0.
Consequently, [βi:βj]=0 for 1≤i, j≤d. Therefore, {βj}j=1d satisfy the conditions (1) and (2) of Lemma 9.
Note that for eachy∈DomH(τ), it follows that
(151)M((y,χa1)(-∞)⋮(y,χa2da-2n)(-∞))=((y,∑j=12da-2nm1jχaj)(-∞)⋮(y,∑j=12da-2nmdjχaj)(-∞))=((y,ω1)(-∞)⋮(y,ωd)(-∞)),N((y,χb1)(+∞)⋮(y,χb2d-2n)(+∞))=((y,∑j=12d-2nn1jχbj)(+∞)⋮(y,∑j=12d-2nndjχbj)(+∞))=((y,ω1)(+∞)⋮(y,ωd)(+∞)),
where (145) and (146) have been used. Therefore, it follows from Lemma 24 that T can be expressed as
(152)T={(y,gπ)∈H(τ):[(y,gπ):βj]=0,1≤j≤d}.
Hence, T is a J-SSE of H0(τ) by Lemma 9. The sufficiency is proved.
We now show the necessity. Suppose thatT is a J-SSE of H0(τ). By Lemma 9 and Theorem 17, there exists a set of {βj}j=1d⊂H(τ) such that (152) holds. Write βj=(ωj,ρjπ). Then ωaj∈DomHa(τ) and ωbj∈DomHb(τ) for 1≤j≤d. By Theorems 37 and 38, each ωaj and ωbj can be uniquely expressed as
(153)ωaj=yaj,0+∑i=12ncijzai+∑s=12da-2nmjsχas,ωbj=ybj,0+∑i=12ndijzbi+∑s=12db-2nnjsχbs,1≤j≤d,
where ybj,0∈DomHb,0(τ), yaj,0∈DomHa,0(τ), and dij,cij,njs,mjs∈ℂ. Set
(154)M=(mjs)d×(2da-2n),N=(njs)d×(2db-2n).
First, we want to show thatM and N satisfy condition (1). Otherwise, suppose that rank(M,N)<d. Then there exists C=(c1,c2,…,cd)∈ℂd with C≠0 such that
(155)C(M,N)=0.
Then CM=0 and CN=0. Let ω=∑j=1d‍cjωj. We then have
(156)((ω,χa1)(-∞),…,(ω,χa2da-2n)(-∞))=CMGa=0((ω,χb1)(+∞),…,(ω,χb2d-2n)(+∞))=CNGb=0.
For each y∈DomH(τ), yb can be uniquely expressed as (134) by Theorem 37 and ya can be uniquely expressed as (141) by Theorem 38. So it follows from (156), (78), (79), and (2) and (3) of Theorem 27 that for any y∈DomH(τ), (ω,y)(+∞)=(ω,y)(-∞)=0, which yields that ω∈DomH0(τ) by (1) of Theorem 27. Then ω1,ω2,…,ωd are linearly dependent in DomH(τ) (modulo DomH0(τ)), and consequently β1,β2,…,βd are linearly dependent in H(τ) (modulo H0(τ)). This is a contradiction. Hence, rank(M,N)=d.
Next, we prove thatM and N satisfy condition (2). It can be easily verified that
(157)((ωi,ωj)(-∞))1≤i,j≤d=MGaMT,((ωi,ωj)(+∞))1≤i,j≤d=NGbNT.
Hence, by Lemma 14 and [βi:βj]=0 for 1≤i, j≤d, M and N satisfy condition (2).
In addition, it follows from (78), (79), (153), and (2) and (3) of Theorem 27 that
(158)((y,ω1)(-∞)⋮(y,ωd)(-∞))=((y,∑j=12da-2nm1jχaj)(-∞)⋮(y,∑j=12da-2nmdjχaj)(-∞))=M((y,χa1)(-∞)⋮(y,χa2da-2n)(-∞)),((y,ω1)(+∞)⋮(y,ωd)(+∞))=((y,∑j=12d-2nn1jχbj)(+∞)⋮(y,∑j=12d-2nndjχbj)(+∞))=N((y,χb1)(+∞)⋮(y,χb2d-2n)(+∞)).
Hence, T in (152) can be expressed as (143). The necessity is proved.
The entire proof is complete.To end this subsection, we give characterizations ofJ-SSEs of H0(τ) in four special cases of defect indices: da=db=n; da=n,db=2n; da=2n,db=n; da=db=2n.In the case thatda=db=n, that is, d=0 by Theorem 32, the following result is derived from Lemma 11 and Theorem 17.Theorem 40.
Assume that(A1), (Aa,2), (Ab,2), and (A3) hold. If d=0, then H0(τ) is a J-self-adjoint subspace.In the case thatda=n, db=2n, it follows from Theorem 32 that d=n. Let χbi, (1≤i≤2n), be 2n linearly independent solutions in ℒW2(ℐb) of (1λ) satisfying (χb1(c0),χb2(c0),…,χb2n(c0))=I2n. Then, by (3) of Lemma 14 one has that
(159)Gb=((χbi,χbj)(c0))1≤i,j≤2n=(J0Y)*(c0)𝒥Y(c0)=𝒥.
The following result can be directly derived from Theorem 39.Theorem 41.
Assume that(A1), (A2), (Aa,2), (Ab,2), and (A3) hold. If (1λ) is in l.p.c. at t=-∞ and in l.c.c. at t=+∞. Let χbi(1≤i≤2n) be 2n linearly independent solutions in ℒW2(ℐb) of (1λ) satisfying (χ1b(c0),χ2b(c0),…,χ2nb(c0))=I2n. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exists a matrix Nn×2n such that
(160)rankN=n,N𝒥NT=0,T={(y,gπ)∈H(τ):N((y,χb1)(+∞)⋮(y,χb2n)(+∞))=0}.In the case thatda=2n,db=n, a similar result can be easily given. So we omit the details in this case.In the case thatda=db=2n, it follows from Theorem 32 that d=2n. Let χi(1≤i≤2n) be 2n linearly independent solutions in ℒW2(ℐ) of (1λ) satisfying
(161)(χ1(c0),χ2(c0),…,χ2n(c0))=I2n.
Then, by Lemma 14, Gb and Ga, defined by (133) and (140), satisfy
(162)Ga=Gb=𝒥.
The following result is a direct consequence of Theorem 39.Theorem 42.
Assume that(A1), (Aa,2), (Ab,2), and (A3) hold. If (1λ) is in l.c.c. at both t=+∞ and t=-∞, then for any given λ∈Γ(H0(τ)), let χi(1≤i≤2n) be 2n linearly independent solutions in ℒW2(ℐ) of (1λ) satisfying (χ1(c0),χ2(c0),…,ϕ2n(c0))=I2n. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two matrices M2n×2n and N2n×2n such that
(163)rank(M,N)=2n,M𝒥MT=N𝒥NT,(164)T={(y,gπ)∈H(τ):M((y,χ1)(-∞)⋮(y,χ2n)(-∞))-N((y,χ1)(+∞)⋮(y,χ2n)(+∞))=0}.Remark 43.
As we have seen, there is no boundary condition at the endpoints at which system(1λ) is in l.p.c., and the matrix Ga or Gb can be replaced by 𝒥 in the case that system (1λ) is in l.c.c. at t=a or t=b.


### 5.2. At Least One of the Two Endpoints Is Finite

In this subsection, we characterize theJ-SSEs of H0(τ) in the special case that at least one of the two endpoints a and b is finite. We first consider the case that a is finite, and b is finite or infinite.We point out that in this case, characterizations of all theJ-SSEs of H0(τ) can also be given by the proof of Theorem 39, provided that assumptions (Ab,2), (Aa,2), and (A3) hold. But, if there does not exist a c0∈ℐ such that both (Aa,2) and (Ab,2) are satisfied, then Theorem 39 fails. We will remark again that the division of ℐ is not necessary in the case that one of the two endpoints is finite, and characterizations of all the J-SSEs of H0(τ) can still be given provided that (A3) and (A3) hold.In the case thata is finite, ℐ can be regarded as ℐb with a=c0, and (A2) is equivalent to (Ab,2). So all the characterizations for Hb,0(τ) and Hb(τ) given in Sections 3 and 4 are available to H0(τ) and H(τ), respectively, with c0 replaced by a. Assume that (A2) holds. Then for any given λ∈Γ(H0(τ)), as discussed in Section 4, let χ1,…,χ2d-2n be 2d-2n linearly independent solutions in ℒW2(ℐ) of (1λ) such that G is invertible, where G is defined by (133) with χbj is replaced by χj, 1≤j≤2d-2n. Then all the results of Theorem 37 hold with Gb and χbj replaced by G and χj, respectively.Theorem 44.
Assume that the left endpointa is finite, and (A1)–(A3) hold. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two matrices Md×2n and Nd×(2d-2n) such that  (1)
rank(M,N)=d,(2)
M𝒥MT-NGNT=0,  and
(165)T={((y,χ1)(b+1)⋮(y,χ2d-2n)(b+1))(y,gπ)∈H(τ):My(a)-N((y,χ1)(b+1)⋮(y,χ2d-2n)(b+1))=0}.Proof.
The main idea of the proof is similar to that of Theorem39.
We first show the sufficiency. Denote(166)𝒥MT=(ξ1,ξ2,…,ξd),N=(nij)d×(2d-2n),
and set
(167)wi:=∑j=12d-2nnijχj,1≤i≤d.
Clearly, wi∈DomH(τ) for 1≤i≤d. By Remark 26, there exist βi:=(ωi,ρπ)∈H(τ) (1≤i≤d) such that
(168)ωi(a)=ξi,ωi(t)=wi(t),t≥t0+1,
where t0 is specified by (A2). It can be verified by the method used in the proof of Theorem 39 that the set {βj}j=1d satisfies the conditions (1) and (2) in Lemma 9. Note that for each y∈DomH(τ), it follows that
(169)My(a)=(𝒥MT)T𝒥y(a)=(ω1T(a)⋮ωdT(a))𝒥y(a)=((y,ω1)(a)⋮(y,ωd)(a)),N((y,χ1)(b+1)⋮(y,χ2d-2n)(b+1))=((y,∑j=12d-2n‍n1jχj)(b+1)⋮(y,∑j=12d-2n‍ndjχj)(b+1))=((y,ω1)(b+1)⋮(y,ωd)(b+1)).
Therefore, it follows from Lemma 24 that T can be expressed as
(170)T={(y,gπ)∈H(τ):[(y,gπ):βj]=0,1≤j≤d}.
Hence, T is a J-SSE of H0(τ) by Lemma 9. The sufficiency is proved.
We now show the necessity. Suppose thatT is a J-SSE of H0(τ). By Lemma 9 and Theorem 17, there exists a set of {βj}j=1d⊂H(τ) for {H0(τ),H(τ)} such that conditions (1) and (2) in Lemma 9 hold, and T can be expressed as (152). Write βi:=(ωi,ρπ). Then ωj∈DomH(τ) for 1≤j≤d. By Theorem 37, each ωi can be uniquely expressed as
(171)ωi=ybi,0+∑j=12naijzbj+∑s=12d-2nnisχbs,1≤i≤d,
where ybi,0∈DomH0(τ) and aij,nis∈ℂ. Set
(172)Md×2n=(ω1(a),…,ωd(a))T𝒥,Nd×(2d-2n)=(nis)1≤i≤d,1≤s≤2d-2n.
With a similar argument to that used in the proof of Theorem39, we can prove that M and N satisfy conditions (1) and (2). In addition, it is clear that T in (170) can be expressed as (165). The necessity is proved, and then the entire proof is complete.At the end of this subsection, we give the characterizations ofJ-SSEs of H0(τ) in two special cases of defect indices.In the special case thatd=n, Theorem 44 can be described in the following simpler form.Theorem 45.
Assume that the left endpointa is finite, and (A1)–(A3) hold. If (1λ) is in l.p.c. at t=b, then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exists a matrix Mn×2n satisfying the self-adjoint condition
(173)rankM=n,M𝒥MT=0
such that T can be defined by
(174)T={(y,gπ)∈H(τ):My(a)=0}.In the other special case thatd=2n, the following result is a direct consequence of Theorem 44.Theorem 46.
Assume that the left endpointa is finite, and (A1)–(A3) hold. If (1λ) is in l.c.c. at t=b, let χi(1≤i≤2n) be 2n linearly independent solutions in ℒW2(ℐ) of (1λ) satisfying (χ1(c0),χ2(c0),…,χ2n(c0))=I2n. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two 2n×2n matrices M and N such that  (1)
rank(M,N)=2n,(2)
M𝒥MT=N𝒥NT,  and
(175)T={((y,χ1)(b+1)⋮(y,χ2n)(b+1))(y,gπ)∈H(τ):My(a)-N((y,χ1)(b+1)⋮(y,χ2n)(b+1))=0}.For the case thatb is finite and a=-∞, it can be considered by a similar method. Here we only give the following basic result.Theorem 47.
Assume that the right endpointb is finite, and (A1)–(A3) hold. Let χj, 1≤j≤2d-2n, be linearly independent solutions of (1λ) in ℒW2(ℐ) such that G is invertible, where G is defined as Ga in Theorem 38 with χaj replaced by χj. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two matrices Mn×2n and Nd×(2d-2n) such that (1)
rank(M,N)=d,(2)
MGMT-N𝒥NT=0, and
(176)T={(y,gπ)∈H(τ):M((y,χ1)(a)⋮(y,χ2d-2n(a)))-Ny(b+1)=0(y,gπ)((y,χ1)(a)⋮(y,χ2d-2n(a)))}.In the case that both the two endpointsa and b are finite, that is, ℐ=[a,b], it is clear that d=2n by (A2). The characterization of J-SSEs given in Theorem 44 can be simplified as follows.Theorem 48.
Letℐ=[a,b]. Assume that (A1)–(A3) hold. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two 2n×2n matrices M and N such that  (1)
rank(M,N)=2n,(2)
N𝒥NT=M𝒥MT,  and
(177)T={(y,gπ)∈H(τ):My(a)-Ny(b+1)=0}.Proof.
Letχj∈ℒW2(ℐ), 1≤j≤2n, be defined as those in Theorem 46. Then it follows that
(178)N((y,χ1)(b+1)⋮(y,χ2n)(b+1))=N(χ1T(b+1)⋮χ2nT(b+1))𝒥y(b+1)=N1y(b+1),
where N1=N(χ1,…,χ2n)T(b+1)𝒥. It is evident (χ1,…,χ2n)(b+1)=I2n. So by Lemma 14, one has that
(179)rank(M,N)=rank(M,N1)=2n,N1𝒥N1T=N𝒥NT.
Hence, the assertion follows from Theorem 46. This completes the proof.


## 5.1. Both the Endpoints Are Infinite

Letℐ=(-∞,+∞), and assume that (A1), (Aa,2), and (Ab,2) hold, and Γ(H0(τ))≠∅ in this subsection. It follows from Theorem 32 that
(142)da+db-2n=d.
In addition, for some λ∈Γ(H0(τ), let χb1,…,χb2db-2n given in Theorems 37 and χa1,…,χa2da-2n be given in Theorems 38.Theorem 39.
Assume that(A1), (Aa,2), (Ab,2), and (A3) hold. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two matrices Md×(2da-2n) and Nd×(2db-2n) such that  (1)
rank(M,N)=d,(2)
NGbNT-MGaMT=0, and
(143)T={(y,gπ)∈H(τ):M((y,χa1)(-∞)⋮(y,χa2da-2n)(-∞))-N((y,χb1)(+∞)⋮(y,χb2db-2n)(+∞))=0},where Gb and Ga are the same as those in Theorems 37 and 38.Proof.
We first show the sufficiency.
Suppose that there exist two matricesMd×(2da-2n) and Nd×(2db-2n) such that conditions (1) and (2) hold and T is defined by (143). We now prove that T is a J-self-adjoint subspace extension of H0(τ) by Lemma 9.
Denote(144)M=(mij)d×(2da-2n),N=(nij)d×(2db-2n),
and set
(145)wai:=∑j=12da-2nmijχaj,wbi:=∑j=12db-2nnijχbj,1≤i≤d.
Clearly, wai∈DomHa(τ) and wbi∈DomHb(τ) for 1≤i≤d. By Remark 26, there exist βi:=(ωi,ρiπ)∈H(τ)(1≤i≤d) such that
(146)ωi(t)=wai(t),t≤sa0,ωi(t)=wbi(t),t≥tb0+1,
where sa0 and tb0 are specified by (Aa,2) and (Ab,2), respectively.
SinceH(τ) and H0(τ) are liner subspaces, β1,β2,…,βd are linearly independent in H(τ) (modulo H0(τ)) if and only if ω1,ω2,…,ωd are linearly independent in DomH(τ) (modulo DomH0(τ)). So it suffices to show that ω1,ω2,…,ωd are linearly independent in DomH(τ) (modulo DomH0(τ)). Suppose that there exists C=(c1,c2,…,cd)∈ℂd such that
(147)ω=∑j=1dcjωj∈DomH0(τ).
It follows from (145), (146), and (2) and (3) of Theorem 27 that
(148)0=((ω,χa1)(-∞),…,(ω,χa2da-2n)(-∞))=CMGa,0=((ω,χb1)(+∞),…,(ω,χb2d-2n)(+∞))=CNGb.
Since Gb and Ga are invertible, we get from (148) that CM=CN=0. Then C=0 by condition (1). So, ω1,ω2,…,ωd are linearly independent in DomH(τ) (modulo DomH0(τ)), and consequently β1,β2,…,βd are linearly independent in H(τ) (modulo H0(τ)).
Next, we show that[βi:βj]=0 for 1≤i,j≤d. It follows from (145) and (146) that
(149)((ωi,ωj)(-∞))1≤i,j≤d=MGaMT,((ωi,ωj)(+∞))1≤i,j≤d=NGbNT,
which, together with Lemma 24 and condition (2), implies that
(150)([βi:βj])1≤i,j≤d=NGbNT-MGaMT=0.
Consequently, [βi:βj]=0 for 1≤i, j≤d. Therefore, {βj}j=1d satisfy the conditions (1) and (2) of Lemma 9.
Note that for eachy∈DomH(τ), it follows that
(151)M((y,χa1)(-∞)⋮(y,χa2da-2n)(-∞))=((y,∑j=12da-2nm1jχaj)(-∞)⋮(y,∑j=12da-2nmdjχaj)(-∞))=((y,ω1)(-∞)⋮(y,ωd)(-∞)),N((y,χb1)(+∞)⋮(y,χb2d-2n)(+∞))=((y,∑j=12d-2nn1jχbj)(+∞)⋮(y,∑j=12d-2nndjχbj)(+∞))=((y,ω1)(+∞)⋮(y,ωd)(+∞)),
where (145) and (146) have been used. Therefore, it follows from Lemma 24 that T can be expressed as
(152)T={(y,gπ)∈H(τ):[(y,gπ):βj]=0,1≤j≤d}.
Hence, T is a J-SSE of H0(τ) by Lemma 9. The sufficiency is proved.
We now show the necessity. Suppose thatT is a J-SSE of H0(τ). By Lemma 9 and Theorem 17, there exists a set of {βj}j=1d⊂H(τ) such that (152) holds. Write βj=(ωj,ρjπ). Then ωaj∈DomHa(τ) and ωbj∈DomHb(τ) for 1≤j≤d. By Theorems 37 and 38, each ωaj and ωbj can be uniquely expressed as
(153)ωaj=yaj,0+∑i=12ncijzai+∑s=12da-2nmjsχas,ωbj=ybj,0+∑i=12ndijzbi+∑s=12db-2nnjsχbs,1≤j≤d,
where ybj,0∈DomHb,0(τ), yaj,0∈DomHa,0(τ), and dij,cij,njs,mjs∈ℂ. Set
(154)M=(mjs)d×(2da-2n),N=(njs)d×(2db-2n).
First, we want to show thatM and N satisfy condition (1). Otherwise, suppose that rank(M,N)<d. Then there exists C=(c1,c2,…,cd)∈ℂd with C≠0 such that
(155)C(M,N)=0.
Then CM=0 and CN=0. Let ω=∑j=1d‍cjωj. We then have
(156)((ω,χa1)(-∞),…,(ω,χa2da-2n)(-∞))=CMGa=0((ω,χb1)(+∞),…,(ω,χb2d-2n)(+∞))=CNGb=0.
For each y∈DomH(τ), yb can be uniquely expressed as (134) by Theorem 37 and ya can be uniquely expressed as (141) by Theorem 38. So it follows from (156), (78), (79), and (2) and (3) of Theorem 27 that for any y∈DomH(τ), (ω,y)(+∞)=(ω,y)(-∞)=0, which yields that ω∈DomH0(τ) by (1) of Theorem 27. Then ω1,ω2,…,ωd are linearly dependent in DomH(τ) (modulo DomH0(τ)), and consequently β1,β2,…,βd are linearly dependent in H(τ) (modulo H0(τ)). This is a contradiction. Hence, rank(M,N)=d.
Next, we prove thatM and N satisfy condition (2). It can be easily verified that
(157)((ωi,ωj)(-∞))1≤i,j≤d=MGaMT,((ωi,ωj)(+∞))1≤i,j≤d=NGbNT.
Hence, by Lemma 14 and [βi:βj]=0 for 1≤i, j≤d, M and N satisfy condition (2).
In addition, it follows from (78), (79), (153), and (2) and (3) of Theorem 27 that
(158)((y,ω1)(-∞)⋮(y,ωd)(-∞))=((y,∑j=12da-2nm1jχaj)(-∞)⋮(y,∑j=12da-2nmdjχaj)(-∞))=M((y,χa1)(-∞)⋮(y,χa2da-2n)(-∞)),((y,ω1)(+∞)⋮(y,ωd)(+∞))=((y,∑j=12d-2nn1jχbj)(+∞)⋮(y,∑j=12d-2nndjχbj)(+∞))=N((y,χb1)(+∞)⋮(y,χb2d-2n)(+∞)).
Hence, T in (152) can be expressed as (143). The necessity is proved.
The entire proof is complete.To end this subsection, we give characterizations ofJ-SSEs of H0(τ) in four special cases of defect indices: da=db=n; da=n,db=2n; da=2n,db=n; da=db=2n.In the case thatda=db=n, that is, d=0 by Theorem 32, the following result is derived from Lemma 11 and Theorem 17.Theorem 40.
Assume that(A1), (Aa,2), (Ab,2), and (A3) hold. If d=0, then H0(τ) is a J-self-adjoint subspace.In the case thatda=n, db=2n, it follows from Theorem 32 that d=n. Let χbi, (1≤i≤2n), be 2n linearly independent solutions in ℒW2(ℐb) of (1λ) satisfying (χb1(c0),χb2(c0),…,χb2n(c0))=I2n. Then, by (3) of Lemma 14 one has that
(159)Gb=((χbi,χbj)(c0))1≤i,j≤2n=(J0Y)*(c0)𝒥Y(c0)=𝒥.
The following result can be directly derived from Theorem 39.Theorem 41.
Assume that(A1), (A2), (Aa,2), (Ab,2), and (A3) hold. If (1λ) is in l.p.c. at t=-∞ and in l.c.c. at t=+∞. Let χbi(1≤i≤2n) be 2n linearly independent solutions in ℒW2(ℐb) of (1λ) satisfying (χ1b(c0),χ2b(c0),…,χ2nb(c0))=I2n. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exists a matrix Nn×2n such that
(160)rankN=n,N𝒥NT=0,T={(y,gπ)∈H(τ):N((y,χb1)(+∞)⋮(y,χb2n)(+∞))=0}.In the case thatda=2n,db=n, a similar result can be easily given. So we omit the details in this case.In the case thatda=db=2n, it follows from Theorem 32 that d=2n. Let χi(1≤i≤2n) be 2n linearly independent solutions in ℒW2(ℐ) of (1λ) satisfying
(161)(χ1(c0),χ2(c0),…,χ2n(c0))=I2n.
Then, by Lemma 14, Gb and Ga, defined by (133) and (140), satisfy
(162)Ga=Gb=𝒥.
The following result is a direct consequence of Theorem 39.Theorem 42.
Assume that(A1), (Aa,2), (Ab,2), and (A3) hold. If (1λ) is in l.c.c. at both t=+∞ and t=-∞, then for any given λ∈Γ(H0(τ)), let χi(1≤i≤2n) be 2n linearly independent solutions in ℒW2(ℐ) of (1λ) satisfying (χ1(c0),χ2(c0),…,ϕ2n(c0))=I2n. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two matrices M2n×2n and N2n×2n such that
(163)rank(M,N)=2n,M𝒥MT=N𝒥NT,(164)T={(y,gπ)∈H(τ):M((y,χ1)(-∞)⋮(y,χ2n)(-∞))-N((y,χ1)(+∞)⋮(y,χ2n)(+∞))=0}.Remark 43.
As we have seen, there is no boundary condition at the endpoints at which system(1λ) is in l.p.c., and the matrix Ga or Gb can be replaced by 𝒥 in the case that system (1λ) is in l.c.c. at t=a or t=b.


## 5.2. At Least One of the Two Endpoints Is Finite

In this subsection, we characterize theJ-SSEs of H0(τ) in the special case that at least one of the two endpoints a and b is finite. We first consider the case that a is finite, and b is finite or infinite.We point out that in this case, characterizations of all theJ-SSEs of H0(τ) can also be given by the proof of Theorem 39, provided that assumptions (Ab,2), (Aa,2), and (A3) hold. But, if there does not exist a c0∈ℐ such that both (Aa,2) and (Ab,2) are satisfied, then Theorem 39 fails. We will remark again that the division of ℐ is not necessary in the case that one of the two endpoints is finite, and characterizations of all the J-SSEs of H0(τ) can still be given provided that (A3) and (A3) hold.In the case thata is finite, ℐ can be regarded as ℐb with a=c0, and (A2) is equivalent to (Ab,2). So all the characterizations for Hb,0(τ) and Hb(τ) given in Sections 3 and 4 are available to H0(τ) and H(τ), respectively, with c0 replaced by a. Assume that (A2) holds. Then for any given λ∈Γ(H0(τ)), as discussed in Section 4, let χ1,…,χ2d-2n be 2d-2n linearly independent solutions in ℒW2(ℐ) of (1λ) such that G is invertible, where G is defined by (133) with χbj is replaced by χj, 1≤j≤2d-2n. Then all the results of Theorem 37 hold with Gb and χbj replaced by G and χj, respectively.Theorem 44.
Assume that the left endpointa is finite, and (A1)–(A3) hold. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two matrices Md×2n and Nd×(2d-2n) such that  (1)
rank(M,N)=d,(2)
M𝒥MT-NGNT=0,  and
(165)T={((y,χ1)(b+1)⋮(y,χ2d-2n)(b+1))(y,gπ)∈H(τ):My(a)-N((y,χ1)(b+1)⋮(y,χ2d-2n)(b+1))=0}.Proof.
The main idea of the proof is similar to that of Theorem39.
We first show the sufficiency. Denote(166)𝒥MT=(ξ1,ξ2,…,ξd),N=(nij)d×(2d-2n),
and set
(167)wi:=∑j=12d-2nnijχj,1≤i≤d.
Clearly, wi∈DomH(τ) for 1≤i≤d. By Remark 26, there exist βi:=(ωi,ρπ)∈H(τ) (1≤i≤d) such that
(168)ωi(a)=ξi,ωi(t)=wi(t),t≥t0+1,
where t0 is specified by (A2). It can be verified by the method used in the proof of Theorem 39 that the set {βj}j=1d satisfies the conditions (1) and (2) in Lemma 9. Note that for each y∈DomH(τ), it follows that
(169)My(a)=(𝒥MT)T𝒥y(a)=(ω1T(a)⋮ωdT(a))𝒥y(a)=((y,ω1)(a)⋮(y,ωd)(a)),N((y,χ1)(b+1)⋮(y,χ2d-2n)(b+1))=((y,∑j=12d-2n‍n1jχj)(b+1)⋮(y,∑j=12d-2n‍ndjχj)(b+1))=((y,ω1)(b+1)⋮(y,ωd)(b+1)).
Therefore, it follows from Lemma 24 that T can be expressed as
(170)T={(y,gπ)∈H(τ):[(y,gπ):βj]=0,1≤j≤d}.
Hence, T is a J-SSE of H0(τ) by Lemma 9. The sufficiency is proved.
We now show the necessity. Suppose thatT is a J-SSE of H0(τ). By Lemma 9 and Theorem 17, there exists a set of {βj}j=1d⊂H(τ) for {H0(τ),H(τ)} such that conditions (1) and (2) in Lemma 9 hold, and T can be expressed as (152). Write βi:=(ωi,ρπ). Then ωj∈DomH(τ) for 1≤j≤d. By Theorem 37, each ωi can be uniquely expressed as
(171)ωi=ybi,0+∑j=12naijzbj+∑s=12d-2nnisχbs,1≤i≤d,
where ybi,0∈DomH0(τ) and aij,nis∈ℂ. Set
(172)Md×2n=(ω1(a),…,ωd(a))T𝒥,Nd×(2d-2n)=(nis)1≤i≤d,1≤s≤2d-2n.
With a similar argument to that used in the proof of Theorem39, we can prove that M and N satisfy conditions (1) and (2). In addition, it is clear that T in (170) can be expressed as (165). The necessity is proved, and then the entire proof is complete.At the end of this subsection, we give the characterizations ofJ-SSEs of H0(τ) in two special cases of defect indices.In the special case thatd=n, Theorem 44 can be described in the following simpler form.Theorem 45.
Assume that the left endpointa is finite, and (A1)–(A3) hold. If (1λ) is in l.p.c. at t=b, then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exists a matrix Mn×2n satisfying the self-adjoint condition
(173)rankM=n,M𝒥MT=0
such that T can be defined by
(174)T={(y,gπ)∈H(τ):My(a)=0}.In the other special case thatd=2n, the following result is a direct consequence of Theorem 44.Theorem 46.
Assume that the left endpointa is finite, and (A1)–(A3) hold. If (1λ) is in l.c.c. at t=b, let χi(1≤i≤2n) be 2n linearly independent solutions in ℒW2(ℐ) of (1λ) satisfying (χ1(c0),χ2(c0),…,χ2n(c0))=I2n. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two 2n×2n matrices M and N such that  (1)
rank(M,N)=2n,(2)
M𝒥MT=N𝒥NT,  and
(175)T={((y,χ1)(b+1)⋮(y,χ2n)(b+1))(y,gπ)∈H(τ):My(a)-N((y,χ1)(b+1)⋮(y,χ2n)(b+1))=0}.For the case thatb is finite and a=-∞, it can be considered by a similar method. Here we only give the following basic result.Theorem 47.
Assume that the right endpointb is finite, and (A1)–(A3) hold. Let χj, 1≤j≤2d-2n, be linearly independent solutions of (1λ) in ℒW2(ℐ) such that G is invertible, where G is defined as Ga in Theorem 38 with χaj replaced by χj. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two matrices Mn×2n and Nd×(2d-2n) such that (1)
rank(M,N)=d,(2)
MGMT-N𝒥NT=0, and
(176)T={(y,gπ)∈H(τ):M((y,χ1)(a)⋮(y,χ2d-2n(a)))-Ny(b+1)=0(y,gπ)((y,χ1)(a)⋮(y,χ2d-2n(a)))}.In the case that both the two endpointsa and b are finite, that is, ℐ=[a,b], it is clear that d=2n by (A2). The characterization of J-SSEs given in Theorem 44 can be simplified as follows.Theorem 48.
Letℐ=[a,b]. Assume that (A1)–(A3) hold. Then a subspace T⊂(LW2(ℐ))2 is a J-SSE of H0(τ) if and only if there exist two 2n×2n matrices M and N such that  (1)
rank(M,N)=2n,(2)
N𝒥NT=M𝒥MT,  and
(177)T={(y,gπ)∈H(τ):My(a)-Ny(b+1)=0}.Proof.
Letχj∈ℒW2(ℐ), 1≤j≤2n, be defined as those in Theorem 46. Then it follows that
(178)N((y,χ1)(b+1)⋮(y,χ2n)(b+1))=N(χ1T(b+1)⋮χ2nT(b+1))𝒥y(b+1)=N1y(b+1),
where N1=N(χ1,…,χ2n)T(b+1)𝒥. It is evident (χ1,…,χ2n)(b+1)=I2n. So by Lemma 14, one has that
(179)rank(M,N)=rank(M,N1)=2n,N1𝒥N1T=N𝒥NT.
Hence, the assertion follows from Theorem 46. This completes the proof.



---

*Source: 904976-2013-05-28.xml*