# Stability and Coexistence of a Diffusive Predator-Prey System with Nonmonotonic Functional Response and Fear Effect

Authors: Xiaozhou Feng; Hao Sun; Yangfan Xiao; Feng Xiao
Journal: Complexity
 (2020)

---

## Abstract

This paper investigates the diffusive predator-prey system with nonmonotonic functional response and fear effect. Firstly, we discussed the stability of the equilibrium solution for a corresponding ODE system. Secondly, we established a priori positive upper and lower bounds for the positive solutions of the PDE system. Thirdly, sufficient conditions for the local asymptotical stability of two positive equilibrium solutions of the system are given by using the method of eigenvalue spectrum analysis of linearization operator. Finally, the existence and nonexistence of nonconstant positive steady states of this reaction-diffusion system are established by the Leray–Schauder degree theory and Poincaré inequality.


---

## Body

## 1. Introduction

In order to describe the evolution of biological populations in the ecosystem, some mathematical theories and methods have been used to establish the corresponding biological mathematical model, which has become a research hotspot. In recent years, the research on biological models such as the predator-prey model has aroused the attention of many scientists and biologists. The predator-prey model of PDE forms is an important branch of reaction-diffusion equations. The dynamic relationship between predator and their prey is one of the dominant themes in ecology and mathematical ecology. During these thirty years, the investigation on the prey-predator models has been developed, and more realistic models are derived in view of laboratory experiments. Moreover, the research on the prey-predator models has been studied from various views and obtained many good results (see [1–22] and the references therein).However, many studies have shown that only the presence of predators in front of the prey can affect the size of the prey population, and the effect is even greater than the effect of direct predation. Although some biologists have realized that the relationship between the prey and the predator cannot be simply described as direct killing, we should take the fear of the prey population into account. At present, there are few research studies on establishing corresponding mathematical models to explain this phenomenon.For every specific prey-predator system, we know that the functional response of the predator to the prey density is very important, which represents the specific transformation rule of the two organisms. In [8], Pang and Wang considered a predator-prey model incorporating a nonmonotonic functional response which is called the Monod–Haldane or Holling type IV function:(1)ut−d1Δu=ru1−uk−puvd+u2,x∈Ω,t>0,vt−d2Δv=−mv+cpuvd+u2,x∈Ω,t>0,∂nu=∂nv=0,x∈∂Ω,t>0,where ∂n is the outward directional derivative normal to ∂Ω. Model (1) describes a prey population u which serves as food for a predator with population v. The parameters r,d,m,p,c,andk are assumed to be only positive values: the positive constant k is the carrying capacity of the prey and the positive constant m is the death rate of the predator; r is the growth rate of prey u; and the positive constants d1andd2 are the diffusion coefficients.In this paper, based on the above model, in order to describe the evolution law of the population in the ecosystem more specifically, we will consider the natural mortality and fear effect of the prey population and establish the corresponding PDE model within a fixed bounded domainΩ⊂RN with smooth boundary at any given time and the natural tendency of each species to diffuse to areas of smaller population concentration [7–10]. Hence, we will investigate the following reaction-diffusion system under the homogeneous Neumann boundary conditions as follows:(2)ut−d1Δu=ru1+kv−au−bu2−puvd+u2,x∈Ω,t>0,vt−d2Δv=−mv+cpuvd+u2,x∈Ω,t>0,∂u∂n=∂u∂n=0,x∈∂Ω,t>0,ux,0=u0x≥0,vx,0=v0x≥0,x∈Ω,where u0andv0 are continuous functions of x. u and v stand for the densities of prey and predators, respectively. The parameters a,b,c,d,r,k,p,andm are assumed to be only positive constants. a and m denote the intrinsic death rate of prey u and predator v, respectively. k stands for the fear factor of prey to predator. The remaining parameters refer to (1). Here, fu,v=uv/d+u2 stands for Monod–Haldane functional response.The main aim of this paper is to study the nonconstant positive steady states of (2), that is, the existence and nonexistence of nonconstant positive classical solutions of the following elliptic system:(3)−d1Δu=ru1+kv−au−bu2−puvd+u2,x∈Ω,−d2Δv=−mv+cpuvd+u2,x∈Ω,∂u∂n=∂u∂n=0,x∈∂Ω.The rest of this paper is arranged as follows. In Section2, we discuss the stability of the equilibrium of the ODE system which corresponds to system (2). In Section 3, we establish a priori positive upper and lower bounds for the positive solutions of the PDE system. In Section 4, sufficient conditions for the local asymptotical stability of two positive equilibrium solutions of the system are established by using the method of eigenvalue spectrum analysis of linearization operator. In Section 5, the existence and nonexistence of nonconstant positive steady states of this reaction-diffusion system are established by using the Leray–Schauder degree theory, which demonstrates the effect of large diffusivity.


## 2. Stability of the ODE Model

The goal of this section is to discuss the stability of the ODE model; we give the ordinary differential equation of system (3) as follows:(4)dudt=ru1+kv−au−bu2−puvd+u2,dvdt=−mv+cpuvd+u2,u0=u0,v0=v0.By the similar method to [7], for (4), we can get the following result.Lemma 1.
Under initial conditionsu0>0andv0>0, the solution of system (4) is nonnegative and ultimately bounded which implies(5)limsupt⟶∞Lt≤maxcr−a+m4bm,c0u+v0,hereL≐cu+v.Next, we will calculate the equilibrium point of system (4), and the result is given as follows.Theorem 1.
System (4) always has an extinction equilibrium point E0=0,0. If r>a, then system (4) has only the equilibrium point E1=r−a/b,0. If r>a, cp−2md>0, and u˜i<r−a/b, then system (4) has two positive constant equilibrium points E2,i=u˜i,v˜i, i = 1 and 2.Proof.
It is easy to see that all equilibrium points of system (4) satisfy the following equations:(6)ru1+kv−au−bu2−puvd+u2=0,−mv+cpuvd+u2=0.
It follows that system (4) obviously has equilibrium points E0=0,0 and E1=r−a/b,0 with r>a. Next, we consider the existence of positive constant equilibrium point E2,i. By calculating the second equation of (6), we directly get(7)u˜1=cp+c2p2−4m2d2m,u˜2=cp−c2p2−4m2d2m,where cp−2md>0 ensures that u˜i>0i=1,2. Substituting u˜i>0i=1,2 into (6) and combining the two equations of system (6), we can obtain the following equation:(8)ru˜i1+kv˜−au˜i−bu˜i2−pu˜iv˜d+u˜i2=0,i=1,2.
Through the same solution deformation calculation, we can get(9)hv≐a2v˜2−a1v˜+a0=0,i=1,2,where a0=cu˜ir−a−bu˜i;a1=−cau˜ik+cbu˜i2k+m;a2=−mk. According to the Vieta theorem, we get(10)v˜1+v˜2=−a1a2<0,v˜1v˜2=a0a2.
Obviously, ifa0≤0, then fv=0 has no positive constant solution; if a0>0, then fv=0 has only one positive constant solution. Thanks to the same sign a0 and r−a−bu˜i, a0>0 implies r−a−bu˜i>0, which ensures that fv=0 has only one positive constant solution denoted by vi. Thus, system (4) has two positive constant equilibrium points E2,i=u˜i,v˜i, i = 1 and 2. The proof is complete.Theorem 2.
Ifr≤a, then E0=0,0 is globally asymptotically stable. If r>a, then E0=0,0 is unstable.Proof.
The proof of Theorem2 is similar to that of Theorem 2 of [9]; hence, we omit it.Theorem 3.
Assumer>a. If r−acpb+ma−mr−mb2d<0, then E1=r−a/b,0 is locally asymptotically stable. If r−acpb+ma−mr−mb2d>0, then E1=r−a/b,0 is unstable.Proof.
Through mathematical calculation, we obtain the Jacobian matrix of system (5) at the equilibrium point E1=r−a/b,0 as follows:(11)JE1=a−r−krui−puid+ui20r−acpb−mr+ma−mb2db2d+r−a2.
Obviously, whenr−acpb+ma−mr−mb2d<0 and both eigenvalues of JE1 have negative real parts, then E1=r−a/b,0 is locally asymptotically stable; when r−acpb+ma−mr−mb2d>0 and JE1 has a positive eigenvalue, then E1=r−a/b,0 is unstable. The proof is complete.Theorem 4.
Assumer>a. If 2m2vi/c2pb<ui<c2p2/2m2−d (i = 1 and 2), then E2,i=u˜i,v˜ii=1and2 is locally asymptotically stable. If ui<2m2vi/c2pb (i = 1 and 2), then E2,i is unstable.Proof.
ForE2,i=u˜i,v˜i≐ui,vi, the corresponding Jacobian matrix is given by(12)JE2,i=−bui+2pui2vid+ui22−krui1+kvi2−puid+ui2cpvid+ui2−2cpui2vid+ui220≐cpuid+ui2−m,i=1,2.
By simplifying, we can get(13)JE2,i=−bui+2m2vic2p−krui1+kvi2−mccpvid+ui2−2m2vicp0,i=1,2,detμI−JE2,i=μ2−μ⋅tracJE2,i+detJE2,i,where(14)detJE2,i=krui1+kvi2+mc⋅cpvid+ui2−2m2vicp,tracJE2,i=−bui+2m2vic2p.
It is easy to get thatdetJE2,i>0 and tracJE2,i<0 under these conditions 2m2vi/c2pb<ui<c2p2/2m2−d. Then, two eigenvalues of the matrix JE2,i have negative real parts. Therefore, the equilibrium E2,i=u˜i,v˜i is locally asymptotically stable. If ui<2m2vi/c2pb and the matrix JE2,i has one positive eigenvalue, then E2,i is unstable.


## 3. A Priori Estimates on Equation (3)

The main purpose of this section is to give a priori upper and lower bounds for the positive solutions. To this aim, we first recall the following maximum principle due to [23, 24].Lemma 2.
Supposegx,ω∈CΩ×R1.
Ifωx∈C2Ω×C1Ω¯ satisfies(15)Δωx+gx,ωx≥0≤0,x∈Ω,∂ω∂n≤0≥0,x∈∂Ω,and ωx0=maxΩ¯ω, then gx0,ωx0≥0≤0.Theorem 5.
Ifr>a and ackd1<bd2, u,v=ux,vx is a positive solution of (3). Then, the solution ux,vx of (3) yields(16)0<u<r−ab,0<v<ℵ,maxx∈Ω¯v>δ>0,where δ≐r−a1/ak−cd1/b and ℵ≐cd1+cd2r−a/mr−a/b.Proof.
By Lemma2, if u reaches its maximum at x¯∈Ω¯, it follows from the first equation of (3) that(17)r−a−bux¯≥r1+kvx¯−a−bux¯−pvx¯d+u2x¯≥0.
Hence,0<u≤r−a/b. Setting ϖ=cd1u+d2v and combining two equations of system (3), we obtain(18)−Δϖ=−cd1Δu−d2Δv=cru1+kv−acu−bcu2−mv,that is,(19)Δϖ+cru1+kv−acu−bcu2−mv=0.
Ifϖ reaches its maximum at x0∈Ω¯, then(20)crux01+kvx0−acux0−bcu2x0−mvx0≥0,which results in(21)mvx0≤crux01+kvx0−acux0−bcu2x0,<crux0−acux0−bcu2x0,<crux0−acux0.
Thus,(22)vx0<cr−amux0≤cr−a2mb.
Thanks toϖ=cd1u+d2v, we know that(23)d2maxΩ¯v<maxΩ¯ϖ=ϖx0=cd1ux0+d2vx0≤cd1+cd2r−amr−ab.
Letφ=−cd1u+d2v, then,(24)−Δφ=cd1Δu−d2Δv=2cpuvd+u2−cru1+kv+acu+bcu2−mv,that is,(25)Δφ+2cpuvd+u2−cru1+kv+acu+bcu2−mv=0.
Ifφ reaches its maximum at x1∈Ω¯, then(26)crux11+kvx1−acux1−bcu2x1≤2cpux1vx1d+u2x1−mvx1,<2cpux1vx1d−mvx1<N,where N=2cpr−a/bdℵ, which means that(27)1+kvx1>crux1acux1+bcu2x1+N.
Lettinghu=cru/acu+bcu2+N, it is easy to get the maximum of hu, that is, hu<r/a. Thus, vx1>r−a/ak:(28)d2maxΩ¯v<maxΩ¯φ=ϖx1=−cd1ux1+d2vx1≥r−ad2ak−cd1b.
By (23) and (28), we have proved Theorem 5.
According to Theorems5 and 1, we can easily get the following conclusion.Theorem 6.
Ifr>a, r−acp−2md>0, and u<r−a/b, then system (3) has two positive constant solutions E2,i=u˜i,v˜i, i = 1 and 2.Theorem 7.
Suppose thatu,v is a nonnegative classical solution of (3). If r≤a, then u,v is always zero solution.Proof.
Integrating the equation foru in (3) over Ω by parts, we get(29)0=−d1∫Ω∂u∂ndx=−∫Ωd1Δudx=∫Ωur1+kv−a−bu−pvd+u2dx.
Thus,(30)0≤∫Ωbu2dx=∫Ωur1+kv−a−pvd+u2dx≤∫Ωr−audx≤0.
Hence,u≡0. Substituting u=0 into the second equation of (3), we get(31)0=−∫Ωd2Δvdx=−m∫Ωvdx,and then, v≡0. The proof is complete.


## 4. Stability of the Equilibrium of Equation (3)

The goal of this section is to investigate the local and global stability of the positive constant steady stateu˜i,v˜i=U˜. We first discuss the local stability of U˜i. To this end, we need to introduce some notations for developing our result.Let(32)fu,v=ru1+kv−au−bu2−puvd+u2,gu,v=−mv+cpuvd+u2.Therefore, system (3) becomes the following forms:(33)ut=d1Δu+fu,v,vt=d1Δv+gu,v.It follows that two positive solutionsu˜i,v˜ii=1and2 satisfy(34)fu˜i,v˜i=gu˜i,v˜i=0,u˜1,v˜1=cp+c2p2−4m2d2m,v˜1,u˜2,v˜2=cp−c2p2−4m2d2m,v˜2,where v˜ii=1,2 satisfies(35)r1+kv−a−bu˜i−pvd+u˜i2=0,i=1,2.In order to get the linearization operator of (3) at the positive constant solution u˜i,v˜i, for (33), we calculate the partial derivatives with respect to u and v, respectively, at the equilibrium point u˜i,v˜i, as follows:(36)fuu˜i,v˜i=−bu˜i+2pu˜i2v˜id+u˜i22≐f1,fvu˜i,v˜i=−kru˜i1+kv˜i2pu˜id+u˜i2≐f2<0,(37)guu˜i,v˜i=cpv˜id−u˜i2d+u˜i22≐g1<0asi=1;g1>0asi=2,guu˜i,v˜i=cpu˜id+u˜i2−m≐g2=0.Next, give some results as follows:(i)
0=μ0<μ1<μ2<μ3<⋯<μi⋯<∞ are the eigenvalues of −Δ on Ω under homogeneous Neumann boundary condition, and mi is the algebraic multiplicity of eigenvalue μi.(ii)
ϕij,1≤j≤mi, are the normalized eigenfunctions corresponding to μi, and then ϕiji≥0,1≤j≤mi are the orthonormal basis of L2Ω.Ifd1μ1<f1, then there exists iα≐iαα,Ω satisfying(38)d1μi<f1,i<iα,1≤iα<+∞.Defining(39)d˜2i=d˜2iα,Ω≐min1≤i≤iαd2i,d2i=f2g1d1μi−f1μi.Theorem 8.
(1)
Iff1<0 and d2>f2g1/d1μi−f1μi, then the positive constant steady state u˜1,v˜1 of (3) is locally asymptotically stable. If f1<0 and d2<f2g1/d1μi−f1μi, then the positive constant steady state u˜1,v˜1 of (3) is unstable. If f1>d1μ1, then the positive constant steady state u˜1,v˜1 of (3) is unstable.(2)
Iff1<0, then the positive constant steady state u˜2,v˜2 of (3) is locally asymptotically stable; if f1>0, then the positive constant steady state u˜2,v˜2 of (3) is unstable.Proof.
The linearization operator of (3) at the positive constant solution u˜i,v˜ii=1and2 can be written as(40)Lσ=d1Δ+f1f2g1d2Δ+g2,where f1,f2,g1,andg2 are defined in (36)-(37). According to the linear stability theory, if the real parts of all eigenvalues of Lσ are negative, then u˜i,v˜i is locally asymptotically stable; if there exists the positive real part of the eigenvalue of Lσ, then u˜i,v˜i is unstable.
Letϕxandψx be the eigenfunctions corresponding to the eigenvalue λ. Then,(41)Lσϕx,ψx=λϕx,ψx,that is,(42)d1Δϕx+f1−λϕx+f2ψx,d2Δψx+g2−λψx+g1ϕx=0,0.
Let(43)ϕ=∑0≤i<∞,1≤j≤miai,jϕij,ψ=∑0≤i<∞,1≤j≤mibi,jϕij,Bi=f1−d1μi−λf2g1g2−λ−d2ui.
Thus, the eigenvalue equation of system (3) is equivalent to(44)∑0≤i<∞,1≤j≤miBiaij,bijTϕij=0.
λ is an eigenvalue of Lσ if and only if there exists i≥0 such as detBi=0, which is equivalent to(45)λ2+P˜iλ+Q˜i=0,where(46)P˜i=d1+d2μi−f1,Q˜i=−d2μif1−d1μi−f2g1.
Next, we check the stability ofu˜1,v˜1 and u˜2,v˜2, respectively.(1)
For the caseu˜1,v˜1. If f1>d1μ1, then Q˜0<0 with i<iα. Hence, u˜1,v˜1 are unstable. If f1<0 and d2>f2g1/d1μi−f1μi, then P˜i,Q˜i>0. Thus, Reλ<0 and u˜1,v˜1 are locally asymptotically stable. If f1<0 and d2<f2g1/d1μi−f1μi, then Q˜i<0. Thus, Reλ>0 and u˜1,v˜1 are unstable.(2)
For the caseu˜2,v˜2, if f1<0, then P˜iandQ˜i>0. Thus, Reλ<0 and u˜2,v˜2 are locally asymptotically stable. If f1>0, then P˜0=−f1<0andQ˜0>0. Thus, there exists some unstable Reλ>0 and u˜2,v˜2. The proof is complete.


## 5. Nonconstant Positive Steady States of Equation (3)

The main purpose of this section is to provide some sufficient conditions for the existence and nonexistence of a nonconstant positive solution of (3) by using the Leray–Schauder degree theory [12, 24, 25]. Next, we will establish these results by dividing into two sections.


### 5.1. Nonexistence

The goal of this part is to establish some sufficient conditions for the nonexistence of nonconstant positive solutions of (3) by the energy norm method. Some related research studies can refer to [8–10]. For the ease of notation, we set(47)u¯=1Ω∫Ωudx,v¯=1Ω∫Ωvdx,where u,v is a positive solution of (3).Theorem 9.
Ifd1>r−a/μ1 and cp<2md, then system (3) has no nonconstant positive classical solution.Proof.
Letω=u−u¯andχ=v−v¯, then ∫Ωωdx=∫Ωχdx=0.
Multiplying the second equation ofv by χ and integrating over Ω by parts, we obtain(48)d2∫Ω∇χ2dx=∫Ω−mv+cpuvd+u2χdx,=∫Ωcpuvd+u2−cpuv¯d+u2χdx+∫Ωcpuv¯d+u2−cpu¯v¯d+u¯2χdx+m∫Ωv¯χdx−m∫Ωvχdx,=∫Ωcpud+u2χ2dx+∫Ωcpv¯d−uu¯d+u2d+u¯2ωχdx−m∫Ωχ2dx,=∫Ωcpv¯d−uu¯d+u2d+u¯2ωχdx−∫Ωm−cpud+u2χ2dx.
Thanks to the boundary ofuandv (see in Theorem 5), we get(49)d2∫Ω∇χ2dx≤C1∫Ωωχdx−C2∫Ωχ2dx,where C1=cpdℵ>0andC2=m−cp/2d. Applying Cauchy inequality, we obtain(50)ωχ≤ω24C2/C1+C2C1χ2=C1ω24C2+C2C1χ2.
Substituting (50) into (49) and using Poincaré inequality, we get(51)d2∫Ω∇χ2dx≤C124C2∫Ωω2dx≤C124C2μ1∫Ω∇ω2dx.
Becauseu and v are nonnegative, we obtain(52)−d1Δu≤ur−a.
Multiplying the above equation ofu by ω and integrating over Ω by parts, using Poincaré inequality again, we obtain(53)d1∫Ω∇ω2dx≤r−a∫Ωω2dx≤r−aμ1∫Ω∇ω2dx,that is,(54)∫Ω∇ω2dx≤r−ad1μ1∫Ω∇ω2dx.
Ifd1>r−a/μ1, then ∫Ω∇ωdx≡0. Substituting ∫Ω∇ωdx≡0 into (51), we get(55)∫Ω∇ωdx=∫Ω∇χdx≡0,which implies that ωandχ are always constant. The proof is complete.


### 5.2. Global Existence

The goal of this section is to establish the global existence of nonconstant positive classical solutions to (3) when the diffusion coefficients d1 and d2 vary while the parameters r,a,b,c,d,m,p, and k are fixed.For simplicity, we only consider the existence of nonconstant positive classical solutions nearu˜2,v˜2 which are denoted by u˜,v˜. Letting u^=u−u˜,v^=v−v˜, system (3) can be written as follows:(56)−d1Δu=f1u^+f2v^+f3,−d2Δv=g1u^+g3,where f3≐ou^,v^ and g3≐ou^,v^.Define the spaceS and E as follows:(57)S=u^,v^:−u^<u^<r−ab−u^,−v˜<v^<ℵ−v˜,,E=u,v:u,v∈C1+βΩ,∂u∂n=∂v∂n=0,x∈∂Ω.SetU=u^,v^, and then, (56) becomes(58)U=KU+HU,where(59)KU=2f1Gd1u^+f2Gd1v^,g1Gd2u^+g1Gd2v^,HU=Gd1f3,Gd2g3=oU,Gd1=−d1Δ+f1−1,Gd2=−d2Δ+g1−1.Theorem 10.
Supposecp<2md and d1μ1<f1<min−f2/2,d1μ2. If the principal eigenvalue μ1 has an odd multiple eigenfunction and d2>d21, then system (3) has at least one nonconstant positive solution.Proof.
It is easy to see that system (3) has no solution on the boundary of the space S. According to Homotopy invariance of degree theory, for all d1>0, degI−K−H,E∩S,0 is well defined and constant. Next, we will prove(60)degI−K−H,E∩S,0=1.
Assume that0,0 is an isolated fixed point of I−K−H, then(61)degI−K−H,E,0=indexI−Kd1,0,0=−1τ,where τ is the sum of algebraic multiplicity of all eigenvalues greater than 0. Assume that λ is the eigenvalue of K−I and the corresponding eigenfunction is denoted by ξ,η, then(62)−d1λ+1Δξ=1−λf1ξ+f2η,−d2λ+1Δη=g1ξ+λg1η.
Let(63)ξ=∑0≤i<∞,1≤j≤miaijϕij,η=∑0≤i<∞,1≤j≤mibijϕij.
Thus, the eigenvalue equation of system (3) is equivalent to(64)∑0≤i<∞,1≤j≤miBiaij,bijTϕij=0,where(65)Bi=1−λf1−d11+λμif2g1−λg1−d21+λμi.
Thus, all eigenvalues ofK−I satisfy(66)f1g1+d1μig1+d1d2μi2+d2μif1λ2+2d1d2μi2+d1μig1−f1g1λ+d1d2μi2−d2f1μi−f2g1=0.
Set(67)Pi=2d1d2μi2+d1μig1−f1g1andQi=d1d2μi2−d2f1μi−f2g1.
Notice thatf2<0,g1>0,andf1>0 are defined in (36)-(37), and it follows that Lσ and Kd1−I have the same number of eigenvalues. Thanks to Theorem 9, let d1=maxr−a/μ1,f1/μ1+1, then(68)degI−K−H,E∩S,0=indexI−Kd1,0,0=1.
Next, we will calculate the sum of algebraic multiplicity of all eigenvalues ofK−I greater than 0.
Owing tod1μ1<f1andd2>d21, it is easy to see that(69)Qi=−d2μ1f1−d1μ1−f2g1<−d21μ1f1−d1μ1−f2g1=0,P0<0,Q0>0.
Sinced1μ2>f1, then(70)Qi>0,Pi=2d1d2μi2+d1μig1−f1g1>0,i≥2.
Hence,K−I has positive eigenvalues such as λ1,λ2,andλ3, where λ1 satisfies(71)f1g1+d1μ1g1+d1d2μ12+d2μ1f1λ2+2d1d2μ12+d1μ1g1−f1g1λ+d1d2μ12−d2f1μ1−f2g1=0,and λ2andλ3 satisfy(72)f1g1+d1μ0g1+d1d2μ02+d2μ0f1λ2+2d1d2μ02+d1μ0g1−f1g1λ+d1d2μ02−d2f1μ0−f2g1=0.
Therefore, we denote the algebraic multiplicity ofλj by τj with d2>d21, and then(73)τ=∪i=13τi,τj=dim∪i=1∞kerAji,Aj=K−λj+1I,j=1,2,3.
We notice thatτ1=m1 if and only if kerA1∩RA1=0. It is easy to see that(74)kerA1=f2,d1μ11+λ1+f1λ1−1Tϕ1j,1≤j≤m1.
Thanks toRA1=kerA1∗⊥, suppose ξ,η=kerA1∗, then(75)K∗ξ,η⊥=λ1+1ξ,η⊥,K∗ξ,η⊥=2f1Gd1ξ+g1Gd2η,f2Gd1ξ+g1Gd2η,that is,(76)2f1Gd1ξ+g1Gd2η=λ1+1ξ,f2Gd1ξ+g1Gd2η=λ1+1η.
According to the definition ofGd1 and Gd2, we obtain(77)d1d2f2−2f1λ1+1Δξ=fξξ−fηη,d1d2f2−2f1λ1+1Δη=gξξ−gηη,where(78)fξ=2d2f11−λ1f1−f2+d1g1f2λ1+1,fη=2d2f121+λ1−d1g1f2+2λ1f1,gξ=d2f21−λ1f1−f2+d1g1f2λ1+1,gη=d2f1f21+λ1−g1f2+2λ1f1.
By calculating, it follows that(79)Bi∗=fξ+d1d2f2−2f11+λ1μ1−fηgξ−gη+d1d2f2−2f11+λ1μ1,kerA1∗=1+λ1f1+d1μ1,f2+λ1−1f1+λ1+1d1μ1Tϕ1j,1≤j≤m1.
Next, we will prove that(80)f21+λ1f1+d1μ1+d1μ11+λ1+λ1−1f1×f2+λ1−1f1+λ1+1d1μ1≠0.
Becauseλ1 satisfies(81)f1g1+d1μig1+d1d2μi2+d2μif1λ2+2d1d2μi2+d1μig1−f1g1λ+d1d2μi2−d2f1μi−f2g1=0,define the function Hλ as follows:(82)Hλ=f1g1+d1μig1+d1d2μi2+d2μif1λ2+2d1d2μi2+d1μig1−f1g1λ+d1d2μi2−d2f1μi−f2g1.
It is easy to see thatH1>0, so λ1<1. Notice that(83)f21+λ1f1+d1μ1+d1μ11+λ1+λ1−1f1×f2+λ1−1f1+λ1+1d1μ1<f21+λ1f1+d1μ1+2d1μ1f2+2d1μ1<f21+λ1f1+d1μ1+2d1μ1f2+2f1<,≠0.
It follows thatkerA1∩kerA1∗⊥=0, so τ1=m1. Because λ0 is a simple eigenvalue, by the similar method, we can get the following result:(84)kerA2=f2,f1λ1−1Tϕ11,kerA2∗=λ1−1f1+d1μ1,f2+λ1−1f1Tϕ11.
It is easy to get(85)f2λ1+1f1+d1μ1+f1λ1−1f2+λ1−1f1<0.
ThenkerA2∩kerA2∗⊥=0, and it follows that τ2=1. Similarly, we can get τ3=1. Combining the above results, we get(86)τ=τ1+τ2+τ3=m1+1+1.
Soτ is an odd number, and we get(87)degI−K−H,E∩S,0=−1τ=−1.
The proof is complete.


## 5.1. Nonexistence

The goal of this part is to establish some sufficient conditions for the nonexistence of nonconstant positive solutions of (3) by the energy norm method. Some related research studies can refer to [8–10]. For the ease of notation, we set(47)u¯=1Ω∫Ωudx,v¯=1Ω∫Ωvdx,where u,v is a positive solution of (3).Theorem 9.
Ifd1>r−a/μ1 and cp<2md, then system (3) has no nonconstant positive classical solution.Proof.
Letω=u−u¯andχ=v−v¯, then ∫Ωωdx=∫Ωχdx=0.
Multiplying the second equation ofv by χ and integrating over Ω by parts, we obtain(48)d2∫Ω∇χ2dx=∫Ω−mv+cpuvd+u2χdx,=∫Ωcpuvd+u2−cpuv¯d+u2χdx+∫Ωcpuv¯d+u2−cpu¯v¯d+u¯2χdx+m∫Ωv¯χdx−m∫Ωvχdx,=∫Ωcpud+u2χ2dx+∫Ωcpv¯d−uu¯d+u2d+u¯2ωχdx−m∫Ωχ2dx,=∫Ωcpv¯d−uu¯d+u2d+u¯2ωχdx−∫Ωm−cpud+u2χ2dx.
Thanks to the boundary ofuandv (see in Theorem 5), we get(49)d2∫Ω∇χ2dx≤C1∫Ωωχdx−C2∫Ωχ2dx,where C1=cpdℵ>0andC2=m−cp/2d. Applying Cauchy inequality, we obtain(50)ωχ≤ω24C2/C1+C2C1χ2=C1ω24C2+C2C1χ2.
Substituting (50) into (49) and using Poincaré inequality, we get(51)d2∫Ω∇χ2dx≤C124C2∫Ωω2dx≤C124C2μ1∫Ω∇ω2dx.
Becauseu and v are nonnegative, we obtain(52)−d1Δu≤ur−a.
Multiplying the above equation ofu by ω and integrating over Ω by parts, using Poincaré inequality again, we obtain(53)d1∫Ω∇ω2dx≤r−a∫Ωω2dx≤r−aμ1∫Ω∇ω2dx,that is,(54)∫Ω∇ω2dx≤r−ad1μ1∫Ω∇ω2dx.
Ifd1>r−a/μ1, then ∫Ω∇ωdx≡0. Substituting ∫Ω∇ωdx≡0 into (51), we get(55)∫Ω∇ωdx=∫Ω∇χdx≡0,which implies that ωandχ are always constant. The proof is complete.


## 5.2. Global Existence

The goal of this section is to establish the global existence of nonconstant positive classical solutions to (3) when the diffusion coefficients d1 and d2 vary while the parameters r,a,b,c,d,m,p, and k are fixed.For simplicity, we only consider the existence of nonconstant positive classical solutions nearu˜2,v˜2 which are denoted by u˜,v˜. Letting u^=u−u˜,v^=v−v˜, system (3) can be written as follows:(56)−d1Δu=f1u^+f2v^+f3,−d2Δv=g1u^+g3,where f3≐ou^,v^ and g3≐ou^,v^.Define the spaceS and E as follows:(57)S=u^,v^:−u^<u^<r−ab−u^,−v˜<v^<ℵ−v˜,,E=u,v:u,v∈C1+βΩ,∂u∂n=∂v∂n=0,x∈∂Ω.SetU=u^,v^, and then, (56) becomes(58)U=KU+HU,where(59)KU=2f1Gd1u^+f2Gd1v^,g1Gd2u^+g1Gd2v^,HU=Gd1f3,Gd2g3=oU,Gd1=−d1Δ+f1−1,Gd2=−d2Δ+g1−1.Theorem 10.
Supposecp<2md and d1μ1<f1<min−f2/2,d1μ2. If the principal eigenvalue μ1 has an odd multiple eigenfunction and d2>d21, then system (3) has at least one nonconstant positive solution.Proof.
It is easy to see that system (3) has no solution on the boundary of the space S. According to Homotopy invariance of degree theory, for all d1>0, degI−K−H,E∩S,0 is well defined and constant. Next, we will prove(60)degI−K−H,E∩S,0=1.
Assume that0,0 is an isolated fixed point of I−K−H, then(61)degI−K−H,E,0=indexI−Kd1,0,0=−1τ,where τ is the sum of algebraic multiplicity of all eigenvalues greater than 0. Assume that λ is the eigenvalue of K−I and the corresponding eigenfunction is denoted by ξ,η, then(62)−d1λ+1Δξ=1−λf1ξ+f2η,−d2λ+1Δη=g1ξ+λg1η.
Let(63)ξ=∑0≤i<∞,1≤j≤miaijϕij,η=∑0≤i<∞,1≤j≤mibijϕij.
Thus, the eigenvalue equation of system (3) is equivalent to(64)∑0≤i<∞,1≤j≤miBiaij,bijTϕij=0,where(65)Bi=1−λf1−d11+λμif2g1−λg1−d21+λμi.
Thus, all eigenvalues ofK−I satisfy(66)f1g1+d1μig1+d1d2μi2+d2μif1λ2+2d1d2μi2+d1μig1−f1g1λ+d1d2μi2−d2f1μi−f2g1=0.
Set(67)Pi=2d1d2μi2+d1μig1−f1g1andQi=d1d2μi2−d2f1μi−f2g1.
Notice thatf2<0,g1>0,andf1>0 are defined in (36)-(37), and it follows that Lσ and Kd1−I have the same number of eigenvalues. Thanks to Theorem 9, let d1=maxr−a/μ1,f1/μ1+1, then(68)degI−K−H,E∩S,0=indexI−Kd1,0,0=1.
Next, we will calculate the sum of algebraic multiplicity of all eigenvalues ofK−I greater than 0.
Owing tod1μ1<f1andd2>d21, it is easy to see that(69)Qi=−d2μ1f1−d1μ1−f2g1<−d21μ1f1−d1μ1−f2g1=0,P0<0,Q0>0.
Sinced1μ2>f1, then(70)Qi>0,Pi=2d1d2μi2+d1μig1−f1g1>0,i≥2.
Hence,K−I has positive eigenvalues such as λ1,λ2,andλ3, where λ1 satisfies(71)f1g1+d1μ1g1+d1d2μ12+d2μ1f1λ2+2d1d2μ12+d1μ1g1−f1g1λ+d1d2μ12−d2f1μ1−f2g1=0,and λ2andλ3 satisfy(72)f1g1+d1μ0g1+d1d2μ02+d2μ0f1λ2+2d1d2μ02+d1μ0g1−f1g1λ+d1d2μ02−d2f1μ0−f2g1=0.
Therefore, we denote the algebraic multiplicity ofλj by τj with d2>d21, and then(73)τ=∪i=13τi,τj=dim∪i=1∞kerAji,Aj=K−λj+1I,j=1,2,3.
We notice thatτ1=m1 if and only if kerA1∩RA1=0. It is easy to see that(74)kerA1=f2,d1μ11+λ1+f1λ1−1Tϕ1j,1≤j≤m1.
Thanks toRA1=kerA1∗⊥, suppose ξ,η=kerA1∗, then(75)K∗ξ,η⊥=λ1+1ξ,η⊥,K∗ξ,η⊥=2f1Gd1ξ+g1Gd2η,f2Gd1ξ+g1Gd2η,that is,(76)2f1Gd1ξ+g1Gd2η=λ1+1ξ,f2Gd1ξ+g1Gd2η=λ1+1η.
According to the definition ofGd1 and Gd2, we obtain(77)d1d2f2−2f1λ1+1Δξ=fξξ−fηη,d1d2f2−2f1λ1+1Δη=gξξ−gηη,where(78)fξ=2d2f11−λ1f1−f2+d1g1f2λ1+1,fη=2d2f121+λ1−d1g1f2+2λ1f1,gξ=d2f21−λ1f1−f2+d1g1f2λ1+1,gη=d2f1f21+λ1−g1f2+2λ1f1.
By calculating, it follows that(79)Bi∗=fξ+d1d2f2−2f11+λ1μ1−fηgξ−gη+d1d2f2−2f11+λ1μ1,kerA1∗=1+λ1f1+d1μ1,f2+λ1−1f1+λ1+1d1μ1Tϕ1j,1≤j≤m1.
Next, we will prove that(80)f21+λ1f1+d1μ1+d1μ11+λ1+λ1−1f1×f2+λ1−1f1+λ1+1d1μ1≠0.
Becauseλ1 satisfies(81)f1g1+d1μig1+d1d2μi2+d2μif1λ2+2d1d2μi2+d1μig1−f1g1λ+d1d2μi2−d2f1μi−f2g1=0,define the function Hλ as follows:(82)Hλ=f1g1+d1μig1+d1d2μi2+d2μif1λ2+2d1d2μi2+d1μig1−f1g1λ+d1d2μi2−d2f1μi−f2g1.
It is easy to see thatH1>0, so λ1<1. Notice that(83)f21+λ1f1+d1μ1+d1μ11+λ1+λ1−1f1×f2+λ1−1f1+λ1+1d1μ1<f21+λ1f1+d1μ1+2d1μ1f2+2d1μ1<f21+λ1f1+d1μ1+2d1μ1f2+2f1<,≠0.
It follows thatkerA1∩kerA1∗⊥=0, so τ1=m1. Because λ0 is a simple eigenvalue, by the similar method, we can get the following result:(84)kerA2=f2,f1λ1−1Tϕ11,kerA2∗=λ1−1f1+d1μ1,f2+λ1−1f1Tϕ11.
It is easy to get(85)f2λ1+1f1+d1μ1+f1λ1−1f2+λ1−1f1<0.
ThenkerA2∩kerA2∗⊥=0, and it follows that τ2=1. Similarly, we can get τ3=1. Combining the above results, we get(86)τ=τ1+τ2+τ3=m1+1+1.
Soτ is an odd number, and we get(87)degI−K−H,E∩S,0=−1τ=−1.
The proof is complete.


## 6. Conclusion

This paper investigates the diffusive predator-prey system with nonmonotonic functional response and fear effect under homogeneous Neumann boundary conditions. Firstly, we discussed the stability of the equilibrium of the ODE system which corresponds to system (2). Secondly, we established a priori positive upper and lower bounds for the positive solutions of the PDE system by maximum principle (see Theorems 5–7), which means that the density of the two organisms must be in a bounded range if they can coexist in the system. Thirdly, sufficient conditions for the local asymptotical stability of two positive equilibrium solutions of the system are proved by using the method of eigenvalue spectrum analysis of linearization operator (see Theorem 8), which shows that the density values of the two organisms are locally stable at the positive equilibrium point when the model parameters meet certain conditions. Finally, the existence and nonexistence of nonconstant positive steady states of this reaction-diffusion system are established by using the Leray–Schauder degree theory (see Theorems 9-10). The results of Theorem 9 show that the two organisms cannot coexist in the biological system when the diffusion rate of the prey satisfies some specific conditions. However, the results of Theorem 10 show that two species can coexist in a biological system if their diffusivity satisfies certain conditions at the same time. In fact, we have used different methods to study the similar dynamic behavior of the solution on another predator-prey model in reference [26], and one can refer to it for more detailed results.



---

*Source: 8899114-2020-12-29.xml*