数理解析研究所講究録 1445 巻 2005 年 99-109

99

Homoclinic orbits in predator-prey systems
with an Allee effect
島根大学総合理工学部
島根大学総合理工学研究科

杉江 実郎 (Jitsuro Sugie)
木本 恭子 (Kyoko Kimoto)Department of Mathematics
Shimane University

1 Introduction
A class of Gause-type predator-prey systems is one of the most important mathematical models
in population ecology. The following is a typical form of this model:$u’=ur(u)-p(u)v$ ,$v^{\ell}=v(q(u)-D)$

.

(1.1)

Here $u(t)$ and $v(t)$ are prey and predator densities, respectively; $’=d/dt;p(u)$ is the capture
rate of Prey per predator (the rate is called a functional response of predators to prey); $q(u)$ is
the growth rate of the predator enhanced by an amount proportional to the prey density; $r(u)$is the density-dependent growth rate of the Prey in the absence of any predators; $D>0$ is thedeath rate of the predator in the absence of any preys (for details, see [3, Chap. 4]).
System (1.1) includes various kinds of famous models (say, Lotka-Volterra type, Hollingtype, Ivlev type) as specific cases. It naturally has been studied by a number of authors; for example, results can be found in [1-17, 20-22] and the references cited therein. In those literatureand satisfy
it is assumed that the rates $r(u)$ , $q(u)$ and $p(u)$ are sufficiently smooth on$[0, \infty)$(i) $r(0)$ $>0$ and there exists a $K>0$ such that $r(K)=0$ and $(u-K)r(u)<0$ for $u\neq K$ ;(ii) $q(0)=0$ , $\frac{d}{du}q(u)>0$ for all $u>0$ and there exists a $u^{*}\in(0, K)$ such that $q(u^{*})=D$ ;(iii) $p(0)=0$ and $\frac{d}{du}p(u)>0$ for all $u>0$,

respectively. Let $v^{*}=u^{*}r(u^{*})/p(u^{*})$ . Then, from the assumptions above, the point
theis a unique equilibrium in the first quadrant $Q=$ { ( , ) $:u>0$ and $v>0$ } . We callpositive equilibrium hereafter.
Main subjects of system (1.1) are the existence of a unique limit cycle which surrounds the(see, [1, 9-11, 13, 16, 17, 21, 22]) and the global asymptotic stability ofpositive equilibrium
is said to be(refer to [2, 6, 12, 20]). The positive equilibrium
the positive equilibrium
is stable and if every orbit tends to .
globally asymptotically stable if$E^{*}(u^{*}, v^{*})$$u$$E^{*}$$v$$E^{*}$$E^{*}$$E^{*}$$E^{*}$$E^{*}$

ioo
It is easy to show that all solutions of (1.1) are bounded in the future and remain in the first
quadrant . Hence, the Poincar\’e-Bendixson theorem shows that if system (1.1) has a unique
is locally repulsive; that is, there exists somehas to be unstable. Indeed,
limit cycle, thensuch that each orbit of (1.1) starting at every point in goes away fromneighborhood ofdoes not enter into . The smoothness
and each orbit of (1.1) starting at every point inis locallyof the rates $r(u)$ , $q(u)$ and $p(u)$ in system (1.1) plays a major role in showing that
repulsive when system (1.1) has a unique limit cycle.
In a general dynamical system, however, an equilibrium is not always locally repulsive evenand -limit sets are the same equilibif it is unstable. An orbit is said to be homoclinic if its
rium. If a homoclinic orbit exists, then the corresponding equilibrium is unstable, but it is not$Q$$E^{*}$$E^{*}$$U$$U$$E^{*}$$U$$Q\backslash U$$U$$E^{*}$$\alpha-$$\omega$

locally repulsive.
and theSugie and Katayama [20] have shown that under the above assumptions
smoothness condition on $r(u)$ , $q(u)$ and $p(u)$ , system (1.1) has no homoclinic orbits. Fromtheir result we see that there is only two possibilities: either system (1.1) has at least one limit
is globally asymptotically stable. It is safe to say thatcycle or the positive equilibrium
system (1.1) has relatively simple global phase portraits because of the smooth rates.
A question naturally arises as to what will happen in the case of non-smooth rates. Does
homoclinic orbit appear in system (1.1) for lack of smoothness of the rates? To give an answer
to this question, we consider the predator-prey system with a non-smooth prey growth rate:$(\mathrm{i})-(\mathrm{i}\mathrm{i}\mathrm{i})$$E^{*}$$\mathrm{a}$$u’=u(\beta-\gamma|u-\alpha|)-uv$ ,where$\alpha$,$\beta$and$(u’, v^{*})=(\alpha, \beta)$$\gamma$$v’=v(u^{2}-\alpha^{2})$

,

(1.2)

are positive constants. System (1.2) has the positive equilibrium$E^{*}$

at. Since$p(u)=u$ ,$q(u)=u^{2}$,$r(u)=\beta-\gamma|u-\alpha|$

and$D=\alpha^{2}$in system (1.2), assumption (ii) holds and assumption (iii) is satisfied with $u^{*}=\alpha$ . Also,. Otherwise $r(u)$ does not satisfy theassumption (i) is satisfied with $K=\alpha+\beta/\gamma$ if
assumption (i). Note that $r(u)$ is continuous for all $u\geq 0$ , but is not differentiate at $u=\alpha$ .
Our main result is stated as follows:$\beta>\alpha\gamma$

Theorem 1.1. System (1.2) has homoclinic orbits if and only if $0<\beta\leq\gamma^{2}/8$ .
When any predators are absent, system (1.2) is reduced to$u’=u(\beta-\gamma|u-\alpha|)$

.

(1.3)

In this model, the growth rate of the prey increases with the prey density up to the peak ! andthen the rate decreases as the density increases. Hence, Eq. (1.3) exhibits the so-called “Allee, a low density leads to extinction. To be precise, if the density
effect.” In the case that, then the prey is strictly decreasing and will die out. The Allee effect isis lower than aan important phenomenon in the population dynamics$\beta<\alpha\gamma$$-\beta/\gamma$

101
From the above-mentioned relation between the prey growth rate and its density, we may saythat system (1.2) is an ecological significant model simulating the effects of underpopulationand overpopulation of the prey. In system (1.2), the Prey and the predator may coexist or they
may die together. Only one side of species cannot survive. For further details, see Sec. 4.

2 Proof of the main theorem
Consider a generalized Lienard system of the form$\dot{x}=h(y)-F(x)$ ,$\dot{y}=-g(x)$

,

(2.1)

where $=d/ds;F(x)$ and $g(x)$ are continuous on an open interval I which contains 0, and$h(y)$ is continuous and strictly increasing on R. The functions $F(x)$ , $g(x)$ and $h(y)$ satisfy
smoothness conditions for uniqueness of solutions of initial value problems. We assume that
$F(0)=0$ ,$xg(x)>0$

if $x\neq 0$$yh(y)>0$

if $y\neq 0$ .

andThen the origin is the unique critical point of (2.1). Define the nonn egative function $G(x)$ by$G(x)= \int_{0}^{x}g(\xi),d\xi$.

Under the assumption that there exists an $m>0$ such that$h(y)\leq my$(2.2)

for $y>0$ sufficiently small, Sugie [18] has presented the following sufficient conditions for
system (2.1) to have homoclinic orbits (see also [19]).
Theorem A. Assume (2.2) and suppose that
$F(x)\geq\sqrt{m}\{2\sqrt{2G(x)}-\phi(\sqrt{2G(x)})\}$

for

$|x|$

sufficiently small, where$\phi(\sigma)$(2.3)

is a nonnegative continuous function satisfying$\frac{\phi(\sigma)}{\sigma}$

is nondecreasing(2.4)

and there exists a constant $a>4$ such that$a( \int_{0}^{\sigma}\frac{\phi(\xi)}{\xi^{2}}d\xi)^{2}\leq\frac{\phi(\sigma)}{\sigma}$for$\sigma>0$

sufficiently small. Then system (2.1) has homoclinic orbits

(2.5)

102
satisfying conditions
Remark 2.1, We can easily find a nonnegative continuous fiinction
for some $>0;\phi(\sigma)=b\sigma/(\log\sigma)^{2}$(2.4) and (2.5). For example, we may takewith $0<b<1/4$ .
$\phi(u)$$\phi(\sigma)=\sigma^{1+\epsilon}$$\epsilon$

Using Theorem A, we can prove ‘if -part of Theorem 1.1.
Proof of’-part of Theorem 1.1. Changing variables$x=u-\alpha$ ,$y=\log\beta-\log v$

and

$ds$

$=udt$ ,we can transform system (1.2) into system (2.1) with
$F(x)=\gamma|x|$,$g(x)=x+ \alpha-\frac{\alpha^{2}}{x+\alpha}$

and$h(y)=\beta(1-e^{-y})$ ,, and $h(y)$ is defined onwhere $F(x)$ and $g(x)$ are defined on$xg(x)>0$ if $x\neq 0$ and $yh(y)>0$ if $y\neq 0$ . We get$(-\alpha, \infty)$$G(x)= \frac{1}{2}x^{2}+\alpha x$ $- \alpha^{2}\log(\frac{x+\alpha}{\alpha})$$\mathbb{R}$(2.6). Note that $F(0)=0$ ,.

Let $m=\beta$ . Then it is easy to check that$h(y)\leq my_{\backslash }$namely, condition (2.2). We takecontinuous for $\sigma>0$ . Since$\phi(\sigma)=\sigma\sqrt{\sigma}$. Then it is clear that$\phi(\sigma)$

is nonnegative and, condition (2.4) holds. We have$\phi(\sigma)/\sigma=\sqrt{\sigma}$$a( \int_{0}^{\sigma}\frac{\phi(\xi)}{\xi^{2}}d\xi)^{2}=4a\sigma<\sqrt{\sigma}=\frac{\phi(\sigma)}{\sigma}$

for $\sigma>0$ sufficiently small. Thus, condition (2.5) is also satisfied with arbitrary $a>4$ .
sufficiently small. Since
We will show that condition $(2,3)$ holds forenough to show that
$F(x) \geq\frac{\gamma}{2}\{2\sqrt{G(x)}-\sqrt{G(x)}\sqrt[4]{2G(x)}\}$ ,$\beta\leq\gamma^{2}/8$$|x|$, it isnamely,$|x| \geq\sqrt{G(x)}\{1-\frac{1}{2}\sqrt[4]{2G(x)}\}$(2.7)

in some neighborhood of $x=0$ . We first estimate that$g(x)<2x$for$g(x)>3x$$x>-\mathrm{a}$and $x\neq 0$ ;for $- \frac{\alpha}{2}<x<0$ .

In fact, we have$\frac{d}{dx}g(x)=1+\frac{\alpha^{2}}{(x+\alpha)^{2}}$and$\frac{d^{2}}{dx^{2}}g(x)=-\frac{2\alpha^{2}}{(x+\alpha)^{3}}<0$

(2.8)

103
for $x>-\alpha$ , and therefore,$\frac{d}{dx}g(x)|_{x=0}=2$

and $g(x)$ is a convex function. We also have$g(- \frac{\alpha}{2})=-\frac{3}{2}\alpha=3(-\frac{\alpha}{2})$.

From (2.8) we see thatfor $x>0$ ;$\sqrt{G(x)}<|x|$(2.9)

for $- \frac{\alpha}{2}<x<0$ .

$|x|<\sqrt{G(x)}<\sqrt{\frac{3}{2}}|x|$Hence, it is clear that (2.7) holds for $x\geq 0$ . Let us examine whether (2.7) is also true for $x<0$

or not. To this end, we define$f_{1}(x)= \frac{2}{\alpha}x^{2}-x+\alpha-\frac{\alpha^{2}}{x+\alpha}$.

Then, from a straightforward calculation, we conclude that there exists a$\mathrm{f}\mathrm{i}\{\mathrm{x}$$\delta$$>0$ such that) $>0$ for $-\delta<x<0$ ; that is,for$\frac{1}{\alpha}x-1<\frac{g(x)}{-2x}<0$

We may assume without loss of generality that
(2.9), we obtain$\delta$$\frac{1}{\alpha}x-1<\frac{g(x)}{2\sqrt{G(x)}}<0$$-\delta<x<0$ .

$< \min\{\alpha/2, \alpha^{2}/4\}$

for. Hence, together with$-\delta<x<0$ .

We next define(2.10).

$f_{2}(x)=-x- \sqrt{G(x)}+\frac{1}{2}\ovalbox{\tt\small REJECT} G(x)\sqrt[4]{2G(x)}$Then we have$f_{2}(0)=0$

and$\frac{d}{dx}f_{2}(x)=-1-\frac{g(x)}{2\sqrt{G(x)}}+\frac{3g(x)}{8\sqrt{G(x)}}\sqrt[4]{2G(x)}$From $(2.8\mathrm{M}2.10)$ and the fact that$\delta<\alpha^{2}/4$.

, we see that$\frac{d}{dx}f_{2}(x)<-1-(\frac{1}{\alpha}x-1)-\frac{3}{8}\frac{2x}{\sqrt{3/2}x}\sqrt[4]{2G(x)}$$< \frac{1}{\alpha}|x|-\frac{\sqrt{6}\sqrt[4]{2}}{4}\sqrt{|x|}<\frac{1}{\alpha}|x|-\frac{1}{2}\sqrt{|x|}<0$

for $-\delta<x<0$ . Hence, (2.7) holds for $-\delta<x<0$ .

By means of Theorem , we conclude that system (2.1) with (2.6) has homoclinicand so has system (1.1). The proof is complete.
$\mathrm{A}$’$\mathrm{o}\mathrm{r}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}\square$

104
To prove ‘only if’-part of Theorem 1.1 we need the following result which is a slight modification of Theorem 2.5 in [19].
Theorem B. Suppose that there exists an m $>0$ such that
(2.11)$h(y)\geq my$for $y>0$ sufficiently small. Also, suppose that
(2.12)$F(x)\leq\sqrt{m}\{2\sqrt{2G(x)}-\psi$ $(\sqrt{2G(x)})\}$

for $x>0$ or $x<0$ ,$|x|$sufficiently small, whereis a nonnegative continuous function$\psi(\sigma)$satisfying$\frac{\psi(\sigma)}{\sigma}$and there exists a$\sigma_{0}>0$

is nondecreasing and is not greater than 2(2.13)

such that$\int_{0}^{\sigma_{0}}\frac{\psi(\xi)}{\xi^{2}}d\xi=\infty$.

(2.14)

Then system (2.1) fails to have homoclinic orbits.

Remark 2.2. As simple examples satisfying conditions (2.13) and (2.14), we can citewith $>0;\psi(\sigma)=-\sigma/\log$ a for $\sigma>0$ sufficiently small.
$\epsilon\sigma$$\psi(\sigma)=$$\epsilon$of Theorem 1.1. As in the proof of ‘if -part, we see that system (1.1)
Proof of ‘only, then conditions (2.11)is equivalent to system (2.1) with (2.6). We will show that if(2.14) in Theorem are satisfied.
, there exists a $\rho>0$ such thatSince$\mathrm{i}\mathrm{f}’\cdot \mathrm{p}\mathrm{a}\mathrm{r}\mathrm{t}$$\beta>\gamma^{2}/8$$\mathrm{B}$$\beta>\gamma^{2}/8$$\frac{\gamma}{2}+\rho=\sqrt{2\beta}$

Let be chosen so that$\epsilon$$0<\epsilon$.

$< \min\{1, \gamma/2,2\rho/(\gamma+4)\}$$\frac{d}{dy}h(y)|_{y=0}=\beta$. Since,is nonnegative
condition (2.11) is satisfied with $m=\beta$. Then. Define$\sigma>0$
and continuous forand satisfies conditions (2.13) and (2.14). We have only to show thatcondition (2.12) also holds. From the same manner as in the proof of ‘if-part, we can estimatethat$|x|<\sqrt{G(x)}$for $-\alpha<x<0$ .
-$\epsilon^{2}/2$$\psi(\sigma)=\epsilon\sigma$$\psi(\sigma)$Hence, we have$\sqrt{m}\{2\sqrt{2G(x)}-\psi$ $(\sqrt{2G(x)})\}=\sqrt{m}(2-\epsilon)\sqrt{2G(x)}=\sqrt{2\beta-\epsilon^{2}}(2-\epsilon)\sqrt{G(x)}$$>(\sqrt{2\beta}-\epsilon)(2-\epsilon)\sqrt{G(x)}>(\gamma/2+\rho-\in)(2-\epsilon)|x|$$>\{(2-\epsilon)\rho-(\gamma/2+2)\epsilon+\gamma\}|x|>(\rho-\rho+\gamma)|x|$$=\gamma|x|=F(x)$105, namely, condition (2.12) is satisfied for $x<0$ , $|x|$ sufficiently small.

$\mathrm{f}\mathrm{o}\mathrm{r}-\alpha<x<0$

Thus, by Theorem we see that system (2.1) with (2.6) has no homoclinic orbits, and neither
has system (1.1). This completes the proof.
$\mathrm{B}$$\square$

3Discussion
Let us return to the Gause model (1.1). Recall that system (1.1) has three density-dependent
rates $p(u)$ , $q(u)$ and $r(u)$ . We here suppose that $p(u)$ and $q(u)$ satisfy the assumptions (iii)with $r(u^{*})>0$ , whereand (ii) in Sec. 1, respectively; and $r(u)$ is continuous on
is a constant given in the assumption (ii). Note that this condition of $r(u)$ is easier than theassumption (i) in Sec. 1. Let $v’=u^{*}r(u^{*})/p(u^{*})$ . Then system (1.1) has a unique positive
equilibrium $E’(u^{*}, v^{*})$ .
By putting$y=\log v^{*}-\log v$$x=u-u^{*}$ ,and $ds=p(u)dt$ ,$[0, \infty)$$u^{*}$

system (1.1) is transformed into the system.

$= \frac{(x+u^{*})r(x+u^{*})}{p(x+u^{*})}-v^{*}e^{-y}$,$\dot{y}=-\frac{q(x+u^{*})-D}{p(x+u^{*})}$

.

(3.1)

Let
$F(x)=v^{*}- \frac{(x+u^{*})r(x+u^{*})}{p(x+u^{*})}$

and$g(x)= \frac{q(x+u^{*})-D}{p(x+u^{*})}$

for $x>-u^{*}$ and let$h(y)=v^{*}(1-e^{-y})$

for $y\in$ R. It is clear that $F(0)=0$ and $yh(y)>0$ if $y\neq 0$ . From the assumption (ii) we also
see that $xg(x)>0$ for $x>-u$ ’ and $x\neq 0$ . Hence, system (3.1) is of Li\’enard type.
For the sake of simplicity, we denote$\Gamma(x)=\frac{(x+u^{*})r(x+u^{*})}{\sqrt{v^{*}}p(x+u^{*})}$

and$\Delta(x)=\sqrt{2\int_{0}^{x}\frac{q(\xi+u^{*})-D}{p(\xi+u^{*})}d\xi}$

for $x>-u^{*}$ . Then, from Theorems A and , we obtain the following results.
$\mathrm{B}$

Theorem 3.1. Suppose that$\Gamma(0)-\Gamma(x)\geq 2\Delta(x)-\phi(\Delta(x))$(3.2)

is a nonnegative continuous function satisfying conditions
sufficiently small, where(2.4) and (2.5). Then system (3.1) has homoclinic orbits, and so has system (1.1).

for

$|x|$

$\phi(\sigma)$

10G
Theorem 3.2. Suppose that there exists a constant c with $0<c<1$ such that
(3.3)$\Gamma(0)-\Gamma(x)\leq c\{2\Delta(x)-\psi(\Delta(x))\}$

is a nonnegative continuous fiAnction
sufficiently small, wheresatisfying conditions (2.13) and (2.14). Then system (3.1) does not have homoclinic orbits, andfor $x>0$ or $x<0$ ,$\psi(\sigma)$$|x|$

neither has system (1.1).
In single-species population dynamics, the equation(3.4)$u’= \lambda u(1-\frac{u}{K})$is very famous as the logistic growth model. The parameters A and $K$ are positive, and they arecalled the intrinsic rate of increase and the carrying capacity for the Prey density, respectively.
The AUee effect is taken no account in . (3.4).
As mentioned in Sec. 1 we can reduce system (1.2) to Eq. (1.3) in case of absence of any
predators. To contrast with system (1.2) and Eq. (1.3), we consider the system$\mathrm{B}\mathrm{q}$$u’=u(\beta-\gamma(u-\alpha))-uv$ ,(3.5)$v’=v(u^{2}-\alpha^{2})$and the equation$u’=u(\beta-\gamma(u-\alpha))$

.

(3.6)

Arranging the right-hand side of (3.6), we see that this equation becomes the logistic-type equaay and $K=\alpha+\beta/\gamma$ . System (3.5) naturally belongs to the Gausetion (3.4) with$\lambda=\beta+$

model (1.1). In case of (3.5),$\Gamma(x)=\sqrt{\beta}-\frac{\gamma}{\sqrt{\beta}}x$and

0$\Delta(x)=$.
andfor $x>-\alpha$ , because $p(u)=u$ , $q(u)=u^{2}$ , $r(u)=\beta+\alpha\gamma-\gamma u$ ,Since $\Gamma(0)-\Gamma(x)=\gamma x/\sqrt{\beta}$ for $x>-u’$, it turns out that for -a $<x<0$ , conditions (2.13),, but condition (3.2) fails to hold for an(2.14) and (3.3) are satisfied with $c=1/2$ andsatisfying conditions (2.4) and (2.5). Thus, by virtue of Theorem 3.2,
arbitrary function
we conclude that system (3.5) does not have homoclinic orbits. As a matter of fact, it is easy toprove that there exist no limit cycles in system (3.5). Hence, the positive equilibriumis always globally asymptotically stable for any parameters or, and . But, to argue this point
would carry us too far away from the purpose of this paper, and so we omit the details.
$D=\alpha^{2}$$(u^{*}, v^{*})=(\alpha, \beta)$$\psi(\sigma)=\sigma$$\phi(\sigma)$$E^{*}(\alpha, \beta)$$\beta$$\gamma$

4 Numerical Example
To illustrate our results, we give an example and a series of figures. System (1.2) has threeparameters. We fix the value of at 2 and consider the system$\gamma$$u’=u(\beta-2|u-\alpha|)-uv$ ,$v’=v(u^{2}-\alpha^{2})$

,

(3.1)

107
and the point. The origin
is the point
whose positive equilibrium$\beta/2,0)$ are also equilibria of (4.1). We say that an equilibrium is a boundary equilibrium if it ison the u-axis.
In Figures 1-4 below, we classify some orbits in five types according to the properties asfollows:$E^{*}$$E_{1}(\alpha+$$E_{0}(\mathrm{O}, 0)$$(\alpha, \beta)$(i) an orbit starting at a point $P_{i}(\mathrm{i}=1,2, \ldots , 6)$ rotates in a counterclockwise directionabout$E^{*}$

infinitely many times and approaches$E^{*};$(ii) an orbit starting at a point $Q_{i}(\mathrm{i}=1,2, 3)$ tends to the origin as increases;$t$(iii) an orbit starting at a point $R_{i}(\mathrm{i}=1_{7}2, \ldots, 6)$ approaches$E^{*}$

without rotating about$E^{*}$

perpetually ;(iv) an orbit passing through a point $S_{i}(\mathrm{i}=1,2, \ldots, 6)$ makes a loop together with,$E^{*}$

namely, a homoclinic orbit;(v) an orbit passing through a point $T_{i}(\mathrm{i}=1,2,3)$ connects

words, its -limit set is$\alpha$$E^{*}$$E^{*}$and the origin, in otherand its -limit set is the origin.
$\omega$An orbit of type (v) is said to be heteroclinic. The broken line in each figure indicates the graphof $v=\beta-2|u-\alpha|$ . For this reason, if an orbit meets the broken line, then it passes across theline in the vertical direction.
From Theorem 1.1 we see that system (4.1) has homoclinic orbits if and only if $0<\beta\leq$$1/2$ . In Figures 1-4, we describe global phase portraits of (4.1) in the cases that$(1/8,3/4)$ , 1/2. 3/4), (1/8, 1/3), (1/2, 1/3), respectively. Hence, homoclinic orbits exist onlyin Figures 3 and 4.
The first case: In Figure 1 we draw orbits of tyPe (i). Each of them starts from one of thepoints $P_{1}(0.6,0.12)$ , $P_{2}(0.6,0.27)$ , $P_{3}(0.6,0.42)$ , $P_{4}(0.6,0.57)$ , $P_{5}(0.6,0.72)$ and $P_{6}(0.6,0.87)$ .
are unstable and the positive equiand
This figure shows that two boundary equilibria
is globally asymptotically stable. This means that the Prey and the predator always
librium$(\alpha, \beta)=$$E_{1}$$E_{0}$$E^{*}$

exist together.
The second case: As sketched in Figure 2, another boundary equilibrium appears at the point
and theare unstable, and the origin$E_{2}(\alpha-\beta/2,0)$ . The boundary equilibria
andare locally asymptotically stable. There are orbits of two types (i) andpositive equilibrium(ii); each orbit starting at the point $P_{1}(4.1)$ $0.08)$ , $P_{2}(1.2,0.18)$ or $P_{3}(1.2,0.28)$ approaches.
and each orbit starting at the point $Q_{1}(1.2_{7}0.32)$ , $Q_{2}(1.2,0.42)$ or $Q_{3}(1.2,0.52)$ tends toHence,
is not globally asymptotically stable. Both species may become extinct.
The third case: There are nine orbits in Figure 3. Each orbit passing through the point
other orbits are not ho(0.125, 0.05), (0.125, 0.14) or (0.125, 0.23) is homoclinic.
. Each of them starts at the point
moclinic, but they also tend to the positive equilibrium(0.5, 0.02), $R_{2}(0.5,0.1)$ , $R_{3}(0.5,0.18)$ , $R_{4}(0.5,0.26)$ , $R_{5}(0.5, 0.34)$ or $R_{6}(0.5, 0.42)$ . The
are unstable Since homoclinic orbits exist, needless to say, theandboundary equilibria$E_{1}$$E_{0}$$E_{2}$$E^{*}$$E^{*}$$E_{0}$$E^{*}$$\mathfrak{M}\mathrm{e}$$S_{1}$$S_{3}$$S_{2}$$E^{*}$$R_{1}$$E_{0}$$E_{1}$

108
is unstable, it is globally attractive
positive equilibrium
is not even stable. Althoughbecause every orbit approaches it. Hence, the prey coexists with the predator.
The fourth case: Figure 4 is more complicated than Figures 1-3. A remarkable featureand theof this case is the appearance of heteroclinic orbits joining the positive equilibriumboundary equilibrium . Such heteroclinic orbit passes through one of the points $T_{1}(0.2,0.03)$ ,and $T_{3}(0.2,0.15)$ . Each orbit passing through the point $S_{1}(0.5,0.05)$ , $S_{2}(0.5,0.15)$
or $S_{3}(4.1)0.25)$ is homoclinic. There are orbits of two types (ii) and (iii) besides hetero(1, 0.21), $Q_{2}(10.27)\}$clinic orbits and homoclinic orbits; each orbit starting at the pointand each orbit starting at the point $R_{1}(1, 0.02)$ , $R_{2}(1,0.08)$ oror $Q_{3}(1,0.33)$ tends to$R_{3}(1,0.14)$ approaches. As in the second case, the boundary equilibria and E2 areunstable. Since orbits of type (ii) or (v) appear, there is a high risk of extinction of the prey and$E^{*}$$E^{*}$$E^{*}$$E_{0}$$T_{2}(0.2,0\backslash 09)$

$Q_{1}$$E_{0}$$E_{1}$$E^{*}$

the predator.
$v$

$u$

Fig. 1. Phase portrait for system (44) with$\alpha=1/8$ and $\beta=3/4$ .
$v$

Fig. 2. Phase portrait for system (4.1) with$\alpha=1/2$ and $\beta=3/4$ .
$v$

Fig. 4. Phase portrait for system (4.1 ) with$\alpha=1/2$ arrd $\beta=1/3$ .

109
References[1] K.-S. Cheng, Uniqueness of a limit cycle for a predator-prey system, SIAM J. Math. Anal

12 (1981) 541-548.

[2] K.-S. Cheng, S.-B. Hsu and S.-S. Lin, Some results on global stability of a predator-preysystem. J. Math. Biol 12 (19Sl) 115-126.
[3] H. I. Freedman, Deterministic Mathematical Models in Population Ecology (Marcel
Dekker, 1980).
[4] G. F. Gause, The Struggle for Existence (Williams and Wilkins, 1934).

[5] C. S. Holling, The components of predation as revealed by a study of small-mammal
predation of the European pine sawfly, Canad. Entomol 91 (1959) 293-320.
[6] S.-B. Hsu, On global stability of a predator-prey system, Math Bioscl 39 (1978) 1-10.
[7] V. S. Ivlev, Experimental Ecology of the Feeding of Fishes (Yale University Press, 1961).

[8] N. D. Kazarinoff and P. van der Driessche, A model predator-prey system with functional

response, Math Bioscl 39 (1978) 125-134.

[9] R. E. Kooij and A. Zegeling, A predator-prey model with Ivlev’s functional response, J.
Math. Anal AppL 19S (1996) 473-489.
[10] R. E. Kooij and A. Zegeling, Qualitative properties of two-dimensional predator-prey systems, Nonlinear Anal 29 (1997) 693-715.
[11] Y. Kuang, Nonuniqueness of limit cycles of Guase-type predator-prey systems, Appl Anal
29 (1988) 269-287.
[12] Y. Kuang, Global stability of Guase-type predator-prey systems, J. Math. Biol 28 (1990)

463-474.
[13] Y. Kuang and H. I. Freedman, Uniqueness of limit cycles of Guase-type models of
predator-prey systems, Math. Bioscl 88 (1988) 67-84.
[14] R. M. May, Stability and Complexity in Model Ecosystems (Princeton University Press,
1974).

[15] M. L. Rosenzweig and R. H. MacArthur, Graphical representation and stability conditionsof predator-prey interactions, Am. Naturalist. 97 (1963) 209-223.
[16] J. Sugie, Two-parameter bifurcation in a predator-prey system of Ivlev type, J. Math. Anal.
Appl 217 (1998) 349-371.

[17] J. Sugie, Uniqueness of limit cycles in a predator-prey system with Holling-type functional
response, Quart. Appl Math 58 (2000) 577-590.
[18] J. Sugie, Homoclinic orbits in generalized Li\’enard systems, J. Math. Anal AppL (2005),

in press.
[19] J. Sugie and T. Hara, Existence and non-existence of homoclinic trajectories of the Lienard
system, Discrete Contin. Dynam. Systems 2 (1996) 237-254.
[20] J. Sugie and M. Katayama, Global asymptotic stability of a predator-prey system ofHolling type, Nonlinear Anal 38 (1999) 105-121.
[21] J. Sugie, R. Kohno and R. Miyazaki, On a predator-prey system of Holling type, Proc.
Amen Math. Soc. 125 (1997) 2041-2050.
[22] J. Sugie, K. Miyamoto and K. Morino, Absence of limit cycles of a predator-prey systemwith a sigmoid functional response, Appl Math. Lett 9 (1996) 85-90.