# Vector fields near the boundary of a 3-manifold

J. Sotomayor, M. Teixeira

► To cite this version:

J. Sotomayor, M. Teixeira. Vector fields near the boundary of a 3-manifold. Dynamical Systems Valparaiso 1986, pp.169-195, 1988, 10.1007/BFb0083073. hal-02558845

HAL Id: hal-02558845

https://hal.science/hal-02558845

Submitted on 6 May 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
VECTOR FIELDS NEAR THE BOUNDARY
OF A 3-MANIFOLD

by

J. Sotomayor and M.A. Teixeira

ABSTRACT. The simplest patterns of qualitative changes -bifurcations- located around a compact two dimensional submanifold, that occur on smooth one-parameter families of vector fields on a three dimensional manifold, are studied here.

# 1. INTRODUCTION.

Let $M$ be a compact 3-dimensional $C^\infty$ manifold. Denote by $\chi^r$ the space of $C^r$ vector field tangent to $M$, endowed with the $C^r$-topology where $r$ is finite and sufficiently large. Let $S$ be a compact 2-dimensional submanifold of $M$ which, in general, will be the boundary of a compact region in $M$.

## 1.1. DEFINITION. 
A vector field $X$ in $\chi^r$ is said to be $S$-stable if for a neighborhood $T(S)$ of $S$ there are neighborhoods $V$ of $X$ in $\chi^r$ and $V$ of $S$ in $M$ such that $V$ is contained in $T(S)$ and for every $Y$ in $V$ there is a homeomorphism $h(Y)$ mapping $V$ onto itself, preserving $S$, sending orbits of $X|_V$ onto orbits of $Y|_V$. A homeomorphism such as $h(Y)$ will be referred to as an $S$-equivalence between $X$ and $Y$.

The first theorem of this paper characterizes the set $\Sigma_o = \Sigma_o(S)$ of $S$-stable vector fields in $\chi^r$.

THEOREM 1. i) $X$ belongs to $\Sigma_o$ if and only if:

a) $X(p) \neq 0$ for every $p$ in $S$;

b) for every local implicit definition, $f$, of $S$ at $p$, either
$b_1) \ Xf(p) \neq 0 \ , \ b_2) \ Xf(p) = 0 \text{ and } X^2f(p) \neq 0 \text{ or } b_3) \ Xf(p) = X^2f(p) = 0$ and $\{df(p), dXf(p), dX^2f(p)\}$ are linearly independent.

ii) $\Sigma_o$ is open and dense in $\chi^r$.

The points in $S$ at which $Xf \neq 0$ (resp. $Xf = 0$) are called $S$-regular (resp. $S$-singular) points of $X$. The points of $S$ where $b_2$ is satisfied are called *fold singularities*; they form smooth curves in $S$, along which $X$ has qua-dratic contact with $S$. The set where $b_3$ is satisfied is the union of isolated points of cubic contact between $X$ and $S$, located at the extremes of the curves of fold singularities, called *cusp singularities*.

In fact, by projecting $S$ along the orbits of $X$ into a surface $N$ transverse to the orbit through a singularity, we get a Whitney singularity of fold (case $b_2$) or cuspidal type (case $b_3$) [$W$].

Call $\chi_1^r = \chi_1^r(S)$ the complement of $\Sigma_o(S)$ in $X^r$, which is the union of the sets of vector fields $\chi_1^r(a)$ and $\chi_1^r(b)$ which violate respectively conditions a) and b) in Theorem 1. We can assume that the vector fields in $\chi_1^r(b)$ have no critical points on $S$ and at least one point $p$ at which either $c_1) X^2f(p) = 0$ and $X^3f(p) = 0$ or $c_2) X^2f(p) = 0$, $X^3f(p) \neq 0$ but $df(p), d(Xf)(p), d(X^2f)(p)$ are linearly dependent.

## 1.2. DEFINITION. An $S$-hyperbolic critical point of $X$ is a critical point $p$ in $S$ such that i) it is a hyperbolic critical point of $X$ (the eigenvalues of $DX(p)$ have non vanishing real parts); ii) the eigenvalues of $DX(p)$ are pairwise distinct and the corresponding eigenspaces are transversal to $T_pS$ and iii) each pair of non complex conjugate eigenvalues of $DX(p)$ have distinct real parts.

We will denote by $\Sigma_1(a)$ the set of vector fields $X$ in $\chi_1^r(a)$ such that $X$ has a unique critical point $p$ in $S$ which is $S$-hyperbolic, while at all the other points in $S$, conditions b) of Theorem 1 are satisfied.

We may also refer to an $S$-hyperbolic critical point as a singularity of type $\Sigma_1(a)$.
1.3. **DEFINITION.** Call $\Sigma_1(b)$ the set of vector fields $X$ in $\chi_1^r(b)$ which have a unique point $p$ in $S$ such that $X(p) \neq 0$, $Xf(p) = 0$, $X^2f(p) = 0$ and one of the following conditions hold:

$Q_1: X^3f(p) \neq 0$, rank $\{df(p), d(Xf)(p), d(X^2f)(p)\} = 2$ and the function $Xf|_S$ has at $p$ a non-degenerate (Morse) critical point.

$Q_2: X^3f(p) = 0$, $X^4f(p) \neq 0$ and $p$ is a regular point of $Xf|_S$.

A point $p \in S$ which satisfies Definition 1.3 is called a singularity of $X$ of type $\Sigma_1(b)$.

1.4. **DEFINITION.** A vector field $X$ in $\chi_1^r$ is said to be $S$-stable with respect to $\chi_1^r$ if for some neighborhood $T(S)$ of $S$, there are neighborhoods $V$ of $X$ in $X^r$ and $V$ of $S$ in $M$, contained in $T(S)$, such that for every $Y$ in $V \cap \chi_1^r$ there is an $S$-equivalence between $X|_V$ and $Y|_V$.

We have the following result:

THEOREM 2. i) $X$ in $\chi_1^r$ is $S$-stable relative to $\chi_1^r$ if and only if it belongs to $\Sigma_1 = \Sigma_1(a) \cup \Sigma_1(b)$.

ii) $\Sigma_1$ is a codimension one submanifold of $X^r$.

iii) $\Sigma_1$ is open in $\chi_1^r$, in the topology induced from $X^r$.

iv) For a residual set of smooth curves $\gamma: R \to X^r$, $\gamma$ meets $\Sigma_1$ transversally and $\gamma^{-1}(\chi_2^r) = \emptyset$, $\chi_2^r = \chi_1^r - \Sigma_1$.

The background on the study of stability and bifurcation properties of vector fields on manifolds with boundary can be found in [A, P-P, Pe, $S_2$, T, V].

This paper is organized as follows. In Sections 2 and 3 the singularities of vector fields, of critical and tangential types are studied. Section 4 is devoted to the study of local structural stability of families of vector fields. In Section 5 some considerations concerning Transient Vector Fields are made. These sections prepare the way for the proof of Theorems 1 and 2 given in section 6.
## § 2. SINGULARITIES OF TANGENTIAL TYPE.

Let $p$ be a singularity of $X \in \mathcal{X}^r$ of type $\Sigma_1(b)$.

Consider coordinates $x = (x_1, x_2, x_3)$ around $p \in S$ such that $X = \frac{\partial}{\partial x_1}$ and we may solve $f(x_1, x_2, x_3) = 0$ by $x_3 = g(x_1, x_2)$ with $g(0,0) = 0$.

Fix $N = \{x_1 = 0\}$ as being the section transversal to $X$ at $p$. Define

$$ \sigma_X : S, p \to N, p $$

by

$$ \sigma_X(x_1, x_2, g(x_1, x_2)) = (0, x_2, g(x_1, x_2)) $$

and

$$ Xf(x) = \frac{\partial g}{\partial x_1}(x) = g_1(x) , \quad X^2 f(x) = g_{11}(x) , \quad X^3 f(x) = g_{111}(x) $$

and

$$ X^4 f(x) = g_{1111}(x) . $$

The mapping $\sigma_X$ is of the same class of differentiability as $X$ and it is called the projection of $S$ along the orbits of $X$ onto a transverse surface $N$. See more details in $[S_1]$.

All results obtained in the rest of this section follow directly from the Theory of Singularity of Mappings (the work of Chincaro [C] plays a fundamental role in the development of this work) by using standard known techniques.

### 2.1. LEMMA.
The point $p \in S$ is a singularity of $X$ of type $\Sigma_1(b)$ if and only if $\sigma_X$ is $C^r$ equivalent (in the usual sense of the theory of singularity of mappings) to one of the following mappings:

$$ Q_{11} : \sigma_1(x,y) = (x^2 + xy^2, y) \quad (\text{Lips}) $$

$$ Q_{12} : \sigma_2(x,y) = (x^3 - xy^2, y) \quad (\text{Bec to Bec}) $$

$$ Q_2 : \sigma_3(x,y) = (x^4 + xy, y) \quad (\text{Dove's Tail}) $$
2.2. LEMMA. There exist neighborhoods B of X in $X^r$, V of p in M and a $C^{r-3}$-function G : B → $\mathbb{R}$ such that:

a) $G(Y) = 0$ if and only if Y has a unique point $p_Y \in L_Y \cap V$ which is a singular-
ity of Y of the same type as p ; if $G(Y) \neq 0$ then $L_Y$ contains only gen -
eric singularities of Y ;

b) $dG(X) \neq 0$.

2.3. PROPOSITION. There exist neighborhoods B of X in X^r, V of p in M
such that:

i) $X|_V$ is $C^r$-equivalent to $Y|_V$ if and only if $G(Y) = 0$;

ii) If $Y_1, Y_2 \in B$ and $(G(Y_1)G(Y_2)) > 0$ then $Y_1|_V$ is $C^r$-equivalent to $Y_2|_V$.

2.4. REMARK. In the $C^\infty$ class, the equivalence obtained follows directly from Chin-
caro's work [C] and it is clear that they have to respect the stratification as
given in § 3.

2.5. NORMAL FORMS FOR A SINGULARITY OF TYPE $\Sigma_1(b)$. For simplicity's sake we give the normal forms in terms of a "straightened" vector field and a "twisted" boundary.

$$
Q_1 : X(x_1, x_2, x_3) = \frac{\partial}{\partial x_1} \quad \text{and} \quad f(x_1, x_2, x_3) = x_3 - (x_1^3 + x_1 x_2^2)
$$

$$
Q_2 : X(x_1, x_2, x_3) = \frac{\partial}{\partial x_1} \quad \text{and} \quad f(x_1, x_2, x_3) = x_3 - (x_1^3 - x_1 x_2^2)
$$

$$
Q_3 : X(x_1, x_2, x_3) = \frac{\partial}{\partial x_1} \quad \text{and} \quad f(x_1, x_2, x_3) = x_3 - (x_1^4 + x_1 x_2^2) .
$$

2.6. NORMAL FORMS FOR THE UNFOLDING OF A SINGULARITY OF TYPE $\Sigma_1(b)$.

$$
\text{i)} \quad X(x_1, x_2, x_3) = \frac{\partial}{\partial x_1} \quad \text{and} \quad f_{\lambda}(x_1, x_2, x_3) = x_3 - (x_1^{\pm} x_1 x_2 + \lambda x_1)
$$

$$
\text{ii)} \quad X(x_1, x_2, x_3) = \frac{\partial}{\partial x_1} \quad \text{and} \quad f_{\lambda}(x_1, x_2, x_3) = x_3 - (x_1^4 + x_1 x_2^2 + \lambda x_1^2).
$$

Note that the above normal forms are also obtained from the Theory of Singular-
ities of Mappings, via Lemma 2.1.
§ 3. S-HYPERBOLIC CRITICAL POINTS.

Let $p$ in $S$ be an S-hyperbolic critical point of $X$. In what follows we will study the local behavior of trajectories near such a point and their relationship with $S$.

We choose coordinates $x = (x_1, x_2, x_3)$ around $p$ such that $f(x) = x_1$,
$X = (X^1, X^2, X^3)$, $X^1(x) = x_2$. Then we have:

$$
i) \quad Xf(x) = x_2, \quad X^2f(x) = X^2(x) \quad \text{and} \quad L_X = \{x_1 = x_2 = 0\};
$$

ii) If $\alpha \neq 0$ then the order of tangency of the trajectory of $X$ through $q = (0,0,\alpha)$ with $L_X$ (at $q$) is two (this means that $q$ is a fold singularity of $X$); $\frac{\partial x^2}{\partial x_3}(0) \neq 0$, provided the eigenspaces of $DX(p)$ are transverse to $S$ at $p$.

3.1. LEMMA. If $q \in L_X - \{p\}$, then in a neighborhood of $p$, $\phi_t(q)$ does not meet $L_X$ for $t \neq 0$.

PROOF. We have to consider the following cases:

a) All eigenvalues of $DX(p)$ are real.

In this case there is, at least, a pair of invariant two dimensional manifolds
of $X$ in general position which are transverse to $L_X$ at $p$. Moreover all trajec-
tories through any point of $L_X - \{p\}$ tend asymptotically to directions transverse
to $L_X$. These facts permit us to conclude the proof in this case.

b) The eigenvalues of $DX(p)$ are $\lambda \in \mathbb{R}$ and $\lambda' = a \pm ib$ with $a \neq 0$ and $b \neq 0$.

Assume first that the vector field is linear. We may choose canonical coordinates $y = (y_1, y_2, y_3)$ around $p$ such that the orbits of $X$ are given by $\phi_t(y) = (u(t), z(t))$ where

$$
u(t) = e^{a \int (-\sin bt \quad \cos bt)} u_0
$$

and

$$
z(t) = e^{\lambda t} z_0 \quad \text{with} \quad u_0 = (y_1, y_2) \quad \text{and} \quad z_0 = y_3.
$$
Assume $z_o \neq 0$; then all trajectories of the vector field lie in the set $\pi$
given by the equation

$$|u| = K|z|^\alpha \quad \text{with} \quad K = |u_o|^\lambda |z_o|^{-a} \quad \text{and} \quad \alpha = a/\lambda.$$

Observe that:

i) if $\alpha > 0$ then the trajectories of $X$ lie in branches of a "hyperbole" (see Figure 3.1);

ii) if $0 < \alpha < 1$ then the trajectories tend (for $t \to -\infty$) to $p$, asymptotically tangent to the $(y_1, y_2)$-plane (Figure 3.2);

iii) if $\alpha > 1$ then the trajectories tend to $p$ asymptotically tangent to the $y_3$-axis (Figure 3.3).

iv) Moreover $L_X$ and $\pi$ are in general position. We check this by proving that $L_X$ is projected regularly on the $y_3$-axis and observing that the points of $L_X - \{p\}$ are fold singularities of $X$. Next we are going to prove the first assertion.

Let $G: (M, p) \to (\mathbb{R}^2, 0)$ be given by $G = (f, Xf)$ with $S = f^{-1}(0)$. Then $L_X = G^{-1}(0)$ and in the above coordinates we have

$$\frac{\partial G}{\partial y_1 \partial y_2} (0) = \begin{vmatrix} \frac{\partial f}{\partial y_1} (0) & \frac{\partial f}{\partial y_2} (0) \\ a \frac{\partial f}{\partial y_1} (0) - b \frac{\partial f}{\partial y_2} (0) & b \frac{\partial f}{\partial y_1} (0) + a \frac{\partial f}{\partial y_2} (0) \end{vmatrix}$$

Observe that $\text{Det}\left(\frac{\partial G}{\partial y_1 \partial y_2} (0)\right)$ is non zero provided that the eigenspaces of $DX_p$ are transverse to $S$ at $p$. This implies that $L_X$ projects regularly on the $y_3$-axis.
<figure><img src="image_2.png" /><figcaption>Figure 3.1. $\alpha < 0$</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 3.2. $0 < \alpha < 1$</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Figure 3.3. $\alpha > 1$</figcaption></figure>

The conclusion of the lemma in this particular case is now immediate.

We are going to prove that this situation persists in the general nonlinear
case.

We can introduce a $C^1$ change of coordinates in a full neighborhood of $p$ in
such a way that $X$ is represented by the equations:

$$ (3.2) \qquad \begin{aligned} \dot{u} &= (a + ib)u + uh_1(u,z) \\ \dot{z} &= \lambda z + zh_2(u,z) \end{aligned} \qquad \text{where } u \in \mathbb{C} \text{ and such that} $$

$$ \lim_{(u,z) \to (0,0)} h_j(u,z) = 0, \quad j = 1,2. $$

Assume for instance that $a < \lambda < 0$ (the other cases are treated in a similar way).

The solutions of (3.2.1) through $(u_0, z_0)$ satisfy

$$ \begin{aligned} u(t) &= u_0 \exp\left[a + ib\right]t + \int_0^t h_1(u(s), z(s))ds \\ z(t) &= z_0 \exp\left[\lambda t + \int_0^t h_2(u(s), z(s))ds\right]. \end{aligned} $$
So, for $z_o \neq 0$,

$$ \zeta(t) = \left| \frac{u(t)}{z(t)} \right| = \left| \frac{u_o}{z_o} \right| (\exp(a - \lambda)t) \exp\left(\int_0^t (h_1 - h_2)ds\right) $$

tends to 0, when $t \to \infty$.

Arguments similar to those used in the linear case can be used to finish the proof. □

**3.3. REMARK.** a) A proof of the above lemma can also be obtained by performing a spherical blowing up of the critical point; b) using the above notation, we observe that if $\alpha = 1$ then there exists a trajectory tending to $p$ (in positive or negative time) tangent to $S$.

**3.4. LEMMA.** There exist neighborhoods $V$ of $p$ in $M$, $B$ of $X$ in $\chi^r$ such that:

i) Each $Y \in B$ has a unique hyperbolic critical point $P_Y \in V$. Moreover, the correspondence $Y \to P_Y$ is $C^r$;

ii) There exists a $C^r$ function $G: B \to \mathbb{R}$ such that $G(Y) = 0$ if and only if $P_Y \in S$. Moreover $DG(X) \neq 0$.

**PROOF.** Part i) is immediate.

Let $N$ be a one dimensional section transverse to $S$ at $p$ and $\rho: M$, $p \to N$, $p$ be a projection "parallel" to $S$. The required function is defined by $G(Y) = \rho(P_Y)$.

**3.5. FUNDAMENTAL LEMMA.** There exist neighborhoods $V$ of $p$ in $M$, $B$ of $X$ in $\chi^r$ and $C^{r-2}$ function $\tau: B \to V$ such that $\tau(X) = p$ and $f(\tau(Y)) = 0$. Moreover:

i) $Y^2f(q) = 0$ if and only if $q = \tau(Y)$;

ii) if $P_Y \notin S$ then $df, d(Yf), dY^2f$, at $\tau(Y)$, are linearly independent (this means that $\tau(Y)$ is a cusp stable singularity of $Y$);

iii) if $P_Y \in S$ then $P_Y = \tau(Y)$ and it is an S-hyperbolic critical point of $Y$.
PROOF. Call $C(I,S) = \{h : I = (-\epsilon, \epsilon) \to S \text{ of class } C^{r-1}\}$.

Let $h_X \in C(I,S)$ be a $C^{r-1}$-parametrization of $L_X$ with $h_X(0) = p$. We can pick a neighborhood $B$ of $X$ in $X^r$ and a $C^{r-1}$-mapping $h : (B,X) \to (C(I,S), h_X)$ such that $h(Y) = h_Y$ is a $C^{r-1}$-parametrization of $L_Y$.

We have $Yf(h_Y(\alpha)) = 0$ for every $\alpha \in I$.

Define now a $C^{r-2}$-mapping $F : B \times I \to \mathbb{R}$ by

$$F(Y, \alpha) = Y^2 f(h_Y(\alpha)).$$

This mapping satisfies $F(X,0) = X^2 f(p) = 0$ and $\frac{\partial F}{\partial \alpha}(X,0) = \frac{d}{d\alpha}[X^2 f(h_X(\alpha))]_{\alpha=0} \neq 0$. (Here we applied the assertions contained in Remark 4.1).

So we can find a $C^{r-2}$-mapping $\alpha = \alpha(Y)$ which is a solution of $F(Y,\alpha) = 0$.

The required mapping is defined by $\tau(Y) = h_Y(\tau(Y))$.

Part i) follows directly from the definition of $\tau$.

We mention that in the coordinates given at the beginning of this section we have

$$\mathrm{Det} [df(p), d(Xf)(p), d(X^2f)(p)] = \frac{\partial X^2}{\partial x_3} (0) \neq 0.$$

The conclusion follows by continuity. $\square$

### 3.6. PROPOSITION. There exist neighborhoods $V$ of $p$ in $M$ and $B$ of $X$ in $X^r$ such that:

i) $(X,p)$ is $C^0$-S-equivalent to $(Y,p_Y) \in B \times V$ if and only if $G(Y) = 0$;

ii) if $Y_1, Y_2 \in B$ and $(G(Y_1) \cdot G(Y_2)) > 0$ then $Y_1|_V$ is $C^0$-S-equivalent to $Y_2|_V$ where $G$ is defined in Lemma 3.4.

In what follows we define a stratification of a neighborhood of a critical point $p \in S$ which is essential for the proof of this proposition.
### 3.7. DISTINGUISHED SETS.

Following Lemma 3.5, let $Y \in \mathcal{B}$.

First of all, we shall distinguish some subsets in $V$. We have to elaborate the following list:

$$1) \quad S, L_Y, I_Y = \bigcup_{q \in L_Y} (\phi_t(q)), I_Y \cap S$$

2) $\tau(Y)$ and $p_Y$ must be distinguished as well as the trajectories of $Y$ passing through them.

Consider the following possibilities:

3) $p_Y$ is a node; this means that the corresponding eigenvalues $\lambda_i$ are real; $i = 1,2,3$ and $\lambda_1 < \lambda_2 < \lambda_3 < 0$ (resp. $0 < \lambda_3 < \lambda_2 < \lambda_1$). We list:

3a) $W_1^S$: the invariant one dimensional manifold of $Y$ tangent to $T_1$ (resp. $W_1^U$).

3b) $W_{12}^S$: the invariant two dimensional manifold of $Y$ tangent to the linear space generated by $T_1$ and $T_2$ (resp. $W_{12}^U$).

4) $p_Y$ is a nodal-focus: this means that $\lambda_1$ is real, $\lambda_2 = a \pm ib$ with $\lambda_1 < a < 0$ (resp. $0 < a < \lambda_1$).

We list the following distinguished sets:

$W_1^S$: the invariant one dimensional manifold tangent to $T_1$ (resp. $W_1^U$).

5) $p_Y$ is a focal node: this means that $\lambda_1$ is real, $\lambda_2 = a \pm ib$ with $a < \lambda_1 < 0$ (resp. $0 < \lambda_1 < 2$). We list

$W_2^S$: the invariant two dimensional manifold tangent to $T_2$ (resp. $W_2^U$).

6) $p_Y$ is a nodal-saddle: this means that $\lambda_i$ are real, $i = 1,2,3$ and $\lambda_3 < 0 < \lambda_2 < \lambda_1$ (resp. $\lambda_1 < \lambda_2 < 0 < \lambda_3$). Using the above notations the following sets are distinguished: $W_3^S$, $W_{12}^U$ and $W_1^U$ (resp. $W_1^S$, $W_{12}^S$ and $W_3^U$).

7) $p_Y$ is a focal saddle: this means that $\lambda_1$ is real and $\lambda_2 = a \pm ib$ with $\lambda_1 < 0 < a$ (resp. $a < 0 < \lambda_1$). We list $W_1^S$ and $W_2^U$ (resp. $W_1^U$ and $W_2^S$).

8) The boundary of $V$.
9) We include in our list the intersections between each two distinguished sets listed above.

3.8. REMARK. If $\tau(Y) \neq p_Y$ then:

a) the trajectory of $Y$ passing through $\tau(Y)$ never meets the distinguished sets given in 3, 4, 5, 6 and 7 (see above). This can be seen immediately by considering the restriction of $X$ to some suitable two dimensional invariant set and using the results of [T];

b) in a similar way we can check that no trajectory of $Y$ contained in $I_Y$ is a distinguished one dimensional invariant manifold of $Y$ listed in 3, 4, 5, 6 and 7 of (3.7).

3.9. STRATIFICATION OF A NEIGHBORHOOD OF $p_Y$.

Denote by $E_o(Y)$ the union of all 0-dimensional distinguished sets defined above. If $p_Y = \tau_Y$ we observe that $E_o \subset S$. If $p_Y \neq \tau_Y$ then $p_Y \in E_o(Y)$ and $p_Y \notin S$. In this way we define $E_1(Y)$, $E_2(Y)$ and $E_3(Y) = V$. We have to consider the following cases:

a) $p_Y$ is not a saddle. In this case we may consider $X$ transverse to $\partial V$.

b) $p_Y$ is a saddle. In this case there is a one dimensional submanifold of $\partial V$ which is the set of external tangencies between $Y$ and $\partial V$. This submanifold must be included in the list of distinguished sets and of course it is far away from $L_Y$.

We see that $E_o \subset E_1 \subset E_2 \subset E_3$ define a stratification on $V$ in such a way that Whitney's Conditions are naturally satisfied [Th].

Let us now indicate how Proposition 3.6 can be obtained from the above stratification.

3.10. PROOF OF PROPOSITION 3.6.

Let $p \in S$ be a singularity of $X$ of type $\Sigma_1(a)$.

This proof will be done in a geometrical way. We will use results and techniques contained in [S.1] and [T].
PROOF OF PART i). To prove the sufficiency condition we proceed as follows:

a) we have to study each of the cases listed in 3, 4, 5, 6 and 7 of (3.7);

b) we analyze the behavior of $I_X \cap \partial V$ with respect to the intersection between $\partial V$ and the distinguished invariant manifolds of X;

c) because of the definition of a singularity of type $\Sigma_1(a)$, this behavior persists for small perturbations of X in $X_1^r$;

d) the equivalence is first defined on $\partial V$ and then extended to the full neighborhood as a stratified mapping. From Lemma 3.2 it follows that any trajectory passing through a point $q \in I_X \cap \partial V$ encounters $L_X$ just once.

Consider the following cases:

1) The eigenvalues of $DX(p)$ are real with $\lambda_1 < \lambda_2 < \lambda_3 < 0$ (Node).

Assuming the above notations, call $v_1 = w_1^S \cap \partial V = \{q_1\} \cup \{q_2\}$, $v_2 = w_{12}^S \cap \partial V$ and $U = I_X \cap \partial V = U_1 \cup U_2$ (see Figure 3.4).

<figure><img src="image_5.png" /><figcaption>Figure 3.4. The boundary of a neighborhood of a Node</figcaption></figure>

We see that, both $v_1$ and $U$ have two connected components consisting of isolated points and semi-intervals respectively. Moreover, $C\ell(U)$ and $v_2$ meet
transversally.

For $Y \in \mathcal{B}$ we have the analogous objects $\tilde{V}_1 = \{\tilde{q}_1\} \cup \{\tilde{q}_2\}$, $V_2$ and
$\tilde{U} = \tilde{U}_1 \cup \tilde{U}_2$.

We define the equivalence $h : \partial V \to \partial V$ by imposing $q_i \to \tilde{q}_i$, $U_i \to \tilde{U}_i$,
$V_2 \to \tilde{V}_2$ and then extending to $V$ (for example by preserving the rate of arc length)
respecting the stratifications.

II) The eigenvalues of $DX(p)$ are real with $\lambda_3 < 0 < \lambda_1 < \lambda_2$ (Nodal-Saddle).

Consider the sets $V_1 = W_1^u \cap V = \{p_1\} \cup \{q_1\}$, $V_2 = W_{12}^u \cap \partial V$ and $V_3 = W_3^s \cap \partial V = \{p_3\} \cup \{q_3\}$.

We orient $W_3^s$ and consider that $W_{12}^u$ separates $V$ in two regions
$V = V^+ \cup V^-$. Observe that the set of external tangencies between $X$ and $\partial V$ have
two connected components $T = T^+ \cup T^-$ with $T^+ \subset V^+$ and $T^- \subset V^-$. Moreover
$U = U_X \cap \partial V$ has four connected components $U = U_1^+ \cup U_2^+ \cup U_1^- \cup U_2^-$ where
$U_1^+ \cup U_2^+ \subset V^+$, $U_1^- \cup U_2^- \subset V^-$ and such that $X$ enters $V$ on $U_1^+ \cup U_1^-$ and leaves $V$
on $U_2^+ \cup U_2^-$ (see Figure 3.5).

<figure><img src="image_6.png" /><figcaption>Figure 3.5. The boundary of a neighborhood of a Nodal-Saddle</figcaption></figure>
We add that each component of $U$ is a semi-interval $[a,b)$ where $b$ is one of the following points $p_1, q_1, p_2$ and $q_2$ and $a$ is a point belonging to the trajectories of $X$ passing through $L_X \cap \partial V$.

Call $U_1^+ = [a_1, p_2)$, $U_2^+ = [a_2, p_1)$, $U_1^- = [a_3, q_2)$ and $U_2^- = [a_4, q_1)$.

If $Y$ is a small $C^r$-perturbation of $X$ in $X^r$, we have similar objects $\tilde{p}_1, \tilde{p}_2, \tilde{q}_1, \tilde{q}_2, \tilde{a}_1, \tilde{a}_2, \tilde{a}_3, \tilde{a}_4, \tilde{u}_1^+, \tilde{u}_2^+, \tilde{u}_1^-, \tilde{u}_2^-, \tilde{v}_1, \tilde{v}_2, \tilde{v}_3$ and $\tilde{T}$.

The required homeomorphism on $\partial V$ must preserve the above distinguished sets.

Now the conclusion of the proof in this case is straightforward.

### III) The eigenvalues of $DX(p)$ are $\lambda_1 \in \mathbb{R}$ and $\lambda_2 = a \pm ib$ with $a < \lambda_1 < 0$ (Focal-Node).

Let $V_1 = W_2^S \cap \partial V$ and $U = I_X \cap \partial V$. As before, $W_2^S$ determines on $V$ two connected components $V = V^+ \cup V^-$ and $U = U^+ \cup U^-$. With $U^+ \subset V^+$ and $U^- \subset V^-$.

Observe now that $U^+$ (resp. $U^-$) is a semi-interval with an end point belonging to $L_X \cap \partial V$ and $V_1 \subset C\ell(U^+)$ (resp. $V_1 \subset C\ell(U^-)$). This means that $U$ accumulates on $V_1$, spiraling as illustrated in Figure 3.6.

<figure><img src="image_7.png" /><figcaption>Figure 3.6. The boundary of a neighborhood of a Focal-Saddle</figcaption></figure>
We choose a small one dimensional section $\Gamma$ (in $\partial V$) transverse $V_1$ at some point $p \in V_1$.

For a small perturbation $Y$ of $X$ in $\chi_1^\Gamma$ there exist similar objects $\tilde{V}_1$, $\tilde{U}$ and $\tilde{\Gamma}$.

We have to start the construction of the equivalence by imposing that $\Gamma$ and $U$ must be sent to $\tilde{\Gamma}$ and $\tilde{U}$ respectively. Notice that:

a) $U$ (resp. $\tilde{U}$) is transverse to $S \cap \partial V$;

b) any trajectory of $X$ (resp. $Y$) meets $\partial V$ exactly once;

c) we can use the rate of arc length with respect to $S$ and $\partial V$ to extend the homeomorphism.

**PROOF OF PART ii).** Let $Y_1, Y_2 \in B$ such that $G(Y_1) \cdot G(Y_2) > 0$. We will proceed to construct the equivalence between $Y_1$ and $Y_2$ as in Part i).

Notice that any $Y \in B$ with $G(Y) \neq 0$ has also the following sets to be distinguished: $\tau(Y)$, $\alpha_Y = L_Y \cap W$ where $W$ is either some distinguished two dimensional invariant manifold of $Y$ or $\partial V$, as well as the trajectories of $X$ passing through them. Moreover:

a) if $p_Y$ is not a saddle then $U = I_Y \cap \partial V$ is a closed interval in $\partial V$ with end points on $L_Y$, transverse to the strong 2-dimensional invariant manifold of $X$ (see Figure 4.6);

b) if $p_Y$ is a saddle then $U = I_Y \cap \partial V$ is composed of two semi-intervals $U_1 = [a_1, b_1)$, $U_3 = [a_3, b_3)$ and one closed interval $U_2 = [a_2, b_2]$ where $b_1, b_3$ are in the 1-dimensional invariant manifold of $Y$ and $a_1, a_2, a_3, b_2$ are in the trajectories of $Y$ passing through $L_Y \cap \partial V$ (see Figure 3.7). It should be mentioned that $X$ enters (or leaves) $V$ in $U_1$ and $U_3$ and leaves (or enters) $V$ in $U_2$. In addition the trajectories of $Y$ passing through $\tau(Y)$ and $\alpha_Y$ meet $\partial V$ in the interior of $U_2$. We may classify the trajectories of $Y$ in the following way:

I) The trajectories of $Y$ which enter or leave $V$ in a finite time, and do not meet $L_Y$.
II) The trajectories which have a unique external tangency with $∂V$.

III) The trajectories which either enter or leave V passing through points distinct from $α_V$.

IV) The trajectories passing through $U_1, U_2, U_3$.

V) This class is composed by $p_V$.

Now we follow the same lines as in Part i) to finish the proof.

☐

The following result is an immediate consequence of the Fundamental Lemma and Proposition 3.6:

3.11. COROLLARY. If $G(Y) \neq 0$ then $Y|_V$ is S-structurally stable for perturbations of Y in $\chi^r$.

<figure><img src="image_8.png" /><figcaption>Figure 3.7. $G(Y) \neq 0$ and $p_V$ is not a saddle</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>Figure 3.8. $G(Y) \neq 0$ and $p_Y$ is a saddle</figcaption></figure>

### 3.12. REMARK. NORMAL FORMS OF S-HYPERBOLIC CRITICAL POINTS.

1. $X(x_1, x_2, x_3) = (\lambda_1 x_1, \lambda_2 x_2, \lambda_3 x_3)$ and $f(x_1, x_2, x_3) = x_1 + x_2 + x_3$ with $\lambda_1, \lambda_2, \lambda_3$ are real, nonzero and distinct.

2. $X(x_1, x_2, x_3) = (ax_1 + x_2, x_1 - ax_2, \lambda x_3)$ and $f(x_1, x_2, x_3) = x_1 + x_2 + x_3$ with $a \neq 0$, $\lambda \neq 0$ and $a \neq \lambda$.

<figure><img src="image_10.png" /><figcaption>Figure 3.9. Node</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Figure 3.10. Nodal Focus</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 3.11. Nodal-Saddle</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 3.12. Focal-Saddle</figcaption></figure>
### 3.13. REMARK. UNFOLDING OF A SINGULARITY OF TYPE $\Sigma^1(a)$-NORMAL FORMS.

1. $X(x_1, x_2, x_3) = (\lambda_1 x_1, \lambda_2 x_2, \lambda_3 x_3)$ and $f_\mu(x_1, x_2, x_3) = \mu + x_1 + x_2 + x_3$ with $\lambda_1, \lambda_2, \lambda_3$ are real, nonzero and distinct and $\mu \in \mathbb{R}$.

2. $X(x_1, x_2, x_3) = (ax_1 + x_2, x_1 - ax_2, \lambda x_3)$ and $f_\mu(x_1, x_2, x_3) = \mu + x_1 + x_2 + x_3$ with $a \neq 0$, $\lambda \neq 0$, $a \neq \lambda$ and $\mu \in \mathbb{R}$.

The above normal forms are obtained directly from Lemmas 3.4, 3.5 and Proposition 3.6.

## § 4. LOCAL STABILITY.

In this section we discuss the local structural stability (at S) of $X \in \mathcal{X}^r$.

Call $\Sigma_o(p)$ the set of all $X \in \mathcal{X}^r$ which are locally S-stable at $p \in S$. It is known (see $[S_2]$) that:

a) $X \in \Sigma_o(p)$ if and only if $p$ is either a regular point or a stable singularity of $X$;

b) $\Sigma_o(p)$ is open and dense in $\mathcal{X}^r$.

Denote $\chi_1^r(p) = \mathcal{X}^r - \Sigma_o(p)$.

We may refer to a singularity of type $\Sigma_1(a)$ or $\Sigma_1(b)$ simply as an S-quasi generic singularity.

Let $\Sigma_1(p)$ be the set of all $X \in \mathcal{X}^r(p)$ which are locally S-stable (at $p \in S$) relative to $\chi_1^r(p)$. This set is characterized by the following proposition.

### 4.1. PROPOSITION. $X \in \Sigma_1(p)$ if and only if $p$ is a quasigeneric singularity of $X$.

**PROOF.** The sufficiency condition follows from Propositions 3.6 and 2.3.

Let $p \in S$ be a singularity of $X \in \chi_1^r(p)$. We are assuming that $p$ is not a quasigeneric singularity of $X$.

If $X(p) \neq 0$ then we use results in [S.3] to find a sequence $(\chi_n)$ in $\chi_1^r(p)$ converging to $X$ such that each $\chi_n$ is never $C^\alpha$-S equivalent to $X$ (at $p$).

If $X(p) = 0$ and one of the conditions i), ii), iii) and iv) of Definition 1.2 is violated then $X \notin \Sigma_1(p)$. We will further discuss more completely the condi-
tions ii) and iii).

Assume that $X$, at $p$, satisfies the above conditions i) and iv).

If the Condition ii) is dropped, this means that two eigenvalues of $DX(p)$ are equal then we can pick a suitable invariant 2-dimensional manifold $W$ and restrict our study to $X|_W$. In this way we select a sequence $(x_n)$ in $\chi_1^r(p)$ such that $W$ is still an invariant 2-dimensional manifold of each $x_n$ and the eigenvalues of $D(x_n|_W)(p)$ are distinct. Now by using the same techniques and arguments as [T] we prove directly that $X \notin \Sigma_1(p)$.

To illustrate the last situation, suppose that the eigenvalues of $DX(p)$ are $\lambda_1 = \lambda_2 < \lambda_3 < 0$. We choose $W$ as the invariant 2-dimensional manifold of $X$ associated to $\lambda_1$. In addition we take local coordinates around $p$ such that $X(x,y,z) = (\lambda_1 x + h_1(x,y,z), \alpha x + \lambda_1 y + h_2(x,y,z), \lambda_3 z + h_3(x,y,z))$ for some nonzero scalar $\alpha$. We can choose $(x_n)$, given by

$$x_n(x,y,z) = \left\{ \lambda_1 x - \left(\frac{1}{n}\right)^2 y + h_1(x,y,z), \alpha x + \lambda_1 y + h_2(x,y,z), \lambda_3 z + h_3(x,y,z) \right\}.$$

If the eigenvalues of $DX(p)$ are $\lambda_1 = a \in \mathbb{R}$ and $\lambda_2 = a \pm ib$ with $b \neq 0$, then following Remark 3.3 we see that this situation contradicts Condition iv) of Definition 2.4. This finishes the proof. $\square$

<figure><img src="image_14.png" /></figure>

The following result is an immediate consequence of Proposition 2.3, Lemma 3.4, Proposition 3.6 and Proposition 4.1.

### 4.2. PROPOSITION. a) $\Sigma_1(p)$ is open and dense in $\chi_1^r(p)$

b) $\Sigma_1(p)$ is a codimension one submanifold of $\chi^r$ and

c) $\chi_2^r = \chi_1^r - \Sigma_1(p)$ is a closed set of codimension greater than one.

### 4.3. REMARK. Let $I = [-\epsilon, \epsilon]$ be a closed interval. Denote by $\phi^r$ the space of $C^1$-mappings $\zeta: I \to \chi^r(p)$ with the $C^1$-topology. We say $\lambda_o \in I$ is an ordinary value of $\zeta \in \phi^r$ if there is a neighborhood $N$ of $\lambda_o$ such that $\zeta(\lambda)$ is $C^0$-equivalent
to $\zeta(\lambda_o)$ (at $p$) for every $\lambda \in \mathbb{N}$; if $\lambda_o$ is not an ordinary value of $\zeta$, it is called a bifurcation value of $\zeta$. We say that $\zeta_1$ and $\zeta_2$ of $\phi^r$ are $C^0$-S-equivalent if there is a homeomorphism $h: I \to I$ and a map $H: I \to \text{Homeo}(M)$ such that $H(\lambda)$ is a $C^0$-S-equivalence (at $p$) between $\zeta_1(\lambda)$ and $\zeta_2(h(\lambda))$. With this concept we get naturally the structural stability definition in $\phi^r$. Let us denote by $A^r(p)$ the collection of the elements $\zeta \in \phi^r$ such that:

1) $\zeta(I) \subset \Sigma_o(p) \cup \Sigma_1(p)$,

2) $\zeta$ is transversal to $\Sigma_1(p)$,

3) $\zeta(-\epsilon)$ and $\zeta(\epsilon)$ are in $\Sigma_o(p)$. We have that $\zeta$ is structurally stable in $\phi^r$ if and only if $\zeta \in A^r(p)$.

§ 5. TRANSIENT VECTOR FIELDS.

We clarify the intrinsic features of the proofs of Theorem 1 and Theorem 2 by giving in this section some constructions and preparatory results concerning transient vector fields in suitable neighborhoods of $S$ in $M$. In $[P]$ the structural stability of transient vector fields on a manifold has been studied.

For any compact connected $C^\infty$ 3-manifold $P$ with a non-empty 2-dimensional boundary we say that a vector field is transient in $P$ if each integral curve of it leaves the manifold in finite positive and negative time.

Let $p \in S$ be either a S-stable or a S-quasigeneric singularity of $X \in \chi^r$, $B$ a small neighborhood of $X$ in $\chi^r$ and $F_\varepsilon$ be a fundamental neighborhood system of $p_Y$ in $S$ ($p_Y$ given either in Proposition 3.5 or in Lemma 4.4) for each $Y \in B$.

5.1. LEMMA. Let $X(p) \neq 0$. Then for each $Y \in B$ there exists a system of fundamental neighborhoods $V_\varepsilon$ around $p_Y$, diffeomorphic to $F_\varepsilon \times [-1,1]$ such that:

i) $Y$ is transient in $V_\varepsilon$;

ii) $Y$ is transverse to $F_\varepsilon \times \{\pm 1\}$ (this implies in particular that the endpoints of each trajectory of $Y$ in $V_\varepsilon$ are outside of $L_Y$).

PROOF. Consider $I_\varepsilon \times I_\varepsilon \times I_\varepsilon$ an $\varepsilon$-flow box around $p \in S$ with respect to $X$ such
that $I = [-c, c]$, $p = (0,0,0)$ and $X(x,y,z) = (1,0,0)$. The surface can be given locally by the grafic of $z = g(x,y)$ with $(x,y) \in I_c \times I_c$. This proves the asser - tions of the lemma for the original vector field $X$. Now the proof follows by con- tinuity.

☐

The following lemma is immediate.

**5.2. LEMMA.** If $X$ satisfies the hypotheses of Theorem 2 and $X(q) \neq 0$ for every $q \in S$ then there exists a neighborhood $B$ of $X$ in $X^r$ such that for each $Y \in B$ there is a fundamental system of neighborhoods $w_Y$ of $S$ where $X|_{w_Y}$ is transient.

**5.3. REMARK.** We recall from [P] that a vector field is transient if and only if it is a gradient field (for some metric) with no critical points.

<figure><img src="image_15.png" /><figcaption>Figure 5.1. The fold singularity or the Dove's Tail singularity</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 5.2. The cusp singularity</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>Figure 5.3. The Lips singularity</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Figure 5.4. The Bec to Bec singularity</figcaption></figure>
Assume for instance that $p \in S$ is a singularity of $X$ of type $\Sigma_1(a)$.
We are considering $V$ the neighborhood of $p$ as in 3.9.

As before, call $L_X(p)$ the connected component of $L_X$ containing $p$.

If $q \in L_X(p) \cap \partial V$ then consider $V_\epsilon(q)$ as in Lemma 5.2 and define the sets

$$ \Omega_\epsilon(q,p) = V - \{V_\epsilon(q) \cap V\} $$

and

$$ \Omega_{\epsilon}(p) = \bigcup_{q \in L_X \cap V} (\Omega_{\epsilon}(p,q)). $$

If $p$ is a saddle point of $X$ then $\epsilon$ must be choosen such that $0 < \epsilon < d(L_X, T_X)$ where $T_X$ is the set of external tangencies between $X$ and $\partial V$.

The proof of the next lemma is straightforward.

**5.5. LEMMA.** Assume that $X$ satisfies the hypotheses of Theorem 2 with $X(p) = 0$ and $p \in S$. Then there exists a neighborhood $B$ of $X$ in $X^r$ such that for each $Y \in B$ there is a system of fundamental neighborhood $W$ of $S - \{\Omega_\epsilon(p) \cap S\}$ such that $Y|_{W_\epsilon}$ is transient.

## § 6. PROOF OF THEOREMS.

**6.1. PROOF OF THEOREM 1.** This proof follows directly from the characterization of $\Sigma_0(p)$ for every $S$-stable singularity of a vector field, from Proposition 2.3 and from Lemma 5.1.

**6.2. PROOF OF THEOREM 2.** Part i) follows from Proposition 4.1. and Lemma 5.2.

Part ii) and iii) are immediate consequence of Proposition 4.2.

Part iv) follows from Proposition 2.3, Proposition 3.6. and Corollary 3.11.
REFERENCES

[C] E. Chincaro, Bifurcação de Aplicações de Whitney. Tese IMPA, Rio. (1979).

[P] P. Percell., Structural stability on manifolds with boundary, Topology 12 (1973), 123-144.

[P-P] Peixoto e Peixoto., Structural Stability in the plane with enlarged boundary conditions, Anais da Acad. Bras. de Cs. 31, 1959.

[S₁] J. Sotomayor., Generic one parameter families on two dimensional manifolds, Publ. Math. IHES., vol. 43 (1974).

[S₂] J. Sotomayor., Structural stability in manifolds with boundary, in "Global Analysis and its Applications", vol. 3, pp. 167-176, IEAA, Vienna (1974).

[T] M.A. Teixeira., Generic bifurcations in manifolds with boundary, J. Differen- tial Equqtions 25 (1977), 65-89.

[Th] R. Thom., Ensembles et Morphismes Stratifiés, Bull. of the American Math. Soc., v. 75, n° 2 (1969).

[V] S.M. Vishik., Vector fields near the boundary of a manifold, Vestnik Moskov. Univ. Serv. I, Mat. Meh., 27, 1 (1972), 21-28.

[W] H. Whitney., Elementary Estructure of Real Algebraic Varieties, Ann. of Math., 66, (1957).

[A] V.I. Arnold, On local problems of analysis, Vestnik Moskov. Univ. Ser. I. Mat. Meh., 25 (1970), 52-56.

J. Sotomayor
IMPA
Estrada Dona Castorina 110
Jardim Botanico
22.460 Rio de Janeiro, RJ
BRASIL

M.A. Teixeira
IMECC-UNICAMP
Caixa Postal 6065
13.081 Campinas, SP
BRASIL