# p-th Order Optimality Conditions for Singular Lagrange Problem in Calculus of Variations. Elements of p-Regularity Theory

Agnieszka Prusińska, Ewa Szczepanik, Alexey Tret'yakov

► To cite this version:

Agnieszka Prusińska, Ewa Szczepanik, Alexey Tret'yakov. p-th Order Optimality Conditions for Singular Lagrange Problem in Calculus of Variations. Elements of p-Regularity Theory. 25th System Modeling and Optimization (CSMO), Sep 2011, Berlin, Germany. pp.528-537, 10.1007/978-3-642-36062-6_53. hal-01347581

HAL Id: hal-01347581

https://inria.hal.science/hal-01347581

Submitted on 21 Jul 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
p-th order optimality conditions for singular
Lagrange problem in calculus of variations.
Elements of p-regularity theory

Agnieszka Prusińska<fnref n="1" />, Ewa Szczepanik<fnref n="1" />, and Alexey Tret'yakov<fnref n="1" />,<fnref n="2" />

**Abstract.** This paper is devoted to singular calculus of variations problems with constraints which are not regular mappings at the solution point, e.i. its derivatives are not surjective. We pursue an approach based on the constructions of the *p*-regularity theory. For *p*-regular calculus of variations problem we present necessary conditions for optimality in singular case and illustrate our results by classical example of calculus of variations problem.

**Keywords:** singular variational problem, necessary condition of optimality, *p*-regularity, *p*-factor operator.

# 1 Introduction

Let us consider the following Lagrange problem:

$$J_0(x) = \int_{t_1}^{t_2} F(t, x(t), x'(t))dt \rightarrow \min \quad (1)$$

subject to the subsidiary conditions

$$H(t, x(t), x'(t)) = 0, \quad Ax(t_1) + Bx(t_2) = 0 \quad (2)$$

where $x \in C_n^2[t_1, t_2]$, $H(t, x(t), x'(t)) = (H_1(t, x(t), x'(t)), \dots, H_m(t, x(t), x'(t)))^T$, $H_i : \mathbb{R} \times \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$, $i = 1, \dots, m$, $F : \mathbb{R} \times \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$, $t \in [t_1, t_2]$, $A, B$ — $n \times n$ matrices, $C_n^l([t_1, t_2])$ — Banach spaces of $n$-dimensional $l$-times continuously differentiable vector functions with usual norms.

Let us introduce a mapping $G(x) = H(\cdot, x(\cdot), x'(\cdot))$ such that $G : X \to Y$, where $X = \{x(\cdot) \in C_n^2[t_1, t_2] : Ax(t_1) + Bx(t_2) = 0\}$, $Y = C_m[t_1, t_2]$. It means that $G$ acts as follows $G(x)t = H(t, x(t), x'(t))$. Then the system of equations (2) can be replaced by the following operator equation $G(x) = 0_Y$ (or $G(x(\cdot)) = 0_Y$). We assume that all the functions and their derivatives in (1)–(2) are $p+1$-times continuously differentiable with respect to the corresponding variables $t, x, x'$.

<footnote n="1">¹ Siedlce University of Natural Sciences and Humanities, Siedlce, Poland,
aprus@uph.edu.pl</footnote>

<footnote n="2">² System Research Institute of the Polish Academy of Sciences, Warsaw, Poland,
Dorodnicyn Computing Center of the Russian Academy of Sciences, Moscow, Russia</footnote>
Under these assumptions: $G(x) \in C^{p+1}(X)$, where by $C^{p+1}(X)$ we mean a set of $p+1$-times continuously differentiable mappings on $X$.

Let us denote $\lambda(t) = (\lambda_1(t), \dots, \lambda_m(t))^T$, $\lambda(t)H = \lambda_1(t)H_1 + \dots + \lambda_m(t)H_m$, $\lambda(t)H_x = \lambda_1(t)H_{1x} + \dots + \lambda_m(t)H_{mx}$, $\lambda(t)H_{x'} = \lambda_1(t)H_{1x'} + \dots + \lambda_m(t)H_{mx'}$.

If $\mathrm{Im} G'(\hat{x}) = Y$, where $\hat{x}(t)$ is a solution to (1)-(2), then necessary conditions of Euler-Lagrange $F_x + \lambda(t)H_x - \frac{d}{dt}(F_{x'} + \lambda(t)H_{x'}) = 0$ hold. Here, $F_x, H_x, F_{x'}$, $H_{x'}$ are partial derivatives of the functions $F(t, x(t), x'(t))$ and $H(t, x(t), x'(t))$ with respect to $x$ and $x'$, respectively.

In singular (nonregular or degenerate) case when $\mathrm{Im} G'(\hat{x}) \neq Y$, we can only guarantee that the following equations

$$ \lambda_0 F_x + \lambda(t) H_x - \frac{d}{dt}(\lambda_0 F_{x'} + \lambda(t) H_{x'}) = 0 \quad (3) $$

hold, where $\lambda_0^2 + \|\lambda(t)\|^2 = 1$, i.e. $\lambda_0$ might be equal to 0, and then we have not constructive information of the functional $F(t, x(t), x'(t))$.

**Example 1.** Consider the problem

$$ J_0(x) = \int_0^{2\pi} (x_1^2(t) + x_2^2(t) + x_3^2(t) + x_4^2(t) + x_5^2(t))dt \to \min \quad (4) $$

subject to

$$ H(t, x(t), x'(t)) = \begin{aligned} & \left( x_1'(t) - x_2(t) + x_3^2(t)x_1(t) + x_4^2(t)x_2(t) - x_5^2(t)(x_1(t) + x_2(t)) \right) \\ & \left( x_2'(t) + x_1(t) + x_3^2(t)x_2(t) - x_4^2(t)x_1(t) - x_5^2(t)(x_2(t) - x_1(t)) \right) = 0, \end{aligned} \quad (5) $$

$$ x_i(0) - x_i(2\pi) = 0, \quad i = 1, \dots, 5. $$

Here $F(t, x(t), x'(t)) = x_1^2(t) + x_2^2(t) + x_3^2(t) + x_4^2(t) + x_5^2(t)$, $A = -B = I_5$, where $I_5$ is the unit matrix of size 5 and

$$ G(x) = \begin{pmatrix} x_1'(\cdot) - x_2(\cdot) + x_3^2(\cdot)x_1(\cdot) + x_4^2(\cdot)x_2(\cdot) - x_5^2(\cdot)(x_1(\cdot) + x_2(\cdot)) \\ x_2'(\cdot) + x_1(\cdot) + x_3^2(\cdot)x_2(\cdot) - x_4^2(\cdot)x_1(\cdot) - x_5^2(\cdot)(x_2(\cdot) - x_1(\cdot)) \end{pmatrix} = 0. $$

The solution of (1)-(2) is $\hat{x}(t) = 0$. At this point $G'(0)$ is singular. Later we explain this in more details.

The corresponding Euler-Lagrange equation (see (3)) in this case is as follows:

$$ \begin{aligned} & 2\lambda_0 x_1 + \lambda_2 - \lambda'_1 + \lambda_1 x_3^2 + \lambda_1 x_5^2 - \lambda_2 x_5^2 - \lambda_2 x_4^2 = 0 \\ & 2\lambda_0 x_2 - \lambda_1 - \lambda'_2 + \lambda_1 x_4^2 + \lambda_2 x_3^2 - \lambda_1 x_5^2 - \lambda_2 x_5^2 = 0 \\ & 2\lambda_0 x_3 + 2\lambda_1 x_1 x_3 + 2\lambda_2 x_2 x_3 = 0 \\ & 2\lambda_0 x_4 + 2\lambda_1 x_2 x_4 - 2\lambda_2 x_1 x_4 = 0 \\ & 2\lambda_0 x_5 - 2\lambda_1 x_5 x_1 - 2\lambda_1 x_2 x_5 - 2\lambda_2 x_2 x_5 + 2\lambda_2 x_1 x_5 = 0 \\ & \lambda_i(0) - \lambda_i(2\pi) = 0, \quad i = 1, 2. \end{aligned} \quad (6) $$

(to simplify formulas we omit dependence of $t$ here and further in the paper).
If $\lambda_0 = 0$ we obtain the series of spurious solutions to the system (4)-(5):

$$ 
\begin{gathered}
x_1 = a \sin t, \quad x_2 = a \cos t, \quad x_3 = x_4 = x_5 = 0, \\
\lambda_1 = b \sin t, \quad \lambda_2 = b \cos t, \quad a, b \in \mathbb{R}.
\end{gathered}
$$

## 2 Elements of p-regularity theory

Let us recall the p-order necessary and sufficient optimality conditions for degenerate optimization problems (see [1]-[5]):

$$ \min \varphi(x) \qquad (7) $$

subject to

$$ f(x) = 0, \qquad (8) $$

where $f: X \to Y$ and $X, Y$ are Banach spaces, $\varphi: X \to \mathbb{R}$, $f \in C^{p+1}(X)$, $\varphi \in C^2(X)$ and at the solution point $\hat{x}$ of (7)-(8) we have: $\mathrm{Im} f'(\hat{x}) \neq Y$ i.e. $f'(\hat{x})$ is singular.

Let us recall the basic constructions of p-regularity theory which is used in investigation of singular problems.

Suppose that the space Y is decomposed into a direct sum

$$ Y = Y_1 \oplus \dots \oplus Y_p, \qquad (9) $$

where $Y_1 = \overline{\mathrm{Im} f'(\hat{x})}$, $Z_1 = Y$. Let $Z_2$ be closed complementary subspace to $Y_1$ (we assume that such closed complement exists), and let $P_{Z_2}: Y \to Z_2$ be the projection operator onto $Z_2$ along $Y_1$. By $Y_2$ we mean the closed linear span of the image of the quadratic map $P_{Z_2}f^{(2)}(\hat{x})[\cdot]^2$. More generally, define inductively,

$$ Y_i = \overline{\mathrm{span} \, \mathrm{Im} \, P_{Z_i} f^{(i)}(\hat{x})[\cdot]^i} \subseteq Z_i, \quad i = 2, \dots, p-1, $$

where $Z_i$ is a chosen closed complementary subspace for $(Y_1 \oplus \dots \oplus Y_{i-1})$ with respect to $Y$, $i = 2, \dots, p$ and $P_{Z_i}: Y \to Z_i$ is the projection operator onto $Z_i$ along $(Y_1 \oplus \dots \oplus Y_{i-1})$ with respect to $Y$, $i = 2, \dots, p$. Finally, $Y_p = Z_p$. The order p is chosen as the minimum number for which (9) holds. Let us define the following mappings

$$ f_i(x) = P_i f(x), \quad f_i: X \to Y_i, \quad i = 1, \dots, p, $$

where $P_i := P_{Y_i}: Y \to Y_i$ is the projection operator onto $Y_i$ along $(Y_1 \oplus \dots \oplus Y_{i-1} \oplus Y_{i+1} \oplus \dots \oplus Y_p)$ with respect to $Y, i = 1, \dots, p$.

**Definition 1** The linear operator $\Psi_p(\hat{x}, h) \in \mathcal{L}(X, Y_1 \oplus \dots \oplus Y_p)$, $h \in X$, $h \neq 0$

$$ \Psi_p(\hat{x}, h) = f_1'(\hat{x}) + f_2''(\hat{x})h + \dots + f_p^{(p)}(\hat{x})[h]^{p-1}, $$

is called the p-factor operator.
**Definition 2** We say that the mapping $f$ is $p$-regular at $\hat{x}$ along an element $h$, if $\mathrm{Im} \Psi_p(\hat{x}, h) = Y$.

**Remark 1** The condition of $p$-regularity of the mapping $f(x)$ at the point $\hat{x}$ along $h$ is equivalent to $\mathrm{Im} f_p^{(p)}(\hat{x})[h]^{p-1} \circ \mathrm{Ker} \Psi_{p-1}(\hat{x}, h) = Y_p$, where
$$ \Psi_{p-1}(\hat{x}, h) = f_1'(\hat{x}) + f_2''(\hat{x})h + \dots + f_{p-1}^{(p-1)}(\hat{x})[h]^{p-2} $$

**Definition 3** We say that the mapping $f$ is $p$-regular at $\hat{x}$ if it is $p$-regular along any $h$ from the set

$$ H_p(\hat{x}) = \bigcap_{k=1}^{p} \mathrm{Ker}^k f_k^{(k)}(\hat{x}) \setminus \{\mathbf{0}\}, $$

where

$$ \mathrm{Ker}^k f_k^{(k)}(\hat{x}) = \{\xi \in X : f_k^{(k)}(\hat{x})[\xi]^k = 0\}. $$

is $k$-kernel of the $k$-order mapping $f_k^{(k)}(\hat{x})[\xi]^k$.

For a linear surjective operator $\Psi_p(\hat{x}, h) : X \to Y$ between Banach spaces we denote by $\{\Psi_p(\hat{x}, h)\}^{-1}$ its right inverse. Therefore $\{\Psi_p(\hat{x}, h)\}^{-1} : Y \to 2^X$ and we have $\{\Psi_p(\hat{x}, h)\}^{-1}(y) = \{x \in X : \Psi_p(\hat{x}, h)x = y\}$. We define the norm of $\{\Psi_p(\hat{x}, h)\}^{-1}$ via the formula

$$ \|\{\Psi_p(\hat{x}, h)\}^{-1}\| = \sup_{\|y\|=1} \inf\{\|x\| : x \in \{\Psi_p(\hat{x}, h)\}^{-1}(y)\}. $$

We say that $\{\Psi_p(\hat{x}, h)\}^{-1}$ is bounded if $\|\{\Psi_p(\hat{x}, h)\}^{-1}\| < \infty$.

**Definition 4** The mapping $f$ is called strongly $p$-regular at the point $\hat{x}$ if there exists $\gamma > 0$ such that

$$ \sup_{h \in H_{\gamma}} \left\| \{\Psi_p(\hat{x}, h)\}^{-1} \right\| < \infty $$

where $H_\gamma = \{h \in X : \left\| f_k^{(k)}(\hat{x})[h]^k \right\|_{Y_k} \le \gamma, k = 1, \dots, p, \|h\| = 1 \}$.

### 3 Optimality conditions for $p$-regular optimization problems

We define $p$-factor Lagrange function

$$ \mathcal{L}_p(x, \lambda, h) = \varphi(x) + \left\langle \sum_{k=1}^{p} f_k^{(k-1)}(x)[h]^{k-1}, \lambda \right\rangle, $$

where $\lambda \in Y^*$, $f_1^{(0)}(x) = f(x)$ and

$$ \bar{\mathcal{L}}_p(x, \lambda, h) = \varphi(x) + \left\langle \sum_{k=1}^{p} \frac{2}{k(k+1)} f_k^{(k-1)}(x)[h]^{k-1}, \lambda \right\rangle. $$

Let us recall the following basic theorems on optimality conditions in non-regular case.
**Theorem 1 (Necessary and sufficient conditions for optimality) (see [1])**

Let $X$ and $Y$ be Banach spaces, $\varphi \in C^2(X)$, $f \in C^{p+1}(X)$, $f: X \to Y$, $\varphi: X \to \mathbb{R}$. Suppose that $h \in H_p(\hat{x})$ and $f$ is $p$-regular along $h$ at the point $\hat{x}$. If $\hat{x}$ is a local solution to the problem (7)–(8) then there exist multipliers, $\hat{\lambda}(h) \in Y^*$ such that

$$ \mathcal{L}'_{px}(\hat{x}, \hat{\lambda}(h), h) = 0 \Leftrightarrow \varphi'(\hat{x}) + (f_1'(\hat{x}) + \dots + f_p^{(p)}(\hat{x})[h]^{(p-1)})^* \hat{\lambda}(h) = 0. \quad (10) $$

Moreover, if $f$ is strongly $p$-regular at $\hat{x}$, there exist $\alpha > 0$ and a multipliers $\hat{\lambda}(h)$ such that (10) is fulfilled and $\mathcal{L}_{pxx}(\hat{x}, \hat{\lambda}(h), h)[h]^2 \geq \alpha \|h\|^2$ for every $h \in H_p(\hat{x})$, then $\hat{x}$ is a strict local minimizer to the problem (7)–(8).

For our purposes, the following modification of Theorem 1 will be useful (see [3]).

**Theorem 2** Let $X$ and $Y$ be Banach spaces, $\varphi \in C^2(X)$, $f \in C^{p+1}(X)$, $f: X \to Y$, $\varphi: X \to \mathbb{R}$, $h \in H_p(\hat{x})$, and $f$ is $p$-regular along $h$ at the point $\hat{x}$. If $\hat{x}$ is a solution to the problem (7)–(8), then there exist multipliers $\bar{\lambda}_i(h) \in Y_i^*$, $i = 1, \dots, p$ such that

$$ \varphi'(\hat{x}) + (f'(\hat{x}))^* \bar{\lambda}_1(h) + \dots + (f^{(p)}(\hat{x})[h]^{(p-1)})^* \bar{\lambda}_p(h) = 0, \quad (11) $$

and

$$ (f^{(k)}(\hat{x})[h]^{(k-1)})^* \bar{\lambda}_i(h) = 0, \quad k = 1, \dots, i-1, \quad i = 2, \dots, p. \quad (12) $$

Moreover, if $f$ is strongly $p$-regular at $\hat{x}$, there exist $\alpha > 0$ and multipliers $\bar{\lambda}_i(h)$, $i = 1, \dots, p$ such that (11)–(12) hold, and

$$ \left( \varphi''(\hat{x}) + \frac{1}{3} f''(\hat{x}) \bar{\lambda}_1(h) + \dots + \frac{2}{p(p+1)} f^{(p+1)}(\hat{x})[h]^{p-1} \bar{\lambda}_p(h) \right) [h]^2 \geq \\ \geq \alpha \|h\|^2, $$

for every $h \in H_p(\hat{x})$, then $\hat{x}$ is a strict local minimizer to the problem (7)–(8).

**Proof.**

We need to prove only the formula (12). From (10) we obtain

$$ \varphi'(\hat{x}) + (P_1 f'(\hat{x}) + \dots + P_p f^{(p)}(\hat{x})[h]^{(p-1)})^* \hat{\lambda}(h) = 0. $$

This expression can be transformed as follows

$$ \varphi'(\hat{x}) + f'(\hat{x})^* P_1^* \hat{\lambda}(h) + \dots + (f^{(p)}(\hat{x})[h]^{(p-1)})^* P_p^* \hat{\lambda}(h) = 0. $$

Let $\bar{\lambda}_i(h) := P_i^* \hat{\lambda}(h)$, $i = 1, \dots, p$. Then, for $k < i$, $i = 1, \dots, p$, $(f^{(k)}(\hat{x})[h]^{(k-1)})^* \bar{\lambda}_i(h) = (f^{(k)}(\hat{x})[h]^{(k-1)})^* P_i^* \hat{\lambda}(h) = (P_i f^{(k)}(\hat{x})[h]^{(k-1)})^* \hat{\lambda}(h) = 0$, which proves (12).

Now we are ready to apply this theorem to singular calculus of variations problems. Let us introduce $p$-factor Euler-Lagrange function

$$ S(x) = F(x) + \langle \lambda(t), (g_1(x) + g_2'(x)[h] + \dots + g_p^{(p-1)}(x)[h]^{p-1}) \rangle = \\ = F(x) + \lambda(t)G^{(p-1)}(x)[h]^{p-1}, $$
where $G^{(p-1)}(x)[h]^{p-1} = g_1(x) + g'_2(x)[h] + \cdots + g^{(p-1)}_p(x)[h]^{p-1}$,
$\lambda(t) = (\lambda_1(t), \dots, \lambda_m(t))^T$ and $g_k(x)$, for $k = 1, \dots, p$ are determined for the
mapping $G(x)$ similarly like $f_k(x)$, $k = 1, \dots, p$ for the mapping $f(x)$, i.e.
$g_k(x) = P_{Y_k} G(x)$, $k = 1, \dots, p$. Denote

$$
g_k^{(k-1)}(x)[h]^{k-1} = \sum_{i+j=k-1} C_{k-1}^i g_{kx^i(x')^j}^{(k-1)}(x) h^i(h')^j, \quad k=1,\dots,p,
$$

where

$$
g_{k x^i (x')^j}^{(k-1)}(x) = g_k^{(k-1)} \underbrace{x \dots x}_{i} \underbrace{x' \dots x'}_{j}(x).
$$

**Definition 5** We say that the problem (1)–(2) is *p*-regular at $\hat{x}$ along

$$
h \in \bigcap_{k=1}^{p} \mathrm{Ker}^{k} g_{k}^{(k)}(\hat{x}), \quad \|h\| \neq 0 \text{ if}
$$

$$
\mathrm{Im} \left( g'_{1}(\hat{x}) + \dots + g_{p}^{(p)}(\hat{x})[h]^{p-1} \right) = C_{m}[t_{1}, t_{2}].
$$

The following theorem holds.

**Theorem 3** Let $\hat{x}(t)$ be a solution of the problem (7)–(8) and assume that the problem is $p$-regular at $\hat{x}$ along $h \in \bigcap_{k=1}^p \text{Ker}^k g_k^{(k)}(\hat{x})$. Then there exists a multiplier $\hat{\lambda}(t) = (\hat{\lambda}_1(t), \dots, \hat{\lambda}_m(t))^T$ such that the following $p$-factor Euler-Lagrange equation

$$
\begin{equation}
\begin{split}
& S_x(\hat{x}) - \frac{d}{dt} S_{x'}(\hat{x}) = F_x(\hat{x}) + \\
& \qquad \left\langle \hat{\lambda}, \sum_{k=1}^{p} \sum_{i+j=k-1} C_{k-1}^{i} g_{x^i(x')^j}^{(k-1)}(\hat{x}) h^i(h')^j \right\rangle_{x} - \\
& \qquad - \frac{d}{dt} \left[ F_{x'}(\hat{x}) + \left\langle \hat{\lambda}(t), \sum_{k=1}^{p} \sum_{i+j=k-1} C_{k-1}^{i} g_{x^i(x')^j}^{(k-1)}(\hat{x}) h^i(h')^j \right\rangle_{x'} \right]
\end{split}
\tag{13}
\end{equation}
$$

holds.

The proof of this theorem is very similar to the one of analogous result for
the singular isoperimetric problem, see in [3], [4].

Consider again the Example 1 and (4)-(5). Here $p=2$, $\hat{x}=0$. At the beginning we substantiate that $G$ is singular at the points $\bar{x} = (a \sin t, a \cos t, 0, 0, 0)^T$. Indeed, $G'(\bar{x}) = \begin{pmatrix} (\cdot)'_1 - (\cdot)'_2 \\ (\cdot)'_2 + (\cdot)'_1 \end{pmatrix}$, where $G'(\bar{x})x(t) = \begin{pmatrix} x'_1(t) - x'_2(t) \\ x'_2(t) + x'_1(t) \end{pmatrix}$. Let us denote $\begin{pmatrix} x'_1 - x'_2 \\ x'_2 + x'_1 \end{pmatrix}$ by $x' + Lx$, where $L = \begin{pmatrix} 0 & -1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 \end{pmatrix}$.
Then $G'(\bar{x}) = (\cdot)' + L(\cdot)$ and

$\mathrm{Ker}G'(\bar{x}) = \mathrm{span}\{(\Phi_1(t), 0, 0, 0)^T, (\Phi_2(t), 0, 0, 0)^T\} \oplus \{(0, 0, x_3(t), x_4(t), x_5(t))^T,$
$x_i \in \mathcal{C}^2[0, 2\pi], i = 3, 4, 5\},$ where $\Phi_1(t) = (\sin t, \cos t)^T, \Phi_2(t) = (\cos t, -\sin t)^T,$
and moreover $\mathrm{Im} G'(\bar{x}) = (\mathrm{Ker}(G'(\bar{x})^*)^{\perp} = (\mathrm{Ker}(-\frac{d}{dt}(\cdot)' + L^T(\cdot)))^{\perp} =$
$= \{\xi \in \mathcal{C}_2[0, 2\pi] : \langle \xi, \psi_i \rangle = 0, i = 1, 2, \psi_1(t) = (\sin t, \cos t)^T, \psi_2(t) =$
$= (\cos t, -\sin t)^T\} \neq \mathcal{C}_2[0, 2\pi].$

It means that the mapping $G(x)$ is non-regular at the points $\bar{x}$. From the last relation we obtain that $Y_2 = (\mathrm{Im} G'(\bar{x}))^{\perp} = \mathrm{span}\{\psi_1, \psi_2\}$ where $\psi_1' = \psi_2$, $\psi_2' = -\psi_1$ and $\langle \Phi_i, \psi_j \rangle = \delta_{ij}$, $\langle \zeta, \eta \rangle = \int_0^{2\pi} \zeta(\tau)\eta(\tau)d\tau$.

The projection operator $P_{Y_2}$ is defined as

$$P_2\begin{pmatrix} y_1 \\ y_2 \end{pmatrix} = P_2 y = \bar{y}_1 \psi_1 + \bar{y}_2 \psi_2,$$

where $y = (y_1, y_2)^T$ and

$$\langle y - (\bar{y}_1 \psi_1 + \bar{y}_2 \psi_2), \psi_1 \rangle = 0,$$

$$\langle y - (\bar{y}_1 \psi_1 + \bar{y}_2 \psi_2), \psi_2 \rangle = 0,$$

i.e. $\frac{1}{2\pi}\langle y, \psi_1 \rangle = \bar{y}_1$, $\frac{1}{2\pi}\langle y, \psi_2 \rangle = \bar{y}_2$.

Let us point out that $P_2(x_1, \psi_1 + x_2\psi_2) = x_1\psi_1 + x_2\psi_2$.

Based on Remark 1 we can verify surjectivity of $P_2G''(\bar{x})h$ only on $\mathrm{Ker}G'(\bar{x})$, for $h \in \mathrm{Ker}G'(\bar{x}) \cap \mathrm{Ker}^2P_2G''(\bar{x})$, $h = (a \sin t, a \cos t, 1, 1, 1)^T$. In order to find $P_2G''(\bar{x})h$ let us determine

$$G''(\bar{x}) = \begin{pmatrix} 0 & 0 & \sin t & 0 & 0 \\ 0 & 0 & 0 & \cos t & 0 \\ 0 & 0 & 0 & 0 & \cos t - \sin t \\ 0 & 0 & \cos t & 0 & 0 \\ 0 & 0 & 0 & -\sin t & 0 \\ 0 & 0 & 0 & 0 & \sin t - \cos t \end{pmatrix}$$

and

$$G''(\bar{x})h = 2a \begin{pmatrix} 0 & 0 & h_3 \sin t & h_4 \cos t & h_5(\cos t - \sin t) \\ 0 & 0 & h_3 \cos t & -h_4 \sin t & h_5(\sin t - \cos t) \end{pmatrix}.$$

It is obvious that $h = (a \sin t, a \cos t, 1, 1, 1)^T$ belongs to $\mathrm{Ker}G'(\bar{x}) \cap \mathrm{Ker}^2G''(\bar{x})$ and consequently belongs to $\mathrm{Ker}G'(\bar{x}) \cap \mathrm{Ker}^2P_2G''(\bar{x})$. We have

$$G''(\bar{x})[h, x] = 2a(x_3 - x_5) \begin{pmatrix} \sin t \\ \cos t \end{pmatrix} + 2a(x_4 - x_5) \begin{pmatrix} \cos t \\ -\sin t \end{pmatrix}.$$

It means that

$P_2G''(\bar{x})[h, x] = G''(\bar{x})[h, x]$ and $G''(\bar{x})[h] \circ \mathrm{Ker}G'(\bar{x}) = \mathrm{span}\{\Phi_1, \Phi_2\} = Y_2$.
Therefore $G''(\bar{x})[h]$ is surjection. Hence, $G(x)$ is 2-regular along $h$ at the points
$\bar{x} = (a \sin t, a \cos t, 0, 0, 0)^T$. Finally, we can apply Theorem 3 with $\lambda_0 = 1$. We have constructed operator

$$ 
\begin{aligned}
G'(\bar{x}) + P_{Y_2} G''(\bar{x})h &= \\
&= \begin{pmatrix} (\cdot)'_1 \\ (\cdot)'_2 \end{pmatrix} + \begin{pmatrix} 0 & -1 & 2a \sin t & 2a \cos t & 2a(\cos t - \sin t) \\ 1 & 0 & 2a \cos t & -2a \sin t & 2a(\sin t - \cos t) \end{pmatrix}
\end{aligned}
$$

which corresponds to the following system ($F_{x'} = 0$):

$$ F_x(\bar{x}) + (G'(\bar{x}) + P_2 G''(\bar{x})h)^* \lambda = 0 \Leftrightarrow F_x(\bar{x}) + G'(\bar{x})^T \lambda + (P_2 G''(\bar{x})h)^T \lambda = 0, \Leftrightarrow $$

$$ \Leftrightarrow \begin{cases} 2\bar{x}_1 - \lambda'_1 + \lambda_2 = 0 \\ 2\bar{x}_2 - \lambda'_2 + \lambda_1 = 0 \\ 2\bar{x}_3 + 2\lambda_1 a \sin t + 2\lambda_2 a \cos t = 0 \\ 2\bar{x}_4 + 2\lambda_1 a \cos t - 2\lambda_2 a \sin t = 0 \\ 2\bar{x}_5 + 2\lambda_1 a(\cos t - \sin t) + 2\lambda_2 a(\sin t - \cos t) = 0 \end{cases} \quad (14) $$

or

$$ \Leftrightarrow \begin{cases} 2a \sin t - \lambda'_1 + \lambda_2 = 0 \\ 2a \cos t - \lambda'_2 + \lambda_1 = 0 \\ \lambda_1 \sin t + \lambda_2 \cos t = 0 \\ \lambda_1 \cos t - \lambda_2 \sin t = 0 \\ \lambda_1(\cos t - \sin t) + \lambda_2(\sin t - \cos t) = 0, \\ \lambda_i(0) - \lambda_i(2\pi) = 0, \quad i = 1, 2. \end{cases} $$

One can verify that the false solutions of (6)

$$ x_1 = a \sin t, x_2 = a \cos t, x_3 = x_4 = x_5 = 0 $$

do not satisfy the system (14) for $a \neq 0$. It means that $x_1 = a \sin t$, $x_2 = a \cos t$, $x_3 = x_4 = x_5$ do not satisfy 2-factor Euler-Lagrange equation (13)

Let us consider the same problem with higher derivatives $x'(t)$, ..., $x^{(r)}$, $r \ge 2$,

$$ J(x) = \int_{t_1}^{t_2} F(t, x(t), x'(t), \dots, x^{(r)}(t)) dt \to \min, \quad x(t) \in C_n^{2r}[t_1, t_2], $$

subject to subsidiary differential relation

$$ H(t, x(t), x'(t), \dots, x^{(r)}(t)) = \begin{pmatrix} H_1(t, x(t), x'(t), \dots, x^{(r)}(t)) \\ \vdots \\ H_m(t, x(t), x'(t), \dots, x^{(r)}(t)) \end{pmatrix} = \begin{pmatrix} 0 \\ \vdots \\ 0 \end{pmatrix}, $$

$A_k x^{(k)}(t_1) + B_k x^{(k)}(t_2) = 0$, where $A_k, B_k$ are $n \times n$ matrices, $k = 1, \dots, r$. Let $G(x) = H(\cdot, x(\cdot), \dots, x^{(r)}(\cdot))$, $G: X \to Y$, where $Y = C_m([t_1, t_2])$ and $X = \{x(\cdot) \in C_n^{2r}[t_1, t_2] : A_k x^{(k)}(t_1) + B_k x^{(k)}(t_2) = 0, k = 1, \dots, r\}$.
Moreover,

$$g_k^{(k-1)}(x)[h]^{k-1} = \sum_{i_1 + \cdots + i_r = k-1} g_{x^{i_1} \cdots x^{(r)}}^{(k-1)}[h + h' + \cdots + h^{(r)}]^{k-1}, k = 1, \ldots, p,$$

and introduce the co called p-factor Euler-Poisson function

$$K(x) = F(x) + \langle \lambda(t), (g_1(x) + g_2'(x)[h] + \cdots + g_p^{(p-1)}(x)[h]^{p-1}) \rangle$$

**Theorem 4** Let $\hat{x}(t)$ be a solution of the problem (1)-(2) and assume that this problem is p-regular at $\hat{x}$ along $h \in \bigcap_{k=1}^p \text{Ker}^k g_k^{(k)}(\hat{x})$. Then there exist a multiplier $\hat{\lambda}(t) = (\hat{\lambda}_1(t), \ldots, \hat{\lambda}_m(t))^T$ such that the following p-factor Euler-Lagrange equation

$$ \begin{aligned} K_x(\hat{x}) - \frac{d}{dt} K_{x'}(\hat{x}) + \frac{d^2}{dt^2} K_{x''}(\hat{x}) - \cdots + (-1)^r K_{x^{(r)}}(\hat{x}) = & \\ = F_x(\hat{x}) + \left\langle \hat{\lambda}(t), \sum_{k=1}^p g_k^{(k-1)}(\hat{x})[h]^{k-1} \right\rangle_x - & \\ - \frac{d}{dt} \left[ F_{x'}(\hat{x}) + \left\langle \hat{\lambda}(t), \sum_{k=1}^p g_k^{(k-1)}(\hat{x})[h]^{k-1} \right\rangle_{x'} \right] + & \\ + \cdots + (-1)^r \frac{d^r}{dt^r} \left[ F_{x^{(r)}}(\hat{x}) + \left\langle \hat{\lambda}(t), \sum_{k=1}^p g_k^{(k-1)}(\hat{x})[h]^{k-1} \right\rangle_{x^{(r)}} \right] = 0 \end{aligned} $$

holds.

The proof of Theorem 4 is similar to that one the reader can find in [4] for isoperimetric problem.

**Example 2.** Consider the following problem

$$J(x) = \int_{0}^{\pi} (x_{1}^{2}(t) + x_{2}^{2}(t) + x_{3}^{2}(t))dt \rightarrow \min \quad (15)$$

subject to

$$H(t, x(t), x'(t), x''(t)) = x_1''(t) + x_1(t) + x_2^2(t)x_1(t) - x_3^2(t)x_1(t) = 0, \quad (16)$$

$x_i(0) - x_i(\pi) = 0, x_i'(0) + x_i''(0) = 0, i = 1, 2, 3$. Here $A_1 = -B_1 = I_3$, $A_2 = B_2 = I_3$, where $I_3$ means the unit matrix of size 3.

The solution of (15)-(16) is $\hat{x}(t) = 0$. The Euler-Poisson equation in this case has the following form

$$\lambda_0 F_x + \lambda(t) H_x - \frac{d}{dt} (\lambda(t) H_{x'}) + \frac{d^2}{dt^2} (\lambda(t) H_{x''}) = 0$$

or

$$ \begin{aligned} & 2\lambda_0 x_1 + \lambda + \lambda x_2^2 - \lambda x_3^2 + \lambda'' = 0 \\ & 2\lambda_0 x_2 + 2\lambda x_2 x_1 = 0 \\ & 2\lambda_0 x_3 - 2\lambda x_3 x_1 = 0, \\ & \lambda(0) - \lambda(\pi) = 0, \lambda'(0) + \lambda'(\pi) = 0 \end{aligned} $$
and gives us the series of spurious solutions $x_1 = a \sin t$, $x_2 = 0$, $x_3 = 0$, $\lambda = b \sin t$, $\lambda_0 = 0$, $a \in \mathbb{R}$. The mapping $G(x)$ is singular at these points $x_1 = a \sin t$, $x_2 = 0$, $x_3 = 0$ and $G'(a \sin t, 0, 0)$ is non surjective.

But $G(x)$ is 2-regular at the points $\bar{x} = (a \sin t, 0, 0)$ along $h = (\sin t, \sin t, -\sin t)$. Indeed, $Y_2 = \text{span}\{\sin t\}$,

$$G'(\bar{x})h + P_{Y_2} G''(\bar{x})[h]^2 = h'' + h + 2a \sin t \int_0^\pi (\sin^2 t - \sin^2 t) \sin^2 tdt = 0.$$

It means that $h \in \text{Ker}G'(\bar{x}) \cap P_{Y_2}\text{Ker}^2G''(\bar{x})$ and

$$P_{Y_2} G''(\bar{x})h = 2a \sin t \int_0^\pi (\sin t(\cdot)_2 + \sin t(\cdot)_3) \sin^2 tdt.$$

We have

$$P_{Y_2} G''(\bar{x})h \begin{pmatrix} b \sin t \\ b \sin t \\ b \sin t \end{pmatrix} = 2ab \sin t \int_0^\pi 2 \sin^4 tdt = Y_2, b \in \mathbb{R}$$

i.e. $G$ is 2-regular at the points $\bar{x} = (a \sin t, 0, 0)$ along $h$. At these points $\bar{x}$ we can guarantee $\lambda_0 = 1$ in the 2-factor Euler-Poisson equation

$$\begin{aligned} 2a \sin t + \lambda'' + \lambda &= 0 \\ 2a \sin t \int_0^\pi \sin^3 \tau \lambda(\tau) d\tau &= 0 \\ 2a \sin t \int_0^\pi \sin^3 \tau \lambda(\tau) d\tau &= 0 \\ \lambda(0) - \lambda(\pi) &= 0, \quad \lambda(0) + \lambda(\pi) = 0 \end{aligned}$$

The first equation has no solutions for $a \neq 0$, which means that the point $\bar{x} = (a \sin t, 0, 0)^T$ is not a local solution of the considered problem.

**Acknowledgements.** Research of the second author is supported by the Russian Foundation for Basic Research Grant No 11-01-00786a and the Council for the State Support of Leading Scientific Schools Grant 5264.2012.1.

## References

1. Brezhneva O.A., Tret'yakov A.A.: Optimality conditions for degenerate extremum problems with equality constraints, SIAM J. Contr. Optim., Vol. 42 (2), 729-745 (2003)

2. Brezhneva O.A., Tret'yakov A.A.: Corrigendum: Optimality Conditions for Degenerate Extremum Problems with Equality Constraints, SIAM J. Contr. Optim. Vol. 48, No. 5, 3670-3673 (2010)

3. Belash K.N., Tret'yakov A. A.: Methods for solving degenerate problems, USSR Comput. Math. and Math. Phys. 28, 90-94 (1988)

4. Korneva I. T., Tret'yakov A. A.: Application of the factor-analysis to the calculus of variations, Proceedings of Simulation and Analysis in Problems of Decision-Making Theory. Computing Center of Russian Academy of Sciences, Moscow, 144-162 (2002). (in Russian)

5. Prusińska A., Tret'yakov A.A.: P-order Necessary and Sufficient Conditions for Optimality in Singular Calculus of Variations, Discussiones Mathematicae, Differential Inclusions, Control and Optimization 30, 269-279 (2010)