Lagrange multipliers on manifolds

Lagrange multipliers on manifolds

We discuss in this article the theoretical aspects of
the

Lagrange multiplier method

.

To enhance understanding,
proofs and intuitive explanations of the Lagrange multipler method
will be given from several different viewpoints,
both

elementary

and advanced.

Contents:

1

Statements of theorem

1.1

Formulation with differential forms

1.2

Formulation with gradients

1.3

Formulation with tangent maps

2

Proofs

2.1

Beautiful abstract proof

2.2

Clumsy, but down-to-earth proof

3

Intuitive interpretations

3.1

Normals to tangent hyperplanes

3.2

With infinitesimals

3.3

As rates of substitution

4

Stationary points

1

Statements of theorem

Let

N

be a

n

-dimensional

differentiable manifold

(without

boundary

), and

f

:

N

→

ℝ

, and

g

i

:

N

→

ℝ

, for

i

=

1

,

…

,

k

, be

continuously differentiable

.
Set

M

=

⋂

i

=

1

k

g

i

-

1

⁢

(

{

0

}

)

.

1.1

Formulation with differential forms

Theorem 1

.

Suppose

d

⁢

g

i

are

linearly independent

at each point of

M

.
If

p

∈

M

is a

local minimum

or

maximum

point of

f

restricted to

M

,
then there exist

Lagrange multipliers

λ

1

,

…

,

λ

k

∈

R

, depending on

p

,
such that

d

⁢

f

⁢

(

p

)

=

λ

1

⁢

d

⁢

g

1

⁢

(

p

)

+

⋯

+

λ

k

⁢

d

⁢

g

k

⁢

(

p

)

.

Here,

d

denotes the

exterior derivative

.

Of course, as in one-dimensional

calculus

, the condition

d

⁢

f

⁢

(

p

)

=

∑

i

λ

i

⁢

d

⁢

g

i

⁢

(

p

)

by itself does not guarantee

p

is a

minimum

or maximum point, even locally.

1.2

Formulation with gradients

The version of Lagrange multipliers typically
used in calculus is the special case

N

=

ℝ

n

in

Theorem

1.
In this case,
the

conclusion

of the
theorem can also be written
in terms of

gradients

instead of

differential forms

:

Theorem 2

.

Suppose

∇

⁡

g

i

are linearly independent
at each point of

M

.
If

p

∈

M

is a local minimum or maximum point of

f

restricted to

M

,
then there exist Lagrange multipliers

λ

1

,

…

,

λ

k

∈

R

, depending on

p

,
such that

∇

⁡

f

⁢

(

p

)

=

λ

1

⁢

∇

⁡

g

1

⁢

(

p

)

+

⋯

+

λ

k

⁢

∇

⁡

g

k

⁢

(

p

)

.

This formulation and the first one
are

equivalent

since
the 1-form

d

⁢

f

can be identified with the gradient

∇

⁡

f

, via the

formula

∇

⁡

f

⁢

(

p

)

⋅

v

=

d

⁢

f

⁢

(

p

;

v

)

=

d

⁢

f

p

⁢

(

v

)

.

1.3

Formulation with tangent maps

The

functions

g

i

can also be coalesced into a

vector-valued function

g

:

N

→

ℝ

k

. Then we have:

Theorem 3

.

Let

g

=

(

g

1

,

…

,

g

k

)

:

N

→

R

k

.
Suppose the

tangent map

D

⁡

g

is

surjective

at each point of

M

.
If

p

∈

M

is a local minimum or maximum point of

f

restricted to

M

,
then
there exists a Lagrange multiplier vector

λ

∈

(

R

k

)

*

,
depending on

p

, such that

D

⁡

f

⁢

(

p

)

=

D

⁡

g

⁢

(

p

)

*

⁢

λ

.

Here,

D

⁡

g

⁢

(

p

)

:

(

R

k

)

*

→

(

T

p

⁢

N

)

*

denotes the

pullback

of the

linear
transformation

(

http://planetmath.org/DualHomomorphism

)

D

⁡

g

⁢

(

p

)

:

T

p

⁢

N

→

R

k

.

If

D

⁡

g

is represented by its

Jacobian matrix

,
then the condition that it be surjective is equivalent to
its Jacobian matrix having full rank.

Note the deliberate use of the space

(

ℝ

k

)

*

instead of

ℝ

k

— to which the former is

isomorphic

to —
for the Lagrange multiplier vector. It turns out that the
Lagrange multiplier vector naturally
lives in the

dual space

and not the original

vector space

ℝ

k

.
This distinction is particularly important in the

infinite-dimensional

generalizations

of Lagrange multipliers.
But even in the finite-dimensional setting,
we do see hints that the dual space
has to be involved, because a

transpose

is involved
in the matrix expression for Lagrange multipliers.

If the expression

D

⁡

g

⁢

(

p

)

*

⁢

λ

is written
out in

coordinates

, then it is apparent that the components

λ

i

of the vector

λ

are exactly those
Lagrange multipliers from Theorems 1 and 2.

2

Proofs

The proof of the Lagrange multiplier theorem is surprisingly short and elegant,
when properly phrased in the

language

of

abstract manifolds

and
differential forms.

However, for the benefit of the readers not versed in these topics,
we provide, in

addition

to the abstract proof, a concrete

translation

of the

arguments

in the more familiar setting

N

=

ℝ

n

.

2.1

Beautiful abstract proof

Proof.

Since

d

⁢

g

i

are linearly independent at each point of

M

=

⋂

i

=

1

k

g

i

-

1

⁢

(

{

0

}

)

,

M

is an

embedded submanifold

of

N

,
of

dimension

m

=

n

-

k

. Let

α

:

U

→

M

, with

U

open in

ℝ

m

, be
a

coordinate chart

for

M

such that

α

⁢

(

0

)

=

p

.
Then

α

*

⁢

f

has a local minimum or maximum at

0

,
and therefore

0

=

d

⁢

(

α

*

⁢

f

)

=

α

*

⁢

d

⁢

f

at

0

.
But

α

*

at

p

is an

isomorphism

(

T

p

⁢

M

)

*

→

(

T

0

⁢

ℝ

m

)

*

,
so the preceding equation says that

d

⁢

f

vanishes

on

T

p

⁢

M

.

Now, by the definition of

g

i

, we have

α

*

⁢

g

i

=

0

,
so

0

=

d

⁢

(

α

*

⁢

g

i

)

=

α

*

⁢

d

⁢

g

i

. So like

d

⁢

f

,

d

⁢

g

i

vanishes on

T

p

⁢

M

.

In other words,

d

⁢

g

i

⁢

(

p

)

is in the

annihilator

(

http://planetmath.org/AnnihilatorOfVectorSubspace

)

(

T

p

⁢

M

)

0

of the

subspace

T

p

⁢

M

⊆

T

p

⁢

N

.
Since

T

p

⁢

M

has dimension

m

=

n

-

k

, and

T

p

⁢

N

has dimension

n

,
the annihilator

(

T

p

⁢

M

)

0

has dimension

k

.
Now

d

⁢

g

i

⁢

(

p

)

∈

(

T

p

⁢

M

)

0

are linearly independent,
so they must in fact be a basis for

(

T

p

⁢

M

)

0

.
But we had argued that

d

⁢

f

⁢

(

p

)

∈

(

T

p

⁢

M

)

0

.
Therefore

d

⁢

f

⁢

(

p

)

may be written as a unique

linear combination

of the

d

⁢

g

i

⁢

(

p

)

:

d

⁢

f

⁢

(

p

)

=

λ

1

⁢

d

⁢

g

1

⁢

(

p

)

+

⋯

+

λ

k

⁢

d

⁢

g

k

⁢

(

p

)

.

∎

The last paragraph of the previous proof can also be rephrased,
based on the same underlying ideas,
to make evident the fact that the Lagrange multiplier vector
lives in the dual space

(

ℝ

k

)

*

.

Alternative argument..

A general theorem in

linear algebra

states that for any linear
transformation

L

,
the image of the pullback

L

*

is the annihilator of the kernel of

L

.
Since

ker

⁡

D

⁡

g

⁢

(

p

)

=

T

p

⁢

M

and

d

⁢

f

⁢

(

p

)

∈

(

T

p

⁢

M

)

0

,
it immediately follows that

λ

∈

(

ℝ

k

)

*

exists such that

d

⁢

f

⁢

(

p

)

=

D

⁡

g

⁢

(

p

)

*

⁢

λ

.
∎

Yet another proof
could be devised by observing
that the result is obvious if

N

=

ℝ

n

and the constraint functions
are just coordinate projections on

ℝ

n

:

g

i

⁢

(

y

1

,

…

,

y

n

)

=

y

i

,

i

=

1

,

…

,

k

.

We clearly must have

∂

⁡

f

/

∂

⁡

y

k

+

1

=

⋯

=

∂

⁡

f

/

∂

⁡

y

n

=

0

at a point

p

that minimizes

f

⁢

(

y

)

over

y

1

=

⋯

=

y

k

=

0

.
The general case can be deduced to this by a coordinate change:

Alternate argument..

Since

d

⁢

g

i

are linearly independent,
we can find a coordinate chart for

N

about the point

p

,
with coordinate functions

y

1

,

…

,

y

n

:

N

→

ℝ

such that

y

i

=

g

i

for

i

=

1

,

…

,

k

.
Then

d

⁢

f

=

∂

⁡

f

∂

⁡

y

1

⁢

d

⁢

y

1

+

⋯

+

∂

⁡

f

∂

⁡

y

n

⁢

d

⁢

y

n

=

∂

⁡

f

∂

⁡

g

1

⁢

d

⁢

g

1

+

⋯

+

∂

⁡

f

∂

⁡

g

k

⁢

d

⁢

g

k

+

∂

⁡

f

∂

⁡

y

k

+

1

⁢

d

⁢

y

k

+

1

+

⋯

+

∂

⁡

f

∂

⁡

y

n

⁢

d

⁢

y

n

,

but

∂

⁡

f

/

∂

⁡

y

k

+

1

=

⋯

=

∂

⁡

f

/

∂

⁡

y

n

=

0

at the point

p

. Set

λ

i

=

∂

⁡

f

/

∂

⁡

g

i

at

p

.
∎

2.2

Clumsy, but down-to-earth proof

Proof.

We assume that

N

=

ℝ

n

.
Consider the

list vector

g

=

(

g

1

,

…

,

g

k

)

discussed earlier,
and its Jacobian matrix

D

⁡

g

in

Euclidean

coordinates.
The

i

th row of this matrix
is

[

∂

⁡

g

i

∂

⁡

x

1

…

∂

⁡

g

i

∂

⁡

x

n

]

=

(

∇

⁡

g

i

)

T

.

So the matrix

D

⁡

g

has full rank (i.e.

rank

⁡

D

⁡

g

=

k

) if and only
if the

k

gradients

∇

⁡

g

i

are linearly independent.

Consider each solution

q

∈

M

of

g

⁢

(

q

)

=

0

.
Since

D

⁡

g

has full rank, we can apply the

implicit function theorem

,
which states that there exist smooth solution parameterizations

α

:

U

→

M

around each point

q

∈

M

. (

U

is an

open set

in

ℝ

m

,

m

=

n

-

k

.)
These

α

are the coordinate charts which give to

M

=

g

-

1

⁢

(

{

0

}

)

a

manifold

structure

.

We now consider specially the point

q

=

p

; without loss of generality, assume

α

⁢

(

0

)

=

p

.
Then

f

∘

α

is a function on

Euclidean space

having a local minimum or maximum at

0

,
so its

derivative

vanishes at

0

.
Calculating by the

chain rule

, we have

0

=

D

⁡

(

f

∘

α

)

⁢

(

0

)

=

D

⁡

f

⁢

(

p

)

⋅

D

⁡

α

⁢

(

0

)

.
In other words,

ker

⁡

D

⁡

f

⁢

(

p

)

⊇

range of

⁢

D

⁡

α

⁢

(

0

)

=

T

p

⁢

M

.
Intuitively, this says that the

directional derivatives

at

p

of

f

lying in the

tangent space

T

p

⁢

M

of the manifold

M

vanish.

By the definition of

g

and

α

, we have

g

∘

α

=

0

.
By the chain rule again, we derive

0

=

D

⁡

g

⁢

(

p

)

⋅

D

⁡

α

⁢

(

0

)

.

Let the columns of

D

⁡

α

⁢

(

0

)

be the

column vectors

v

1

,

…

,

v

m

, which span the

m

-dimensional space

T

p

⁢

M

,
and look at the

matrix equation

0

=

D

⁡

f

⁢

(

p

)

⋅

D

⁡

α

⁢

(

0

)

again.
The equation for each entry of this matrix, which consists of only one row, is:

∇

⁡

f

⁢

(

p

)

⋅

v

j

=

0

,

j

=

1

,

…

,

m

.

In other words,

∇

⁡

f

⁢

(

p

)

is

orthogonal

to

v

1

,

…

,

v

m

,
and hence it is orthogonal to the entire tangent space

T

p

⁢

M

.

Similarly, the matrix equation

0

=

D

⁡

g

⁢

(

p

)

⋅

D

⁡

α

⁢

(

0

)

can be split into

individual

scalar equations:

∇

⁡

g

i

⁢

(

p

)

⋅

v

j

=

0

,

i

=

1

,

…

,

k

,

j

=

1

,

…

,

m

.

Thus

∇

⁡

g

i

⁢

(

p

)

is orthogonal to

T

p

⁢

M

.
But

∇

⁡

g

i

⁢

(

p

)

are, by

hypothesis

, linearly independent,
and there are

k

of these gradients, so they must form a basis for
the

orthogonal complement

of

T

p

⁢

M

, of

n

-

m

=

k

dimensions.
Hence

∇

⁡

f

⁢

(

p

)

can be written as a unique linear combination of

∇

⁡

g

i

⁢

(

p

)

:

∇

⁡

f

⁢

(

p

)

=

λ

1

⁢

∇

⁡

g

1

⁢

(

p

)

+

⋯

+

λ

k

⁢

∇

⁡

g

k

⁢

(

p

)

.

∎

3

Intuitive interpretations

We now discuss the intuitive and geometric

interpretations

of Lagrange multipliers.

3.1

Normals to tangent hyperplanes

Each equation

g

i

=

0

defines a hypersurface

M

i

in

ℝ

n

, a manifold of dimension

n

-

1

.
If we consider the

tangent

hyperplane

at

p

of these hypersurfaces,

T

p

⁢

M

i

, the gradient

∇

⁡

g

i

⁢

(

p

)

gives the

normal vector

to these hyperplanes.

The manifold

M

is the

intersection

of the hypersurfaces

M

i

.
Presumably, the tangent space

T

p

⁢

M

is the intersection of the

T

p

⁢

M

i

, and the subspace

perpendicular

to

T

p

⁢

M

would be spanned by the normals

∇

⁡

g

i

⁢

(

p

)

.
Now, the

direction

derivatives at

p

of

f

with respect to each vector in

T

p

⁢

M

, as we have proved,
vanish. So the direction of

∇

⁡

f

⁢

(

p

)

, the direction
of the greatest change in

f

at

p

, should be perpendicular
to

T

p

⁢

M

. Hence

∇

⁡

f

⁢

(

p

)

can be written as a linear combination of the

∇

⁡

g

i

⁢

(

p

)

.

Note, however, that this geometric picture, and the manipulations with the gradients

∇

⁡

f

⁢

(

p

)

and

∇

⁡

g

i

⁢

(

p

)

, do not carry over to abstract manifolds.
The notions of gradients and normals to

surfaces

depend on the

inner product

structure of

ℝ

n

, which is
not present in an abstract manifold (without a

Riemannian metric

).

On the other hand, this explains the mysterious appearance of annihilators in
the last paragraph of the abstract proof.
Annihilators and dual space theory serve as the proper tools
to formalize the manipulations we made with the matrix equations

0

=

D

⁡

f

⁢

(

p

)

⋅

D

⁡

α

⁢

(

0

)

and

0

=

D

⁡

g

⁢

(

p

)

⋅

D

⁡

α

⁢

(

0

)

, without resorting to Euclidean coordinates, which, of course, are not even defined on an abstract manifold.

3.2

With infinitesimals

If we are willing to interpret the quantities

d

⁢

f

and

d

⁢

g

i

as

infinitesimals

,
even the abstract version of the result has an intuitive explanation.
Suppose we are at the point

p

of the manifold

M

,
and consider an infinitesimal movement

Δ

⁢

p

about this point.
The infinitesimal movement

Δ

⁢

p

is a vector in the tangent space

T

p

⁢

M

, because, near

p

,

M

looks like the

linear space

T

p

⁢

M

.
And as

p

moves, the function

f

changes by a corresponding infinitesimal amount

d

⁢

f

that is approximately linear in

Δ

⁢

p

.

Furthermore, the change

d

⁢

f

may be decomposed
as the sum of a change as

p

moves

along

the manifold

M

,
and a change as

p

moves

out

of the manifold

M

.
But if

f

has a local minimum at

p

, then there cannot be
any change of

f

along

M

; thus

f

only changes
when moving out of

M

.
Now

M

is described by the equations

g

i

=

0

,
so a movement out of

M

is described by the infinitesimal changes

d

⁢

g

i

.
As

d

⁢

f

is linear in the change

Δ

⁢

p

,
we ought to be able to write it as a weighted sum of the changes

d

⁢

g

i

.
The weights are, of course, the Lagrange multipliers

λ

i

.

The linear algebra performed in the abstract proof can be regarded as the precise,

rigorous

translation of the preceding argument.

3.3

As rates of substitution

Observe that the formula for Lagrange multipliers is formally
very

similar

to the standard formula for expressing
a differential form in terms of a basis:

d

⁢

f

⁢

(

p

)

=

∂

⁡

f

∂

⁡

y

1

⁢

d

⁢

y

1

+

⋯

+

∂

⁡

f

∂

⁡

y

k

⁢

d

⁢

y

k

.

In fact, if

d

⁢

g

i

⁢

(

p

)

are linearly independent,
then they do form a basis for

(

T

p

⁢

M

)

0

,
that can be extended to a basis for

(

T

p

⁢

N

)

*

.
By the uniqueness of the basis representation,
we must have

λ

i

=

∂

⁡

f

∂

⁡

g

i

.

That is,

λ

i

is the

differential

of

f

with respect to changes in

g

i

.

In applications of Lagrange multipliers to economic
problems, the multipliers

λ

i

are

rates of substitution

—
they give the rate of improvement in the objective function

f

as the constraints

g

i

are relaxed.

4

Stationary points

In applications,
sometimes we are interested in
finding

stationary points

p

of

f

— defined as
points

p

such that

d

⁢

f

vanishes on

T

p

⁢

M

, or equivalently,
that the

Taylor expansion

of

f

at

p

, under any system of coordinates
for

M

, has no terms of

first order

. Then the Lagrange multiplier method
works for this situation too.

The following theorem incorporates
the more general notion of stationary points.

Theorem 4

.

Let

N

be a

n

-dimensional differentiable manifold (without boundary), and

f

:

N

→

R

,

g

i

:

N

→

R

, for

i

=

1

,

…

,

k

,
be continuously differentiable. Suppose

p

∈

M

=

⋂

i

=

1

k

g

i

-

1

⁢

(

{

0

}

)

, and

d

⁢

g

i

⁢

(

p

)

are linearly independent.

Then

p

is a stationary point (e.g. a

local extremum

point) of

f

restricted to

M

,
if and only if there exist

λ

1

,

…

,

λ

k

∈

R

such that

d

⁢

f

⁢

(

p

)

=

λ

1

⁢

d

⁢

g

1

⁢

(

p

)

+

⋯

+

λ

k

⁢

d

⁢

g

k

⁢

(

p

)

.

The Lagrange multipliers

λ

i

, which depend on

p

, are unique when they exist.

In this formulation,

M

is not necessarily a manifold,
but it is one when intersected with a sufficiently small

neighborhood

about

p

.
So it makes sense to talk about

T

p

⁢

M

, although we are abusing notation here.
The subspace in question can be more accurately described as the

annihilated subspace

of

span

⁡

{

d

⁢

g

i

⁢

(

p

)

}

.

It is also enough that

d

⁢

g

i

be linearly independent
only at the point

p

.
For

d

⁢

g

i

are

continuous

, so they will be
linearly independent for points near

p

anyway,
and we may restrict our viewpoint to a sufficiently small neighborhood
around

p

, and the proofs carry through.

The proof involves only simple modifications to
that of Theorem 1 — for

instance

,
the converse implication follows
because we have already proved that the

d

⁢

g

i

⁢

(

p

)

form a basis for the annihilator of

T

p

⁢

M

,
independently of whether or not

p

is a stationary point of

f

on

M

.

References

1

Friedberg, Insel, Spence.

Linear Algebra

. Prentice-Hall, 1997.

2

David Luenberger.

Optimization

by Vector Space Methods

.
John Wiley & Sons, 1969.

3

James R. Munkres.

Analysis on Manifolds

. Westview Press, 1991.

4

R. Tyrrell Rockafellar. “Lagrange Multipliers and Optimality”.

SIAM Review

. Vol. 35, No. 2, June 1993.

5

Michael Spivak.

Calculus on Manifolds

. Perseus Books, 1998.

Title

Lagrange multipliers on manifolds

Canonical name

LagrangeMultipliersOnManifolds

Date of creation

2013-03-22 15:25:45

Last modified on

2013-03-22 15:25:45

Owner

stevecheng (10074)

Last modified by

stevecheng (10074)

Numerical id

24

Author

stevecheng (10074)

Entry type

Topic

Classification

msc 58C05

Classification

msc 49-00

Related topic

Manifold

Generated on Sat Feb 10 11:50:57 2018 by

LaTeXML