conjugate gradient algorithm

conjugate gradient algorithm

The

conjugate gradient algorithm

is used to solve the quadratic

minimization

problem:

min

⁡

(

1

2

⁢

x

T

⁢

Q

⁢

x

-

b

T

⁢

x

)

or equivalently to solve the

linear system

Q

⁢

x

-

b

=

0

, where

Q

is a given

n

by

n

symmetric

positive definite matrix

and

b

is a given

n

vector.

The

algorithm

requires n

iterations

, starting from an arbitrary initial guess

x

0

(often

x

0

=

0

is used). We will use the following notation:

•

i

— iteration number;

•

x

i

— solution approximation;

•

d

i

— search

direction

;

•

r

i

— residual, which is defined as

b

-

Q

⁢

x

i

.

Algorithm

1.

Initialization. Let

x

0

=

0

(or other starting point). Let

r

0

=

d

0

=

-

Q

⁢

x

0

+

b

. (The initial search direction is set to minus the

gradient

of the quadratic function being minimized, evaluated at the starting point).

2.

For

i

=

0

to

n

-

1

compute

α

=

r

i

T

⁢

d

i

d

i

T

⁢

Q

⁢

d

i

x

i

+

1

=

x

i

+

α

⁢

d

i

r

i

+

1

=

r

i

-

α

⁢

Q

⁢

d

i

If

∥

r

i

∥

<

ε

or

i

=

n

-

1

Stop, the solution has been found. Otherwise, continue:

β

=

r

i

+

1

T

⁢

Q

⁢

d

i

d

i

T

⁢

Q

⁢

d

i

d

i

+

1

=

-

r

i

+

1

+

β

⁢

d

i

(In each iteration the solution estimate is set to the previous estimate plus a multiple of the previous search direction. The next search direction is then set to the gradient plus a multiple of the previous search direction).

Discussion

The

conjugate gradient method

was developed in 1952 by Hestenes and Stiefel as an improvement to the steepest descent method. Whereas steepest descent approaches the solution asymptotically, the conjugate gradient method will find the solution in n iterations (assuming no roundoff error).

Why the name? The search directions

d

i

are

conjugate

in the sense that

d

j

T

⁢

Q

⁢

d

i

=

0

for

j

<

i

. In

addition

these directions are computed from (but are not equal to) the gradient.

The conjugate gradient method has been generalized to the case where the

function

being minimized is only approximately quadratic. In that case the explicit formula for

α

given above is replaced by a line-search procedure; this is a trial and error method in which various values of

α

are tried, and the value that leads to the smallest value of the objective is chosen. Well known generalized c.g. methods include the Fletcher-Reeves method and the Polak-Ribiere method.

Example

Solve

(

1

2

2

1

)

⁢

x

=

(

-

3

0

)

We have

x

0

=

(

0

0

)

then

x

1

=

(

-

3

0

)

and finally

x

2

=

(

1

-

2

)

which is the solution.

References

Luenberger:

Introduction

to Linear and Nonlinear Programming, Addison-Wesley, 1973

Jonathan Richard Shewchuk: An Introduction to the Conjugate Gradient Method Without the Agonizing Pain, August 1994.

\htmladdnormallink

http://www-2.cs.cmu.edu/ jrs/jrspapers.html
http://www-2.cs.cmu.edu/ jrs/jrspapers.html [A detailed

derivation

of the method from first principles].

Press, et al.: Numerical Recipes in C, Cambridge University Press, 1995 [Chapter 10.6 contains an implementation of the generalized conjugate gradient method of Polak and Ribiere].

Title

conjugate gradient algorithm

Canonical name

ConjugateGradientAlgorithm

Date of creation

2013-03-22 14:58:54

Last modified on

2013-03-22 14:58:54

Owner

aplant (12431)

Last modified by

aplant (12431)

Numerical id

16

Author

aplant (12431)

Entry type

Algorithm

Classification

msc 15A06

Classification

msc 90C20

Synonym

method of conjugate gradients

Generated on Fri Feb 9 18:22:36 2018 by

LaTeXML