Project: ArangoDB (App Definition and Development / Database)
File: autodiff.pdf

Autodi
Automatic Dierentiation C++ Library
Matthew Pulver
June 22, 2019
Contents
1 Description 2
2 Examples 2
2.1 Example 1: Single-variable derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1.1 Calculate derivatives of f(x) =x4atx= 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2 Example 2: Multi-variable mixed partial derivatives with multi-precision data type . . . . . . . . . . . 3
2.2.1 Calculate@12f
@w3@x2@y4@z3(11;12;13;14) with a precision of about 50 decimal digits,
where f(w; x; y; z ) = exp
wsin
xlog(y)
z
+q
wz
xy
+w2
tan(z). . . . . . . . . . . . . . . . . . . . . 3
2.3 Example 3: Black-Scholes Option Pricing with Greeks Automatically Calculated . . . . . . . . . . . . 4
2.3.1 Calculate greeks directly from the Black-Scholes pricing function. . . . . . . . . . . . . . . . . . 4
3 Advantages of Automatic Dierentiation 5
4 Mathematics 6
4.1 Truncated Taylor Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.1.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.2 Arithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2.1 Addition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2.2 Subtraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2.3 Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2.4 Division . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.3 General Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.4 Multiple Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.4.1 Declaring Multiple Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5 Writing Functions for Autodi Compatibility 10
5.1 Piecewise-Rational Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.2 Functions That Call Existing Autodi Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.3 New Functions For Which The Derivatives Can Be Calculated . . . . . . . . . . . . . . . . . . . . . . . 11
6 Function Writing Guidelines 12
6.1 Example 1: f(x) = max(0 ; x) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
6.2 Example 2: f(x) = sinc( x) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.3 Example 3: f(x) =pxandf0(0) =1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
7 Acknowledgments 15
1
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )1 Description
Autodi is a header-only C++ library that facilitates the automatic dierentiation (forward mode) of mathematical
functions of single and multiple variables.
This implementation is based upon the Taylor series expansion of an analytic function fat the point x0:
f(x0+") =f(x0) +f0(x0)"+f00(x0)
2!"2+f000(x0)
3!"3+
=NX
n=0f(n)(x0)
n!"n+O
"N+1
:
The essential idea of autodi is the substitution of numbers with polynomials in the evaluation of f(x0). By
substituting the number x0with the rst-order polynomial x0+", and using the same algorithm to compute f(x0+"),
the resulting polynomial in "contains the function's derivatives f0(x0),f00(x0),f000(x0), ... within the coecients.
Each coecient is equal to the derivative of its respective order, divided by the factorial of the order.
In greater detail, assume one is interested in calculating the rst Nderivatives of fatx0. Without loss of
precision to the calculation of the derivatives, all terms O
"N+1
that include powers of "greater than Ncan be
discarded. (This is due to the fact that each term in a polynomial depends only upon equal and lower-order terms
under arithmetic operations.) Under these truncation rules, fprovides a polynomial-to-polynomial transformation:
f : x0+"7!NX
n=0yn"n=NX
n=0f(n)(x0)
n!"n:
C++'s ability to overload operators and functions allows for the creation of a class fvar (forward-mode autodi
variable) that represents polynomials in ". Thus the same algorithm fthat calculates the numeric value of y0=f(x0),
when written to accept and return variables of a generic (template) type, is also used to calculate the polynomialPN
n=0yn"n=f(x0+"). The derivatives f(n)(x0) are then found from the product of the respective factorial n! and
coecient yn:
dnf
dxn(x0) =n!yn:
2 Examples
2.1 Example 1: Single-variable derivatives
2.1.1 Calculate derivatives of f(x) =x4atx= 2.
In this example, make fvar<double, Order>(2.0) instantiates the polynomial 2 + ". The Order=5 means that
enough space is allocated (on the stack) to hold a polynomial of up to degree 5 during the proceeding computation.
Internally, this is modeled by a std::array<double,6> whose elementsf2, 1, 0, 0, 0, 0 gcorrespond to the
6 coecients of the polynomial upon initialization. Its fourth power, at the end of the computation, is a polynomial
with coecients y =f16, 32, 24, 8, 1, 0 g. The derivatives are obtained using the formula f(n)(2) = n!y[n].
#include <boost/math/differentiation/autodiff.hpp>
#include <iostream>
template <typename T>
T fourth_power(T const& x) {
T x4 = x * x; // retval in operator*() uses x4's memory via NRVO.
x4 *= x4; // No copies of x4 are made within operator*=() even when squaring.
return x4; // x4 uses y's memory in main() via NRVO.
}
int main() {
using namespace boost::math::differentiation;
2
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )constexpr unsigned Order = 5; // Highest order derivative to be calculated.
auto const x = make_fvar<double, Order>(2.0); // Find derivatives at x=2.
auto const y = fourth_power(x);
for (unsigned i = 0; i <= Order; ++i)
std::cout << "y.derivative(" << i << ") = " << y.derivative(i) << std::endl;
return 0;
}
/*
Output:
y.derivative(0) = 16
y.derivative(1) = 32
y.derivative(2) = 48
y.derivative(3) = 48
y.derivative(4) = 24
y.derivative(5) = 0
*/
The above calculates
y:derivative (0) = f(2) = x4
x=2= 16
y:derivative (1) = f0(2) = 4x3
x=2= 32
y:derivative (2) =f00(2) = 43x2
x=2= 48
y:derivative (3) =f000(2) =432xjx=2= 48
y:derivative (4) =f(4)(2) = 4321 = 24
y:derivative (5) =f(5)(2) = 0
2.2 Example 2: Multi-variable mixed partial derivatives with multi-precision data
type
2.2.1 Calculate@12f
@w3@x2@y4@z3(11;12;13;14)with a precision of about 50 decimal digits,
where f(w; x; y; z ) = exp
wsin
xlog(y)
z
+q
wz
xy
+w2
tan(z).
In this example, make ftuple<float50, Nw, Nx, Ny, Nz>(11, 12, 13, 14) returns a std::tuple of 4 indepen-
dent fvar variables, with values of 11, 12, 13, and 14, for which the maximum order derivative to be calculated for
each are 3, 2, 4, 3, respectively. The order of the variables is important, as it is the same order used when calling
v.derivative(Nw, Nx, Ny, Nz) in the example below.
#include <boost/math/differentiation/autodiff.hpp>
#include <boost/multiprecision/cpp_bin_float.hpp>
#include <iostream>
using namespace boost::math::differentiation;
template <typename W, typename X, typename Y, typename Z>
promote<W, X, Y, Z> f(const W& w, const X& x, const Y& y, const Z& z) {
using namespace std;
return exp(w * sin(x * log(y) / z) + sqrt(w * z / (x * y))) + w * w / tan(z);
}
int main() {
using float50 = boost::multiprecision::cpp_bin_float_50;
constexpr unsigned Nw = 3; // Max order of derivative to calculate for w
constexpr unsigned Nx = 2; // Max order of derivative to calculate for x
3
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )constexpr unsigned Ny = 4; // Max order of derivative to calculate for y
constexpr unsigned Nz = 3; // Max order of derivative to calculate for z
// Declare 4 independent variables together into a std::tuple.
auto const variables = make_ftuple<float50, Nw, Nx, Ny, Nz>(11, 12, 13, 14);
auto const& w = std::get<0>(variables); // Up to Nw derivatives at w=11
auto const& x = std::get<1>(variables); // Up to Nx derivatives at x=12
auto const& y = std::get<2>(variables); // Up to Ny derivatives at y=13
auto const& z = std::get<3>(variables); // Up to Nz derivatives at z=14
auto const v = f(w, x, y, z);
// Calculated from Mathematica symbolic differentiation.
float50 const answer("1976.319600747797717779881875290418720908121189218755");
std::cout << std::setprecision(std::numeric_limits<float50>::digits10)
<< "mathematica : " << answer << '\n'
<< "autodiff : " << v.derivative(Nw, Nx, Ny, Nz) << '\n'
<< std::setprecision(3)
<< "relative error: " << (v.derivative(Nw, Nx, Ny, Nz) / answer - 1) << '\n';
return 0;
}
/*
Output:
mathematica : 1976.3196007477977177798818752904187209081211892188
autodiff : 1976.3196007477977177798818752904187209081211892188
relative error: 2.67e-50
*/
2.3 Example 3: Black-Scholes Option Pricing with Greeks Automatically Calculated
2.3.1 Calculate greeks directly from the Black-Scholes pricing function.
Below is the standard Black-Scholes pricing function written as a function template, where the price, volatility
(sigma), time to expiration (tau) and interest rate are template parameters. This means that any Greek based on
these 4 variables can be calculated using autodi. The below example calculates delta and gamma where the variable
of dierentiation is only the price. For examples of more exotic greeks, see example/black scholes.cpp .
#include <boost/math/differentiation/autodiff.hpp>
#include <iostream>
using namespace boost::math::constants;
using namespace boost::math::differentiation;
// Equations and function/variable names are from
// https://en.wikipedia.org/wiki/Greeks_(finance)#Formulas_for_European_option_Greeks
// Standard normal cumulative distribution function
template <typename X>
X Phi(X const& x) {
return 0.5 * erfc(-one_div_root_two<X>() * x);
}
enum class CP { call, put };
// Assume zero annual dividend yield (q=0).
template <typename Price, typename Sigma, typename Tau, typename Rate>
promote<Price, Sigma, Tau, Rate> black_scholes_option_price(CP cp,
double K,
Price const& S,
Sigma const& sigma,
Tau const& tau,
4
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )Rate const& r) {
using namespace std;
auto const d1 = (log(S / K) + (r + sigma * sigma / 2) * tau) / (sigma * sqrt(tau));
auto const d2 = (log(S / K) + (r - sigma * sigma / 2) * tau) / (sigma * sqrt(tau));
switch (cp) {
case CP::call:
return S * Phi(d1) - exp(-r * tau) * K * Phi(d2);
case CP::put:
return exp(-r * tau) * K * Phi(-d2) - S * Phi(-d1);
}
}
int main() {
double const K = 100.0; // Strike price.
auto const S = make_fvar<double, 2>(105); // Stock price.
double const sigma = 5; // Volatility.
double const tau = 30.0 / 365; // Time to expiration in years. (30 days).
double const r = 1.25 / 100; // Interest rate.
auto const call_price = black_scholes_option_price(CP::call, K, S, sigma, tau, r);
auto const put_price = black_scholes_option_price(CP::put, K, S, sigma, tau, r);
std::cout << "black-scholes call price = " << call_price.derivative(0) << '\n'
<< "black-scholes put price = " << put_price.derivative(0) << '\n'
<< "call delta = " << call_price.derivative(1) << '\n'
<< "put delta = " << put_price.derivative(1) << '\n'
<< "call gamma = " << call_price.derivative(2) << '\n'
<< "put gamma = " << put_price.derivative(2) << '\n';
return 0;
}
/*
Output:
black-scholes call price = 56.5136
black-scholes put price = 51.4109
call delta = 0.773818
put delta = -0.226182
call gamma = 0.00199852
put gamma = 0.00199852
*/
3 Advantages of Automatic Dierentiation
The above examples illustrate some of the advantages of using autodi:
Elimination of code redundancy. The existence of Nseparate functions to calculate derivatives is a form of
code redundancy, with all the liabilities that come with it:
{Changes to one function require Nadditional changes to other functions. In the 3rdexample above,
consider how much larger and inter-dependent the above code base would be if a separate function were
written for each Greek value.
{Dependencies upon a derivative function for a dierent purpose will break when changes are made to the
original function. What doesn't need to exist cannot break.
{Code bloat, reducing conceptual integrity. Control over the evolution of code is easier/safer when the code
base is smaller and able to be intuitively grasped.
Accuracy of derivatives over nite dierence methods. Single-iteration nite dierence methods always include
a xfree variable that must be carefully chosen for each application. If  xis too small, then numerical errors
become large. If  xis too large, then mathematical errors become large. With autodi, there are no free
5
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )variables to set and the accuracy of the answer is generally superior to nite dierence methods even with the
best choice of  x.
4 Mathematics
In order for the usage of the autodi library to make sense, a basic understanding of the mathematics will help.
4.1 Truncated Taylor Series
Basic calculus courses teach that a real analytic function f:D!Ris one which can be expressed as a Taylor series
at a point x02DR:
f(x) =f(x0) +f0(x0)(xx0) +f00(x0)
2!(xx0)2+f000(x0)
3!(xx0)3+
One way of thinking about this form is that given the value of an analytic function f(x0) and its derivatives
f0(x0); f00(x0); f000(x0); :::evaluated at a point x0, then the value of the function f(x) can be obtained at any other
point x2Dusing the above formula.
Let us make the substitution x=x0+"and rewrite the above equation to get:
f(x0+") =f(x0) +f0(x0)"+f00(x0)
2!"2+f000(x0)
3!"3+
Now consider "asan abstract algebraic entity that never acquires a numeric value , much like one does in basic algebra
with variables like xory. For example, we can still manipulate entities like xyand (1 + 2 x+ 3x2) without having
to assign specic numbers to them.
Using this formula, autodi goes in the other direction. Given a general formula/algorithm for calculating
f(x0+"), the derivatives are obtained from the coecients of the powers of "in the resulting computation. The
general coecient for "nis
f(n)(x0)
n!:
Thus to obtain f(n)(x0), the coecient of "nis multiplied by n!.
4.1.1 Example
Apply the above technique to calculate the derivatives of f(x) =x4atx0= 2.
The rst step is to evaluate f(x0+") and simply go through the calculation/algorithm, treating "as an abstract
algebraic entity:
f(x0+") =f(2 +")
= (2 + ")4
=
4 + 4"+"22
= 16 + 32 "+ 24"2+ 8"3+"4:
Equating the powers of "from this result with the above "-taylor expansion yields the following equalities:
f(2) = 16 ; f0(2) = 32 ;f00(2)
2!= 24;f000(2)
3!= 8;f(4)(2)
4!= 1;f(5)(2)
5!= 0:
Multiplying both sides by the respective factorials gives
f(2) = 16 ; f0(2) = 32 ; f00(2) = 48 ; f000(2) = 48 ; f(4)(2) = 24 ; f(5)(2) = 0 :
These values can be directly conrmed by the power rule applied to f(x) =x4.
6
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )4.2 Arithmetic
What was essentially done above was to take a formula/algorithm for calculating f(x0) from a number x0, and
instead apply the same formula/algorithm to a polynomial x0+". Intermediate steps operate on values of the form
x=x0+x1"+x2"2++xN"N
and the nal return value is of this polynomial form as well. In other words, the normal arithmetic operators
+;;;applied to numbers xare instead applied to polynomials x. Through the overloading of C++ operators
and functions, oating point data types are replaced with data types that represent these polynomials. More
specically, C++ types such as double are replaced with std::array<double,N+1> , which hold the above N+ 1
coecients xi, and are wrapped in a class that overloads all of the arithmetic operators.
The logic of these arithmetic operators simply mirror that which is applied to polynomials. We'll look at each of
the 4 arithmetic operators in detail.
4.2.1 Addition
The addition of polynomials xandyis done component-wise:
z=x+y
= NX
i=0xi"i!
+ NX
i=0yi"i!
=NX
i=0(xi+yi)"i
zi=xi+yi fori2f0;1;2; :::; Ng:
4.2.2 Subtraction
Subtraction follows the same form as addition:
z=xy
= NX
i=0xi"i!
 NX
i=0yi"i!
=NX
i=0(xiyi)"i
zi=xiyi fori2f0;1;2; :::; Ng:
4.2.3 Multiplication
Multiplication produces higher-order terms:
7
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )z=xy
= NX
i=0xi"i! NX
i=0yi"i!
=x0y0+ (x0y1+x1y0)"+ (x0y2+x1y1+x2y0)"2++0
@NX
j=0xjyNj1
A"N+O
"N+1
=NX
i=0iX
j=0xjyij"i+O
"N+1
zi=iX
j=0xjyij fori2f0;1;2; :::; Ng:
In the case of multiplication, terms involving powers of "greater than N, collectively denoted by O
"N+1
, are
simply discarded. Fortunately, the values of ziforiNdo not depend on any of these discarded terms, so there is
no loss of precision in the nal answer. The only information that is lost are the values of higher order derivatives,
which we are not interested in anyway. If we were, then we would have simply chosen a larger value of Nto begin
with.
4.2.4 Division
Division is not directly calculated as are the others. Instead, to nd the components of z=xywe require that
x=yz. This yields a recursive formula for the components zi:
xi=iX
j=0yjzij
=y0zi+iX
j=1yjzij
zi=1
y00
@xiiX
j=1yjzij1
A fori2f0;1;2; :::; Ng:
In the case of division, the values for zimust be calculated sequentially, since zidepends on the previously calculated
values z0; z1; :::; z i1.
4.3 General Functions
Calling standard mathematical functions such as log() ,cos() , etc. should return accurate higher order derivatives.
For example, exp(x) may be written internally as a specic 14th-degree polynomial to approximate exwhen 0 <
x <1. This would mean that the 15thderivative, and all higher order derivatives, would be 0, however we know that
d15
dx15ex=ex. How should such functions whose derivatives are known be written to provide accurate higher order
derivatives? The answer again comes back to the function's Taylor series.
To simplify notation, for a given polynomial x=x0+x1"+x2"2++xN"Ndene
x"=x1"+x2"2++xN"N=NX
i=1xi"i:
This allows for a concise expression of a general function fofx:
8
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )f(x) =f(x0+x")
=f(x0) +f0(x0)x"+f00(x0)
2!x2
"+f000(x0)
3!x3
"++f(N)(x0)
N!xN
"+O
"N+1
=NX
i=0f(i)(x0)
i!xi
"+O
"N+1
where "has been substituted with x"in the "-taylor series for f(x). This form gives a recipe for calculating f(x) in
general from regular numeric calculations f(x0),f0(x0),f00(x0), ... and successive powers of the epsilon terms x".
For an application in which we are interested in up to Nderivatives in xthe data structure to hold this information
is an ( N+ 1)-element array vwhose general element is
v[i] =f(i)(x0)
i!fori2f0;1;2; :::; Ng:
4.4 Multiple Variables
In C++, the generalization to mixed partial derivatives with multiple independent variables is conveniently achieved
with recursion. To begin to see the recursive pattern, consider a two-variable function f(x; y). Since xandyare
independent, they require their own independent epsilons "xand"y, respectively.
Expand f(x; y) for x=x0+"x:
f(x0+"x; y) =f(x0; y) +@f
@x(x0; y)"x+1
2!@2f
@x2(x0; y)"2
x+1
3!@3f
@x3(x0; y)"3
x++1
M!@Mf
@xM(x0; y)"M
x+O
"M+1
x
=MX
i=01
i!@if
@xi(x0; y)"i
x+O
"M+1
x
:
Next, expand f(x0+"x; y) for y=y0+"y:
f(x0+"x; y0+"y) =NX
j=01
j!@j
@yj MX
i=0"i
x1
i!@if
@xi!
(x0; y0)"j
y+O
"M+1
x
+O
"N+1
y
=MX
i=0NX
j=01
i!j!@i+jf
@xi@yj(x0; y0)"i
x"j
y+O
"M+1
x
+O
"N+1
y
:
Similar to the single-variable case, for an application in which we are interested in up to Mderivatives in xand
Nderivatives in y, the data structure to hold this information is an ( M+ 1)(N+ 1) array vwhose element at
(i; j) is
v[i][j] =1
i!j!@i+jf
@xi@yj(x0; y0) for ( i; j)2f0;1;2; :::; Mgf 0;1;2; :::; Ng:
The generalization to additional independent variables follows the same pattern.
4.4.1 Declaring Multiple Variables
Internally, independent variables are represented by vectors within orthogonal vector spaces. Because of this, one
must be careful when declaring more than one independent variable so that they do not end up in parallel vector
spaces. This can easily be achieved by following one rule:
When declaring more than one independent variable, call make ftuple<>() once and only once.
The tuple of values returned are independent. Though it is possible to achieve the same result with multiple calls to
make fvar , this is an easier and less error-prone method. See Section 2.2 for example usage.
9
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )5 Writing Functions for Autodi Compatibility
In this section, a general procedure is given for writing new, and transforming existing, C++ mathematical functions
for compatibility with autodi.
There are 3 categories of functions that require dierent strategies:
1. Piecewise-rational functions. These are simply piecewise quotients of polynomials. All that is needed is to
turn the function parameters and return value into generic (template) types. This will then allow the function
to accept and return autodi's fvar types, thereby using autodi's overloaded arithmetic operators which
calculate the derivatives automatically.
2. Functions that call existing autodi functions. This is the same as the previous, but may also include calls to
functions that are in the autodi library. Examples: exp() ,log() ,tgamma() , etc.
3. New functions for which the derivatives can be calculated. This is the most general technique, as it allows for
the development of a function which do not fall into the previous two categories.
Functions written in any of these ways may then be added to the autodi library.
5.1 Piecewise-Rational Functions
f(x) =1
1 +x2
By simply writing this as a template function, autodi can calculate derivatives for it:
#include <boost/math/differentiation/autodiff.hpp>
#include <iostream>
template <typename T>
T rational(T const& x) {
return 1 / (1 + x * x);
}
int main() {
using namespace boost::math::differentiation;
auto const x = make_fvar<double, 10>(0);
auto const y = rational(x);
std::cout << std::setprecision(std::numeric_limits<double>::digits10)
<< "y.derivative(10) = " << y.derivative(10) << std::endl;
return 0;
}
/*
Output:
y.derivative(10) = -3628800
*/
As simple as f(x) may seem, the derivatives can get increasingly complex as derivatives are taken. For example, the
10thderivative has the form
f(10)(x) =3628800155x2+ 330 x4462x6+ 165 x811x10
(1 +x2)11:
Derivatives of f(x) are useful, and in fact used, in calculating higher order derivatives for arctan( x) for instance,
since
arctan(n)(x) =d
dxn11
1 +x2for 1n:
10
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )5.2 Functions That Call Existing Autodi Functions
Many of the standard library math function are overloaded in autodi. It is recommended to use argument-dependent
lookup (ADL) in order for functions to be written in a way that is general enough to accommodate standard types
(double ) as well as autodi types ( fvar ).
Example:
#include <boost/math/constants/constants.hpp>
#include <cmath>
using namespace boost::math::constants;
// Standard normal cumulative distribution function
template <typename T>
T Phi(T const& x)
{
return 0.5 * std::erfc(-one_div_root_two<T>() * x);
}
Though Phi(x) is general enough to handle the various fundamental oating point types, this will not work if xis
an autodi fvar variable, since std::erfc does not include a specialization for fvar . The recommended solution is
to remove the namespace prex std:: from erfc :
#include <boost/math/constants/constants.hpp>
#include <boost/math/differentiation/autodiff.hpp>
#include <cmath>
using namespace boost::math::constants;
// Standard normal cumulative distribution function
template <typename T>
T Phi(T const& x)
{
using std::erfc;
return 0.5 * erfc(-one_div_root_two<T>() * x);
}
In this form, when xis of type fvar , the C++ compiler will search for and nd a function erfc within the same
namespace as fvar , which is in the autodi library, via ADL. Because of the using-declaration, it will also call
std::erfc when xis a fundamental type such as double .
5.3 New Functions For Which The Derivatives Can Be Calculated
Mathematical functions which do not fall into the previous two categories can be constructed using autodi helper
functions. This requires a separate function for calculating the derivatives. In case you are asking yourself what
good is an autodi library if one needs to supply the derivatives, the answer is that the new function will t in with
the rest of the autodi library, thereby allowing for the creation of additional functions via all of the arithmetic
operators, plus function composition, which was not readily available without the library.
The example given here is for cos:
template <typename RealType, size_t Order>
fvar<RealType, Order> cos(fvar<RealType, Order> const& cr) {
using std::cos;
using std::sin;
using root_type = typename fvar<RealType, Order>::root_type;
constexpr size_t order = fvar<RealType, Order>::order_sum;
root_type const d0 = cos(static_cast<root_type>(cr));
if constexpr (order == 0)
return fvar<RealType, Order>(d0);
else {
11
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )root_type const d1 = -sin(static_cast<root_type>(cr));
root_type const derivatives[4]{d0, d1, -d0, -d1};
return cr.apply_derivatives(order,
[&derivatives](size_t i) { return derivatives[i & 3]; });
}
}
This uses the helper function fvar::apply derivatives which takes two parameters:
1. The highest order derivative to be calculated.
2. A function that maps derivative order to derivative value.
The highest order derivative necessary to be calculated is generally equal to fvar::order sum. In the case of sin
and cos, the derivatives are cyclical with period 4. Thus it is sucient to store only these 4 values into an array,
and take the derivative order modulo 4 as the index into this array.
A second helper function, not shown here, is apply coefficients . This is used the same as apply derivatives
except that the supplied function calculates coecients instead of derivatives. The relationship between a coecient
Cnand derivative Dnfor derivative order nis
Cn=Dn
n!:
Internally, fvar holds coecients rather than derivatives, so in case the coecient values are more readily available
than the derivatives, it can save some unnecessary computation to use apply coefficients . See the denition of
atan for an example.
Both of these helper functions use Horner's method when calculating the resulting polynomial fvar . This works
well when the derivatives are nite, but in cases where derivatives are innite, this can quickly result in NaN values as
the computation progresses. In these cases, one can call non-Horner versions of both function which better \isolate"
innite values so that they are less likely to evolve into NaN values.
The four helper functions available for constructing new autodi functions from known coecients/derivatives
are:
1.fvar::apply coefficients
2.fvar::apply coefficients nonhorner
3.fvar::apply derivatives
4.fvar::apply derivatives nonhorner
6 Function Writing Guidelines
At a high level there is one fairly simple principle, loosely and intuitively speaking, to writing functions for which
autodi can eectively calculate derivatives:
Autodi Function Principle (AFP)
A function whose branches in logic correspond to piecewise analytic calculations over non-singleton inter-
vals, with smooth transitions between the intervals, and is free of indeterminate forms in the calculated
value and higher order derivatives, will work ne with autodi.
Stating this with greater mathematical rigor can be done. However what seems to be more practical, in this case, is
to give examples and categories of examples of what works, what doesn't, and how to remedy some of the common
problems that may be encountered. That is the approach taken here.
6.1 Example 1: f(x) = max(0 ; x)
One potential implementation of f(x) = max(0 ; x) is:
12
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )template<typename T>
T f(const T& x)
{
return 0 < x ? x : 0;
}
Though this is consistent with Section 5, there are two problems with it:
1.f(nan) = 0 . This problem is independent of autodi, but is worth addressing anyway. If there is an indetermi-
nate form that arises within a calculation and is input into f, then it gets \covered up" by this implementation
leading to an unknowingly incorrect result. Better for functions in general to propagate NaN values, so that
the user knows something went wrong and doesn't rely on an incorrect result, and likewise the developer can
track down where the NaN originated from and remedy it.
2.f0(0) = 0 when autodi is applied. This is because freturns 0 as a constant when x==0 , wiping out any of
the derivatives (or sensitivities) that xwas holding as an autodi variable. Instead, let us apply the AFP and
identify the two intervals over which fis dened: (1;0][(0;1). Though the function itself is not analytic
atx= 0, we can attempt somewhat to smooth out this transition point by averaging the calculation of f(x)
atx= 0 from each interval. If x <0 then the result is simply 0, and if 0 < xthen the result is x. The average
is1
2(0 +x) which will allow autodi to calculate f0(0) =1
2. This is a more reasonable answer.
A better implementation that resolves both issues is:
template<typename T>
T f(const T& x)
{
if (x < 0)
return 0;
else if (x == 0)
return 0.5*x;
else
return x;
}
6.2 Example 2: f(x) = sinc( x)
The denition of sinc : R!Ris
sinc(x) =(
1 if x= 0
sin(x)
xotherwise.
A potential implementation is:
template<typename T>
T sinc(const T& x)
{
using std::sin;
return x == 0 ? 1 : sin(x) / x;
}
Though this is again consistent with Section 5, and returns correct non-derivative values, it returns a constant when
x==0 thereby losing all derivative information contained in xand contributions from sinc. For example, sinc00(0) =1
3,
however y.derivative(2) == 0 when y = sinc(make fvar<double,2>(0)) using the above incorrect implementa-
tion. Applying the AFP, the intervals upon which separate branches of logic are applied are ( 1;0)[[0;0][(0;1).
The violation occurs due to the singleton interval [0 ;0], even though a constant function of 1 is technically analytic.
The remedy is to dene a custom sinc overload and add it to the autodi library. This has been done. Mathematically,
it is well-dened and free of indeterminate forms, as is the 3rdexpression in the equalities
1
xsin(x) =1
x1X
n=0(1)n
(2n+ 1)!x2n+1=1X
n=0(1)n
(2n+ 1)!x2n:
13
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )The autodi library contains helper functions to help write function overloads when the derivatives of a function are
known. This is an advanced feature and documentation for this may be added at a later time.
For now, it is worth understanding the ways in which indeterminate forms can occur within a mathematical
calculation, and avoid them when possible by rewriting the function. Table 1 compares 3 types of indeterminate
forms. Assume the product a*bis a positive nite value.
f(x) =a
x
(bx2) g(x) =a
x
(bx) h(x) =a
x2
(bx)
Mathematical
Limitlim
x!0f(x) = 0 lim
x!0g(x) =ab lim
x!0h(x) =1
Floating Point
Arithmeticf(0) = inf*0 = nan g(0) = inf*0 = nan h(0) = inf*0 = nan
Table 1: Automatic dierentiation does not compute limits. Indeterminate forms must be simplied manually.
(These cases are not meant to be exhaustive.)
Indeterminate forms result in NaN values within a calculation. Mathematically, if they occur at locally isolated
points, then we generally prefer the mathematical limit as the result, even if it is innite. As demonstrated in Table 1,
depending upon the nature of the indeterminate form, the mathematical limit can be 0 (no matter the values of
aorb), or ab, or1, but these 3 cases cannot be distinguished by the oating point result of nan. Floating point
arithmetic does not perform limits (directly), and neither does the autodi library. Thus it is up to the diligence of
the developer to keep a watchful eye over where indeterminate forms can arise.
6.3 Example 3: f(x) =pxandf0(0) =1
When working with functions that have innite higher order derivatives, this can very quickly result in nans in higher
order derivatives as the computation progresses, as inf-inf ,inf/inf , and 0*inf result in nan. See Table 2 for an
example.
f(x) f(0) f0(0) f00(0) f000(0)
sqrt(x) 0 inf -inf inf
sqr(sqrt(x)+1) 1 inf nan nan
x+2*sqrt(x)+1 1 inf -inf inf
Table 2: Indeterminate forms in higher order derivatives. sqr(x) == x*x .
Calling the autodi-overloaded implementation of f(x) =pxat the value x==0 results in the 1strow (after the
header row) of Table 2, as is mathematically correct. The 2ndrow shows f(x) = (px+ 1)2resulting in nanvalues
forf00(0) and all higher order derivatives. This is due to the internal arithmetic in which infis added to -inf
during the squaring, resulting in a nanvalue for f00(0) and all higher orders. This is typical of infvalues in autodi.
Where they show up, they are correct, however they can quickly introduce nanvalues into the computation upon the
addition of oppositely signed infvalues, division by inf, or multiplication by 0. It is worth noting that the infection
ofnanonly spreads upward in the order of derivatives, since lower orders do not depend upon higher orders (which
is also why dropping higher order terms in an autodi computation does not result in any loss of precision for lower
order terms.)
The resolution in this case is to manually perform the squaring in the computation, replacing the 2ndrow with
the 3rd:f(x) =x+ 2px+ 1. Though mathematically equivalent, it allows autodi to avoid nanvalues sincepxis
more \isolated" in the computation. That is, the infvalues that unavoidably show up in the derivatives of sqrt(x)
forx==0 do not have the chance to interact with other infvalues as with the squaring.
6.4 Summary
The AFP gives a high-level unied guiding principle for writing C++ template functions that autodi can eectively
evaluate derivatives for.
Examples have been given to illustrate some common items to avoid doing:
1. It is not enough for functions to be piecewise continuous. On boundary points between intervals, consider
returning the average expression of both intervals, rather than just one of them. Example: max(0 ; x) atx= 0.
14
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )In cases where the limits from both sides must match, and they do not, then nanmay be a more appropriate
value depending on the application.
2. Avoid returning individual constant values (e.g. sinc(0) = 1.) Values must be computed uniformly along with
other values in its local interval. If that is not possible, then the function must be overloaded to compute the
derivatives precisely using the helper functions from Section 5.3.
3. Avoid intermediate indeterminate values in both the value (sinc( x) atx= 0) and derivatives ((px+ 1)2at
x= 0). Try to isolate expressions that may contain innite values/derivatives so that they do not introduce
NaN values into the computation.
7 Acknowledgments
Kedar Bhat | C++11 compatibility, Boost Special Functions compatibility testing, codecov integration, and
feedback.
Nick Thompson | Initial feedback and help with Boost integration.
John Maddock | Initial feedback and help with Boost integration.
References
[1]https://en.wikipedia.org/wiki/Automatic_differentiation
[2] Andreas Griewank, Andrea Walther. Evaluating Derivatives . SIAM, 2nd ed. 2008.
15
Copyright cMatthew Pulver 2018{2019. Distributed under the Boost Software License, Version 1.0.
(See accompanying le LICENSE 10.txt or copy at https://www.boost.org/LICENSE_1_0.txt )