# Calculus/Derivatives of multivariate functions
## The matrix of a linear transformation
### Theorem

A linear transformation $L:\mathbb {R} ^{n}\to \mathbb {R} ^{m}$ amounts to multiplication by a uniquely defined matrix; that is, there exists a unique matrix $A\in \mathbb {R} ^{m\times n}$ such that
    $\forall {\vec {v}}\in \mathbb {R} ^{n}:L({\vec {v}})=A{\vec {v}}$

### Proof

We set the column vectors
    ${\begin{pmatrix}a_{1,j}\\a_{2,j}\\\vdots \\a_{n,j}\end{pmatrix}}:=L({\vec {e}}_{j})$
where $\{{\vec {e}}_{1},\ldots ,{\vec {e}}_{n}\}$ is the standard basis of $\mathbb {R} ^{n}$ . Then we define from this
    $A:={\begin{pmatrix}a_{1,1}&\cdots &a_{1,n}\\\vdots &\ddots &\vdots \\a_{n,1}&\cdots &a_{n,n}\end{pmatrix}}$
and note that for any vector ${\vec {v}}=(v_{1},\ldots ,v_{n})^{t}$ of $\mathbb {R} ^{n}$ we obtain
    $A{\vec {v}}=A\left(\sum _{j=1}^{n}v_{j}{\vec {e}}_{j}\right)=\sum _{j=1}^{n}Av_{j}{\vec {e}}_{j}=\sum _{j=1}^{n}v_{j}L({\vec {e}}_{j})=L\left(\sum _{j=1}^{n}v_{j}{\vec {e}}_{j}\right)=L({\vec {v}})$
Thus, we have shown existence. To prove uniqueness, suppose there were any other matrix $B\in \mathbb {R} ^{m\times n}$ with the property that $\forall {\vec {v}}\in \mathbb {R} ^{n}:L({\vec {v}})=B{\vec {v}}$ . Then in particular,
    $B{\vec {e}}_{j}=L({\vec {e}}_{j})$
which already implies that $A=B$ (since all the columns of both matrices are identical).$\Box$

## How to generalise the derivative

It is not immediately straightforward how one would generalize the derivative to higher dimensions. For, if we take the definition of the derivative at a point $x_{0}$
    $\lim _{h\to 0}{\frac {f(x_{0}+h)-f(x_{0})}{h}}$
and insert vectors for $h$ and $x_{0}$ , we would divide the whole thing by a vector. But this is not defined.
Hence, we shall rephrase the definition of the derivative a bit and cast it into a form where it can be generalized to higher dimensions.

### Theorem

Let $f:\mathbb {R} \to \mathbb {R}$ be a one-dimensional function and let $x_{0}\in \mathbb {R}$ . Then $f$ is differentiable at $x_{0}$ if and only if there exists a linear function $l:\mathbb {R} \to \mathbb {R}$ such that
    $\lim _{h\to 0}{\frac {{\Big |}f(x_{0}+h)-{\bigf(x_{0})+l(h){\big )}{\Big |}}{|h|}}=0$
We note that according to the above, linear functions $l:\mathbb {R} \to \mathbb {R}$ are given by multiplication by a $1\times 1$-matrix, that is, a scalar.

### Proof

First assume that $f$ is differentiable at $x_{0}$. We set $l(h):=f'(x_{0})\cdot h$ and obtain
    ${\frac {{\Big |}f(x_{0}+h)-{\bigf(x_{0})+l(h){\big )}{\Big |}}{|h|}}=\left|{\frac {f(x_{0}+h)-f(x_{0})}{h}}-f'(x_{0})\right|$
which converges to 0 due to the definition of $f'(x_{0})$ .
Assume now that we are given an $l:\mathbb {R} \to \mathbb {R}$ such that
    $\lim _{h\to 0}{\frac {{\Big |}f(x_{0}+h)-{\bigf(x_{0})+l(h){\big )}{\Big |}}{|h|}}=0$
Let $c$ be the scalar associated to $l$ . Then by an analogous computation $f'(x_{0})=c$ .$\Box$
With the latter formulation of differentiability from the above theorem, we may readily generalize to higher dimensions, since division by the Euclidean norm of a vector is defined, and linear mappings are also defined in higher dimensions.

### Definition

A function $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ is called differentiable or totally differentiable at a point $x_{0}\in \mathbb {R} ^{m}$ if and only if there exists a linear function $L:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ such that
    $\lim _{{\vec {h}}\to 0}{\frac {{\Big \|}f(x_{0}+{\vec {h}})-{\bigf(x_{0})+L({\vec {h}}){\big )}{\Big \|}}{\|{\vec {h}}\|}}=0$
We have already proven that this definition coincides with the usual one in the one-dim. case (that is $m=n=1$).
We have the following theorem:

### Theorem

Let $S\subseteq \mathbb {R} ^{m}$ be a set, let $x_{0}\in {\overset {\circ }{S}}$ be an interior point of $S$ , and let $f:S\to \mathbb {R} ^{m}$ be a function differentiable at $x_{0}$ . Then the linear map $L$ such that
    $\lim _{{\vec {h}}\to 0}{\frac {{\Big \|}f(x_{0}+{\vec {h}})-{\bigf(x_{0})+L({\vec {h}}){\big )}{\Big \|}}{\|{\vec {h}}\|}}=0$
is unique; that is, there exists only one such map $L$ .

### Proof

Since $x_{0}$ is an interior point of $S$, we find $r>0$ such that $B_{r}(x_{0})\subseteq S$ . Let now $K:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be any other linear mapping with the property that
    $\lim _{{\vec {h}}\to 0}{\frac {{\Big \|}f(x_{0}+{\vec {h}})-{\bigf(x_{0})+K({\vec {h}}){\big )}{\Big \|}}{\|{\vec {h}}\|}}=0$
We note that for all vectors of the standard basis $\{e_{1},\ldots ,e_{n}\}$ , the numbers $\lambda e_{j}$ for $0\leq \lambda <r$ are contained within $S$ . Hence, we obtain by the triangle inequality
    ${\Big \|}L({\vec {e}}_{j})-K({\vec {e}}_{j}){\Big \|}={\frac {{\bigl \|}L(\lambda {\vec {e}}_{j})-K(\lambda {\vec {e}}_{j}){\bigr \|}}{\|\lambda {\vec {e}}_{j}\|}}\leq {\frac {{\Big \|}f(x_{0}+\lambda {\vec {e}}_{j})-{\bigf(x_{0})+L(\lambda {\vec {e}}_{j}){\big )}{\Big \|}}{\|\lambda {\vec {e}}_{j}\|}}+{\frac {{\Big \|}f(x_{0}+\lambda {\vec {e}}_{j})-{\bigf(x_{0})+K(\lambda {\vec {e}}_{j}){\big )}{\Big \|}}{\|\lambda {\vec {e}}_{j}\|}}$
Taking $\lambda \to 0$ , we see that $L({\vec {e}}_{j})=K({\vec {e}}_{j})$ . Thus, $L$ and $K$ coincide on all basis vectors, and since every other vector can be expressed as a linear combination of those, by linearity of $L$ and $K$ we obtain $L=K$ .$\Box$
Thus, the following definition is justified:

### Definition

Let $f:S\to \mathbb {R} ^{n}$ be a function (where $S\subseteq \mathbb {R} ^{m}$ is a subset of $\mathbb {R} ^{m}$), and let $x_{0}$ be an interior point of $S$ such that $f$ is differentiable at $x_{0}$ . Then the unique linear function $L$ such that
    $\lim _{{\vec {h}}\to 0}{\frac {{\Big \|}f(x_{0}+{\vec {h}})-{\bigf(x_{0})+L({\vec {h}}){\big )}{\Big \|}}{\|{\vec {h}}\|}}=0$
is called the differential of $f$ at $x_{0}$ and is denoted $f(x_{0}):=L$ .

## Directional and partial derivatives

We shall first define directional derivatives.

### Definition

Let $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be a function, and let ${\vec {v}}\in \mathbb {R} ^{m}$ be a vector. If the limit
    $\lim _{h\to 0}{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{h}}$
exists, it is called directional derivative of $f$ in direction ${\vec {v}}$ . We denote it by $D_{\vec {v}}f(x_{0})$ .
The following theorem relates directional derivatives and the differential of a totally differentiable function:

### Theorem

Let $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be a function that is totally differentiable at $x_{0}$, and let ${\vec {v}}\in \mathbb {R} ^{m}\setminus \{0\}$ be a nonzero vector. Then $D_{\vec {v}}f(x_{0})$ exists and is equal to $f'(x_{0}){\vec {v}}$ .

### Proof

According to the very definition of total differentiability,
    $\lim _{h\to 0}\left\|{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{|h|\cdot \|{\vec {v}}\|}}-{\frac {f'(x_{0}){\vec {v}}}{|h|\cdot \|{\vec {v}}\|}}\right\|=0$
Hence,
    $\lim _{h\to 0}\left\|{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{|h|}}-{\frac {f'(x_{0}){\vec {v}}}{|h|}}\right\|=0$
by multiplying the above equation by $\|{\vec {v}}\|$ . Noting that
    $\left\|{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{|h|}}-{\frac {f'(x_{0}){\vec {v}}}{|h|}}\right\|=\left\|{\frac {f(x_{0}+h{\vec {v}})-f(x_{0})}{h}}-{\frac {f'(x_{0}){\vec {v}}}{h}}\right\|$
the theorem follows.$\Box$
A special case of directional derivatives are partial derivatives:

### Definition

Let $\{{\vec {e}}_{1},\ldots ,{\vec {e}}_{m}\}$ be the standard basis of $\mathbb {R} ^{m}$ , let $x_{0}\in \mathbb {R} ^{m}$ and let $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be a function such that the directional derivatives $D_{{\vec {e}}_{j}}f(x_{0})$ all exist. Then we set
    ${\frac {\partial f}{\partial x_{j}}}:=D_{{\vec {e}}_{j}}f(x_{0})$
and call it the partial derivative in the direction of $x_{j}$ .
In fact, by writing down the definition of $D_{{\vec {e}}_{j}}f(x_{0})$ , we see that the partial derivative in the direction of $x_{j}$ is nothing else than the derivative of the function $y\mapsto f(x_{0,1},\ldots ,x_{0,j-1},y,x_{0,j+1},\ldots ,x_{0,m})$ in the variable $y$ at the place $x_{0,j}$ . That is, for instance, if
    $f(x,y,z)=x^{2}+4z^{3}+3xy$
then
    ${\frac {\partial f}{\partial x}}=2x+3y\ ,\ {\frac {\partial f}{\partial y}}=3x\ ,\ {\frac {\partial f}{\partial z}}=12z^{2}$
that is, when forming a partial derivative, we regard the other variables as constant and derive only with respect to the variable we are considering.

## The Jacobian matrix

From the above, we know that the differential of a function $f'(x_{0})$ has an associated matrix representing the linear map thus defined. Under a condition, we can determine this matrix from the partial derivatives of the component functions.

### Theorem

Let $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ be a function such that all partial derivatives exist at $x_{0}$ and are continuous in each component on $B_{r}(x_{0})$ for a possibly very small, but positive $r>0$ . Then $f$ is totally differentiable at $x_{0}$ and the differential of $f$ is given by left multiplication by the matrix
    $J_{f}(x_{0}):={\begin{pmatrix}{\dfrac {\partial f_{1}}{\partial x_{1}}}&\cdots &{\dfrac {\partial f_{1}}{\partial x_{m}}}\\\vdots &\ddots &\vdots \\{\dfrac {\partial f_{n}}{\partial x_{1}}}&\cdots &{\dfrac {\partial f_{n}}{\partial x_{m}}}\end{pmatrix}}$
where $f=(f_{1},\ldots ,f_{n})$ .
The matrix $J_{f}(x_{0})$ is called the Jacobian matrix.

### Proof

    ${\frac {{\Big \|}f(x_{0}+{\vec {h}})-(f(x_{0})+J_{f}(x_{0}){\vec {h}}){\Big \|}}{\|{\vec {h}}\|}}$ $={\frac {\left\|\displaystyle \sum _{j=1}^{n}f_{j}(x_{0}+{\vec {h}}){\vec {e}}_{j}-\sum _{j=1}^{n}\left(f_{j}(x_{0})+\sum _{k=1}^{m}h_{k}{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0})\right){\vec {e}}_{j}\right\|}{\|{\vec {h}}\|}}$ $\leq \sum _{j=1}^{n}{\frac {\left\|f_{j}(x_{0}+h)-\left(f_{j}(x_{0})+\displaystyle \sum _{k=1}^{m}h_{k}{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0})\right)\right\|}{\|{\vec {h}}\|}}$
We shall now prove that all summands of the last sum go to 0.
Indeed, let $j\in \{1,\ldots ,n\}$ . Writing again ${\vec {h}}=(h_{1},\ldots ,h_{m})$ , we obtain by the one-dimensional mean value theorem, first applied in the first variable, then in the second and so on, the succession of equations
    $f_{j}(x_{0}+h_{1}{\vec {e}}_{1})-f_{j}(x_{0})=\overbrace {(x_{0,1}+h_{1}-x_{0,1})} ^{=h_{1}}{\frac {\partial f_{j}}{\partial x_{1}}}(x_{0}+t_{1}{\vec {e}}_{1})$
    $f_{j}(x_{0}+h_{1}{\vec {e}}_{1}+h_{2}{\vec {e}}_{2})-f_{j}(x_{0}+h_{1}{\vec {e}}_{1})=\overbrace {(x_{0,2}+h_{2}-x_{0,2})} ^{=h_{2}}{\frac {\partial f_{j}}{\partial x_{2}}}(x_{0}+h_{1}{\vec {e}}_{1}+t_{2}{\vec {e}}_{2})$
    $\vdots$
    $f_{j}(x_{0}+h_{1}{\vec {e}}_{1}+\cdots +h_{m}{\vec {e}}_{m})-f_{j}(x_{0}+h_{1}{\vec {e}}_{1}+\cdots +h_{m-1}{\vec {e}}_{m-1})=\overbrace {(x_{0,m}+h_{m}-x_{0,m})} ^{=h_{m}}{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0}+h_{1}{\vec {e}}_{1}+\cdots +h_{m-1}{\vec {e}}_{m-1}+t_{n}{\vec {e}}_{m})$
for suitably chosen $t_{k}\in [x_{0,k},x_{0,k}+h_{k}]$ . We can now sum all these equations together to obtain
    $f_{j}(x_{0}+{\vec {h}})-f(x_{0})=\sum _{k=1}^{m}h_{k}{\frac {\partial f_{j}}{\partial x_{k}}}\left(x_{0}+\sum _{l=1}^{k-1}h_{l}{\vec {e}}_{l}+t_{k}{\vec {e}}_{k}\right)$
Let now $\delta >0$ . Using the continuity of the ${\frac {\partial f_{j}}{\partial x_{k}}}$ on $B_{r}(x_{0})$ , we may choose $\delta _{k}>0$ such that
    $\left|{\frac {\partial f_{j}}{\partial x_{k}}}\left(x_{0}+\sum _{l=1}^{k-1}h_{l}{\vec {e}}_{l}+t_{k}e_{k}\right)-{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0})\right|<{\frac {\epsilon }{m}}$
for $|h_{k}|<\delta _{k}$ , given that ${\vec {h}}\in B_{r}(0)$ (which we may assume as ${\vec {h}}\to {\vec {0}}$). Hence, we obtain
    ${\frac {\left\|f_{j}(x_{0}+h)-\left(f_{j}(x_{0})+\displaystyle \sum _{k=1}^{m}h_{k}{\frac {\partial f_{j}}{\partial x_{m}}}(x_{0})\right)\right\|}{\|{\vec {h}}\|}}\leq {\frac {\|{\vec {h}}\|\cdot m\cdot {\frac {\epsilon }{m}}}{\|{\vec {h}}\|}}$
and thus the theorem.$\Box$

### Corollary

If $f:\mathbb {R} ^{m}\to \mathbb {R} ^{n}$ is continuously differentiable at $x_{0}\in \mathbb {R} ^{m}$ and ${\vec {v}}\in \mathbb {R} ^{m}\setminus \{0\}$ , then
    $D_{\vec {v}}f(x_{0})=\sum _{j=1}^{m}v_{j}{\frac {\partial f}{\partial x_{j}}}(x_{0})$

### Proof

    $D_{\vec {v}}f(x_{0})=f'(x_{0})({\vec {v}})=J_{f}(x_{0}){\vec {v}}=\sum _{j=1}^{m}v_{j}{\frac {\partial f}{\partial x_{j}}}(x_{0})$$\Box$