Title: Automatic Differentiation Part 1: Understanding the Math - PyImageSearch

URL Source: https://pyimagesearch.com/2022/12/05/automatic-differentiation-part-1-understanding-the-math/

Published Time: 2022-12-05T14:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Automatic Differentiation Part 1: Understanding the Math](#h2BPTitle)

*   [Introduction](#h3Introduction)

*   [Jacobian](#h4Jacobian)
*   [Chain Rule](#h4Chain)
*   [Mix the Jacobian and Chain Rule](#h4Mix)
*   [Forward and Reverse Accumulations](#h4Accumulation)
*   [Forward Accumulation](#h4Forward)
*   [Reverse Accumulation](#h4Reverse)

*   [Summary](#h2Summary)

*   [References](#h3References)
*   [Citation Information](#h3Citation)

* * *

In this tutorial, you will learn **the math behind automatic differentiation needed for backpropagation**.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/12/Automatic-Differentiation-Part-1-featured.png)

This lesson is the 1st of a 2-part series on **Autodiff 101 — Understanding Automatic Differentiation from Scratch**:

1.  [**_Automatic Differentiation Part 1: Understanding the Math_**](https://pyimg.co/pyxml) **(this tutorial)**
2.  _Automatic Differentiation Part 2: Implementation Using Micrograd_

**To learn about automatic differentiation,** **_just keep reading._**

* * *

[**Automatic Differentiation Part 1: Understanding the Math**](#TOC)
--------------------------------------------------------------------

Imagine you are trekking down a hill. It is dark, and there are a lot of bumps and turns. You have no way of knowing how to reach the center. Now imagine every time you progress, you have to pause, take out the topological map of the hill and calculate your direction and speed for the next set. Sounds painfully less fun, right?

If you have been a reader of our tutorials, you would know what that analogy refers to. The hill is your loss landscape, the topological map is the set of rules for multivariate calculus, and you are the parameters of the neural network. The objective is to reach the global minimum.

And that brings us to the question:

> _Why do we use a Deep Learning Framework today?_

The first thing that pops into the mind is **automatic differentiation**. We write the forward pass, and that is it; no need to worry about the backward pass. Every operator is automatically differentiated and is waiting to be used in an optimization algorithm (like stochastic gradient descent).

Today in this tutorial, we will walk through the valleys of automatic differentiation.

* * *

### [**Introduction**](#TOC)

In this section, we will lay out the foundation necessary for understanding `autodiff`.

* * *

#### [**Jacobian**](#TOC)

Let’s consider a function ![Image 2: F \colon \mathbb{R}^{n} \to \mathbb{R}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2082%2011'%3E%3C/svg%3E). ![Image 3: F](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E) is a multivariate function that simultaneously depends on multiple variables. Here the multiple variables can be ![Image 4: x = \{x_{1}, x_{2}, \ldots, x_{n}\}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20140%2017'%3E%3C/svg%3E). The output of the function is a **scalar value**. This can be considered as a neural network that takes an image and outputs the probability of a dog’s presence in the image.

**_Note_****:** Let us recall that in a neural network, we compute gradients with respect to the parameters (weights and biases) and not the inputs (the image). Thus the domain of the function is the parameters and not the inputs, which helps keep the gradient computation accessible. We need to now think of everything we do in this tutorial from the perspective of making it **simple** and **efficient** to obtain the gradients with respect to the weights and biases (_parameters_). This is illustrated in **Figure 1**.

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20500'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/12/autograd1.png)

**Figure 1:** Domain of the function from the perspective of a neural network (source: image by the authors).

A neural network is a composition of many sublayers. So let’s consider our function ![Image 6: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E) as a composition of multiple functions (primitive operations).

![Image 7: F(x) \ = \ D \circ C \circ B \circ A](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20166%2018'%3E%3C/svg%3E)

The function ![Image 8: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E) is composed of four primitive functions, namely ![Image 9: D, C, B, \text{ and } A](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20110%2016'%3E%3C/svg%3E). For anyone new to composition, we can call ![Image 10: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E) to be a function where ![Image 11: D(C(B(A(x))))](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20110%2018'%3E%3C/svg%3E) is equal to ![Image 12: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E).

The next step would be to find the gradient of ![Image 13: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E). However, before diving into the gradients of the function, let us revisit Jacobian matrices. It turns out that the derivatives of a multivariate function are a Jacobian matrix consisting of partial derivatives of the function w.r.t. all the variables upon which it depends.

Consider two multivariate functions, ![Image 14: u](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%207'%3E%3C/svg%3E) and ![Image 15: v](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%207'%3E%3C/svg%3E), which depend on the variables ![Image 16: x](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%207'%3E%3C/svg%3E) and ![Image 17: y](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2010'%3E%3C/svg%3E). The Jacobian would look like this:

![Image 18: \displaystyle\frac{\partial{(u, v)}}{\partial{x, y}} \ = \ \begin{bmatrix} \displaystyle\frac{\partial u}{\partial x} & \displaystyle\frac{\partial u}{\partial y}\\  \\ \displaystyle\frac{\partial v}{\partial x} & \displaystyle\frac{\partial v}{\partial y} \end{bmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20156%2095'%3E%3C/svg%3E)

Now let’s compute the Jacobian of our function ![Image 19: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E). We need to note here that the function depends of ![Image 20: n](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%207'%3E%3C/svg%3E) variables ![Image 21: x = \{x_{1}, x_{2}, \ldots, x_{n}\}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20140%2017'%3E%3C/svg%3E), and outputs a scalar value. This means that the Jacobian will be a row vector.

![Image 22: F^\prime(x) \ = \ \displaystyle\frac{\partial{y}}{\partial{x}} \ = \ \begin{bmatrix} \displaystyle\frac{\partial y}{\partial x_{1}} & \ldots  & \displaystyle\frac{\partial y}{\partial x_{n}} \end{bmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20245%2040'%3E%3C/svg%3E)

* * *

#### [**Chain Rule**](#TOC)

Remember how our function ![Image 23: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E) is composed of many primitive functions? The derivative of such a composed function is done with the help of the chain rule. To help our way into the chain rule, let us first write down the composition and then define the intermediate values.

![Image 24: F(x) = D(C(B(A(x))))](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20166%2018'%3E%3C/svg%3E) is composed of:

*   ![Image 25: y = D(c)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2062%2018'%3E%3C/svg%3E)
*   ![Image 26: c = C(b)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2060%2018'%3E%3C/svg%3E)
*   ![Image 27: b = B(a)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2062%2018'%3E%3C/svg%3E)
*   ![Image 28: a = A(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2062%2018'%3E%3C/svg%3E)

Now that the composition is spelled out, let’s first get the derivatives of the intermediate values.

*   ![Image 29: D^\prime(c) = \displaystyle\frac{\partial{y}}{\partial{c}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2080%2036'%3E%3C/svg%3E)
*   ![Image 30: C^\prime(b) = \displaystyle\frac{\partial{c}}{\partial{b}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2077%2036'%3E%3C/svg%3E)
*   ![Image 31: B^\prime(a) = \displaystyle\frac{\partial{b}}{\partial{a}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2081%2036'%3E%3C/svg%3E)
*   ![Image 32: A^\prime(x) = \displaystyle\frac{\partial{a}}{\partial{x}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2080%2036'%3E%3C/svg%3E)

Now with the help of the chain rule, we derive the derivative of the function ![Image 33: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E).

![Image 34: F^\prime(x) \ = \ \displaystyle\frac{\partial{y}}{\partial{c}} \displaystyle\frac{\partial{c}}{\partial{b}} \displaystyle\frac{\partial{b}}{\partial{a}} \displaystyle\frac{\partial{a}}{\partial{x}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20154%2036'%3E%3C/svg%3E)

* * *

#### [**Mix the Jacobian and Chain Rule**](#TOC)

After knowing about the Jacobian and the Chain Rule, let us visualize the two together. Shown in **Figure 2**.

![Image 35: F^\prime(x) \ = \ \displaystyle\frac{\partial{y}}{\partial{x}} \ = \ \begin{bmatrix} \displaystyle\frac{\partial y}{\partial x_{1}} & \ldots  & \displaystyle\frac{\partial y}{\partial x_{n}} \end{bmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20245%2040'%3E%3C/svg%3E)

![Image 36: F^\prime(x) \ = \ \displaystyle\frac{\partial{y}}{\partial{c}} \displaystyle\frac{\partial{c}}{\partial{b}} \displaystyle\frac{\partial{b}}{\partial{a}} \displaystyle\frac{\partial{a}}{\partial{x}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20154%2036'%3E%3C/svg%3E)

[![Image 37](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20615%20500'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/12/grad_jacobian.png)

**Figure 2:** Jacobian and chain rule together (source: image by the authors).

The derivative of our function ![Image 38: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E) is just the matrix multiplication of the Jacobian matrices of the intermediate terms.

Now, this is where we ask the question:

> _Does it matter the order in which we do the matrix multiplication?_

* * *

#### [**Forward and Reverse Accumulations**](#TOC)

In this section, we try to understand the answer to the question of ordering the Jacobian matrix multiplication.

There are two extremes in which we could order the multiplications: the forward accumulation and the reverse accumulation.

* * *

#### [**Forward Accumulation**](#TOC)

If we order the multiplication from right to left in the same order in which the function ![Image 39: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E) was evaluated, the process is called forward accumulation. The best way to think about the ordering is to place brackets in the equation, as shown in **Figure 3**.

![Image 40: F^\prime(x) \ = \ \displaystyle\frac{\partial{y}}{\partial{c}} \left(\frac{\partial{c}}{\partial{b}} \left(\frac{\partial{b}}{\partial{a}} \displaystyle\frac{\partial{a}}{\partial{x}}\right)\right)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20207%2041'%3E%3C/svg%3E)

[![Image 41](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20412'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/12/forward.png)

**Figure 3:** Forward accumulation of gradients (source: image by the authors).

With the function ![Image 42: F : \mathbb{R}^{n} \to \mathbb{R}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2084%2011'%3E%3C/svg%3E), the forward accumulation process is matrix multiplication in all the steps. This is more [FLOPs](https://stackoverflow.com/a/60275432/19623137).

**_Note:_** Forward accumulation is beneficial when we want to get the derivative of a function ![Image 43: F: \mathbb{R} \to \mathbb{R}^{n}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2083%2011'%3E%3C/svg%3E).

Another way to understand forwarding accumulation is to think of a Jacobian-Vector Product (JVP). Consider a Jacobian ![Image 44: F^\prime(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2037%2018'%3E%3C/svg%3E) and a vector ![Image 45: v](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%207'%3E%3C/svg%3E). The Jacobian-Vector Product would look to be ![Image 46: F^\prime(x)v](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2046%2018'%3E%3C/svg%3E)

![Image 47: F^\prime(x)v \ = \ \displaystyle\frac{\partial{y}}{\partial{c}} \left(\displaystyle\frac{\partial{c}}{\partial{b}} \left(\displaystyle\frac{\partial{b}}{\partial{a}} \left(\displaystyle\frac{\partial{a}}{\partial{x}} v\right)\right)\right)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20251%2041'%3E%3C/svg%3E)

This is done for us to have matrix-vector multiplication at all the stages (which makes the process more efficient).

➤ **Question:** If we have a Jacobian-Vector Product, how can we obtain the Jacobian from it?

➤ **Answer:** We pass a one-hot vector and get each column of the Jacobian one at a time.

So we can think of forwarding accumulation as a process in which we build the Jacobian per column.

* * *

#### [**Reverse Accumulation**](#TOC)

Suppose we order the multiplication from left to right, in the opposite direction to which the function was evaluated. In that case, the process is called reverse accumulation. The diagram of the process is illustrated in **Figure 4**.

![Image 48: F^\prime(x) \ = \ \left(\left(\displaystyle\frac{\partial{y}}{\partial{c}} \displaystyle\frac{\partial{c}}{\partial{b}}\right) \displaystyle\frac{\partial{b}}{\partial{a}} \right)\displaystyle\frac{\partial{a}}{\partial{x}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20209%2041'%3E%3C/svg%3E)

[![Image 49](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20362'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/12/backward.png)

**Figure 4:** Reverse accumulation of gradients (source: image by the authors).

As it turns out, with reverse accumulation deriving the derivative of a function ![Image 50: F : \mathbb{R}^{n} \to \mathbb{R}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2084%2011'%3E%3C/svg%3E) is a vector to matrix multiplication at all steps. This means that for the particular function, reverse accumulation has lesser FLOPs than forwarding accumulation.

Another way to understand forwarding accumulation is to think of a Vector-Jacobian Product (VJP). Consider a Jacobian ![Image 51: F^\prime(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2037%2018'%3E%3C/svg%3E) and a vector ![Image 52: v](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%207'%3E%3C/svg%3E). The Vector-Jacobian Product would look to be ![Image 53: v^{T}F^\prime(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2054%2018'%3E%3C/svg%3E)

![Image 54: v^{T}F^\prime(x) \ = \ \left(\left(\left(v^{T} \displaystyle\frac{\partial{y}}{\partial{c}}\right) \displaystyle\frac{\partial{c}}{\partial{b}}\right) \displaystyle\frac{\partial{b}}{\partial{a}}\right)\displaystyle\frac{\partial{a}}{\partial{x}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20271%2041'%3E%3C/svg%3E)

This allows us to have vector-matrix multiplication at all stages (which makes the process more efficient).

➤ **Question:** If we have a Vector-Jacobian Product, how can we obtain the Jacobian from it?

➤ **Answer:** We pass a one-hot vector and get each row of the Jacobian one at a time.

So we can think of reverse accumulation as a process in which we build the Jacobian per row.

Now, if we consider our previously mentioned function ![Image 55: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E), we know that the Jacobian ![Image 56: F^\prime(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2037%2018'%3E%3C/svg%3E) is a row vector. Therefore, if we apply the reverse accumulation process, which means the Vector-Jacobian Product, we can obtain the row vector in one shot. On the other hand, if we apply the forward accumulation process, the Jacobian-Vector Product, we will obtain a single element as a column, and we would need to iterate to build the entire row.

This is why reverse accumulation is used more often in the Neural Network literature.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

In this tutorial, we studied the math of automatic differentiation and how it is applied to the parameters of a Neural Network. The next tutorial will expand on this and see how we can implement automatic differentiation using a python package. The implementation will involve a step-by-step walkthrough of creating a python package and using it to train a neural network.

Did you enjoy a math-heavy tutorial on the fundamentals of automatic differentiation? Let us know.

**Twitter:** [@PyImageSearch](https://twitter.com/pyimagesearch)

* * *

### [**References**](#TOC)

*   [Automatic Differentiation — VideoLectures.NET](http://videolectures.net/deeplearning2017_johnson_automatic_differentiation/)

* * *

### [**Citation Information**](#TOC)

**A. R. Gosthipaty and R. Raha.** “Automatic Differentiation Part 1: Understanding the Math,” _PyImageSearch_, P. Chugh, S. Huot, K. Kidriavsteva, and A. Thanki, eds., 2022, [https://pyimg.co/pyxml](https://pyimg.co/pyxml)

@incollection{ARG-RR\_2022\_autodiff1,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {Automatic Differentiation Part 1: Understanding the Math},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva and Abhishek Thanki},
  year = {2022},
  note = {https://pyimg.co/pyxml},
}

* * *

![Image 57: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

![Image 58](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.