Title: What's New in PyTorch 2.0? torch.compile - PyImageSearch

URL Source: https://pyimagesearch.com/2023/03/27/whats-new-in-pytorch-2-0-torch-compile/

Published Time: 2023-03-27T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [What’s New in PyTorch 2.0? `torch.compile`](#h2BPTitle)

*   [Configuring Your Development Environment](#h3Configuring)
*   [Installation](#h3Installation)
*   [Verification](#h3Verification)
*   [Overview of PyTorch 2.0](#h3Overview)
*   [What’s New in PyTorch 2.0? `torch.compile`](#h3WhatsNew)

*   [`torch.compile` Definition](#h4Definition)

*   [Accelerating DNNs with PyTorch 2.0](#h3AcceleratingDNNs)
*   [Project Structure](#h3Project)
*   [Accelerating Convolutional Neural Networks](#h3AcceleratingCNN)

*   [Parsing Command Line Arguments and Running a Model](#h4Parsing)
*   [Evaluating Convolutional Neural Networks](#h4EvaluatingCNN)

*   [Accelerating Vision Transformers](#h3AcceleratingViT)

*   [Evaluating Vision Transformers](#h4EvaluatingViT)

*   [Accelerating BERT](#h3AcceleratingBERT)

*   [Evaluating BERT](#h4EvaluatingBERT)

*   [Miscellaneous](#h3Miscellaneous)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

Over the last few years, PyTorch has evolved as a popular and widely used framework for training deep neural networks (DNNs). The success of PyTorch is attributed to its simplicity, first-class Python integration, and imperative style of programming. Since the launch of PyTorch in 2017, it has strived for high performance and eager execution. It has provided some of the best abstractions for distributed training, data loading, and automatic differentiation.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201024%20575'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/pt-2-1-featured.png)

With continuous innovation from the PyTorch team, PyTorch has moved from version 1.0 to the most recent version, 1.13. However, over all these years, hardware accelerators like GPUs have become 15x and 2x faster in compute and memory access, respectively. Thus, to leverage these resources and deliver high-performance eager execution, the team moved substantial parts of PyTorch internals to C++.

On December 2, 2022, the team announced the launch of PyTorch 2.0, a next-generation release that will make training deep neural networks much faster and support dynamic shapes. The stable release of PyTorch 2.0 is planned for March 2023. This blog series aims to understand and test the capabilities of PyTorch 2.0 via its beta release.

In this series, you will learn about **Accelerating Deep Learning Models with PyTorch 2.0**.

This lesson is the 1st of a 2-part series on **Accelerating Deep Learning Models with PyTorch 2.0**:

1.  **_[What’s New in PyTorch 2.0? `torch.compile`](https://pyimg.co/fh15d)_ (today’s tutorial)**
2.  _What’s Behind PyTorch 2.0? TorchDynamo and TorchInductor (primarily for developers)_

**To learn what’s new in PyTorch 2.0,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

* * *

[**What’s New in PyTorch 2.0? `torch.compile`**](#TOC)
------------------------------------------------------

We start this lesson by learning to install PyTorch 2.0.

* * *

### [**Configuring Your Development Environment**](#TOC)

* * *

### [**Installation**](#TOC)

Like previous versions, PyTorch 2.0 is available as a Python `pip` package. However, to successfully install PyTorch 2.0, your system should have installed the latest CUDA (Compute Unified Device Architecture) versions (11.6 and 11.7). Here’s how you can install the PyTorch 2.0 nightly version via pip:

For CUDA version 11.7

$ pip3 install numpy --pre torch --force-reinstall --extra-index-url https://download.pytorch.org/whl/nightly/cu117

For CUDA version 11.6

$ pip3 install numpy --pre torch --force-reinstall --extra-index-url https://download.pytorch.org/whl/nightly/cu116

However, if you don’t have CUDA 11.6 or 11.7 installed on your system, you can download all the required dependencies in the PyTorch nightly binaries with docker.

$ sudo apt install -y nvidia-docker2
$ sudo systemctl restart docker
$ docker pull ghcr.io/pytorch/pytorch-nightly
$ docker run --gpus all -it ghcr.io/pytorch/pytorch-nightly:latest /bin/bash

Be sure to specify `--gpus all` so that your container can access all your GPUs.

* * *

### [**Verification**](#TOC)

Optionally, you can verify your installation via:

$ git clone https://github.com/pytorch/pytorch
$ cd tools/dynamo
$ python verify\_dynamo.py

* * *

### [**Overview of PyTorch 2.0**](#TOC)

Before understanding what’s new in PyTorch 2.0, let us first understand the fundamental difference between _eager_ and _graph_ executions (**Figure 1**).

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20631%20385'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/Eager_vs_Graph.jpg)

**Figure 1:** Eager vs. Graph execution (source: image by the author).

**Eager Execution:** An eager execution evaluates the operations immediately and at run time. The programs are generally easy to write, test, and debug with a natural Python-like syntax design. However, because of its nature, it fails to fully leverage the capabilities of hardware accelerators like GPUs. PyTorch is a common example that follows eager execution.

**Graph Execution:** Graph execution, on the other hand, builds a graph of all operations and operands before running. Such an execution is much faster than an eager one, as the graph formed can be optimized to leverage the capabilities of hardware accelerators. However, such programs take more work to write and debug. TensorFlow is a typical example that follows graph execution.

PyTorch has always strived for high performance and eager execution while delivering some of the best abstractions for distributed learning, data loading, and automatic differentiation. To make PyTorch programs faster, the team moved its internals to C++, making the executions faster and less hackable without compromising the flexibility offered by eager mode.

The [PyTorch 2.0](https://pytorch.org/get-started/pytorch-2.0/) release aims to make the training of deep neural networks faster with low memory usage, along with supporting dynamic shapes. In addition, PyTorch 2.0 aims to leverage the capabilities of hardware accelerators and offers better speedups in eager mode.

The backbone of PyTorch 2.0 is four new technologies (TorchDynamo, AOT Autograd, PrimTorch, and TorchInductor) aiming to make PyTorch programs run faster and with less memory.

*   [TorchDynamo](https://github.com/pytorch/torchdynamo) safely captures the PyTorch programs using a new CPython feature called [Frame Evaluation API](https://peps.python.org/pep-0523/) introduced in [PEP 523](https://peps.python.org/pep-0523/). TorchDynamo can [acquire graphs 99%](https://dev-discuss.pytorch.org/t/torchdynamo-update-8-torchdynamo-passed-correctness-check-on-7k-github-models/663) safely, without errors, and with negligible overhead.
*   [AOT Autograd](https://pytorch.org/functorch/stable/notebooks/aot_autograd_optimizations.html) is the new PyTorch autograd engine that generates ahead-of-time (AOT) backward traces.
*   With the [PrimTorch](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-2/645) project, the team could canonicalize 2000+ PyTorch operations (which used to make its backend challenging) to a set of 250 primitive operators that cover the complete PyTorch backend. This makes it easy to implement any new feature in the PyTorch backend.
*   The new [OpenAI Triton-based](https://github.com/openai/triton) deep learning compiler (TorchInductor) can generate fast code for multiple accelerators and backends.

We will discuss more on these new technologies in a future lesson. This high-level overview should set the background and context on what makes PyTorch 2.0 programs faster.

* * *

### [**What’s New in PyTorch 2.0? `torch.compile`**](#TOC)

The core of the PyTorch 2.0 is a `torch.compile` function that wraps your standard PyTorch model, optimizes it under the hood, and returns a compiled version.

* * *

#### [**`torch.compile` Definition**](#TOC)

def torch.compile(model: Callable,
  \*,
  mode: Optional\[str\] = "default",
  dynamic: bool = False,
  fullgraph:bool = False,
  backend: Union\[str, Callable\] = "inductor",
  # advanced backend options go here as kwargs
  \*\*kwargs
) -> torch.\_dynamo.NNOptimizedModule

Here:

*   On **Line 1**, `model` is your `nn.Module` instance. In other words, your standard PyTorch model instance.
*   On **Line 3**, `mode` specifies how much the compiler should optimize while compiling. There are three types:
    *   `default` mode: compiles your model efficiently without taking too much time to compile.
    *   `reduce-overhead` mode: reduces the framework overhead by a lot more but consumes a small amount of extra memory.
    *   `max-autotune` mode: compiles for a long time, giving you the fastest code it can generate.
*   On **Line 4**, `dynamic` specifies where the optimization should be done for dynamic shapes. Since specific compiler optimizations are not applicable for dynamic shapes, it is important to specify this before compiling.
*   On **Line 5**, `fullgraph` compiles the entire program into a single graph. Most users don’t need it unless they are very performance specific.
*   On **Line 6**, `backend` specifies which compiler backend to use. By default, TorchInductor is used, but a few others are available, like `aot_cudagraphs` and `nvfuser`.

`torch.compile`, in its default, is intended to provide you with most of the speedups PyTorch 2.0 has to offer. Hence you only need to use other modes if you are keen on getting the best speed. Based on our discussion, here (**Figure 2**) are the three execution modes you can run your program on.

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20936%20294'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/execution-models.png)

**Figure 2:** Mental model of different execution models supported by PyTorch 2.0 (source: [PyTorch 2.0](https://pytorch.org/get-started/pytorch-2.0/#pytorch-2x-faster-more-pythonic-and-as-dynamic-as-ever)).

Here’s a quick differentiation between the three optimization modes offered by `torch.compile`:

**Table 1:** Comparing different modes present in `torch.compile` (source: table by the author).

<table><tbody><tr><td><strong>Default Mode</strong></td><td><strong>Reduce Overhead Mode</strong></td><td><strong>Max Autotune Mode</strong></td></tr><tr><td>Optimizes for large models</td><td>Optimizes for small models</td><td>Optimizes to produce the fastest models</td></tr><tr><td>Low compile time</td><td>Low compile time</td><td>Very high compile time</td></tr><tr><td>No extra memory usage</td><td>Uses some extra memory</td><td>—</td></tr></tbody></table>

Since `torch.compile` is backward compatible, all other operations (e.g., reading and updating attributes, serialization, distributed learning, inference, and export) would work just as PyTorch 1.x.

Whenever you wrap your model under `torch.compile`, the model goes through the following steps before execution (**Figure 3**):

1.  **Graph Acquisition:** The model is broken down and re-written into subgraphs. Subgraphs that can be compiled/optimized are flattened, whereas other subgraphs which can’t be compiled fall back to the eager model.
2.  **Graph Lowering:** All PyTorch operations are decomposed into their chosen backend-specific kernels.
3.  **Graph Compilation:** All the backend kernels call their corresponding low-level device operations.

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20936%20474'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/compilation-process.jpg)

**Figure 3:** The PyTorch compilation process (source: [PyTorch 2.0](https://pytorch.org/get-started/pytorch-2.0/#pytorch-2x-faster-more-pythonic-and-as-dynamic-as-ever)).

Now, let’s start some experimentation.

* * *

### [**Accelerating DNNs with PyTorch 2.0**](#TOC)

* * *

### [**Project Structure**](#TOC)

We first need to review our project directory structure.

Start by accessing the **_“Downloads”_** section of this tutorial to retrieve the source code.

From there, take a look at the directory structure:

├── cnn.py
├── vit.py
├── bert.py
├── utils.py

The project directory contains four files. The `utils.py` file implements basic utility functions to parse command line arguments and run/report a model’s speed. The `cnn.py`, `vit.py`, and `bert.py` files load a specified CNN (convolutional neural network), ViT (vision transformer), or a BERT (bidirectional encoder representations from transformers) model, compile it with `torch.compile`, and report its speed on a random input. We will discuss these files in detail in subsequent sections.

* * *

### [**Accelerating Convolutional Neural Networks**](#TOC)

Using `torch.compile` is easy and is expected to provide 30%-200% speedups on most models you run daily. However, first, we will look into some utility functions in `utils.py` to parse command line arguments and run a model on a given input.

* * *

#### [**Parsing Command Line Arguments and Running a Model**](#TOC)

import torch
import time
import numpy as np
import argparse

# command line arguments
def parse\_args():
    parser = argparse.ArgumentParser()
    parser.add\_argument('--model', type=str, help='model type', default='resnet50')
    parser.add\_argument('--batch\_size', type=int, help='Batch size', default=128)
    parser.add\_argument('--steps', type=int, help='Steps', default=10)
    parser.add\_argument('--mode', type=str, help='Mode', default='default')
    parser.add\_argument('--backend', type=str, help='Backend', default='inductor')

    args = parser.parse\_args()
    return args

# running a model
def run\_model(model, inputs, steps=20):
    # load model on GPU
    model = model.cuda() 
    # define an optimizer
    optimizer = torch.optim.Adam(model.parameters()) 
    times = \[\]
    for step in range(steps):
        begin = time.time()
        # zero gradients
        optimizer.zero\_grad() 
        # forward pass
        output = model(inputs.cuda()) 
        # back propagate 
        if not isinstance(output, torch.Tensor):
            output = output.logits
        output.sum().backward() 
        # optimize weights
        optimizer.step() 
        end = time.time()
        # calcuate step time
        times.append(float(end - begin)) 
        print(f"Time for {step}-th forward pass is {end - begin}")
    
    # calcuate median step time
    median = np.median(times) 
    print("Median step time is {:.3f} seconds".format(median))

On **Lines 1-4**, we import the `torch`, `time`, `argparse`, and `numpy` libraries. Then, on **Lines 7-16**, we define the `parse_args()` function that parses the following command line arguments:

*   `--model`: specifies the model to load (default set to `resnet50`)
*   `--batch_size`: specifies the batch size of the inputs (default set to `128`)
*   `--steps`: specifies the number of steps to run the model (default is set to `10`)
*   `--mode`: specifies whether to use `default`, `reduce-overhead`, or `original` mode for compilation. We won’t experiment with the `max-autotune` mode as it takes very long to compile and doesn’t always work.
*   `--backend`: specifies the compiler backend (default set to `inductor`)

Then on **Lines 19-44**, we define the `run_model()` function that takes `model`, `inputs`, and `steps` as arguments and runs the model training on `inputs` for a given number of `steps`. First, on **Line 21**, we load the `model` on the GPU. Then on **Line 23**, we define `optimizer` over model parameters.

Finally, on **Lines 25-40**, we run the model training for given `steps` wherein we pass the given `inputs`, backpropagate gradients, and update network weights in each step. We print and store the time taken by each step in a list of `times`. Finally, on **Lines 43 and 44**, we calculate and print the median step time taken by our compiled model.

Now let’s start experimenting with convolutional neural networks.

* * *

#### [**Evaluating Convolutional Neural Networks**](#TOC)

import torch
from utils import parse\_args, run\_model

args = parse\_args()

# loading pretrained resnet50
model = torch.hub.load('pytorch/vision:v0.10.0', args.model, pretrained=True) 

# compile your model
if args.mode in \['default', 'reduce-overhead'\]:
        model = torch.compile(model, mode=args.mode, backend=args.backend)   

# random input image
inputs = torch.randn(args.batch\_size, 3, 224, 224) 
run\_model(model, inputs, args.steps)

We start by loading the `torch` library and utilities from `utils.py` (**Lines 1 and 2**). On **Line 4**, we read the command line arguments. Then on **Line 7**, we load the given `args.model` from TorchHub. If you are unfamiliar with TorchHub, we highly recommend watching our [tutorials](https://pyimagesearch.com/2021/12/20/torch-hub-series-1-introduction-to-torch-hub/).

On **Lines 10 and 11**, we compile the `model` using `torch.compile` with specified mode `args.mode`. Note that if a user specifies any other mode apart from `default` and `reduce-overhead`, we return the original model. By default, we use the `inductor` backend. On **Line 14**, we define a random input image with a given batch size `args.batch_size`. Finally, on **Line 15**, we run and report the time taken by the model using the `run_model` utility function.

Here is a sample command to run the above code snippet. The following command tests a ResNet-50 model with default mode and batch size `256`.

$ python cnn.py --model resnet50 --batch\_size 256 --mode default --steps 10

**Figure 4** displays how the output should look. Note that your numbers might differ depending on your GPU specs.

[![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20974%20356'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/output.png)

**Figure 4:** Running ResNet-50 model using `torch.compile` (source: image by the author).

When you run the above code snippet, you will notice that the first step takes an abnormally long time while the subsequent steps are faster. This is because the `torch.compile` is a lazy wrapper and only compiles the model during the first forward pass.

To notice the speedup, you will need to compare the speed of the compiled model with the original model (by using `--mode original` in the command).

In **Figure 5,** we compare several convolutional models like ResNets, GoogleNet, AlexNet, SqueezeNet, DenseNet, MobileNet, and Wide ResNet. On average, CNNs give a 10% training speedup on NVIDIA A6000s. Among all the models, MobileNetV2 and SqueezeNet provide close to 20% speedup, while AlexNet and Wide ResNet give <10% speedup. Please note that the speedup might differ depending on your hardware accelerator. You are likely to see more significant speedups with newer GPUs like A100s.

[![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20831%20386'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/cnns-2.png)

**Figure 5:** Speedup in CNN training with PyTorch 2.0 on NVIDIA A6000 GPUs (source: image by the author).

* * *

### [**Accelerating Vision Transformers**](#TOC)

Similarly, using the `torch.compile` wrapper, one can speed up a vision transformer for image classification tasks. We will use the PyTorch image models (`timm`) library that can be installed via pip:

$ pip install timm

For this example, we will refer to the `vit.py` file in our project directory.

* * *

#### [**Evaluating Vision Transformers**](#TOC)

import torch
import timm
from utils import parse\_args, run\_model

args = parse\_args()

# loading pretrained ViT model
model = timm.create\_model(args.model, pretrained=True) 

# compile your model
if args.mode in \['default', 'reduce-overhead'\]:
        model = torch.compile(model, mode=args.mode, backend=args.backend)   

# random input image
inputs = torch.randn(args.batch\_size, 3, 224, 224) 
run\_model(model, inputs, args.steps)

Like our previous example, we start by loading `torch`, the `timm` library, and utilities from `utils.py` (**Lines 1-3**). Next, on **Line 5**, we read the command line arguments. Then on **Line 8**, we load the given `args.model` from TIMM. The remainder of the code is the same.

Here’s a sample command to run the above code snippet. The following command tests a ViT-B/16 (vision transformer base architecture and patch size 16) model with default mode and batch size 256. You can check out the list of available models using `timm.list_models()`.

$ python vit.py --model vit\_base\_patch16\_224 --batch\_size 256 --mode default --steps 10

In **Figure 6**, we compare several state-of-the-art vision transformers. We notice that, on average, transformers give only 2%-3% speedup compared to >10% speedup for CNNs. This is likely because of the self-attention module, which operates on the global view of the image rather than the local view (e.g., convolutions in CNNs). Hence are difficult to optimize. Models (e.g., MLP-Mixer) give negative speedup, on the other hand.

[![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20825%20371'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/vision-transformers-2.png)

**Figure 6:** Speedup in ViT training with PyTorch 2.0 on NVIDIA A6000 GPUs (source: image by the author).

* * *

### [**Accelerating BERT**](#TOC)

A similar concept works for natural language processing (NLP) models like BERT. We will use the [Hugging Face](https://huggingface.co/docs/transformers/index) transformers library that can be installed via pip:

$ pip install transformers==4.26.1

For this example, we will refer to the `bert.py` file in our project directory.

* * *

#### [**Evaluating BERT**](#TOC)

import torch
from transformers import AutoConfig, AutoTokenizer, AutoModelForSequenceClassification
from utils import parse\_args, run\_model

args = parse\_args()

# loading pretrained BERT model
config = AutoConfig.from\_pretrained(args.model)
tokenizer = AutoTokenizer.from\_pretrained(args.model)
model = AutoModelForSequenceClassification.from\_config(config)

# compile your model
if args.mode in \['default', 'reduce-overhead'\]:
        model = torch.compile(model, mode=args.mode, backend=args.backend)   

# random input text
text = ", ".join(\["This is a very long text" for i in range(20)\])
inputs = tokenizer(text, return\_tensors='pt')
inputs = inputs\["input\_ids"\].repeat(args.batch\_size, 1)
run\_model(model, inputs, args.steps)

On **Lines 1-3**, we import the `torch`, `transformers`, and `utils.py` libraries. Next, on **Line 5**, we read the command line arguments. Then, on **Lines 8-10**, we load the given `args.model` and its `config` and `tokenizer` from the Hugging Face transformers library.

On **Lines 13 and 14**, we compile the `model` using `torch.compile` with specified mode `args.mode`. Then on **Lines 17-19**, we define a dummy tokenized input text with a given batch size `args.batch_size`. Finally, on **Line 20**, we run and report the time taken by the model using the `run_model` utility function.

Here’s a sample command to run the above code snippet. The following command tests a BERT model with default mode and batch size 256.

$ python bert.py --model bert-base-uncased --batch\_size 256 --mode default --steps 10

**Figure 7** compares some of the state-of-the-art NLP models (e.g., BERT, DistillBERT, and XLM-RoBERTa) from the [Hugging Face](https://huggingface.co/docs/hub/models-libraries) library. On average, PyTorch 2.0 provides a 5%-6% speedup on these models. On the other hand, DistillBERT achieves a maximum speedup of 8.5%.

[![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20563%20371'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/nlp-transformers-2.png)

**Figure 7:** Speedup in NLP models with PyTorch 2.0 on NVIDIA A6000 GPUs (source: image by the author).

* * *

### [**Miscellaneous**](#TOC)

**Different Benchmarks: Figure 8** shows the speedup of PyTorch 2.0 on NVIDIA A100 GPUs across 163 open source models from different libraries (e.g., [TIMM](https://timm.fast.ai/), [TorchBench](https://github.com/pytorch/benchmark), and [Hugging Face](https://huggingface.co/)). At Float32 precision, it runs 21% faster on average, and at AMP (automatic mixed precision), it runs 51% faster on average. The figure reports the uneven weighted average speedup of `0.75 * AMP + 0.25 * float32` since we find AMP is more common in practice.

[![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20561%20321'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/a100-speedup-2.png)

**Figure 8:** Speedup across different open source benchmarks with PyTorch 2.0 on NVIDIA A100 GPUs (source: [PyTorch](https://pytorch.org/get-started/pytorch-2.0/#pytorch-2x-faster-more-pythonic-and-as-dynamic-as-ever)).

**Different Backends:** By default, we have used the “inductor” compiler backend in our experiments so far. However, there are plenty of backends supported by PyTorch 2.0. You can find the list of supported backends using `torch._dynamo.list_backends()`. **Figure 9** compares a few different compiler backends with the default TorchInductor backend for the ResNet-50 model. We can see that TorchInductor, by default, gives the maximum speedup.

[![Image 11](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20725%20371'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/03/compiler-backends-2.png)

**Figure 9:** Speedup across different compiler backends with PyTorch 2.0 on NVIDIA A6000 GPUs (source: image by the author).

You can experiment with other backends. Note that these backends are hardware-dependent; some might not work on your hardware.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

PyTorch has been one of the most popular and widely used frameworks for training and deploying deep learning models. Continuous innovation in PyTorch has resulted in an elegant, high-performance, and eager execution framework. PyTorch 2.0, a next-generation release, brings significant speedup in eager execution by leveraging the best of hardware accelerators through the latest technologies (e.g., TorchDynamo, TorchInductor, PrimTorch, and AOT Autograd).

At the core, PyTorch 2.0 introduces `torch.compile`, a function that wraps your `nn.Module` instances, optimizes its graph, and provides a fast model for several backends and architectures. Besides being easy to use, `torch.compile` is backward compatible. All other operations (e.g., reading and updating attributes, serialization, distributed learning, inference, export, etc.) would work just as in PyTorch 1.x.

On 163 open source models from different libraries (e.g., [TIMM](https://timm.fast.ai/), [TorchBench](https://github.com/pytorch/benchmark), and [Hugging Face](https://huggingface.co/)), `torch.compile` provided 30%-200% speedups on NVIDIA A100s. Moreover, at Float32 precision, it runs 21% faster on average, and at AMP (automatic mixed precision), it runs 51% faster on average. We also experimented on NVIDIA A6000s and observed that PyTorch 2.0 could provide up to 20% speedup on vision architectures (e.g., SqueezeNet, DenseNet, etc.).

PyTorch has always strived for high performance and eager execution while delivering some of the best abstractions for distributed learning, data loading, and automatic differentiation. With this new release, training deep neural networks in eager modes will become much faster!

* * *

### [**Citation Information**](#TOC)

**Mangla, P.** “What’s New in PyTorch 2.0? `torch.compile`,” _PyImageSearch_, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, R. Raha, and A. Thanki, eds., 2023, [https://pyimg.co/fh15d](https://pyimg.co/fh15d)

@incollection{Mangla\_2023\_PT2TC,
  author = {Puneet Mangla},
  title = {What's New in PyTorch 2.0? torch.compile},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha and Abhishek Thanki},
  year = {2023},
  url = {https://pyimg.co/fh15d},
}

* * *

![Image 12: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 13](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!