For example, in the end-to-end example, we load in a pre-trained model as qat_model, then we simply perform quantization-aware training using:
   python
   # specify quantization config for QAT
   qat_model.qconfig=torch.quantization.get_default_qat_qconfig('fbgemm')
   # prepare QAT
   torch.quantization.prepare_qat(qat_model, inplace=True)
   # convert to quantized version, removing dropout, to check for accuracy on each
   epochquantized_model=torch.quantization.convert(qat_model.eval(), inplace=False)
Device and Operator Support
Quantization support is restricted to a subset of available operators, depending on the method being used, for a list of supported operators, please see the documentation at https://pytorch.org/docs/stable/quantization.html.


The set of available operators and the quantization numerics also depend on the backend being used to run quantized models. Currently quantized operators are supported only for CPU inference in the following backends: x86 and ARM. Both the quantization configuration (how tensors should be quantized and the quantized kernels (arithmetic with quantized tensors) are backend dependent. One can specify the backend by doing:
import torchbackend='fbgemm'
# 'fbgemm' for server, 'qnnpack' for mobile
my_model.qconfig = torch.quantization.get_default_qconfig(backend)
# prepare and convert model
# Set the backend on which the quantized kernels need to be run
torch.backends.quantized.engine=backend


torch.backends.quantized.engine=backend
```
However, quantization aware training occurs in full floating point and can run on either GPU or CPU. Quantization aware training is typically only used in CNN models when post training static or dynamic quantization doesn’t yield sufficient accuracy. This can occur with models that are highly optimized to achieve small size (such as Mobilenet).
Integration in torchvision
We’ve also enabled quantization for some of the most popular models in torchvision: Googlenet, Inception, Resnet, ResNeXt, Mobilenet and Shufflenet. We have upstreamed these changes to torchvision in three forms:
1. Pre-trained quantized weights so that you can use them right away.
2. Quantization ready model definitions so that you can do post-training quantization or quantization aware training.


A script for doing quantization aware training — which is available for any of these model though, as you will learn below, we only found it necessary for achieving accuracy with Mobilenet.
We also have a tutorial showing how you can do transfer learning with quantization using one of the torchvision models.

Choosing an approach
The choice of which scheme to use depends on multiple factors:
1. Model/Target requirements: Some models might be sensitive to quantization, requiring quantization aware training.
2. Operator/Backend support: Some backends require fully quantized operators.
Currently, operator coverage is limited and may restrict the choices listed in the table below:
The table below provides a guideline.


.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:black;}
.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:black;}
.tg .tg-c3ow{border-color:inherit;text-align:center;vertical-align:top}
.tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top;font-weight:bold;color:black;}
article.pytorch-article table tr th:first-of-type, article.pytorch-article table tr td:first-of-type{padding-left:5px}

Model Type
Preferred scheme
Why

LSTM/RNN
Dynamic Quantization
Throughput dominated by compute/memory bandwidth for weights


BERT/Transformer
Dynamic Quantization
Throughput dominated by compute/memory bandwidth for weights

CNN
Static Quantization
Throughput limited by memory bandwidth for activations

CNN
Quantization Aware Training
In the case where accuracy can't be achieved with static quantization

Performance Results
Quantization provides a 4x reduction in the model size and a speedup of 2x to 3x compared to floating point implementations depending on the hardware platform and the model being benchmarked. Some sample results are:

Model
Float Latency (ms)


Float Latency (ms)
  Quantized Latency (ms)
  Inference Performance Gain
  Device
  Notes

  BERT
  581
  313
  1.8x
  Xeon-D2191 (1.6GHz)
  Batch size = 1, Maximum sequence length= 128, Single thread, x86-64, Dynamic quantization

  Resnet-50
  214
  103
  2x
  Xeon-D2191 (1.6GHz)
  Single thread, x86-64, Static quantization

  Mobilenet-v2
  97
  17


17
  5.7x
  Samsung S9
  Static quantization, Floating point numbers are based on Caffe2 run-time and are not optimized

Accuracy results
We also compared the accuracy of static quantized models with the floating point models on Imagenet. For dynamic quantization, we compared the F1 score of BERT on the GLUE benchmark for MRPC.
Computer Vision Model accuracy

Model
Top-1 Accuracy (Float)
Top-1 Accuracy (Quantized)
Quantization scheme

Googlenet
69.8
69.7
Static post training quantization


Inception-v3
77.5
77.1
Static post training quantization

ResNet-18
69.8
69.4
Static post training quantization

Resnet-50
76.1
75.9
Static post training quantization

ResNext-101 32x8d
79.3
79
Static post training quantization

Mobilenet-v2
71.9
71.6
Quantization Aware Training

Shufflenet-v2
69.4


69.4
68.4
Static post training quantization

Speech and NLP Model accuracy

Model
F1 (GLUEMRPC) Float
F1 (GLUEMRPC) Quantized
Quantization scheme

BERT
0.902
0.895
Dynamic quantization

Conclusion


Conclusion
To get started on quantizing your models in PyTorch, start with the tutorials on the PyTorch website. If you are working with sequence data start with dynamic quantization for LSTM, or BERT. If you are working with image data then we recommend starting with the transfer learning with quantization tutorial. Then you can explore static post training quantization. If you find that the accuracy drop with post training quantization is too high, then try quantization aware training.


If you run into issues you can get community help by posting in at discuss.pytorch.org, use the quantization category for quantization related issues.
This post is authored by Raghuraman Krishnamoorthi, James Reed, Min Ni, Chris Gottbrath and Seth Weidman. Special thanks to Jianyu Huang, Lingyi Liu and Haixin Liu for producing quantization metrics included in this post.
Further reading:

PyTorch quantization presentation at Neurips: (https://research.fb.com/wp-content/uploads/2019/12/2.-Quantization.pptx)
Quantized Tensors (https://github.com/pytorch/pytorch/wiki/
Introducing-Quantized-Tensor)
Quantization RFC on Github (https://github.com/pytorch/pytorch/
issues/18318)


layout: blog_detail
title: 'What’s New in PyTorch Profiler 1.9?'
author: Sabrina Smai, Program Manager on the AI Framework team at Microsoft

PyTorch Profiler v1.9 has been released! The goal of this new release (previous PyTorch Profiler release) is to provide you with new state-of-the-art tools to help diagnose and fix machine learning performance issues regardless of whether you are working on one or numerous machines. The objective is to target the execution steps that are the most costly in time and/or memory, and visualize the work load distribution between GPUs and CPUs.
Here is a summary of the five major features being released:


Distributed Training View: This helps you understand how much time and memory is consumed in your distributed training job. Many issues occur when you take a training model and split the load into worker nodes to be run in parallel as it can be a black box. The overall model goal is to speed up model training. This distributed training view will help you diagnose and debug issues within individual nodes.
Memory View: This view allows you to understand your memory usage better. This tool will help you avoid the famously pesky Out of Memory error by showing active memory allocations at various points of your program run.
GPU Utilization Visualization: This tool helps you make sure that your GPU is being fully utilized.
Cloud Storage Support: Tensorboard plugin can now read profiling data from Azure Blob Storage, Amazon S3, and Google Cloud Platform.


Jump to Source Code: This feature allows you to visualize stack tracing information and jump directly into the source code. This helps you quickly optimize and iterate on your code based on your profiling results.

Getting Started with PyTorch Profiling Tool
PyTorch includes a profiling functionality called « PyTorch Profiler ». The PyTorch Profiler tutorial can be found here.
To instrument your PyTorch code for profiling, you must:
$ pip install torch-tb-profiler
import torch.profiler as profiler
With profiler.profile(XXXX)

Comments:
• For CUDA and CPU profiling, see below:
with torch.profiler.profile(
activities=[
torch.profiler.ProfilerActivity.CPU,
torch.profiler.ProfilerActivity.CUDA],


torch.profiler.ProfilerActivity.CUDA],
```
•   With profiler.record_function(“$NAME”): allows putting a decorator (a tag associated to a name) for a block of function
•   Profile_memory=True parameter under profiler.profile allows you to profile CPU and GPU memory footprint
Visualizing PyTorch Model Performance using PyTorch Profiler
Distributed Training
Recent advances in deep learning argue for the value of large datasets and large models, which requires you to scale out model training to more computational resources. Distributed Data Parallel (DDP) and NVIDIA Collective Communications Library (NCCL) are the widely adopted paradigms in PyTorch for accelerating your deep learning training.
In this release of PyTorch Profiler, DDP with NCCL backend is now supported.

Computation/Communication Overview


Computation/Communication Overview
In the Computation/Communication overview under the Distributed training view, you can observe the computation-to-communication ratio of each worker and [load balancer](https://en.wikipedia.org/wiki/Load_balancing_(computing) nodes between worker as measured by granularity.
Scenario 1:
If the computation and overlapping time of one worker is much larger than the others, this may suggest an issue in the workload balance or worker being a straggler. Computation is the sum of kernel time on GPU minus the overlapping time. The overlapping time is the time saved by interleaving communications during computation. The more overlapping time represents better parallelism between computation and communication. Ideally the computation and communication completely overlap with each other. Communication is the total communication time minus the overlapping time. The example image below displays how this scenario appears on Tensorboard.


Figure: A straggler example

Scenario 2:
If there is a small batch size (i.e. less computation on each worker) or the data to be transferred is large, the computation-to-communication may also be small and be seen in the profiler with low GPU utilization and long waiting times. This computation/communication view will allow you to diagnose your code to reduce communication by adopting gradient accumulation, or to decrease the communication proportion by increasing batch size. DDP communication time depends on model size. Batch size has no relationship with model size. So increasing batch size could make computation time longer and make computation-to-communication ratio bigger.
Synchronizing/Communication Overview


Synchronizing/Communication Overview
In the Synchronizing/Communication view, you can observe the efficiency of communication. This is done by taking the step time minus computation and communication time. Synchronizing time is part of the total communication time for waiting and synchronizing with other workers. The Synchronizing/Communication view includes initialization, data loader, CPU computation, and so on Insights like what is the ratio of total communication is really used for exchanging data and what is the idle time of waiting for data from other workers can be drawn from this view.

For example, if there is an inefficient workload balance or straggler issue, you’ll be able to identify it in this Synchronizing/Communication view. This view will show several workers’ waiting time being longer than others.


This table view above allows you to see the detailed statistics of all communication ops in each node. This allows you to see what operation types are being called, how many times each op is called, what is the size of the data being transferred by each op, etc.
Memory View:
This memory view tool helps you understand the hardware resource consumption of the operators in your model. Understanding the time and memory consumption on the operator-level allows you to resolve performance bottlenecks and in turn, allow your model to execute faster. Given limited GPU memory size, optimizing the memory usage can:
1. Allow bigger model which can potentially generalize better on end level tasks.
2. Allow bigger batch size. Bigger batch sizes increase the training speed.


The profiler records all the memory allocation during the profiler interval. Selecting the “Device” will allow you to see each operator’s memory usage on the GPU side or host side. You must enable profile_memory=True to generate the below memory data as shown here.
With torch.profiler.profile(
Profiler_memory=True # this will take 1 – 2 minutes to complete.
)

Important Definitions:
•   “Size Increase” displays the sum of all allocation bytes and minus all the memory release bytes.
•   “Allocation Size” shows the sum of all allocation bytes without considering the memory release.
•   “Self” means the allocated memory is not from any child operators, instead by the operator itself.

GPU Metric on Timeline:


GPU Metric on Timeline:
This feature will help you debug performance issues when one or more GPU are underutilized. Ideally, your program should have high GPU utilization (aiming for 100% GPU utilization), minimal CPU to GPU communication, and no overhead.
Overview:
The overview page highlights the results of three important GPU usage metrics at different levels (i.e. GPU Utilization, Est. SM Efficiency, and Est. Achieved Occupancy). Essentially, each GPU has a bunch of SM each with a bunch of warps that can execute a bunch of threads concurrently. Warps execute a bunch because the amount depends on the GPU. But at a high level, this GPU Metric on Timeline tool allows you can see the whole stack, which is useful.
If the GPU utilization result is low, this suggests a potential bottleneck is present in your model. Common reasons:
•Insufficient parallelism in kernels (i.e., low batch size)
•Small kernels called in a loop. This is to say the launch overheads are not amortized


•CPU or I/O bottlenecks lead to the GPU not receiving enough work to keep busy
Looking of the overview page where the performance recommendation section is where you’ll find potential suggestions on how to increase that GPU utilization. In this example, GPU utilization is low so the performance recommendation was to increase batch size. Increasing batch size 4 to 32, as per the performance recommendation, increased the GPU Utilization by 60.68%.


GPU Utilization: the step interval time in the profiler when a GPU engine was executing a workload. The high the utilization %, the better. The drawback of using GPU utilization solely to diagnose performance bottlenecks is it is too high-level and coarse. It won’t be able to tell you how many Streaming Multiprocessors are in use. Note that while this metric is useful for detecting periods of idleness, a high value does not indicate efficient use of the GPU, only that it is doing anything at all. For instance, a kernel with a single thread running continuously will get a GPU Utilization of 100%


Estimated Stream Multiprocessor Efficiency (Est. SM Efficiency) is a finer grained metric, it indicates what percentage of SMs are in use at any point in the trace This metric reports the percentage of time where there is at least one active warp on a SM and those that are stalled (NVIDIA doc). Est. SM Efficiency also has it’s limitation. For instance, a kernel with only one thread per block can’t fully use each SM. SM Efficiency does not tell us how busy each SM is, only that they are doing anything at all, which can include stalling while waiting on the result of a memory load. To keep an SM busy, it is necessary to have a sufficient number of ready warps that can be run whenever a stall occurs


Estimated Achieved Occupancy (Est. Achieved Occupancy) is a layer deeper than Est. SM Efficiency and GPU Utilization for diagnosing performance issues. Estimated Achieved Occupancy indicates how many warps can be active at once per SMs. Having a sufficient number of active warps is usually key to achieving good throughput. Unlike GPU Utilization and SM Efficiency, it is not a goal to make this value as high as possible. As a rule of thumb, good throughput gains can be had by improving this metric to 15% and above. But at some point you will hit diminishing returns. If the value is already at 30% for example, further gains will be uncertain. This metric reports the average values of all warp schedulers for the kernel execution period (NVIDIA doc). The larger the Est. Achieve Occupancy value is the better.


Overview details: Resnet50_batchsize4

Overview details: Resnet50_batchsize32

Kernel View
The kernel has “Blocks per SM” and “Est. Achieved Occupancy” which is a great tool to compare model runs.

Mean Blocks per SM:
Blocks per SM = Blocks of this kernel / SM number of this GPU. If this number is less than 1, it indicates the GPU multiprocessors are not fully utilized. “Mean Blocks per SM” is weighted average of all runs of this kernel name, using each run’s duration as weight.
Mean Est. Achieved Occupancy:


Mean Est. Achieved Occupancy:
Est. Achieved Occupancy is defined as above in overview. “Mean Est. Achieved Occupancy” is weighted average of all runs of this kernel name, using each run’s duration as weight.
Trace View
This trace view displays a timeline that shows the duration of operators in your model and which system executed the operation. This view can help you identify whether the high consumption and long execution is because of input or model training. Currently, this trace view shows GPU Utilization and Est. SM Efficiency on a timeline.


GPU utilization is calculated independently and divided into multiple 10 millisecond buckets. The buckets’ GPU utilization values are drawn alongside the timeline between 0 – 100%. In the above example, the “ProfilerStep5” GPU utilization during thread 28022’s busy time is higher than the following the one during “Optimizer.step”. This is where you can zoom-in to investigate why that is.

From above, we can see the former’s kernels are longer than the later’s kernels. The later’s kernels are too short in execution, which results in lower GPU utilization.
Est. SM Efficiency: Each kernel has a calculated est. SM efficiency between 0 – 100%. For example, the below kernel has only 64 blocks, while the SMs in this GPU is 80. Then its “Est. SM Efficiency” is 64/80, which is 0.8.


Cloud Storage Support
After running pip install tensorboard, to have data be read through these cloud providers, you can now run:
torch-tb-profiler[blob]
torch-tb-profiler[gs]
torch-tb-profiler[s3]

pip install torch-tb-profiler[blob], pip install torch-tb-profiler[gs], or pip install torch-tb-profiler[S3] to have data be read through these cloud providers. For more information, please refer to this README.
Jump to Source Code:
One of the great benefits of having both TensorBoard and the PyTorch Profiler being integrated directly in Visual Studio Code (VS Code) is the ability to directly jump to the source code (file and line) from the profiler stack traces. VS Code Python Extension now supports TensorBoard Integration.


Jump to source is ONLY available when Tensorboard is launched within VS Code. Stack tracing will appear on the plugin UI if the profiling with_stack=True. When you click on a stack trace from the PyTorch Profiler, VS Code will automatically open the corresponding file side by side and jump directly to the line of code of interest for you to debug. This allows you to quickly make actionable optimizations and changes to your code based on the profiling results and suggestions.

Gify: Jump to Source using Visual Studio Code Plug In UI


For how to optimize batch size performance, check out the step-by-step tutorial here. PyTorch Profiler is also integrated with PyTorch Lightning and you can simply launch your lightning training jobs with --trainer.profiler=pytorch flag to generate the traces. Check out an example here.
What’s Next for the PyTorch Profiler?
You just saw how PyTorch Profiler can help optimize a model. You can now try the Profiler by pip install torch-tb-profiler to optimize your PyTorch model.


Look out for an advanced version of this tutorial in the future. We are also thrilled to continue to bring state-of-the-art tool to PyTorch users to improve ML performance. We'd love to hear from you. Feel free to open an issue here.
For new and exciting features coming up with PyTorch Profiler, follow @PyTorch on Twitter and check us out on pytorch.org.
Acknowledgements
The author would like to thank the contributions of the following individuals to this piece. From the Facebook side: Geeta Chauhan, Gisle Dankel, Woo Kim, Sam Farahzad, and Mark Saroufim. On the Microsoft side: AI Framework engineers (Teng Gao, Mike Guo, and Yang Gu), Guoliang Hua, and Thuy Nguyen.


layout: blog_detail
title: 'New PyTorch library releases including TorchVision Mobile, TorchAudio I/O, and more'
author: Team PyTorch

Today, we are announcing updates to a number of PyTorch libraries, alongside the PyTorch 1.8 release. The updates include new releases for the domain libraries including TorchVision, TorchText and TorchAudio as well as new version of TorchCSPRNG. These releases include a number of new features and improvements and, along with the PyTorch 1.8 release, provide a broad set of updates for the PyTorch community to build on and leverage.
Some highlights include:
* TorchVision - Added support for PyTorch Mobile including Detectron2Go (D2Go), auto-augmentation of data during training, on the fly type conversion, and AMP autocasting.


TorchAudio - Major improvements to I/O, including defaulting to sox_io backend and file-like object support. Added Kaldi Pitch feature and support for CMake based build allowing TorchAudio to better support no-Python environments.
TorchText - Updated the dataset loading API to be compatible with standard PyTorch data loading utilities.
TorchCSPRNG - Support for cryptographically secure pseudorandom number generators for PyTorch is now stable with new APIs for AES128 ECB/CTR and CUDA support on Windows.
Please note that, starting in PyTorch 1.6, features are classified as Stable, Beta, and Prototype. Prototype features are not included as part of the binary distribution and are instead available through either building from source, using nightlies or via compiler flag. You can see the detailed announcement here.

TorchVision 0.9.0
[Stable] TorchVision Mobile: Operators, Android Binaries, and Tutorial


We are excited to announce the first on-device support and binaries for a PyTorch domain library. We have seen significant appetite in both research and industry for on-device vision support to allow low latency, privacy friendly, and resource efficient mobile vision experiences. You can follow this new tutorial to build your own Android object detection app using TorchVision operators, D2Go, or your own custom operators and model.

[Stable] New Mobile models for Classification, Object Detection and Semantic Segmentation
We have added support for the MobileNetV3 architecture and provided pre-trained weights for Classification, Object Detection and Segmentation. It is easy to get up and running with these models, just import and load them as you would any torchvision model:
```python
import torch
import torchvision
Classification


import torch
import torchvision
Classification
x = torch.rand(1, 3, 224, 224)
m_classifier = torchvision.models.mobilenet_v3_large(pretrained=True)
m_classifier.eval()
predictions = m_classifier(x)
Quantized Classification
x = torch.rand(1, 3, 224, 224)
m_classifier = torchvision.models.quantization.mobilenet_v3_large(pretrained=True)
m_classifier.eval()
predictions = m_classifier(x)
Object Detection: Highly Accurate High Resolution Mobile Model
x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)]
m_detector = torchvision.models.detection.fasterrcnn_mobilenet_v3_large_fpn(pretrained=True)
m_detector.eval()
predictions = m_detector(x)
Semantic Segmentation: Highly Accurate Mobile Model
x = torch.rand(1, 3, 520, 520)
m_segmenter = torchvision.models.segmentation.deeplabv3_mobilenet_v3_large(pretrained=True)
m_segmenter.eval()
predictions = m_segmenter(x)
```


predictions = m_segmenter(x)
These models are highly competitive with TorchVision’s existing models on resource efficiency, speed, and accuracy. See our [release notes](https://github.com/pytorch/vision/releases) for detailed performance metrics.
### [Stable] AutoAugment
[AutoAugment](https://arxiv.org/pdf/1805.09501.pdf) is a common Data Augmentation technique that can increase the accuracy of Scene Classification models. Though the data augmentation policies are directly linked to their trained dataset, empirical studies show that ImageNet policies provide significant improvements when applied to other datasets. We’ve implemented 3 policies learned on the following datasets: ImageNet, CIFA10 and SVHN. These can be used standalone or mixed-and-matched with existing transforms:
```python
from torchvision import transforms
t = transforms.AutoAugment()
transformed = t(image)
transform=transforms.Compose([
   transforms.Resize(256),
   transforms.AutoAugment(),
   transforms.ToTensor()])


transforms.ToTensor()])
```
Other New Features for TorchVision

[Stable] All read and decode methods in the io.image package now support:
Palette, Grayscale Alpha and RBG Alpha image types during PNG decoding
On-the-fly conversion of image from one type to the other during read
[Stable] WiderFace dataset
[Stable] Improved FasterRCNN speed and accuracy by introducing a score threshold on RPN
[Stable] Modulation input for DeformConv2D
[Stable] Option to write audio to a video file
[Stable] Utility to draw bounding boxes
[Beta] Autocast support in all Operators
Find the full TorchVision release notes here.

TorchAudio 0.8.0
I/O Improvements
We have continued our work from the previous release to improve TorchAudio’s I/O support, including:


[Stable] Changing the default backend to “sox_io” (for Linux/macOS), and updating the “soundfile” backend’s interface to align with that of “sox_io”. The legacy backend and interface are still accessible, though it is strongly discouraged to use them.
[Stable] File-like object support in both "sox_io" backend, “soundfile” backend and sox_effects.
[Stable] New options to change the format, encoding, and bits_per_sample when saving.
[Stable] Added GSM, HTK, AMB, AMR-NB and AMR-WB format support to the “sox_io” backend.
[Beta] A new functional.apply_codec function which can degrade audio data by applying audio codecs supported by “sox_io” backend in an in-memory fashion.
Here are some examples of features landed in this release:
```python

Load audio over HTTP
with requests.get(URL, stream=True) as response:
    waveform, sample_rate = torchaudio.load(response.raw)
Saving to Bytes buffer as 32-bit floating-point PCM
buffer_ = io.BytesIO()
torchaudio.save(
    buffer_, waveform, sample_rate,


buffer_, waveform, sample_rate,
    format="wav", encoding="PCM_S", bits_per_sample=16)
Apply effects while loading audio from S3
client = boto3.client('s3')
response = client.get_object(Bucket=S3_BUCKET, Key=S3_KEY)
waveform, sample_rate = torchaudio.sox_effects.apply_effect_file(
    response['Body'],
    [["lowpass", "-1", "300"], ["rate", "8000"]])
Apply GSM codec to Tensor
encoded = torchaudio.functional.apply_codec(
    waveform, sample_rate, format="gsm")
```
Check out the revamped audio preprocessing tutorial, Audio Manipulation with TorchAudio.
[Stable] Switch to CMake-based build


[Stable] Switch to CMake-based build
In the previous version of TorchAudio, it was utilizing CMake to build third party dependencies. Starting in 0.8.0, TorchaAudio uses CMake to build its C++ extension. This will open the door to integrate TorchAudio in non-Python environments (such as C++ applications and mobile). We will continue working on adding example applications and mobile integrations.
[Beta] Improved and New Audio Transforms


[Beta] Improved and New Audio Transforms
We have added two widely requested operators in this release: the SpectralCentroid transform and the Kaldi Pitch feature extraction (detailed in "A pitch extraction algorithm tuned for automatic speech recognition"). We’ve also exposed a normalization method to Mel transforms, and additional STFT arguments to Spectrogram. We would like to ask  our community to continue to raise feature requests for core audio processing features like these!
Community Contributions


Community Contributions
We had more contributions from the open source community in this release than ever before, including several completely new features. We would like to extend our sincere thanks to the community. Please check out the newly added CONTRIBUTING.md for ways to contribute code, and remember that reporting bugs and requesting features are just as valuable. We will continue posting well-scoped work items as issues labeled “help-wanted” and “contributions-welcome” for anyone who would like to contribute code, and are happy to coach new contributors through the contribution process.
Find the full TorchAudio release notes here.
TorchText 0.9.0
[Beta] Dataset API Updates


TorchText 0.9.0
[Beta] Dataset API Updates
In this release, we are updating TorchText’s dataset API to be compatible with PyTorch data utilities, such as DataLoader, and are deprecating TorchText’s custom data abstractions such as Field. The updated datasets are simple string-by-string iterators over the data. For guidance about migrating from the legacy abstractions to use modern PyTorch data utilities, please refer to our migration guide.
The text datasets listed below have been updated as part of this work. For examples of how to use these datasets, please refer to our end-to-end text classification tutorial.
* Language modeling: WikiText2, WikiText103, PennTreebank, EnWik9


Text classification: AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull, IMDB
Sequence tagging: UDPOS, CoNLL2000Chunking
Translation: IWSLT2016, IWSLT2017
Question answer: SQuAD1, SQuAD2
Find the full TorchText release notes here.

[Stable] TorchCSPRNG 0.2.0
We released TorchCSPRNG in August 2020, a PyTorch C++/CUDA extension that provides cryptographically secure pseudorandom number generators for PyTorch. Today, we are releasing the 0.2.0 version and designating the library as stable. This release includes a new API for encrypt/decrypt with AES128 ECB/CTR as well as CUDA 11 and Windows CUDA support.
Find the full TorchCSPRNG release notes here.


Thanks for reading, and if you are excited about these updates and want to participate in the future of PyTorch, we encourage you to join the discussion forums and open GitHub issues.
Cheers!
Team PyTorch


Torch Distributed ElasticMakes distributed PyTorch fault-tolerant and elastic.
Get Started
===========
Usage
^^^^^
* Quickstart
* Train script
* Examples
Documentation
=============
API
^^^
* torchrun (Elastic Launch)
* Elastic Agent
* Multiprocessing
* Error Propagation
* Rendezvous
* Expiration Timers
* Metrics
* Events
Advanced
^^^^^^^^
* Customization
Plugins
^^^^^^^
* TorchElastic Kubernetes


torch.overridesThis module exposes various helper functions for the
"torch_function" protocol. See Extending torch for more detail on
the "torch_function" protocol.
Functions
=========
torch.overrides.get_ignored_functions()
   Return public functions that cannot be overridden by
   "torch_function".
   Returns:
      A tuple of functions that are publicly available in the torch
      API but cannot be overridden with "torch_function". Mostly
      this is because none of the arguments of these functions are
      tensors or tensor-likes.
   Return type:
      Set[Callable]
   -[ Examples ]-

torch.Tensor.as_subclass in torch.overrides.get_ignored_functions()
   True
torch.add in torch.overrides.get_ignored_functions()
   False
torch.overrides.get_overridable_functions()
   List functions that are overridable via torch_function
   Returns:
      A dictionary that maps namespaces that contain overridable


functions to functions in that namespace that can be overridden.
   Return type:
      Dict[Any, List[Callable]]
torch.overrides.resolve_name(f)
   Get a human readable string name for a function passed to
   torch_function
   Parameters:
      f (Callable) -- Function to resolve the name of.
   Returns:
      Name of the function; if eval'ed it should give back the input
      function.
   Return type:
      str
torch.overrides.get_testing_overrides()
   Return a dict containing dummy overrides for all overridable
   functions
   Returns:
      A dictionary that maps overridable functions in the PyTorch API
      to lambda functions that have the same signature as the real
      function and unconditionally return -1. These lambda functions
      are useful for testing API coverage for a type that defines
      "torch_function".
   Return type:
      Dict[Callable, Callable]
   -[ Examples ]-

import inspect
my_add = torch.overrides.get_testing_overrides()[torch.add]


inspect.signature(my_add)

torch.overrides.handle_torch_function(public_api, relevant_args, args, kwargs)
   Implement a function with checks for "torch_function"
   overrides.
   See torch::autograd::handle_torch_function for the equivalent of
   this function in the C++ implementation.
   Parameters:
      * public_api (function) -- Function exposed by the public
        torch API originally called like "public_api(args, kwargs)"
        on which arguments are now being checked.
      * relevant_args (iterable) -- Iterable of arguments to
        check for torch_function methods.
      * args (tuple) -- Arbitrary positional arguments
        originally passed into "public_api".
      * kwargs (tuple) -- Arbitrary keyword arguments originally
        passed into "public_api".
   Returns:
      Result from calling "implementation" or an "torch_function"
      method, as appropriate.
   Return type:
      object


Return type:
      object
   :raises TypeError : if no implementation is found.:
   -[ Example ]-

def func(a):
   ...     if has_torch_function_unary(a):
   ...         return handle_torch_function(func, (a,), a)
   ...     return a + 0
torch.overrides.has_torch_function()
   Check for torch_function implementations in the elements of an
   iterable or if a torch_function mode is enabled.  Considers
   exact "Tensor" s and "Parameter" s non-dispatchable.  Use this to
   guard a call to "handle_torch_function()"; don't use it to test if
   something is Tensor-like, use "is_tensor_like()" instead. :param
   relevant_args: Iterable or arguments to check for
   torch_function methods. :type relevant_args: iterable
   Returns:
      True if any of the elements of relevant_args have
      torch_function implementations, False otherwise.
   Return type:
      bool
   See also:
     "torch.is_tensor_like"
        Checks if something is a Tensor-like, including an exact
        "Tensor".


"Tensor".
torch.overrides.is_tensor_like(inp)
   Returns "True" if the passed-in input is a Tensor-like.
   Currently, this occurs whenever there's a "torch_function"
   attribute on the type of the input.
   -[ Examples ]-
   A subclass of tensor is generally a Tensor-like.

class SubTensor(torch.Tensor): ...
is_tensor_like(SubTensor([0]))
   True
   Built-in or user types aren't usually Tensor-like.
is_tensor_like(6)
   False
is_tensor_like(None)
   False
class NotATensor: ...
is_tensor_like(NotATensor())
   False
   But, they can be made Tensor-like by implementing
   torch_function.
class TensorLike:
   ...     @classmethod
   ...     def torch_function(cls, func, types, args, kwargs):
   ...         return -1
is_tensor_like(TensorLike())
   True
torch.overrides.is_tensor_method_or_property(func)
   Returns True if the function passed in is a handler for a method or
   property belonging to "torch.Tensor", as passed into


"torch_function".
   Note:
     For properties, their "get" method must be passed in.
   This may be needed, in particular, for the following reasons:
   1. Methods/properties sometimes don't contain a module slot.
   2. They require that the first passed-in argument is an instance of
      "torch.Tensor".
   -[ Examples ]-

is_tensor_method_or_property(torch.Tensor.add)
   True
is_tensor_method_or_property(torch.add)
   False
   Return type:
      bool
torch.overrides.wrap_torch_function(dispatcher)
   Wraps a given function with "torch_function" -related
   functionality.
   Parameters:
      dispatcher (Callable) -- A callable that returns an
      iterable of Tensor-likes passed into the function.
   Note:
     This decorator may reduce the performance of your code.
     Generally, it's enough to express your code as a series of
     functions that, themselves, support torch_function. If you
     find yourself in the rare situation where this is not the case,


e.g. if you're wrapping a low-level library and you also need it
     to work for Tensor-likes, then this function is available.
   -[ Examples ]-

def dispatcher(a): # Must have the same signature as func
   ...     return (a,)
@torch.overrides.wrap_torch_function(dispatcher)
def func(a): # This will make func dispatchable by torch_function
   ...     return a + 0


Quantization Accuracy DebuggingThis document provides high level strategies for improving
quantization accuracy. If a quantized model has error compared to the
original model, we can categorize the error into:
1. data insensitive error - caused by intrinsic model quantization
   error, large portion of input data has large error
2. data sensitive error - caused by outlier input data, small
   portion of input data has large error
3. implementation error - quantized kernel is not matching
   reference implementation
Data insensitive error
======================
General tips

For PTQ, ensure that the data you are calibrating with is
   representative of your dataset. For example, for a classification
   problem a general guideline is to have multiple samples in every
   category, and the overall number of samples should be at least 100.
   There is no penalty for calibrating with more data other than
   calibration time.


calibration time.
2. If your model has Conv-BN or Linear-BN patterns, consider fusing
   them. If you are using FX graph mode quantization, this is done
   automatically by the workflow. If you are using Eager mode
   quantization, you can do this manually with the
   "torch.ao.quantization.fuse_modules" API.
3. Increase the precision of dtype of the problematic ops. Usually,
   fp32 will have the highest accuracy, followed by fp16, followed by
   dynamically quantized int8, followed by statically quantized int8.
   1. Note: this is trading off performance for accuracy.
   2. Note: availability of kernels per dtype per op can vary by
      backend.
   3. Note: dtype conversions add an additional performance cost. For
      example, "fp32_op -> quant -> int8_op -> dequant -> fp32_op ->
      quant -> int8_op -> dequant" will have a performance penalty
      compared to "fp32_op -> fp32_op -> quant -> int8_op -> int8_op
      -> dequant" because of a higher number of required dtype
      conversions.


conversions.
4. If you are using PTQ, consider using QAT to recover some of the
   accuracy loss from quantization.
Int8 quantization tips

If you are using per-tensor weight quantization, consider using
   per-channel weight quantization.
If you are doing inference on fbgemm, ensure that you set the
   reduce_range argument to False if your CPU is Cooperlake or
   newer, and to True otherwise.
Audit the input activation distribution variation across different
   samples. If this variation is high, the layer may be suitable for
   dynamic quantization but not static quantization.
Data sensitive error
====================
If you are using static quantization and a small portion of your input
data is resulting in high quantization error, you can try:
Adjust your calibration dataset to make it more representative of
   your inference dataset.
Manually inspect (using Numeric Suite) which layers have high


quantization error. For these layers, consider leaving them in
   floating point or adjusting the observer settings to choose a
   better scale and zero_point.
Implementation error
====================
If you are using PyTorch quantization with your own backend you may
see differences between the reference implementation of an operation
(such as "dequant -> op_fp32 -> quant") and the quantized
implementation (such as op_int8) of the op on the target hardware.
This could mean one of two things:
1. the differences (usually small) are expected due to specific
   behavior of the target kernel on the target hardware compared to
   fp32/cpu. An example of this is accumulating in an integer dtype.
   Unless the kernel guarantees bitwise equivalency with the reference
   implementation, this is expected.
2. the kernel on the target hardware has an accuracy issue. In this
   case, reach out to the kernel developer.
Numerical Debugging Tooling (prototype)Warning:


Warning:
  Numerical debugging tooling is early prototype and subject to
  change.
* torch.ao.ns._numeric_suite Eager mode numeric suite
* torch.ao.ns._numeric_suite_fx FX numeric suite


JIT Utils - torch.utils.jit


Distributed OptimizersWarning:
  Distributed optimizer is not currently supported when using CUDA
  tensors
"torch.distributed.optim" exposes DistributedOptimizer, which takes a
list of remote parameters ("RRef") and runs the optimizer locally on
the workers where the parameters live.  The distributed optimizer can
use any of the local optimizer Base class to apply the gradients on
each worker.
class torch.distributed.optim.DistributedOptimizer(optimizer_class, params_rref, args, *kwargs)
   DistributedOptimizer takes remote references to parameters
   scattered across workers and applies the given optimizer locally
   for each parameter.
   This class uses "get_gradients()" in order to retrieve the
   gradients for specific parameters.
   Concurrent calls to "step()", either from the same or different
   clients, will be serialized on each worker -- as each worker's
   optimizer can only work on one set of gradients at a time. However,


there is no guarantee that the full forward-backward-optimizer
   sequence will execute for one client at a time. This means that the
   gradients being applied may not correspond to the latest forward
   pass executed on a given worker. Also, there is no guaranteed
   ordering across workers.
   DistributedOptimizer creates the local optimizer with TorchScript
   enabled by default, so that optimizer updates are not blocked by
   the Python Global Interpreter Lock (GIL) in the case of
   multithreaded training (e.g. Distributed Model Parallel). This
   feature is currently enabled for most optimizers. You can also
   follow the recipe in PyTorch tutorials to enable TorchScript
   support for your own custom optimizers.
   Parameters:
      * optimizer_class (optim.Optimizer) -- the class of
        optimizer to instantiate on each worker.
      * params_rref (list[RRef]) -- list of RRefs to local
        or remote parameters to optimize.


or remote parameters to optimize.
      * args -- arguments to pass to the optimizer constructor on
        each worker.
      * kwargs -- arguments to pass to the optimizer constructor
        on each worker.
   Example::
      >>> import torch.distributed.autograd as dist_autograd
      >>> import torch.distributed.rpc as rpc
      >>> from torch import optim
      >>> from torch.distributed.optim import DistributedOptimizer
      >>>
      >>> with dist_autograd.context() as context_id:
      >>>   # Forward pass.
      >>>   rref1 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 3))
      >>>   rref2 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 1))
      >>>   loss = rref1.to_here() + rref2.to_here()
      >>>
      >>>   # Backward pass.
      >>>   dist_autograd.backward(context_id, [loss.sum()])
      >>>
      >>>   # Optimizer.
      >>>   dist_optim = DistributedOptimizer(
      >>>      optim.SGD,
      >>>      [rref1, rref2],
      >>>      lr=0.05,
      >>>   )


 lr=0.05,
  >>>   )
  >>>   dist_optim.step(context_id)

step(context_id)
      Performs a single optimization step.
      This will call "torch.optim.Optimizer.step()" on each worker
      containing parameters to be optimized, and will block until all
      workers return. The provided "context_id" will be used to
      retrieve the corresponding "context" that contains the gradients
      that should be applied to the parameters.
      Parameters:
         context_id -- the autograd context id for which we should
         run the optimizer step.
class torch.distributed.optim.PostLocalSGDOptimizer(optim, averager)
   Wraps an arbitrary "torch.optim.Optimizer" and runs post-local SGD,
   This optimizer runs local optimizer at every step. After the warm-
   up stage, it averages parameters periodically afer the local
   optimizer is applied.
   Parameters:
      * optim (Optimizer) -- The local optimizer.
      * averager (ModelAverager) -- A model averager instance to


run post-localSGD algorithm.
   Example:
      >>> import torch
      >>> import torch.distributed as dist
      >>> import torch.distributed.algorithms.model_averaging.averagers as averagers
      >>> import torch.nn as nn
      >>> from torch.distributed.optim import PostLocalSGDOptimizer
      >>> from torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook import (
      >>>   PostLocalSGDState,
      >>>   post_localSGD_hook,
      >>> )
      >>>
      >>> model = nn.parallel.DistributedDataParallel(
      >>>    module, device_ids=[rank], output_device=rank
      >>> )
      >>>
      >>> # Register a post-localSGD communication hook.
      >>> state = PostLocalSGDState(process_group=None, subgroup=None, start_localSGD_iter=100)
      >>> model.register_comm_hook(state, post_localSGD_hook)
      >>>
      >>> # Create a post-localSGD optimizer that wraps a local optimizer.
      >>> # Note that warmup_steps used in PostLocalSGDOptimizer must be the same as


start_localSGD_iter used in PostLocalSGDState.
  >>> local_optim = torch.optim.SGD(params=model.parameters(), lr=0.01)
  >>> opt = PostLocalSGDOptimizer(
  >>>     optim=local_optim,
  >>>     averager=averagers.PeriodicModelAverager(period=4, warmup_steps=100)
  >>> )
  >>>
  >>> # In the first 100 steps, DDP runs global gradient averaging at every step.
  >>> # After 100 steps, DDP runs gradient averaging within each subgroup (intra-node by default),
  >>> # and post-localSGD optimizer runs global model averaging every 4 steps after applying the local optimizer.
  >>> for step in range(0, 200):
  >>>    opt.zero_grad()
  >>>    loss = loss_fn(output, labels)
  >>>    loss.backward()
  >>>    opt.step()

load_state_dict(state_dict)
      This is the same as "torch.optim.Optimizer" "load_state_dict()",
      but also restores model averager's step value to the one saved
      in the provided "state_dict".


in the provided "state_dict".
      If there is no ""step"" entry in "state_dict", it will raise a
      warning and initialize the model averager's step to 0.
   state_dict()
      This is the same as "torch.optim.Optimizer" "state_dict()", but
      adds an extra entry to record model averager's step to the
      checkpoint to ensure reload does not cause unnecessary warm up
      again.
   step()
      Performs a single optimization step (parameter update).
class torch.distributed.optim.ZeroRedundancyOptimizer(params, optimizer_class, process_group=None, parameters_as_bucket_view=False, overlap_with_ddp=False, **defaults)
   This class wraps an arbitrary "optim.Optimizer" and shards its
   states across ranks in the group as described by ZeRO. The local
   optimizer instance in each rank is only responsible for updating
   approximately "1 / world_size" parameters and hence only needs to
   keep "1 / world_size" optimizer states. After parameters are


updated locally, each rank will broadcast its parameters to all
   other peers to keep all model replicas in the same state.
   "ZeroRedundancyOptimizer" can be used in conjunction with
   "torch.nn.parallel.DistributedDataParallel" to reduce per-rank peak
   memory consumption.
   "ZeroRedundancyOptimizer" uses a sorted-greedy algorithm to pack a
   number of parameters at each rank. Each parameter belongs to a
   single rank and is not divided among ranks. The partition is
   arbitrary and might not match the the parameter registration or
   usage order.
   Parameters:
      params ("Iterable") -- an "Iterable" of "torch.Tensor" s or
      "dict" s giving all parameters, which will be sharded across
      ranks.
   Keyword Arguments:
      * optimizer_class ("torch.nn.Optimizer") -- the class of the
        local optimizer.
      * process_group ("ProcessGroup", optional) --
        "torch.distributed" "ProcessGroup" (default:
        "dist.group.WORLD" initialized by


"dist.group.WORLD" initialized by
        "torch.distributed.init_process_group()").
      * parameters_as_bucket_view (bool, optional) -- if
        "True", parameters are packed into buckets to speed up
        communication, and "param.data" fields point to bucket views
        at different offsets; if "False", each individual parameter is
        communicated separately, and each "params.data" stays intact
        (default: "False").
      * overlap_with_ddp (bool, optional) -- if "True",
        "step()" is overlapped with "DistributedDataParallel" 's
        gradient synchronization; this requires (1) either a
        functional optimizer for the "optimizer_class" argument or one
        with a functional equivalent and (2) registering a DDP
        communication hook constructed from one of the functions in
        "ddp_zero_hook.py"; parameters are packed into buckets
        matching those in "DistributedDataParallel", meaning that the


"parameters_as_bucket_view" argument is ignored. If "False",
        "step()" runs disjointly after the backward pass (per normal).
        (default: "False")
      * *defaults -- any trailing arguments, which are forwarded
        to the local optimizer.
   Example:
      >>> import torch.nn as nn
      >>> from torch.distributed.optim import ZeroRedundancyOptimizer
      >>> from torch.nn.parallel import DistributedDataParallel as DDP
      >>> model = nn.Sequential([nn.Linear(2000, 2000).to(rank) for _ in range(20)])
      >>> ddp = DDP(model, device_ids=[rank])
      >>> opt = ZeroRedundancyOptimizer(
      >>>     ddp.parameters(),
      >>>     optimizer_class=torch.optim.Adam,
      >>>     lr=0.01
      >>> )
      >>> ddp(inputs).sum().backward()
      >>> opt.step()
   Warning:
     Currently, "ZeroRedundancyOptimizer" requires that all of the
     passed-in parameters are the same dense type.
   Warning:
     If you pass "overlap_with_ddp=True", be wary of the following:


Given the way that overlapping "DistributedDataParallel" with
     "ZeroRedundancyOptimizer" is currently implemented, the first two
     or three training iterations do not perform parameter updates in
     the optimizer step, depending on if "static_graph=False" or
     "static_graph=True", respectively. This is because it needs
     information about the gradient bucketing strategy used by
     "DistributedDataParallel", which is not finalized until the
     second forward pass if "static_graph=False" or until the third
     forward pass if "static_graph=True". To adjust for this, one
     option is to prepend dummy inputs.
   Warning:
     ZeroRedundancyOptimizer is experimental and subject to change.
   add_param_group(param_group)
      Add a parameter group to the "Optimizer" 's "param_groups".
      This can be useful when fine tuning a pre-trained network, as
      frozen layers can be made trainable and added to the "Optimizer"
      as training progresses.
      Parameters:


as training progresses.
      Parameters:
         param_group (dict) -- specifies the parameters to be
         optimized and group-specific optimization options.
      Warning:
        This method handles updating the shards on all partitions but
        needs to be called on all ranks. Calling this on a subset of
        the ranks will cause the training to hang because
        communication primitives are called depending on the managed
        parameters and expect all the ranks to participate on the same
        set of parameters.
   consolidate_state_dict(to=0)
      Consolidate a list of "state_dict" s (one per rank) on the
      target rank.
      Parameters:
         to (int) -- the rank that receives the optimizer states
         (default: 0).
      Raises:
         RuntimeError -- if "overlap_with_ddp=True" and this
         method is     called before this "ZeroRedundancyOptimizer"
         instance     has been fully initialized, which happens once


"DistributedDataParallel" gradient buckets have been
         rebuilt.
      Warning:
        This needs to be called on all ranks.
   join_hook(kwargs)
      Returns the ZeRO join hook, which enables training on uneven
      inputs by shadowing the collective communications in the
      optimizer step.
      Gradients must be properly set before this hook is called.
      Parameters:
         kwargs (dict) -- a "dict" containing any keyword
         arguments to modify the behavior of the join hook at run
         time; all "Joinable" instances sharing the same join context
         manager are forwarded the same value for "kwargs".
      This hook does not support any keyword arguments; i.e. "kwargs"
      is unused.
   load_state_dict(state_dict)
      Load the state pertaining to the given rank from the input
      "state_dict", updating the local optimizer as needed.
      Parameters:
         state_dict (dict) -- optimizer state; should be an


object returned from a call to "state_dict()".
      Raises:
         RuntimeError -- if "overlap_with_ddp=True" and this
         method is     called before this "ZeroRedundancyOptimizer"
         instance     has been fully initialized, which happens once
         "DistributedDataParallel" gradient buckets have been
         rebuilt.
   state_dict()
      Returns the last global optimizer state known to this rank.
      Raises:
         RuntimeError -- if "overlap_with_ddp=True" and this
         method is     called before this "ZeroRedundancyOptimizer"
         instance     has been fully initialized, which happens once
         "DistributedDataParallel" gradient buckets have been
         rebuilt; or if this method is called without a preceding call
         to "consolidate_state_dict()".
      Return type:
         Dict[str, Any]
   step(closure=None, **kwargs)
      Performs a single optimizer step and syncs parameters across all
      ranks.
      Parameters:


ranks.
      Parameters:
         closure (Callable) -- a closure that re-evaluates the
         model and returns the loss; optional for most optimizers.
      Returns:
         Optional loss depending on the underlying local optimizer.
      Return type:
         Optional[float]


Distributed Autograd DesignThis note will present the detailed design for distributed autograd
and walk through the internals of the same. Make sure you're familiar
with Autograd mechanics and the Distributed RPC Framework before
proceeding.
Background
==========
Let's say you have two nodes and a very simple model partitioned
across two nodes. This can be implemented using
"torch.distributed.rpc" as follows:
   import torch
   import torch.distributed.rpc as rpc
   def my_add(t1, t2):
     return torch.add(t1, t2)
   # On worker 0:
   t1 = torch.rand((3, 3), requires_grad=True)
   t2 = torch.rand((3, 3), requires_grad=True)
   # Perform some computation remotely.
   t3 = rpc.rpc_sync("worker1", my_add, args=(t1, t2))
   # Perform some computation locally based on remote result.
   t4 = torch.rand((3, 3), requires_grad=True)
   t5 = torch.mul(t3, t4)
   # Compute some loss.
   loss = t5.sum()
The main motivation behind distributed autograd is to enable running a


backward pass on such distributed models with the "loss" that we've
computed and record appropriate gradients for all tensors that require
gradients.
Autograd recording during the forward pass
==========================================
PyTorch builds the autograd graph during the forward pass and this
graph is used to execute the backward pass. For more details see How
autograd encodes the history.
For distributed autograd, we need to keep track of all RPCs during the
forward pass to ensure the backward pass is executed appropriately.
For this purpose, we attach "send" and "recv" functions to the
autograd graph when we perform an RPC.
* The "send" function is attached to the source of the RPC and its
  output edges point to the autograd function for the input tensors of
  the RPC. The input for this function during the backward pass is
  received from the destination as the output of the appropriate
  "recv" function.
* The "recv" function is attached to the destination of the RPC and


its inputs are retrieved from operators executed on the destination
  using the input tensors. The output gradients of this function are
  sent to the source node to the appropriate "send" function during
  the backward pass.
* Each "send-recv" pair is assigned a globally unique
  "autograd_message_id" to uniquely identify the pair. This is useful
  to look up the corresponding function on a remote node during the
  backward pass.
* For RRef, whenever we call "torch.distributed.rpc.RRef.to_here()" we
  attach an appropriate "send-recv" pair for the tensors involved.
As an example, this is what the autograd graph for our example above
would look like (t5.sum() excluded for simplicity):
[image]
Distributed Autograd Context
============================
Each forward and backward pass that uses distributed autograd is
assigned a unique "torch.distributed.autograd.context" and this
context has a globally unique "autograd_context_id". This context is
created on each node as needed.


created on each node as needed.
This context serves the following purpose:
1. Multiple nodes running distributed backward passes might accumulate
   gradients on the same tensor and as a result the ".grad" field of
   the tensor would have gradients from a variety of distributed
   backward passes before we have the opportunity to run the
   optimizer. This is similar to calling "torch.autograd.backward()"
   multiple times locally. In order to provide a way of separating out
   the gradients for each backward pass, the gradients are accumulated
   in the "torch.distributed.autograd.context" for each backward pass.
2. During the forward pass we store the "send" and "recv" functions
   for each autograd pass in this context. This ensures we hold
   references to the appropriate nodes in the autograd graph to keep
   it alive. In addition to this, it is easy to look up the
   appropriate "send" and "recv" functions during the backward pass.
3. In general we also use this context to store some metadata for each


distributed autograd pass.
From the user's perspective the autograd context is setup as follows:
   import torch.distributed.autograd as dist_autograd
   with dist_autograd.context() as context_id:
     loss = model.forward()
     dist_autograd.backward(context_id, loss)
It is important to note that your model's forward pass must be invoked
within the distributed autograd context manager, as a valid context is
needed in order to ensure that all "send" and "recv" functions are
stored properly to run the backward pass across all participating
nodes.
Distributed Backward Pass
=========================
In this section we outline the challenge of computing dependencies
accurately during a distributed backward pass and describe a couple of
algorithms (with tradeoffs) on how we can execute a distributed
backward pass.
Computing dependencies

Consider the following piece of code being run on a single machine
   import torch
   a = torch.rand((3, 3), requires_grad=True)


a = torch.rand((3, 3), requires_grad=True)
   b = torch.rand((3, 3), requires_grad=True)
   c = torch.rand((3, 3), requires_grad=True)
   d = a + b
   e = b * c
   d.sum.().backward()
This is what the autograd graph for the code above would look like:
[image]
The first step the autograd engine performs as part of the backward
pass is computing the number of dependencies for each node in the
autograd graph. This helps the autograd engine know when a node in the
graph is ready for execution. The numbers in brackets for "add(1)" and
"mul(0)" denote the number of dependencies. As you can see, this means
during the backward pass the "add" node needs 1 input and the "mul"
node doesn't need any inputs (in other words doesn't need to be
executed). The local autograd engine computes these dependencies by
traversing the graph from the root nodes ("d" in this case).
The fact that certain nodes in the autograd graph might not be
executed in the backward pass poses a challenge for distributed


autograd. Consider this piece of code which uses RPC.
   import torch
   import torch.distributed.rpc as rpc
   a = torch.rand((3, 3), requires_grad=True)
   b = torch.rand((3, 3), requires_grad=True)
   c = torch.rand((3, 3), requires_grad=True)
   d = rpc.rpc_sync("worker1", torch.add, args=(a, b))
   e = rpc.rpc_sync("worker1", torch.mul, args=(b, c))
   loss = d.sum()
The associated autograd graph for the code above would be:
[image]
Computing dependencies of this distributed autograd graph is much more
challenging and requires some overhead (either in terms of computation
or network communication).
For performance sensitive applications we can avoid a lot of overhead
by assuming every "send" and "recv" function are valid as part of the
backward pass (most applications don't perform RPCs that aren't used).
This simplifies the distributed autograd algorithm and is much more
efficient, but at the cost that the application needs to be aware of


the limitations. This algorithm is called the FAST mode algorithm and
is described in detail below.
In the general case it might not be necessary that every "send" and
"recv" function is valid as part of the backward pass. To address
this, we have proposed a SMART mode algorithm which is described in a
later section. Please note that currently, only the FAST mode
algorithm is implemented.
FAST mode algorithm

The key assumption of this algorithm is that each "send" function has
a dependency of 1 when we run a backward pass. In other words, we
assume we'll receive a gradient over RPC from another node.
The algorithm is as follows:
1. We start from the worker which has the roots for the backward pass
   (all roots must be local).
2. Lookup all the "send" functions for the current Distributed
   Autograd Context.
3. Compute dependencies locally starting from the provided roots and
   all the "send" functions we retrieved.
4. After computing dependencies, kick off the local autograd engine


with the provided roots.
5. When the autograd engine executes the "recv" function, the "recv"
   function sends the input gradients via RPC to the appropriate
   worker. Each "recv" function knows the destination worker id since
   it is recorded as part of the forward pass. The "recv" function
   also sends over the "autograd_context_id" and "autograd_message_id"
   to the remote host.
6. When this request is received on the remote host, we use the
   "autograd_context_id" and "autograd_message_id" to look up the
   appropriate "send" function.
7. If this is the first time a worker has received a request for the
   given "autograd_context_id", it will compute dependencies locally
   as described in points 1-3 above.
8. The "send" function retrieved in 6. is then enqueued for execution
   on the local autograd engine for that worker.
9. Finally, instead of accumulating the gradients on the ".grad" field
   of the Tensor, we accumulate the gradients separately per


Distributed Autograd Context. The gradients are stored in a
   "Dict[Tensor, Tensor]", which is basically a map from Tensor to its
   associated gradient and this map can be retrieved using the
   "get_gradients()" API.
As an example the complete code with distributed autograd would be as
follows:
   import torch
   import torch.distributed.autograd as dist_autograd
   import torch.distributed.rpc as rpc
   def my_add(t1, t2):
     return torch.add(t1, t2)
   # On worker 0:
   # Setup the autograd context. Computations that take
   # part in the distributed backward pass must be within
   # the distributed autograd context manager.
   with dist_autograd.context() as context_id:
     t1 = torch.rand((3, 3), requires_grad=True)
     t2 = torch.rand((3, 3), requires_grad=True)
     # Perform some computation remotely.
     t3 = rpc.rpc_sync("worker1", my_add, args=(t1, t2))
     # Perform some computation locally based on remote result.
     t4 = torch.rand((3, 3), requires_grad=True)


t4 = torch.rand((3, 3), requires_grad=True)
     t5 = torch.mul(t3, t4)
     # Compute some loss.
     loss = t5.sum()
     # Run the backward pass.
     dist_autograd.backward(context_id, [loss])
     # Retrieve the gradients from the context.
     dist_autograd.get_gradients(context_id)
The distributed autograd graph with dependencies would be as follows
(t5.sum() excluded for simplicity):
[image]
The FAST mode algorithm applied to the above example would be as
follows:
1. On "Worker 0" we start from the roots "loss" and "send1" to compute
   dependencies. As a result "send1" is marked with a dependency of 1
   and "mul" on "Worker 0" is marked with a dependency of 1.
2. Now, we kickoff the local autograd engine on "Worker 0". We first
   execute the "mul" function, accumulate its output in the autograd
   context as the gradient for "t4". Then, we execute "recv2" which
   sends the gradients to "Worker 1".
3. Since this is the first time "Worker 1" has heard about this


backward pass, it starts dependency computation and marks the
   dependencies for "send2", "add" and "recv1" appropriately.
4. Next, we enqueue "send2" on the local autograd engine of "Worker
   1", which in turn executes "add" and "recv1".
5. When "recv1" is executed it sends the gradients over to "Worker 0".
6. Since "Worker 0" has already computed dependencies for this
   backward pass, it just enqueues and executes "send1" locally.
7. Finally, gradients for "t1", "t2" and "t4" are accumulated in the
   Distributed Autograd Context.
SMART mode algorithm

Full details of this algorithm are still in the works, but for the
general idea you can refer to Distributed Autograd Algorithm Smart
mode section in the RFC.
Distributed Optimizer
=====================
The "DistributedOptimizer" operates as follows:
1. Takes a list of remote parameters ("RRef") to optimize. These could
   also be local parameters wrapped within a local "RRef".


Takes a "Optimizer" class as the local optimizer to run on all
   distinct "RRef" owners.
The distributed optimizer creates an instance of the local
   "Optimizer" on each of the worker nodes and holds an "RRef" to
   them.
When "torch.distributed.optim.DistributedOptimizer.step()" is
   invoked, the distributed optimizer uses RPC to remotely execute all
   the local optimizers on the appropriate remote workers. A
   distributed autograd "context_id" must be provided as input to
   "torch.distributed.optim.DistributedOptimizer.step()". This is used
   by local optimizers to apply gradients stored in the corresponding
   context.
If multiple concurrent distributed optimizers are updating the same
   parameters on a worker, these updates are serialized via a lock.
Simple end to end example
=========================
Putting it all together, the following is a simple end to end example
using distributed autograd and the distributed optimizer. If the code


is placed into a file called "dist_autograd_simple.py", it can be run
with the command "MASTER_ADDR="localhost" MASTER_PORT=29500 python
dist_autograd_simple.py":
   import torch
   import torch.multiprocessing as mp
   import torch.distributed.autograd as dist_autograd
   from torch.distributed import rpc
   from torch import optim
   from torch.distributed.optim import DistributedOptimizer
   def random_tensor():
       return torch.rand((3, 3), requires_grad=True)
   def _run_process(rank, dst_rank, world_size):
       name = "worker{}".format(rank)
       dst_name = "worker{}".format(dst_rank)
       # Initialize RPC.
       rpc.init_rpc(
           name=name,
           rank=rank,
           world_size=world_size
       )
       # Use a distributed autograd context.
       with dist_autograd.context() as context_id:
           # Forward pass (create references on remote nodes).
           rref1 = rpc.remote(dst_name, random_tensor)
           rref2 = rpc.remote(dst_name, random_tensor)


loss = rref1.to_here() + rref2.to_here()
           # Backward pass (run distributed autograd).
           dist_autograd.backward(context_id, [loss.sum()])
           # Build DistributedOptimizer.
           dist_optim = DistributedOptimizer(
           optim.SGD,
           [rref1, rref2],
           lr=0.05,
           )
           # Run the distributed optimizer step.
           dist_optim.step(context_id)
   def run_process(rank, world_size):
       dst_rank = (rank + 1) % world_size
       _run_process(rank, dst_rank, world_size)
       rpc.shutdown()
   if name == 'main':
     # Run world_size workers
     world_size = 2
     mp.spawn(run_process, args=(world_size,), nprocs=world_size)


torch.utils.mobile_optimizerWarning:
  This API is in beta and may change in the near future.
Torch mobile supports
"torch.utils.mobile_optimizer.optimize_for_mobile" utility to run a
list of optimization pass with modules in eval mode. The method takes
the following parameters: a torch.jit.ScriptModule object, a
blocklisting optimization set, a preserved method list, and a backend.
For CPU Backend, by default, if optimization blocklist is None or
empty, "optimize_for_mobile" will run the following optimizations:
   * Conv2D + BatchNorm fusion (blocklisting option
     mobile_optimizer.MobileOptimizerType.CONV_BN_FUSION):  This
     optimization pass folds "Conv2d-BatchNorm2d" into "Conv2d" in
     "forward" method of this module and all its submodules. The
     weight and bias of the "Conv2d" are correspondingly updated.
   * Insert and Fold prepacked ops (blocklisting option
     mobile_optimizer.MobileOptimizerType.INSERT_FOLD_PREPACK_OPS):


This optimization pass rewrites the graph to replace 2D
     convolutions and linear ops with their prepacked counterparts.
     Prepacked ops are stateful ops in that, they require some state
     to be created, such as weight prepacking and use this state, i.e.
     prepacked weights, during op execution. XNNPACK is one such
     backend that provides prepacked ops, with kernels optimized for
     mobile platforms (such as ARM CPUs). Prepacking of weight enables
     efficient memory access and thus faster kernel execution. At the
     moment "optimize_for_mobile" pass rewrites the graph to replace
     "Conv2D/Linear" with 1) op that pre-packs weight for XNNPACK
     conv2d/linear ops and 2) op that takes pre-packed weight and
     activation as input and generates output activations. Since 1
     needs to be done only once, we fold the weight pre-packing such
     that it is done only once at model load time. This pass of the
     "optimize_for_mobile" does 1 and 2 and then folds, i.e. removes,


weight pre-packing ops.
   * ReLU/Hardtanh fusion: XNNPACK ops support fusion of clamping.
     That is clamping of output activation is done as part of the
     kernel, including for 2D convolution and linear op kernels. Thus
     clamping effectively comes for free. Thus any op that can be
     expressed as clamping op, such as "ReLU" or "hardtanh", can be
     fused with previous "Conv2D" or "linear" op in XNNPACK. This pass
     rewrites graph by finding "ReLU/hardtanh" ops that follow XNNPACK
     "Conv2D/linear" ops, written by the previous pass, and fuses them
     together.
   * Dropout removal (blocklisting option
     mobile_optimizer.MobileOptimizerType.REMOVE_DROPOUT): This
     optimization pass removes "dropout" and "dropout_" nodes from
     this module when training is false.
   * Conv packed params hoisting (blocklisting option
     mobile_optimizer.MobileOptimizerType.HOIST_CONV_PACKED_PARAMS):
     This optimization pass moves convolution packed params to the


root module, so that the convolution structs can be deleted. This
     decreases model size without impacting numerics.
   * Add/ReLU fusion (blocklisting option
     mobile_optimizer.MobileOptimizerType.FUSE_ADD_RELU): This pass
     finds instances of "relu" ops that follow "add" ops and fuses
     them into a single "add_relu".
for Vulkan Backend, by default, if optimization blocklist is None or
empty, "optimize_for_mobile" will run the following optimization:
   * Automatic GPU Transfer (blocklisting option mobile_optimize
     r.MobileOptimizerType.VULKAN_AUTOMATIC_GPU_TRANSFER): This
     optimization pass rewrites the graph so that moving input and
     output data to and from the GPU becomes part of the model.
"optimize_for_mobile" will also invoke freeze_module pass which only
preserves "forward" method. If you have other method to that needed to
be preserved, add them into the preserved method list and pass into
the method.


the method.
torch.utils.mobile_optimizer.optimize_for_mobile(script_module, optimization_blocklist=None, preserved_methods=None, backend='CPU')
   Parameters:
      * script_module (ScriptModule) -- An instance of torch
        script module with type of ScriptModule.
      * optimization_blocklist
        (Optional[Set[_MobileOptimizerType]]) -- A set
        with type of MobileOptimizerType. When set is not passed,
        optimization method will run all the optimizer pass;
        otherwise, optimizer method will run the optimization pass
        that is not included inside optimization_blocklist.
      * preserved_methods (Optional[List]) -- A list of
        methods that needed to be preserved when freeze_module pass is
        invoked
      * backend (str) -- Device type to use for running the
        result model ('CPU'(default), 'Vulkan' or 'Metal').
   Returns:
      A new optimized torch script module
   Return type:
      RecursiveScriptModule


Quantization Backend ConfigurationFX Graph Mode Quantization allows the user to configure various
quantization behaviors of an op in order to match the expectation of
their backend.
In the future, this document will contain a detailed spec of these
configurations.
Default values for native configurations
========================================
Below is the output of the configuration for quantization of ops in
x86 and qnnpack (PyTorch's default quantized backends).
Results:
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_SHARE_OBSERVER_WITH_INPUT,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_SHARE_OBSERVER_WITH_INPUT,
   },
   {


},
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_SHARE_OBSERVER_WITH_INPUT,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'num_tensor_args_to_observation_type': {
       0: ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
       1: ObservationType.OUTPUT_SHARE_OBSERVER_WITH_INPUT,
       2: ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     },
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


},
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_SHARE_OBSERVER_WITH_INPUT,
   },
   {
     'pattern': (, ),
     'dtype_configs': [
       {


'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'fused_module': ,
     'fuser_method': ,
   },
   {


},
   {
     'pattern': (, ),
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,


'root_module': ,
     'reference_quantized_module_for_root': ,
     'fuser_method': ,
   },
   {
     'pattern': (, ),
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.float32, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,


'bias_dtype': torch.float32,
         'is_dynamic': True,
       },
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.float16, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.float32, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.float16, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
         'is_dynamic': True,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,


'fused_module': ,
     'fuser_method': ,
   },
   {
     'pattern': (, ),
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'fused_module': ,
     'fuser_method': ,
   },
   {
     'pattern': (, ),
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'root_module': ,
     'reference_quantized_module_for_root': ,
     'fuser_method': ,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
   },
   {
     'pattern': (, ),
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'fused_module': ,
     'fuser_method': ,
   },
   {
     'pattern': (, ),
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'root_module': ,
     'reference_quantized_module_for_root': ,
     'fuser_method': ,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {


'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
   },
   {


'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_SHARE_OBSERVER_WITH_INPUT,
   },
   {
     'pattern': clamp,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_SHARE_OBSERVER_WITH_INPUT,
   },
   {
     'pattern': contiguous,
     'dtype_configs': [
       {


'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
       },
     ],
     'observation_type': ObservationType.OUTPUT_SHARE_OBSERVER_WITH_INPUT,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'root_module': ,
     'qat_module': ,
     'reference_quantized_module_for_root': ,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {


'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'root_module': ,


'reference_quantized_module_for_root': ,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],


},
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'input_type_to_index': {'weight': 1, 'bias': 2},
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],


},
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'root_module': ,
     'qat_module': ,
     'reference_quantized_module_for_root': ,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'root_module': ,
     'reference_quantized_module_for_root': ,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'input_type_to_index': {'weight': 1, 'bias': 2},
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


},
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'qat_module': ,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'weight_dtype': DTypeWithConstraints(dtype=torch.qint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),


'bias_dtype': torch.float32,
       },
     ],
     'observation_type': ObservationType.OUTPUT_USE_DIFFERENT_OBSERVER_AS_INPUT,
     'root_module': ,
     'reference_quantized_module_for_root': ,
   },
   {
     'pattern': ,
     'dtype_configs': [
       {
         'input_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),
         'output_dtype': DTypeWithConstraints(dtype=torch.quint8, quant_min_lower_bound=None, quant_max_upper_bound=None, scale_min_lower_bound=None, scale_max_upper_bound=None, scale_exact_match=None, zero_point_exact_match=None),