class torch.nn.MSELoss(size_average=None, reduce=None, reduction='mean') [source]

Creates a criterion that measures the mean squared error (squared L2 norm) between each element in the input xx  and target yy . The unreduced (i.e. with reduction set to 'none') loss can be described as:  ℓ(x,y)=L={l1,…,lN}⊤,ln=(xn−yn)2,\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad l_n = \left( x_n - y_n \right)^2,
where NN  is the batch size. If reduction is not 'none' (default 'mean'), then:  ℓ(x,y)={mean(L),if reduction=‘mean’;sum(L),if reduction=‘sum’.\ell(x, y) = \begin{cases} \operatorname{mean}(L), & \text{if reduction} = \text{`mean';}\\ \operatorname{sum}(L), & \text{if reduction} = \text{`sum'.} \end{cases}
xx  and yy  are tensors of arbitrary shapes with a total of nn  elements each. The mean operation still operates over all the elements, and divides by nn . The division by nn  can be avoided if one sets reduction = 'sum'.  Parameters

size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there are multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'
     Shape:

 Input: (N,∗)(N, *)  where ∗*  means, any number of additional dimensions Target: (N,∗)(N, *) , same shape as the input    Examples: >>> loss = nn.MSELoss()
>>> input = torch.randn(3, 5, requires_grad=True)
>>> target = torch.randn(3, 5)
>>> output = loss(input, target)
>>> output.backward()


class torch.nn.MultiheadAttention(embed_dim, num_heads, dropout=0.0, bias=True, add_bias_kv=False, add_zero_attn=False, kdim=None, vdim=None) [source]

Allows the model to jointly attend to information from different representation subspaces. See Attention Is All You Need  MultiHead(Q,K,V)=Concat(head1,…,headh)WO\text{MultiHead}(Q, K, V) = \text{Concat}(head_1,\dots,head_h)W^O
where headi=Attention(QWiQ,KWiK,VWiV)head_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V) .  Parameters

embed_dim – total dimension of the model.
num_heads – parallel attention heads.
dropout – a Dropout layer on attn_output_weights. Default: 0.0.
bias – add bias as module parameter. Default: True.
add_bias_kv – add bias to the key and value sequences at dim=0.
add_zero_attn – add a new batch of zeros to the key and value sequences at dim=1.
kdim – total number of features in key. Default: None.
vdim – total number of features in value. Default: None.    Note that if kdim and vdim are None, they will be set to embed_dim such that query, key, and value have the same number of features. Examples: >>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
>>> attn_output, attn_output_weights = multihead_attn(query, key, value)

forward(query, key, value, key_padding_mask=None, need_weights=True, attn_mask=None) [source]

 Parameters

key, value (query,) – map a query and a set of key-value pairs to an output. See “Attention Is All You Need” for more details.
key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored
need_weights – output attn_output_weights.
attn_mask – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch.     Shapes for inputs:

 query: (L,N,E)(L, N, E)  where L is the target sequence length, N is the batch size, E is the embedding dimension. key: (S,N,E)(S, N, E) , where S is the source sequence length, N is the batch size, E is the embedding dimension. value: (S,N,E)(S, N, E)  where S is the source sequence length, N is the batch size, E is the embedding dimension. key_padding_mask: (N,S)(N, S)  where N is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged.
attn_mask: if a 2D mask: (L,S)(L, S)  where L is the target sequence length, S is the source sequence length. If a 3D mask: (N⋅num_heads,L,S)(N\cdot\text{num\_heads}, L, S)  where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight.    Shapes for outputs:

 attn_output: (L,N,E)(L, N, E)  where L is the target sequence length, N is the batch size, E is the embedding dimension. attn_output_weights: (N,L,S)(N, L, S)  where N is the batch size, L is the target sequence length, S is the source sequence length.


forward(query, key, value, key_padding_mask=None, need_weights=True, attn_mask=None) [source]

 Parameters

key, value (query,) – map a query and a set of key-value pairs to an output. See “Attention Is All You Need” for more details.
key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored
need_weights – output attn_output_weights.
attn_mask – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch.     Shapes for inputs:

 query: (L,N,E)(L, N, E)  where L is the target sequence length, N is the batch size, E is the embedding dimension. key: (S,N,E)(S, N, E) , where S is the source sequence length, N is the batch size, E is the embedding dimension. value: (S,N,E)(S, N, E)  where S is the source sequence length, N is the batch size, E is the embedding dimension. key_padding_mask: (N,S)(N, S)  where N is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged.
attn_mask: if a 2D mask: (L,S)(L, S)  where L is the target sequence length, S is the source sequence length. If a 3D mask: (N⋅num_heads,L,S)(N\cdot\text{num\_heads}, L, S)  where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight.    Shapes for outputs:

 attn_output: (L,N,E)(L, N, E)  where L is the target sequence length, N is the batch size, E is the embedding dimension. attn_output_weights: (N,L,S)(N, L, S)  where N is the batch size, L is the target sequence length, S is the source sequence length.


class torch.nn.MultiLabelMarginLoss(size_average=None, reduce=None, reduction='mean') [source]

Creates a criterion that optimizes a multi-class multi-classification hinge loss (margin-based loss) between input xx  (a 2D mini-batch Tensor) and output yy  (which is a 2D Tensor of target class indices). For each sample in the mini-batch:  loss(x,y)=∑ijmax(0,1−(x[y[j]]−x[i]))x.size(0)\text{loss}(x, y) = \sum_{ij}\frac{\max(0, 1 - (x[y[j]] - x[i]))}{\text{x.size}(0)}
where x∈{0,⋯,x.size(0)−1}x \in \left\{0, \; \cdots , \; \text{x.size}(0) - 1\right\} , y∈{0,⋯,y.size(0)−1}y \in \left\{0, \; \cdots , \; \text{y.size}(0) - 1\right\} , 0≤y[j]≤x.size(0)−10 \leq y[j] \leq \text{x.size}(0)-1 , and i≠y[j]i \neq y[j]  for all ii  and jj . yy  and xx  must have the same size. The criterion only considers a contiguous block of non-negative targets that starts at the front. This allows for different samples to have variable amounts of target classes.  Parameters

size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there are multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'
     Shape:

 Input: (C)(C)  or (N,C)(N, C)  where N is the batch size and C is the number of classes. Target: (C)(C)  or (N,C)(N, C) , label targets padded by -1 ensuring same shape as the input. Output: scalar. If reduction is 'none', then (N)(N) .    Examples: >>> loss = nn.MultiLabelMarginLoss()
>>> x = torch.FloatTensor([[0.1, 0.2, 0.4, 0.8]])
>>> # for target y, only consider labels 3 and 0, not after label -1
>>> y = torch.LongTensor([[3, 0, -1, 1]])
>>> loss(x, y)
>>> # 0.25 * ((1-(0.1-0.2)) + (1-(0.1-0.4)) + (1-(0.8-0.2)) + (1-(0.8-0.4)))
tensor(0.8500)


class torch.nn.MultiLabelSoftMarginLoss(weight=None, size_average=None, reduce=None, reduction='mean') [source]

Creates a criterion that optimizes a multi-label one-versus-all loss based on max-entropy, between input xx  and target yy  of size (N,C)(N, C) . For each sample in the minibatch:  loss(x,y)=−1C∗∑iy[i]∗log((1+exp(−x[i]))−1)+(1−y[i])∗log(exp(−x[i])(1+exp(−x[i])))loss(x, y) = - \frac{1}{C} * \sum_i y[i] * \log((1 + \exp(-x[i]))^{-1}) + (1-y[i]) * \log\left(\frac{\exp(-x[i])}{(1 + \exp(-x[i]))}\right)
where i∈{0,⋯,x.nElement()−1}i \in \left\{0, \; \cdots , \; \text{x.nElement}() - 1\right\} , y[i]∈{0,1}y[i] \in \left\{0, \; 1\right\} .  Parameters

weight (Tensor, optional) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.
size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there are multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'
     Shape:

 Input: (N,C)(N, C)  where N is the batch size and C is the number of classes. Target: (N,C)(N, C) , label targets padded by -1 ensuring same shape as the input. Output: scalar. If reduction is 'none', then (N)(N) .


class torch.nn.MultiMarginLoss(p=1, margin=1.0, weight=None, size_average=None, reduce=None, reduction='mean') [source]

Creates a criterion that optimizes a multi-class classification hinge loss (margin-based loss) between input xx  (a 2D mini-batch Tensor) and output yy  (which is a 1D tensor of target class indices, 0≤y≤x.size(1)−10 \leq y \leq \text{x.size}(1)-1 ): For each mini-batch sample, the loss in terms of the 1D input xx  and scalar output yy  is:  loss(x,y)=∑imax(0,margin−x[y]+x[i]))px.size(0)\text{loss}(x, y) = \frac{\sum_i \max(0, \text{margin} - x[y] + x[i]))^p}{\text{x.size}(0)}
where x∈{0,⋯,x.size(0)−1}x \in \left\{0, \; \cdots , \; \text{x.size}(0) - 1\right\}  and i≠yi \neq y . Optionally, you can give non-equal weighting on the classes by passing a 1D weight tensor into the constructor. The loss function then becomes:  loss(x,y)=∑imax(0,w[y]∗(margin−x[y]+x[i]))p)x.size(0)\text{loss}(x, y) = \frac{\sum_i \max(0, w[y] * (\text{margin} - x[y] + x[i]))^p)}{\text{x.size}(0)}
 Parameters

p (int, optional) – Has a default value of 11 . 11  and 22  are the only supported values.
margin (float, optional) – Has a default value of 11 .
weight (Tensor, optional) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.
size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there are multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'


class torch.nn.NLLLoss(weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean') [source]

The negative log likelihood loss. It is useful to train a classification problem with C classes. If provided, the optional argument weight should be a 1D Tensor assigning weight to each of the classes. This is particularly useful when you have an unbalanced training set. The input given through a forward call is expected to contain log-probabilities of each class. input has to be a Tensor of size either (minibatch,C)(minibatch, C)  or (minibatch,C,d1,d2,...,dK)(minibatch, C, d_1, d_2, ..., d_K)  with K≥1K \geq 1  for the K-dimensional case (described later). Obtaining log-probabilities in a neural network is easily achieved by adding a LogSoftmax layer in the last layer of your network. You may use CrossEntropyLoss instead, if you prefer not to add an extra layer. The target that this loss expects should be a class index in the range [0,C−1][0, C-1]  where C = number of classes; if ignore_index is specified, this loss also accepts this class index (this index may not necessarily be in the class range). The unreduced (i.e. with reduction set to 'none') loss can be described as:  ℓ(x,y)=L={l1,…,lN}⊤,ln=−wynxn,yn,wc=weight[c]⋅1{c≠ignore_index},\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad l_n = - w_{y_n} x_{n,y_n}, \quad w_{c} = \text{weight}[c] \cdot \mathbb{1}\{c \not= \text{ignore\_index}\},
where xx  is the input, yy  is the target, ww  is the weight, and NN  is the batch size. If reduction is not 'none' (default 'mean'), then  ℓ(x,y)={∑n=1N1∑n=1Nwynln,if reduction=‘mean’;∑n=1Nln,if reduction=‘sum’.\ell(x, y) = \begin{cases} \sum_{n=1}^N \frac{1}{\sum_{n=1}^N w_{y_n}} l_n, & \text{if reduction} = \text{`mean';}\\ \sum_{n=1}^N l_n, & \text{if reduction} = \text{`sum'.} \end{cases}
Can also be used for higher dimension inputs, such as 2D images, by providing an input of size (minibatch,C,d1,d2,...,dK)(minibatch, C, d_1, d_2, ..., d_K)  with K≥1K \geq 1 , where KK  is the number of dimensions, and a target of appropriate shape (see below). In the case of images, it computes NLL loss per-pixel.  Parameters

weight (Tensor, optional) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.
size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there are multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

ignore_index (int, optional) – Specifies a target value that is ignored and does not contribute to the input gradient. When size_average is True, the loss is averaged over non-ignored targets.
reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the weighted mean of the output is taken, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'
     Shape:

 Input: (N,C)(N, C)  where C = number of classes, or (N,C,d1,d2,...,dK)(N, C, d_1, d_2, ..., d_K)  with K≥1K \geq 1  in the case of K-dimensional loss. Target: (N)(N)  where each value is 0≤targets[i]≤C−10 \leq \text{targets}[i] \leq C-1 , or (N,d1,d2,...,dK)(N, d_1, d_2, ..., d_K)  with K≥1K \geq 1  in the case of K-dimensional loss. Output: scalar. If reduction is 'none', then the same size as the target: (N)(N) , or (N,d1,d2,...,dK)(N, d_1, d_2, ..., d_K)  with K≥1K \geq 1  in the case of K-dimensional loss.    Examples: >>> m = nn.LogSoftmax(dim=1)
>>> loss = nn.NLLLoss()
>>> # input is of size N x C = 3 x 5
>>> input = torch.randn(3, 5, requires_grad=True)
>>> # each element in target has to have 0 <= value < C
>>> target = torch.tensor([1, 0, 4])
>>> output = loss(m(input), target)
>>> output.backward()
>>>
>>>
>>> # 2D loss example (used, for example, with image inputs)
>>> N, C = 5, 4
>>> loss = nn.NLLLoss()
>>> # input is of size N x C x height x width
>>> data = torch.randn(N, 16, 10, 10)
>>> conv = nn.Conv2d(16, C, (3, 3))
>>> m = nn.LogSoftmax(dim=1)
>>> # each element in target has to have 0 <= value < C
>>> target = torch.empty(N, 8, 8, dtype=torch.long).random_(0, C)
>>> output = loss(m(conv(data)), target)
>>> output.backward()


class torch.nn.PairwiseDistance(p=2.0, eps=1e-06, keepdim=False) [source]

Computes the batchwise pairwise distance between vectors v1v_1 , v2v_2  using the p-norm:  ∥x∥p=(∑i=1n∣xi∣p)1/p.\Vert x \Vert _p = \left( \sum_{i=1}^n \vert x_i \vert ^ p \right) ^ {1/p}.
 Parameters

p (real) – the norm degree. Default: 2
eps (float, optional) – Small value to avoid division by zero. Default: 1e-6
keepdim (bool, optional) – Determines whether or not to keep the vector dimension. Default: False     Shape:

 Input1: (N,D)(N, D)  where D = vector dimension
 Input2: (N,D)(N, D) , same shape as the Input1 Output: (N)(N) . If keepdim is True, then (N,1)(N, 1) .   Examples::

>>> pdist = nn.PairwiseDistance(p=2)
>>> input1 = torch.randn(100, 128)
>>> input2 = torch.randn(100, 128)
>>> output = pdist(input1, input2)


torch.nn.parallel.data_parallel(module, inputs, device_ids=None, output_device=None, dim=0, module_kwargs=None) [source]

Evaluates module(input) in parallel across the GPUs given in device_ids. This is the functional version of the DataParallel module.  Parameters

module (Module) – the module to evaluate in parallel
inputs (Tensor) – inputs to the module
device_ids (list of python:int or torch.device) – GPU ids on which to replicate module
output_device (list of python:int or torch.device) – GPU location of the output Use -1 to indicate the CPU. (default: device_ids[0])   Returns
a Tensor containing the result of module(input) located on output_device


class torch.nn.parallel.DistributedDataParallel(module, device_ids=None, output_device=None, dim=0, broadcast_buffers=True, process_group=None, bucket_cap_mb=25, find_unused_parameters=False, check_reduction=False, gradient_as_bucket_view=False) [source]

Implements distributed data parallelism that is based on torch.distributed package at the module level. This container parallelizes the application of the given module by splitting the input across the specified devices by chunking in the batch dimension. The module is replicated on each machine and each device, and each such replica handles a portion of the input. During the backwards pass, gradients from each node are averaged. The batch size should be larger than the number of GPUs used locally. See also: Basics and Use nn.parallel.DistributedDataParallel instead of multiprocessing or nn.DataParallel. The same constraints on input as in torch.nn.DataParallel apply. Creation of this class requires that torch.distributed to be already initialized, by calling torch.distributed.init_process_group(). DistributedDataParallel is proven to be significantly faster than torch.nn.DataParallel for single-node multi-GPU data parallel training. To use DistributedDataParallel on a host with N GPUs, you should spawn up N processes, ensuring that each process exclusively works on a single GPU from 0 to N-1. This can be done by either setting CUDA_VISIBLE_DEVICES for every process or by calling: >>> torch.cuda.set_device(i)
 where i is from 0 to N-1. In each process, you should refer the following to construct this module: >>> torch.distributed.init_process_group(
>>>     backend='nccl', world_size=N, init_method='...'
>>> )
>>> model = DistributedDataParallel(model, device_ids=[i], output_device=i)
 In order to spawn up multiple processes per node, you can use either torch.distributed.launch or torch.multiprocessing.spawn.  Note Please refer to PyTorch Distributed Overview for a brief introduction to all features related to distributed training.   Note nccl backend is currently the fastest and highly recommended backend when using GPUs. This applies to both single-node and multi-node distributed training.   Note This module also supports mixed-precision distributed training. This means that your model can have different types of parameters such as mixed types of fp16 and fp32, the gradient reduction on these mixed types of parameters will just work fine.   Note If you use torch.save on one process to checkpoint the module, and torch.load on some other processes to recover it, make sure that map_location is configured properly for every process. Without map_location, torch.load would recover the module to devices where the module was saved from.   Note When a model is trained on M nodes with batch=N, the gradient will be M times smaller when compared to the same model trained on a single node with batch=M*N if the loss is summed (NOT averaged as usual) across instances in a batch (because the gradients between different nodes are averaged). You should take this into consideration when you want to obtain a mathematically equivalent training process compared to the local training counterpart. But in most cases, you can just treat a DistributedDataParallel wrapped model, a DataParallel wrapped model and an ordinary model on a single GPU as the same (E.g. using the same learning rate for equivalent batch size).   Note Parameters are never broadcast between processes. The module performs an all-reduce step on gradients and assumes that they will be modified by the optimizer in all processes in the same way. Buffers (e.g. BatchNorm stats) are broadcast from the module in process of rank 0, to all other replicas in the system in every iteration.   Note If you are using DistributedDataParallel in conjunction with the Distributed RPC Framework, you should always use torch.distributed.autograd.backward() to compute gradients and torch.distributed.optim.DistributedOptimizer for optimizing parameters. Example: >>> import torch.distributed.autograd as dist_autograd
>>> from torch.nn.parallel import DistributedDataParallel as DDP
>>> from torch import optim
>>> from torch.distributed.optim import DistributedOptimizer
>>> from torch.distributed.rpc import RRef
>>>
>>> t1 = torch.rand((3, 3), requires_grad=True)
>>> t2 = torch.rand((3, 3), requires_grad=True)
>>> rref = rpc.remote("worker1", torch.add, args=(t1, t2))
>>> ddp_model = DDP(my_model)
>>>
>>> # Setup optimizer
>>> optimizer_params = [rref]
>>> for param in ddp_model.parameters():
>>>     optimizer_params.append(RRef(param))
>>>
>>> dist_optim = DistributedOptimizer(
>>>     optim.SGD,
>>>     optimizer_params,
>>>     lr=0.05,
>>> )
>>>
>>> with dist_autograd.context() as context_id:
>>>     pred = ddp_model(rref.to_here())
>>>     loss = loss_func(pred, loss)
>>>     dist_autograd.backward(context_id, loss)
>>>     dist_optim.step()
   Warning Constructor, forward method, and differentiation of the output (or a function of the output of this module) are distributed synchronization points. Take that into account in case different processes might be executing different code.   Warning This module assumes all parameters are registered in the model by the time it is created. No parameters should be added nor removed later. Same applies to buffers.   Warning This module assumes all parameters are registered in the model of each distributed processes are in the same order. The module itself will conduct gradient allreduce following the reverse order of the registered parameters of the model. In other words, it is users’ responsibility to ensure that each distributed process has the exact same model and thus the exact same parameter registration order.   Warning This module allows parameters with non-rowmajor-contiguous strides. For example, your model may contain some parameters whose torch.memory_format is torch.contiguous_format and others whose format is torch.channels_last. However, corresponding parameters in different processes must have the same strides.   Warning This module doesn’t work with torch.autograd.grad() (i.e. it will only work if gradients are to be accumulated in .grad attributes of parameters).   Warning If you plan on using this module with a nccl backend or a gloo backend (that uses Infiniband), together with a DataLoader that uses multiple workers, please change the multiprocessing start method to forkserver (Python 3 only) or spawn. Unfortunately Gloo (that uses Infiniband) and NCCL2 are not fork safe, and you will likely experience deadlocks if you don’t change this setting.   Warning Forward and backward hooks defined on module and its submodules won’t be invoked anymore, unless the hooks are initialized in the forward() method.   Warning You should never try to change your model’s parameters after wrapping up your model with DistributedDataParallel. Because, when wrapping up your model with DistributedDataParallel, the constructor of DistributedDataParallel will register the additional gradient reduction functions on all the parameters of the model itself at the time of construction. If you change the model’s parameters afterwards, gradient redunction functions no longer match the correct set of parameters.   Warning Using DistributedDataParallel in conjunction with the Distributed RPC Framework is experimental and subject to change.   Warning The gradient_as_bucket_view mode does not yet work with Automatic Mixed Precision (AMP). AMP maintains stashed gradients that are used for unscaling gradients. With gradient_as_bucket_view=True, these stashed gradients will point to communication buckets in the first iteration. In the next iteration, the communication buckets are mutated and thus these stashed gradients will be unexpectedly mutated as well, which might lead to wrong results.   Parameters

module (Module) – module to be parallelized
device_ids (list of python:int or torch.device) – CUDA devices. This should only be provided when the input module resides on a single CUDA device. For single-device modules, the i’th module replica is placed on device_ids[i]. For multi-device modules and CPU modules, device_ids must be None or an empty list, and input data for the forward pass must be placed on the correct device. (default: all visible devices for single-device modules)
output_device (int or torch.device) – Device location of output for single-device CUDA modules. For multi-device modules and CPU modules, it must be None, and the module itself dictates the output location. (default: device_ids[0] for single-device modules)
broadcast_buffers (bool) – Flag that enables syncing (broadcasting) buffers of the module at beginning of the forward function. (default: True)
process_group – The process group to be used for distributed data all-reduction. If None, the default process group, which is created by torch.distributed.init_process_group(), will be used. (default: None)
bucket_cap_mb – DistributedDataParallel will bucket parameters into multiple buckets so that gradient reduction of each bucket can potentially overlap with backward computation. bucket_cap_mb controls the bucket size in MegaBytes (MB). (default: 25)
find_unused_parameters (bool) – Traverse the autograd graph from all tensors contained in the return value of the wrapped module’s forward function. Parameters that don’t receive gradients as part of this graph are preemptively marked as being ready to be reduced. Note that all forward outputs that are derived from module parameters must participate in calculating loss and later the gradient computation. If they don’t, this wrapper will hang waiting for autograd to produce gradients for those parameters. Any outputs derived from module parameters that are otherwise unused can be detached from the autograd graph using torch.Tensor.detach. (default: False)
check_reduction – This argument is deprecated.
gradient_as_bucket_view (bool) – This is a prototype feature and subject to changes. When set to True, gradients will be views pointing to different offsets of allreduce communication buckets. This can reduce peak memory usage, where the saved memory size will be equal to the total gradients size. Moreover, it avoids the overhead of copying between gradients and allreduce communication buckets. When gradients are views, detach_() cannot be called on the gradients. If hitting such errors, please fix it by referring to the zero_grad() function in torch/optim/optimizer.py as a solution.   Variables
~DistributedDataParallel.module (Module) – the module to be parallelized.   Example: >>> torch.distributed.init_process_group(backend='nccl', world_size=4, init_method='...')
>>> net = torch.nn.parallel.DistributedDataParallel(model, pg)

join(divide_by_initial_world_size=True, enable=True) [source]

A context manager to be used in conjunction with an instance of torch.nn.parallel.DistributedDataParallel to be able to train with uneven inputs across participating processes. This context manager will keep track of already-joined DDP processes, and “shadow” the forward and backward passes by inserting collective communication operations to match with the ones created by non-joined DDP processes. This will ensure each collective call has a corresponding call by already-joined DDP processes, preventing hangs or errors that would otherwise happen when training with uneven inputs across processes. Once all DDP processes have joined, the context manager will broadcast the model corresponding to the last joined process to all processes to ensure the model is the same across all processes (which is guaranteed by DDP). To use this to enable training with uneven inputs across processes, simply wrap this context manager around your training loop. No further modifications to the model or data loading is required.  Warning This module works only with the multi-process, single-device usage of torch.nn.parallel.DistributedDataParallel, which means that a single process works on a single GPU.   Warning This module currently does not support custom distributed collective operations in the forward pass, such as SyncBatchNorm or other custom defined collectives in the model’s forward pass.   Parameters

divide_by_initial_world_size (bool) – If True, will divide gradients by the initial world_size DDP training was launched with. If False, will compute the effective world size (number of ranks that have not depleted their inputs yet) and divide gradients by that during allreduce. Set divide_by_initial_world_size=True to ensure every input sample including the uneven inputs have equal weight in terms of how much they contribute to the global gradient. This is achieved by always dividing the gradient by the initial world_size even when we encounter uneven inputs. If you set this to False, we divide the gradient by the remaining number of nodes. This ensures parity with training on a smaller world_size although it also means the uneven inputs would contribute more towards the global gradient. Typically, you would want to set this to True for cases where the last few inputs of your training job are uneven. In extreme cases, where there is a large discrepancy in the number of inputs, setting this to False might provide better results.
enable (bool) – Whether to enable uneven input detection or not. Pass in enable=False to disable in cases where you know that inputs are even across participating processes. Default is True.    Example: >>>  import torch
>>>  import torch.distributed as dist
>>>  import os
>>>  import torch.multiprocessing as mp
>>>  import torch.nn as nn
>>>  # On each spawned worker
>>>  def worker(rank):
>>>      dist.init_process_group("nccl", rank=rank, world_size=2)
>>>      torch.cuda.set_device(rank)
>>>      model = nn.Linear(1, 1, bias=False).to(rank)
>>>      model = torch.nn.parallel.DistributedDataParallel(
>>>          model, device_ids=[rank], output_device=rank
>>>      )
>>>      # Rank 1 gets one more input than rank 0.
>>>      inputs = [torch.tensor([1]).float() for _ in range(10 + rank)]
>>>      with model.join():
>>>          for _ in range(5):
>>>              for inp in inputs:
>>>                  loss = model(inp).sum()
>>>                  loss.backward()
>>>  # Without the join() API, the below synchronization will hang
>>>  # blocking for rank 1's allreduce to complete.
>>>  torch.cuda.synchronize(device=rank)

no_sync() [source]

A context manager to disable gradient synchronizations across DDP processes. Within this context, gradients will be accumulated on module variables, which will later be synchronized in the first forward-backward pass exiting the context. Example: >>> ddp = torch.nn.parallel.DistributedDataParallel(model, pg)
>>> with ddp.no_sync():
>>>   for input in inputs:
>>>     ddp(input).backward()  # no synchronization, accumulate grads
>>> ddp(another_input).backward()  # synchronize grads

register_comm_hook(state, hook) [source]

Registers a communication hook which is an enhancement that provides a flexible hook to users where they can specify how DDP aggregates gradients across multiple workers. This hook would be very useful for researchers to try out new ideas. For example, this hook can be used to implement several algorithms like GossipGrad and gradient compression which involve different communication strategies for parameter syncs while running Distributed DataParallel training.  Parameters

state (object) –
Passed to the hook to maintain any state information during the training process. Examples include error feedback in gradient compression, peers to communicate with next in GossipGrad, etc. It is locally stored by each worker and shared by all the gradient tensors on the worker.
hook (callable) –
Averages gradient tensors across workers and defined as: hook(state: object, bucket: dist._GradBucket) -> torch.futures.Future: This function is called once the bucket is ready. The hook can perform whatever processing is needed and return a Future indicating completion of any async work (ex: allreduce). If the hook doesn’t perform any communication, it can also just return a completed Future. The Future should hold the new value of grad bucket’s tensors. Once a bucket is ready, c10d reducer would call this hook and use the tensors returned by the Future and copy grads to individual parameters. We also provide an API called get_future to retrieve a Future associated with the completion of c10d.ProcessGroup.work.      Warning Grad bucket’s tensors will not be predivided by world_size. User is responsible to divide by the world_size in case of operations like allreduce.   Warning DDP communication hook can only be registered once and should be registered before calling backward.   Warning The Future object that hook returns should contain a result that has the same shape with the tensors inside grad bucket.   Warning DDP communication hook does not support single-process multiple-device mode. Gradbucket tensors should consist of only a single tensor.   Warning get_future API supports only NCCL backend and will return a torch._C.Future which is an internal type and should be used with caution. It can still be used by register_comm_hook API, but it is subject to some subtle differences compared to torch.futures.Future.   Warning DDP communication hook is experimental and subject to change.   Example::

Below is an example of a noop hook that returns the same tensors. >>> def noop(state: object, bucket: dist._GradBucket): -> torch.futures.Future
>>>     fut = torch.futures.Future()
>>>     fut.set_result(bucket.get_tensors())
>>>     return fut
 >>> ddp.register_comm_hook(state = None, hook = noop)
  Example::

Below is an example of a Parallel SGD algorithm where gradients are encoded before allreduce, and then decoded after allreduce. >>> def encode_and_decode(state: object, bucket: dist._GradBucket): -> torch.futures.Future
>>>     tensors = [t / process_group.world_size for t in bucket.get_tensors()]
>>>     encoded_tensors = encode(tensors) # encode gradients
>>>     fut = process_group.allreduce(encoded_tensors).get_future()
>>>     # Define the then callback to decode.
>>>     def decode(fut):
>>>         decoded_tensors = decode(fut.value()) # decode gradients
>>>         return decoded_tensors
>>>     return fut.then(decode)
 >>> ddp.register_comm_hook(state = None, hook = encode_and_decode)


join(divide_by_initial_world_size=True, enable=True) [source]

A context manager to be used in conjunction with an instance of torch.nn.parallel.DistributedDataParallel to be able to train with uneven inputs across participating processes. This context manager will keep track of already-joined DDP processes, and “shadow” the forward and backward passes by inserting collective communication operations to match with the ones created by non-joined DDP processes. This will ensure each collective call has a corresponding call by already-joined DDP processes, preventing hangs or errors that would otherwise happen when training with uneven inputs across processes. Once all DDP processes have joined, the context manager will broadcast the model corresponding to the last joined process to all processes to ensure the model is the same across all processes (which is guaranteed by DDP). To use this to enable training with uneven inputs across processes, simply wrap this context manager around your training loop. No further modifications to the model or data loading is required.  Warning This module works only with the multi-process, single-device usage of torch.nn.parallel.DistributedDataParallel, which means that a single process works on a single GPU.   Warning This module currently does not support custom distributed collective operations in the forward pass, such as SyncBatchNorm or other custom defined collectives in the model’s forward pass.   Parameters

divide_by_initial_world_size (bool) – If True, will divide gradients by the initial world_size DDP training was launched with. If False, will compute the effective world size (number of ranks that have not depleted their inputs yet) and divide gradients by that during allreduce. Set divide_by_initial_world_size=True to ensure every input sample including the uneven inputs have equal weight in terms of how much they contribute to the global gradient. This is achieved by always dividing the gradient by the initial world_size even when we encounter uneven inputs. If you set this to False, we divide the gradient by the remaining number of nodes. This ensures parity with training on a smaller world_size although it also means the uneven inputs would contribute more towards the global gradient. Typically, you would want to set this to True for cases where the last few inputs of your training job are uneven. In extreme cases, where there is a large discrepancy in the number of inputs, setting this to False might provide better results.
enable (bool) – Whether to enable uneven input detection or not. Pass in enable=False to disable in cases where you know that inputs are even across participating processes. Default is True.    Example: >>>  import torch
>>>  import torch.distributed as dist
>>>  import os
>>>  import torch.multiprocessing as mp
>>>  import torch.nn as nn
>>>  # On each spawned worker
>>>  def worker(rank):
>>>      dist.init_process_group("nccl", rank=rank, world_size=2)
>>>      torch.cuda.set_device(rank)
>>>      model = nn.Linear(1, 1, bias=False).to(rank)
>>>      model = torch.nn.parallel.DistributedDataParallel(
>>>          model, device_ids=[rank], output_device=rank
>>>      )
>>>      # Rank 1 gets one more input than rank 0.
>>>      inputs = [torch.tensor([1]).float() for _ in range(10 + rank)]
>>>      with model.join():
>>>          for _ in range(5):
>>>              for inp in inputs:
>>>                  loss = model(inp).sum()
>>>                  loss.backward()
>>>  # Without the join() API, the below synchronization will hang
>>>  # blocking for rank 1's allreduce to complete.
>>>  torch.cuda.synchronize(device=rank)


no_sync() [source]

A context manager to disable gradient synchronizations across DDP processes. Within this context, gradients will be accumulated on module variables, which will later be synchronized in the first forward-backward pass exiting the context. Example: >>> ddp = torch.nn.parallel.DistributedDataParallel(model, pg)
>>> with ddp.no_sync():
>>>   for input in inputs:
>>>     ddp(input).backward()  # no synchronization, accumulate grads
>>> ddp(another_input).backward()  # synchronize grads


register_comm_hook(state, hook) [source]

Registers a communication hook which is an enhancement that provides a flexible hook to users where they can specify how DDP aggregates gradients across multiple workers. This hook would be very useful for researchers to try out new ideas. For example, this hook can be used to implement several algorithms like GossipGrad and gradient compression which involve different communication strategies for parameter syncs while running Distributed DataParallel training.  Parameters

state (object) –
Passed to the hook to maintain any state information during the training process. Examples include error feedback in gradient compression, peers to communicate with next in GossipGrad, etc. It is locally stored by each worker and shared by all the gradient tensors on the worker.
hook (callable) –
Averages gradient tensors across workers and defined as: hook(state: object, bucket: dist._GradBucket) -> torch.futures.Future: This function is called once the bucket is ready. The hook can perform whatever processing is needed and return a Future indicating completion of any async work (ex: allreduce). If the hook doesn’t perform any communication, it can also just return a completed Future. The Future should hold the new value of grad bucket’s tensors. Once a bucket is ready, c10d reducer would call this hook and use the tensors returned by the Future and copy grads to individual parameters. We also provide an API called get_future to retrieve a Future associated with the completion of c10d.ProcessGroup.work.      Warning Grad bucket’s tensors will not be predivided by world_size. User is responsible to divide by the world_size in case of operations like allreduce.   Warning DDP communication hook can only be registered once and should be registered before calling backward.   Warning The Future object that hook returns should contain a result that has the same shape with the tensors inside grad bucket.   Warning DDP communication hook does not support single-process multiple-device mode. Gradbucket tensors should consist of only a single tensor.   Warning get_future API supports only NCCL backend and will return a torch._C.Future which is an internal type and should be used with caution. It can still be used by register_comm_hook API, but it is subject to some subtle differences compared to torch.futures.Future.   Warning DDP communication hook is experimental and subject to change.   Example::

Below is an example of a noop hook that returns the same tensors. >>> def noop(state: object, bucket: dist._GradBucket): -> torch.futures.Future
>>>     fut = torch.futures.Future()
>>>     fut.set_result(bucket.get_tensors())
>>>     return fut
 >>> ddp.register_comm_hook(state = None, hook = noop)
  Example::

Below is an example of a Parallel SGD algorithm where gradients are encoded before allreduce, and then decoded after allreduce. >>> def encode_and_decode(state: object, bucket: dist._GradBucket): -> torch.futures.Future
>>>     tensors = [t / process_group.world_size for t in bucket.get_tensors()]
>>>     encoded_tensors = encode(tensors) # encode gradients
>>>     fut = process_group.allreduce(encoded_tensors).get_future()
>>>     # Define the then callback to decode.
>>>     def decode(fut):
>>>         decoded_tensors = decode(fut.value()) # decode gradients
>>>         return decoded_tensors
>>>     return fut.then(decode)
 >>> ddp.register_comm_hook(state = None, hook = encode_and_decode)


class torch.nn.parameter.Parameter [source]

A kind of Tensor that is to be considered a module parameter. Parameters are Tensor subclasses, that have a very special property when used with Module s - when they’re assigned as Module attributes they are automatically added to the list of its parameters, and will appear e.g. in parameters() iterator. Assigning a Tensor doesn’t have such effect. This is because one might want to cache some temporary state, like last hidden state of the RNN, in the model. If there was no such class as Parameter, these temporaries would get registered too.  Parameters

data (Tensor) – parameter tensor.
requires_grad (bool, optional) – if the parameter requires gradient. See Excluding subgraphs from backward for more details. Default: True


class torch.nn.parameter.UninitializedParameter [source]

A parameter that is not initialized. Unitialized Parameters are a a special case of torch.nn.Parameter where the shape of the data is still unknown. Unlikely a torch.nn.Parameter, uninitialized parameters hold no data and attempting to access some properties, like their shape, will throw a runtime error. The only operations that can be performed on a uninitialized parameter are changing its datatype, moving it to a different device and converting it to a regular torch.nn.Parameter.
materialize(shape, device=None, dtype=None) [source]

Create a Parameter with the same properties of the uninitialized one. Given a shape, it materializes a parameter in the same device and with the same dtype as the current one or the specified ones in the arguments.  Parameters

shape – (tuple): the shape for the materialized tensor.
device (torch.device) – the desired device of the parameters and buffers in this module. Optional.
dtype (torch.dtype) – the desired floating point type of the floating point parameters and buffers in this module. Optional.


materialize(shape, device=None, dtype=None) [source]

Create a Parameter with the same properties of the uninitialized one. Given a shape, it materializes a parameter in the same device and with the same dtype as the current one or the specified ones in the arguments.  Parameters

shape – (tuple): the shape for the materialized tensor.
device (torch.device) – the desired device of the parameters and buffers in this module. Optional.
dtype (torch.dtype) – the desired floating point type of the floating point parameters and buffers in this module. Optional.


class torch.nn.ParameterDict(parameters=None) [source]

Holds parameters in a dictionary. ParameterDict can be indexed like a regular Python dictionary, but parameters it contains are properly registered, and will be visible by all Module methods. ParameterDict is an ordered dictionary that respects  the order of insertion, and in update(), the order of the merged OrderedDict or another ParameterDict (the argument to update()).  Note that update() with other unordered mapping types (e.g., Python’s plain dict) does not preserve the order of the merged mapping.  Parameters
parameters (iterable, optional) – a mapping (dictionary) of (string : Parameter) or an iterable of key-value pairs of type (string, Parameter)   Example: class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.params = nn.ParameterDict({
                'left': nn.Parameter(torch.randn(5, 10)),
                'right': nn.Parameter(torch.randn(5, 10))
        })

    def forward(self, x, choice):
        x = self.params[choice].mm(x)
        return x

clear() [source]

Remove all items from the ParameterDict.

items() [source]

Return an iterable of the ParameterDict key/value pairs.

keys() [source]

Return an iterable of the ParameterDict keys.

pop(key) [source]

Remove key from the ParameterDict and return its parameter.  Parameters
key (string) – key to pop from the ParameterDict

update(parameters) [source]

Update the ParameterDict with the key-value pairs from a mapping or an iterable, overwriting existing keys.  Note If parameters is an OrderedDict, a ParameterDict, or an iterable of key-value pairs, the order of new elements in it is preserved.   Parameters
parameters (iterable) – a mapping (dictionary) from string to Parameter, or an iterable of key-value pairs of type (string, Parameter)

values() [source]

Return an iterable of the ParameterDict values.


clear() [source]

Remove all items from the ParameterDict.


items() [source]

Return an iterable of the ParameterDict key/value pairs.


keys() [source]

Return an iterable of the ParameterDict keys.


pop(key) [source]

Remove key from the ParameterDict and return its parameter.  Parameters
key (string) – key to pop from the ParameterDict


update(parameters) [source]

Update the ParameterDict with the key-value pairs from a mapping or an iterable, overwriting existing keys.  Note If parameters is an OrderedDict, a ParameterDict, or an iterable of key-value pairs, the order of new elements in it is preserved.   Parameters
parameters (iterable) – a mapping (dictionary) from string to Parameter, or an iterable of key-value pairs of type (string, Parameter)


values() [source]

Return an iterable of the ParameterDict values.


class torch.nn.ParameterList(parameters=None) [source]

Holds parameters in a list. ParameterList can be indexed like a regular Python list, but parameters it contains are properly registered, and will be visible by all Module methods.  Parameters
parameters (iterable, optional) – an iterable of Parameter to add   Example: class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.params = nn.ParameterList([nn.Parameter(torch.randn(10, 10)) for i in range(10)])

    def forward(self, x):
        # ParameterList can act as an iterable, or be indexed using ints
        for i, p in enumerate(self.params):
            x = self.params[i // 2].mm(x) + p.mm(x)
        return x

append(parameter) [source]

Appends a given parameter at the end of the list.  Parameters
parameter (nn.Parameter) – parameter to append

extend(parameters) [source]

Appends parameters from a Python iterable to the end of the list.  Parameters
parameters (iterable) – iterable of parameters to append


append(parameter) [source]

Appends a given parameter at the end of the list.  Parameters
parameter (nn.Parameter) – parameter to append


extend(parameters) [source]

Appends parameters from a Python iterable to the end of the list.  Parameters
parameters (iterable) – iterable of parameters to append


class torch.nn.PixelShuffle(upscale_factor) [source]

Rearranges elements in a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W)  to a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r) , where r is an upscale factor. This is useful for implementing efficient sub-pixel convolution with a stride of 1/r1/r . See the paper: Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network by Shi et. al (2016) for more details.  Parameters
upscale_factor (int) – factor to increase spatial resolution by    Shape:

 Input: (∗,Cin,Hin,Win)(*, C_{in}, H_{in}, W_{in}) , where * is zero or more batch dimensions Output: (∗,Cout,Hout,Wout)(*, C_{out}, H_{out}, W_{out}) , where     Cout=Cin÷upscale_factor2C_{out} = C_{in} \div \text{upscale\_factor}^2
 Hout=Hin×upscale_factorH_{out} = H_{in} \times \text{upscale\_factor}
 Wout=Win×upscale_factorW_{out} = W_{in} \times \text{upscale\_factor}
Examples: >>> pixel_shuffle = nn.PixelShuffle(3)
>>> input = torch.randn(1, 9, 4, 4)
>>> output = pixel_shuffle(input)
>>> print(output.size())
torch.Size([1, 1, 12, 12])


class torch.nn.PixelUnshuffle(downscale_factor) [source]

Reverses the PixelShuffle operation by rearranging elements in a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r)  to a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W) , where r is a downscale factor. See the paper: Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network by Shi et. al (2016) for more details.  Parameters
downscale_factor (int) – factor to decrease spatial resolution by    Shape:

 Input: (∗,Cin,Hin,Win)(*, C_{in}, H_{in}, W_{in}) , where * is zero or more batch dimensions Output: (∗,Cout,Hout,Wout)(*, C_{out}, H_{out}, W_{out}) , where     Cout=Cin×downscale_factor2C_{out} = C_{in} \times \text{downscale\_factor}^2
 Hout=Hin÷downscale_factorH_{out} = H_{in} \div \text{downscale\_factor}
 Wout=Win÷downscale_factorW_{out} = W_{in} \div \text{downscale\_factor}
Examples: >>> pixel_unshuffle = nn.PixelUnshuffle(3)
>>> input = torch.randn(1, 1, 12, 12)
>>> output = pixel_unshuffle(input)
>>> print(output.size())
torch.Size([1, 9, 4, 4])


class torch.nn.PoissonNLLLoss(log_input=True, full=False, size_average=None, eps=1e-08, reduce=None, reduction='mean') [source]

Negative log likelihood loss with Poisson distribution of target. The loss can be described as:  target∼Poisson(input)loss(input,target)=input−target∗log(input)+log(target!)\text{target} \sim \mathrm{Poisson}(\text{input}) \text{loss}(\text{input}, \text{target}) = \text{input} - \text{target} * \log(\text{input}) + \log(\text{target!})
The last term can be omitted or approximated with Stirling formula. The approximation is used for target values more than 1. For targets less or equal to 1 zeros are added to the loss.  Parameters

log_input (bool, optional) – if True the loss is computed as exp(input)−target∗input\exp(\text{input}) - \text{target}*\text{input} , if False the loss is input−target∗log(input+eps)\text{input} - \text{target}*\log(\text{input}+\text{eps}) .
full (bool, optional) –
whether to compute full loss, i. e. to add the Stirling approximation term  target∗log(target)−target+0.5∗log(2πtarget).\text{target}*\log(\text{target}) - \text{target} + 0.5 * \log(2\pi\text{target}).

size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there are multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

eps (float, optional) – Small value to avoid evaluation of log(0)\log(0)  when log_input = False. Default: 1e-8
reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'
    Examples: >>> loss = nn.PoissonNLLLoss()
>>> log_input = torch.randn(5, 2, requires_grad=True)
>>> target = torch.randn(5, 2)
>>> output = loss(log_input, target)
>>> output.backward()
  Shape:

 Input: (N,∗)(N, *)  where ∗*  means, any number of additional dimensions Target: (N,∗)(N, *) , same shape as the input Output: scalar by default. If reduction is 'none', then (N,∗)(N, *) , the same shape as the input


class torch.nn.PReLU(num_parameters=1, init=0.25) [source]

Applies the element-wise function:  PReLU(x)=max(0,x)+a∗min(0,x)\text{PReLU}(x) = \max(0,x) + a * \min(0,x)
or  PReLU(x)={x, if x≥0ax, otherwise \text{PReLU}(x) = \begin{cases} x, & \text{ if } x \geq 0 \\ ax, & \text{ otherwise } \end{cases}
Here aa  is a learnable parameter. When called without arguments, nn.PReLU() uses a single parameter aa  across all input channels. If called with nn.PReLU(nChannels), a separate aa  is used for each input channel.  Note weight decay should not be used when learning aa  for good performance.   Note Channel dim is the 2nd dim of input. When input has dims < 2, then there is no channel dim and the number of channels = 1.   Parameters

num_parameters (int) – number of aa  to learn. Although it takes an int as input, there is only two values are legitimate: 1, or the number of channels at input. Default: 1
init (float) – the initial value of aa . Default: 0.25     Shape:

 Input: (N,∗)(N, *)  where * means, any number of additional dimensions Output: (N,∗)(N, *) , same shape as the input     Variables
~PReLU.weight (Tensor) – the learnable weights of shape (num_parameters).    Examples: >>> m = nn.PReLU()
>>> input = torch.randn(2)
>>> output = m(input)


class torch.nn.qat.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', qconfig=None) [source]

A Conv2d module attached with FakeQuantize modules for weight, used for quantization aware training. We adopt the same interface as torch.nn.Conv2d, please see https://pytorch.org/docs/stable/nn.html?highlight=conv2d#torch.nn.Conv2d for documentation. Similar to torch.nn.Conv2d, with FakeQuantize modules initialized to default.  Variables
~Conv2d.weight_fake_quant – fake quant module for weight
classmethod from_float(mod) [source]

Create a qat module from a float module or qparams_dict Args: mod a float module, either produced by torch.quantization utilities or directly from user


classmethod from_float(mod) [source]

Create a qat module from a float module or qparams_dict Args: mod a float module, either produced by torch.quantization utilities or directly from user


class torch.nn.qat.Linear(in_features, out_features, bias=True, qconfig=None) [source]

A linear module attached with FakeQuantize modules for weight, used for quantization aware training. We adopt the same interface as torch.nn.Linear, please see https://pytorch.org/docs/stable/nn.html#torch.nn.Linear for documentation. Similar to torch.nn.Linear, with FakeQuantize modules initialized to default.  Variables
~Linear.weight – fake quant module for weight
classmethod from_float(mod) [source]

Create a qat module from a float module or qparams_dict Args: mod a float module, either produced by torch.quantization utilities or directly from user


class torch.nn.quantized.BatchNorm2d(num_features, eps=1e-05, momentum=0.1) [source]

This is the quantized version of BatchNorm2d.


class torch.nn.quantized.BatchNorm3d(num_features, eps=1e-05, momentum=0.1) [source]

This is the quantized version of BatchNorm3d.


class torch.nn.quantized.Conv1d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros') [source]

Applies a 1D convolution over a quantized input signal composed of several quantized input planes. For details on input arguments, parameters, and implementation see Conv1d.  Note Only zeros is supported for the padding_mode argument.   Note Only torch.quint8 is supported for the input data type.   Variables

~Conv1d.weight (Tensor) – packed tensor derived from the learnable weight parameter.
~Conv1d.scale (Tensor) – scalar for the output scale
~Conv1d.zero_point (Tensor) – scalar for the output zero point    See Conv1d for other attributes. Examples: >>> m = nn.quantized.Conv1d(16, 33, 3, stride=2)
>>> input = torch.randn(20, 16, 100)
>>> # quantize input to quint8
>>> q_input = torch.quantize_per_tensor(input, scale=1.0, zero_point=0,
                                        dtype=torch.quint8)
>>> output = m(q_input)

classmethod from_float(mod) [source]

Creates a quantized module from a float module or qparams_dict.  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by the user


classmethod from_float(mod) [source]

Creates a quantized module from a float module or qparams_dict.  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by the user


class torch.nn.quantized.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros') [source]

Applies a 2D convolution over a quantized input signal composed of several quantized input planes. For details on input arguments, parameters, and implementation see Conv2d.  Note Only zeros is supported for the padding_mode argument.   Note Only torch.quint8 is supported for the input data type.   Variables

~Conv2d.weight (Tensor) – packed tensor derived from the learnable weight parameter.
~Conv2d.scale (Tensor) – scalar for the output scale
~Conv2d.zero_point (Tensor) – scalar for the output zero point    See Conv2d for other attributes. Examples: >>> # With square kernels and equal stride
>>> m = nn.quantized.Conv2d(16, 33, 3, stride=2)
>>> # non-square kernels and unequal stride and with padding
>>> m = nn.quantized.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2))
>>> # non-square kernels and unequal stride and with padding and dilation
>>> m = nn.quantized.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2), dilation=(3, 1))
>>> input = torch.randn(20, 16, 50, 100)
>>> # quantize input to quint8
>>> q_input = torch.quantize_per_tensor(input, scale=1.0, zero_point=0, dtype=torch.quint8)
>>> output = m(q_input)

classmethod from_float(mod) [source]

Creates a quantized module from a float module or qparams_dict.  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by the user


class torch.nn.quantized.Conv3d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros') [source]

Applies a 3D convolution over a quantized input signal composed of several quantized input planes. For details on input arguments, parameters, and implementation see Conv3d.  Note Only zeros is supported for the padding_mode argument.   Note Only torch.quint8 is supported for the input data type.   Variables

~Conv3d.weight (Tensor) – packed tensor derived from the learnable weight parameter.
~Conv3d.scale (Tensor) – scalar for the output scale
~Conv3d.zero_point (Tensor) – scalar for the output zero point    See Conv3d for other attributes. Examples: >>> # With square kernels and equal stride
>>> m = nn.quantized.Conv3d(16, 33, 3, stride=2)
>>> # non-square kernels and unequal stride and with padding
>>> m = nn.quantized.Conv3d(16, 33, (3, 5, 5), stride=(1, 2, 2), padding=(1, 2, 2))
>>> # non-square kernels and unequal stride and with padding and dilation
>>> m = nn.quantized.Conv3d(16, 33, (3, 5, 5), stride=(1, 2, 2), padding=(1, 2, 2), dilation=(1, 2, 2))
>>> input = torch.randn(20, 16, 56, 56, 56)
>>> # quantize input to quint8
>>> q_input = torch.quantize_per_tensor(input, scale=1.0, zero_point=0, dtype=torch.quint8)
>>> output = m(q_input)

classmethod from_float(mod) [source]

Creates a quantized module from a float module or qparams_dict.  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by the user


class torch.nn.quantized.DeQuantize [source]

Dequantizes an incoming tensor  Examples::

>>> input = torch.tensor([[1., -1.], [1., -1.]])
>>> scale, zero_point, dtype = 1.0, 2, torch.qint8
>>> qm = Quantize(scale, zero_point, dtype)
>>> quantized_input = qm(input)
>>> dqm = DeQuantize()
>>> dequantized = dqm(quantized_input)
>>> print(dequantized)
tensor([[ 1., -1.],
        [ 1., -1.]], dtype=torch.float32)


torch.nn.quantized.dynamic Linear
class torch.nn.quantized.dynamic.Linear(in_features, out_features, bias_=True, dtype=torch.qint8) [source]

A dynamic quantized linear module with floating point tensor as inputs and outputs. We adopt the same interface as torch.nn.Linear, please see https://pytorch.org/docs/stable/nn.html#torch.nn.Linear for documentation. Similar to torch.nn.Linear, attributes will be randomly initialized at module creation time and will be overwritten later  Variables

~Linear.weight (Tensor) – the non-learnable quantized weights of the module which are of shape (out_features,in_features)(\text{out\_features}, \text{in\_features}) .
~Linear.bias (Tensor) – the non-learnable floating point bias of the module of shape (out_features)(\text{out\_features}) . If bias is True, the values are initialized to zero.    Examples: >>> m = nn.quantized.dynamic.Linear(20, 30)
>>> input = torch.randn(128, 20)
>>> output = m(input)
>>> print(output.size())
torch.Size([128, 30])

classmethod from_float(mod) [source]

Create a dynamic quantized module from a float module or qparams_dict  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by the user

 LSTM
class torch.nn.quantized.dynamic.LSTM(*args, **kwargs) [source]

A dynamic quantized LSTM module with floating point tensor as inputs and outputs. We adopt the same interface as torch.nn.LSTM, please see https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM for documentation. Examples: >>> rnn = nn.LSTM(10, 20, 2)
>>> input = torch.randn(5, 3, 10)
>>> h0 = torch.randn(2, 3, 20)
>>> c0 = torch.randn(2, 3, 20)
>>> output, (hn, cn) = rnn(input, (h0, c0))

 LSTMCell
class torch.nn.quantized.dynamic.LSTMCell(*args, **kwargs) [source]

A long short-term memory (LSTM) cell. A dynamic quantized LSTMCell module with floating point tensor as inputs and outputs. Weights are quantized to 8 bits. We adopt the same interface as torch.nn.LSTMCell, please see https://pytorch.org/docs/stable/nn.html#torch.nn.LSTMCell for documentation. Examples: >>> rnn = nn.LSTMCell(10, 20)
>>> input = torch.randn(6, 3, 10)
>>> hx = torch.randn(3, 20)
>>> cx = torch.randn(3, 20)
>>> output = []
>>> for i in range(6):
        hx, cx = rnn(input[i], (hx, cx))
        output.append(hx)

 GRUCell
class torch.nn.quantized.dynamic.GRUCell(input_size, hidden_size, bias=True, dtype=torch.qint8) [source]

A gated recurrent unit (GRU) cell A dynamic quantized GRUCell module with floating point tensor as inputs and outputs. Weights are quantized to 8 bits. We adopt the same interface as torch.nn.GRUCell, please see https://pytorch.org/docs/stable/nn.html#torch.nn.GRUCell for documentation. Examples: >>> rnn = nn.GRUCell(10, 20)
>>> input = torch.randn(6, 3, 10)
>>> hx = torch.randn(3, 20)
>>> output = []
>>> for i in range(6):
        hx = rnn(input[i], hx)
        output.append(hx)

 RNNCell
class torch.nn.quantized.dynamic.RNNCell(input_size, hidden_size, bias=True, nonlinearity='tanh', dtype=torch.qint8) [source]

An Elman RNN cell with tanh or ReLU non-linearity. A dynamic quantized RNNCell module with floating point tensor as inputs and outputs. Weights are quantized to 8 bits. We adopt the same interface as torch.nn.RNNCell, please see https://pytorch.org/docs/stable/nn.html#torch.nn.RNNCell for documentation. Examples: >>> rnn = nn.RNNCell(10, 20)
>>> input = torch.randn(6, 3, 10)
>>> hx = torch.randn(3, 20)
>>> output = []
>>> for i in range(6):
        hx = rnn(input[i], hx)
        output.append(hx)


class torch.nn.quantized.dynamic.GRUCell(input_size, hidden_size, bias=True, dtype=torch.qint8) [source]

A gated recurrent unit (GRU) cell A dynamic quantized GRUCell module with floating point tensor as inputs and outputs. Weights are quantized to 8 bits. We adopt the same interface as torch.nn.GRUCell, please see https://pytorch.org/docs/stable/nn.html#torch.nn.GRUCell for documentation. Examples: >>> rnn = nn.GRUCell(10, 20)
>>> input = torch.randn(6, 3, 10)
>>> hx = torch.randn(3, 20)
>>> output = []
>>> for i in range(6):
        hx = rnn(input[i], hx)
        output.append(hx)


class torch.nn.quantized.dynamic.Linear(in_features, out_features, bias_=True, dtype=torch.qint8) [source]

A dynamic quantized linear module with floating point tensor as inputs and outputs. We adopt the same interface as torch.nn.Linear, please see https://pytorch.org/docs/stable/nn.html#torch.nn.Linear for documentation. Similar to torch.nn.Linear, attributes will be randomly initialized at module creation time and will be overwritten later  Variables

~Linear.weight (Tensor) – the non-learnable quantized weights of the module which are of shape (out_features,in_features)(\text{out\_features}, \text{in\_features}) .
~Linear.bias (Tensor) – the non-learnable floating point bias of the module of shape (out_features)(\text{out\_features}) . If bias is True, the values are initialized to zero.    Examples: >>> m = nn.quantized.dynamic.Linear(20, 30)
>>> input = torch.randn(128, 20)
>>> output = m(input)
>>> print(output.size())
torch.Size([128, 30])

classmethod from_float(mod) [source]

Create a dynamic quantized module from a float module or qparams_dict  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by the user


classmethod from_float(mod) [source]

Create a dynamic quantized module from a float module or qparams_dict  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by the user


class torch.nn.quantized.dynamic.LSTM(*args, **kwargs) [source]

A dynamic quantized LSTM module with floating point tensor as inputs and outputs. We adopt the same interface as torch.nn.LSTM, please see https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM for documentation. Examples: >>> rnn = nn.LSTM(10, 20, 2)
>>> input = torch.randn(5, 3, 10)
>>> h0 = torch.randn(2, 3, 20)
>>> c0 = torch.randn(2, 3, 20)
>>> output, (hn, cn) = rnn(input, (h0, c0))


class torch.nn.quantized.dynamic.LSTMCell(*args, **kwargs) [source]

A long short-term memory (LSTM) cell. A dynamic quantized LSTMCell module with floating point tensor as inputs and outputs. Weights are quantized to 8 bits. We adopt the same interface as torch.nn.LSTMCell, please see https://pytorch.org/docs/stable/nn.html#torch.nn.LSTMCell for documentation. Examples: >>> rnn = nn.LSTMCell(10, 20)
>>> input = torch.randn(6, 3, 10)
>>> hx = torch.randn(3, 20)
>>> cx = torch.randn(3, 20)
>>> output = []
>>> for i in range(6):
        hx, cx = rnn(input[i], (hx, cx))
        output.append(hx)


class torch.nn.quantized.dynamic.RNNCell(input_size, hidden_size, bias=True, nonlinearity='tanh', dtype=torch.qint8) [source]

An Elman RNN cell with tanh or ReLU non-linearity. A dynamic quantized RNNCell module with floating point tensor as inputs and outputs. Weights are quantized to 8 bits. We adopt the same interface as torch.nn.RNNCell, please see https://pytorch.org/docs/stable/nn.html#torch.nn.RNNCell for documentation. Examples: >>> rnn = nn.RNNCell(10, 20)
>>> input = torch.randn(6, 3, 10)
>>> hx = torch.randn(3, 20)
>>> output = []
>>> for i in range(6):
        hx = rnn(input[i], hx)
        output.append(hx)


class torch.nn.quantized.ELU(scale, zero_point, alpha=1.0) [source]

This is the quantized equivalent of ELU.  Parameters

scale – quantization scale of the output tensor
zero_point – quantization zero point of the output tensor
alpha – the alpha constant


class torch.nn.quantized.Embedding(num_embeddings, embedding_dim, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False, _weight=None, dtype=torch.quint8) [source]

A quantized Embedding module with quantized packed weights as inputs. We adopt the same interface as torch.nn.Embedding, please see https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding for documentation. Similar to Embedding, attributes will be randomly initialized at module creation time and will be overwritten later  Variables
~Embedding.weight (Tensor) – the non-learnable quantized weights of the module of shape (num_embeddings,embedding_dim)(\text{num\_embeddings}, \text{embedding\_dim}) .    Examples::

>>> m = nn.quantized.Embedding(num_embeddings=10, embedding_dim=12)
>>> indices = torch.tensor([9, 6, 5, 7, 8, 8, 9, 2, 8])
>>> output = m(indices)
>>> print(output.size())
torch.Size([9, 12]

classmethod from_float(mod) [source]

Create a quantized embedding module from a float module  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by user


classmethod from_float(mod) [source]

Create a quantized embedding module from a float module  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by user


class torch.nn.quantized.EmbeddingBag(num_embeddings, embedding_dim, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, mode='sum', sparse=False, _weight=None, include_last_offset=False, dtype=torch.quint8) [source]

A quantized EmbeddingBag module with quantized packed weights as inputs. We adopt the same interface as torch.nn.EmbeddingBag, please see https://pytorch.org/docs/stable/nn.html#torch.nn.EmbeddingBag for documentation. Similar to EmbeddingBag, attributes will be randomly initialized at module creation time and will be overwritten later  Variables
~EmbeddingBag.weight (Tensor) – the non-learnable quantized weights of the module of shape (num_embeddings,embedding_dim)(\text{num\_embeddings}, \text{embedding\_dim}) .    Examples::

>>> m = nn.quantized.EmbeddingBag(num_embeddings=10, embedding_dim=12, include_last_offset=True, mode='sum')
>>> indices = torch.tensor([9, 6, 5, 7, 8, 8, 9, 2, 8, 6, 6, 9, 1, 6, 8, 8, 3, 2, 3, 6, 3, 6, 5, 7, 0, 8, 4, 6, 5, 8, 2, 3])
>>> offsets = torch.tensor([0, 19, 20, 28, 28, 32])
>>> output = m(indices, offsets)
>>> print(output.size())
torch.Size([5, 12]

classmethod from_float(mod) [source]

Create a quantized embedding_bag module from a float module  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by user


classmethod from_float(mod) [source]

Create a quantized embedding_bag module from a float module  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by user


class torch.nn.quantized.FloatFunctional [source]

State collector class for float operations. The instance of this class can be used instead of the torch. prefix for some operations. See example usage below.  Note This class does not provide a forward hook. Instead, you must use one of the underlying functions (e.g. add).  Examples: >>> f_add = FloatFunctional()
>>> a = torch.tensor(3.0)
>>> b = torch.tensor(4.0)
>>> f_add.add(a, b)  # Equivalent to ``torch.add(a, b)``
  Valid operation names:

 add cat mul add_relu add_scalar mul_scalar


torch.nn.quantized.functional.adaptive_avg_pool2d(input, output_size) [source]

Applies a 2D adaptive average pooling over a quantized input signal composed of several quantized input planes.  Note The input quantization parameters propagate to the output.  See AdaptiveAvgPool2d for details and output shape.  Parameters
output_size – the target output size (single integer or double-integer tuple)


torch.nn.quantized.functional.avg_pool2d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None) [source]

Applies 2D average-pooling operation in kH×kWkH \times kW  regions by step size sH×sWsH \times sW  steps. The number of output features is equal to the number of input planes.  Note The input quantization parameters propagate to the output.  See AvgPool2d for details and output shape.  Parameters

input – quantized input tensor (minibatch,in_channels,iH,iW)(\text{minibatch} , \text{in\_channels} , iH , iW)

kernel_size – size of the pooling region. Can be a single number or a tuple (kH, kW)

stride – stride of the pooling operation. Can be a single number or a tuple (sH, sW). Default: kernel_size

padding – implicit zero paddings on both sides of the input. Can be a single number or a tuple (padH, padW). Default: 0
ceil_mode – when True, will use ceil instead of floor in the formula to compute the output shape. Default: False

count_include_pad – when True, will include the zero-padding in the averaging calculation. Default: True

divisor_override – if specified, it will be used as divisor, otherwise size of the pooling region will be used. Default: None


torch.nn.quantized.functional.conv1d(input, weight, bias, stride=1, padding=0, dilation=1, groups=1, padding_mode='zeros', scale=1.0, zero_point=0, dtype=torch.quint8) [source]

Applies a 1D convolution over a quantized 1D input composed of several input planes. See Conv1d for details and output shape.  Parameters

input – quantized input tensor of shape (minibatch,in_channels,iW)(\text{minibatch} , \text{in\_channels} , iW)

weight – quantized filters of shape (out_channels,in_channelsgroups,iW)(\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , iW)

bias – non-quantized bias tensor of shape (out_channels)(\text{out\_channels}) . The tensor type must be torch.float.
stride – the stride of the convolving kernel. Can be a single number or a tuple (sW,). Default: 1
padding – implicit paddings on both sides of the input. Can be a single number or a tuple (padW,). Default: 0
dilation – the spacing between kernel elements. Can be a single number or a tuple (dW,). Default: 1
groups – split input into groups, in_channels\text{in\_channels}  should be divisible by the number of groups. Default: 1
padding_mode – the padding mode to use. Only “zeros” is supported for quantized convolution at the moment. Default: “zeros”
scale – quantization scale for the output. Default: 1.0
zero_point – quantization zero_point for the output. Default: 0
dtype – quantization data type to use. Default: torch.quint8
    Examples: >>> from torch.nn.quantized import functional as qF
>>> filters = torch.randn(33, 16, 3, dtype=torch.float)
>>> inputs = torch.randn(20, 16, 50, dtype=torch.float)
>>> bias = torch.randn(33, dtype=torch.float)
>>>
>>> scale, zero_point = 1.0, 0
>>> dtype_inputs = torch.quint8
>>> dtype_filters = torch.qint8
>>>
>>> q_filters = torch.quantize_per_tensor(filters, scale, zero_point, dtype_filters)
>>> q_inputs = torch.quantize_per_tensor(inputs, scale, zero_point, dtype_inputs)
>>> qF.conv1d(q_inputs, q_filters, bias, padding=1, scale=scale, zero_point=zero_point)


torch.nn.quantized.functional.conv2d(input, weight, bias, stride=1, padding=0, dilation=1, groups=1, padding_mode='zeros', scale=1.0, zero_point=0, dtype=torch.quint8) [source]

Applies a 2D convolution over a quantized 2D input composed of several input planes. See Conv2d for details and output shape.  Parameters

input – quantized input tensor of shape (minibatch,in_channels,iH,iW)(\text{minibatch} , \text{in\_channels} , iH , iW)

weight – quantized filters of shape (out_channels,in_channelsgroups,kH,kW)(\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kH , kW)

bias – non-quantized bias tensor of shape (out_channels)(\text{out\_channels}) . The tensor type must be torch.float.
stride – the stride of the convolving kernel. Can be a single number or a tuple (sH, sW). Default: 1
padding – implicit paddings on both sides of the input. Can be a single number or a tuple (padH, padW). Default: 0
dilation – the spacing between kernel elements. Can be a single number or a tuple (dH, dW). Default: 1
groups – split input into groups, in_channels\text{in\_channels}  should be divisible by the number of groups. Default: 1
padding_mode – the padding mode to use. Only “zeros” is supported for quantized convolution at the moment. Default: “zeros”
scale – quantization scale for the output. Default: 1.0
zero_point – quantization zero_point for the output. Default: 0
dtype – quantization data type to use. Default: torch.quint8
    Examples: >>> from torch.nn.quantized import functional as qF
>>> filters = torch.randn(8, 4, 3, 3, dtype=torch.float)
>>> inputs = torch.randn(1, 4, 5, 5, dtype=torch.float)
>>> bias = torch.randn(8, dtype=torch.float)
>>>
>>> scale, zero_point = 1.0, 0
>>> dtype_inputs = torch.quint8
>>> dtype_filters = torch.qint8
>>>
>>> q_filters = torch.quantize_per_tensor(filters, scale, zero_point, dtype_filters)
>>> q_inputs = torch.quantize_per_tensor(inputs, scale, zero_point, dtype_inputs)
>>> qF.conv2d(q_inputs, q_filters, bias, padding=1, scale=scale, zero_point=zero_point)


torch.nn.quantized.functional.conv3d(input, weight, bias, stride=1, padding=0, dilation=1, groups=1, padding_mode='zeros', scale=1.0, zero_point=0, dtype=torch.quint8) [source]

Applies a 3D convolution over a quantized 3D input composed of several input planes. See Conv3d for details and output shape.  Parameters

input – quantized input tensor of shape (minibatch,in_channels,iD,iH,iW)(\text{minibatch} , \text{in\_channels} , iD , iH , iW)

weight – quantized filters of shape (out_channels,in_channelsgroups,kD,kH,kW)(\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kD , kH , kW)

bias – non-quantized bias tensor of shape (out_channels)(\text{out\_channels}) . The tensor type must be torch.float.
stride – the stride of the convolving kernel. Can be a single number or a tuple (sD, sH, sW). Default: 1
padding – implicit paddings on both sides of the input. Can be a single number or a tuple (padD, padH, padW). Default: 0
dilation – the spacing between kernel elements. Can be a single number or a tuple (dD, dH, dW). Default: 1
groups – split input into groups, in_channels\text{in\_channels}  should be divisible by the number of groups. Default: 1
padding_mode – the padding mode to use. Only “zeros” is supported for quantized convolution at the moment. Default: “zeros”
scale – quantization scale for the output. Default: 1.0
zero_point – quantization zero_point for the output. Default: 0
dtype – quantization data type to use. Default: torch.quint8
    Examples: >>> from torch.nn.quantized import functional as qF
>>> filters = torch.randn(8, 4, 3, 3, 3, dtype=torch.float)
>>> inputs = torch.randn(1, 4, 5, 5, 5, dtype=torch.float)
>>> bias = torch.randn(8, dtype=torch.float)
>>>
>>> scale, zero_point = 1.0, 0
>>> dtype_inputs = torch.quint8
>>> dtype_filters = torch.qint8
>>>
>>> q_filters = torch.quantize_per_tensor(filters, scale, zero_point, dtype_filters)
>>> q_inputs = torch.quantize_per_tensor(inputs, scale, zero_point, dtype_inputs)
>>> qF.conv3d(q_inputs, q_filters, bias, padding=1, scale=scale, zero_point=zero_point)


torch.nn.quantized.functional.hardswish(input, scale, zero_point) [source]

This is the quantized version of hardswish().  Parameters

input – quantized input
scale – quantization scale of the output tensor
zero_point – quantization zero point of the output tensor


torch.nn.quantized.functional.interpolate(input, size=None, scale_factor=None, mode='nearest', align_corners=None) [source]

Down/up samples the input to either the given size or the given scale_factor See torch.nn.functional.interpolate() for implementation details. The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width.  Note The input quantization parameters propagate to the output.   Note Only 2D/3D input is supported for quantized inputs   Note Only the following modes are supported for the quantized inputs:  bilinear nearest    Parameters

input (Tensor) – the input tensor
size (int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int]) – output spatial size.
scale_factor (float or Tuple[float]) – multiplier for spatial size. Has to match input size if it is a tuple.
mode (str) – algorithm used for upsampling: 'nearest' | 'bilinear'

align_corners (bool, optional) – Geometrically, we consider the pixels of the input and output as squares rather than points. If set to True, the input and output tensors are aligned by the center points of their corner pixels, preserving the values at the corner pixels. If set to False, the input and output tensors are aligned by the corner points of their corner pixels, and the interpolation uses edge value padding for out-of-boundary values, making this operation independent of input size when scale_factor is kept the same. This only has an effect when mode is 'bilinear'. Default: False


torch.nn.quantized.functional.linear(input, weight, bias=None, scale=None, zero_point=None) [source]

Applies a linear transformation to the incoming quantized data: y=xAT+by = xA^T + b . See Linear  Note Current implementation packs weights on every call, which has penalty on performance. If you want to avoid the overhead, use Linear.   Parameters

input (Tensor) – Quantized input of type torch.quint8

weight (Tensor) – Quantized weight of type torch.qint8

bias (Tensor) – None or fp32 bias of type torch.float

scale (double) – output scale. If None, derived from the input scale
zero_point (long) – output zero point. If None, derived from the input zero_point     Shape:

 Input: (N,∗,in_features)(N, *, in\_features)  where * means any number of additional dimensions Weight: (out_features,in_features)(out\_features, in\_features)
 Bias: (out_features)(out\_features)
 Output: (N,∗,out_features)(N, *, out\_features)


torch.nn.quantized.functional.max_pool2d(input, kernel_size, stride=None, padding=0, dilation=1, ceil_mode=False, return_indices=False) [source]

Applies a 2D max pooling over a quantized input signal composed of several quantized input planes.  Note The input quantization parameters are propagated to the output.  See MaxPool2d for details.


torch.nn.quantized.functional.upsample(input, size=None, scale_factor=None, mode='nearest', align_corners=None) [source]

Upsamples the input to either the given size or the given scale_factor  Warning This function is deprecated in favor of torch.nn.quantized.functional.interpolate(). This is equivalent with nn.quantized.functional.interpolate(...).  See torch.nn.functional.interpolate() for implementation details. The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width.  Note The input quantization parameters propagate to the output.   Note Only 2D input is supported for quantized inputs   Note Only the following modes are supported for the quantized inputs:  bilinear nearest    Parameters

input (Tensor) – quantized input tensor
size (int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int]) – output spatial size.
scale_factor (float or Tuple[float]) – multiplier for spatial size. Has to be an integer.
mode (string) – algorithm used for upsampling: 'nearest' | 'bilinear'

align_corners (bool, optional) – Geometrically, we consider the pixels of the input and output as squares rather than points. If set to True, the input and output tensors are aligned by the center points of their corner pixels, preserving the values at the corner pixels. If set to False, the input and output tensors are aligned by the corner points of their corner pixels, and the interpolation uses edge value padding for out-of-boundary values, making this operation independent of input size when scale_factor is kept the same. This only has an effect when mode is 'bilinear'. Default: False
     Warning With align_corners = True, the linearly interpolating modes (bilinear) don’t proportionally align the output and input pixels, and thus the output values can depend on the input size. This was the default behavior for these modes up to version 0.3.1. Since then, the default behavior is align_corners = False. See Upsample for concrete examples on how this affects the outputs.


torch.nn.quantized.functional.upsample_bilinear(input, size=None, scale_factor=None) [source]

Upsamples the input, using bilinear upsampling.  Warning This function is deprecated in favor of torch.nn.quantized.functional.interpolate(). This is equivalent with nn.quantized.functional.interpolate(..., mode='bilinear', align_corners=True).   Note The input quantization parameters propagate to the output.   Note Only 2D inputs are supported   Parameters

input (Tensor) – quantized input
size (int or Tuple[int, int]) – output spatial size.
scale_factor (int or Tuple[int, int]) – multiplier for spatial size


torch.nn.quantized.functional.upsample_nearest(input, size=None, scale_factor=None) [source]

Upsamples the input, using nearest neighbours’ pixel values.  Warning This function is deprecated in favor of torch.nn.quantized.functional.interpolate(). This is equivalent with nn.quantized.functional.interpolate(..., mode='nearest').   Note The input quantization parameters propagate to the output.   Note Only 2D inputs are supported   Parameters

input (Tensor) – quantized input
size (int or Tuple[int, int] or Tuple[int, int, int]) – output spatial size.
scale_factor (int) – multiplier for spatial size. Has to be an integer.


class torch.nn.quantized.GroupNorm(num_groups, num_channels, weight, bias, scale, zero_point, eps=1e-05, affine=True) [source]

This is the quantized version of GroupNorm.  Additional args:

scale - quantization scale of the output, type: double.
zero_point - quantization zero point of the output, type: long.


class torch.nn.quantized.Hardswish(scale, zero_point) [source]

This is the quantized version of Hardswish.  Parameters

scale – quantization scale of the output tensor
zero_point – quantization zero point of the output tensor


class torch.nn.quantized.InstanceNorm1d(num_features, weight, bias, scale, zero_point, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) [source]

This is the quantized version of InstanceNorm1d.  Additional args:

scale - quantization scale of the output, type: double.
zero_point - quantization zero point of the output, type: long.


class torch.nn.quantized.InstanceNorm2d(num_features, weight, bias, scale, zero_point, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) [source]

This is the quantized version of InstanceNorm2d.  Additional args:

scale - quantization scale of the output, type: double.
zero_point - quantization zero point of the output, type: long.


class torch.nn.quantized.InstanceNorm3d(num_features, weight, bias, scale, zero_point, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) [source]

This is the quantized version of InstanceNorm3d.  Additional args:

scale - quantization scale of the output, type: double.
zero_point - quantization zero point of the output, type: long.


class torch.nn.quantized.LayerNorm(normalized_shape, weight, bias, scale, zero_point, eps=1e-05, elementwise_affine=True) [source]

This is the quantized version of LayerNorm.  Additional args:

scale - quantization scale of the output, type: double.
zero_point - quantization zero point of the output, type: long.


class torch.nn.quantized.Linear(in_features, out_features, bias_=True, dtype=torch.qint8) [source]

A quantized linear module with quantized tensor as inputs and outputs. We adopt the same interface as torch.nn.Linear, please see https://pytorch.org/docs/stable/nn.html#torch.nn.Linear for documentation. Similar to Linear, attributes will be randomly initialized at module creation time and will be overwritten later  Variables

~Linear.weight (Tensor) – the non-learnable quantized weights of the module of shape (out_features,in_features)(\text{out\_features}, \text{in\_features}) .
~Linear.bias (Tensor) – the non-learnable bias of the module of shape (out_features)(\text{out\_features}) . If bias is True, the values are initialized to zero.
~Linear.scale – scale parameter of output Quantized Tensor, type: double
~Linear.zero_point – zero_point parameter for output Quantized Tensor, type: long    Examples: >>> m = nn.quantized.Linear(20, 30)
>>> input = torch.randn(128, 20)
>>> input = torch.quantize_per_tensor(input, 1.0, 0, torch.quint8)
>>> output = m(input)
>>> print(output.size())
torch.Size([128, 30])

classmethod from_float(mod) [source]

Create a quantized module from a float module or qparams_dict  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by the user


classmethod from_float(mod) [source]

Create a quantized module from a float module or qparams_dict  Parameters
mod (Module) – a float module, either produced by torch.quantization utilities or provided by the user


class torch.nn.quantized.QFunctional [source]

Wrapper class for quantized operations. The instance of this class can be used instead of the torch.ops.quantized prefix. See example usage below.  Note This class does not provide a forward hook. Instead, you must use one of the underlying functions (e.g. add).  Examples: >>> q_add = QFunctional()
>>> a = torch.quantize_per_tensor(torch.tensor(3.0), 1.0, 0, torch.qint32)
>>> b = torch.quantize_per_tensor(torch.tensor(4.0), 1.0, 0, torch.qint32)
>>> q_add.add(a, b)  # Equivalent to ``torch.ops.quantized.add(a, b, 1.0, 0)``
  Valid operation names:

 add cat mul add_relu add_scalar mul_scalar


class torch.nn.quantized.Quantize(scale, zero_point, dtype) [source]

Quantizes an incoming tensor  Parameters

scale – scale of the output Quantized Tensor
zero_point – zero_point of output Quantized Tensor
dtype – data type of output Quantized Tensor   Variables
zero_point, dtype (`scale`,) –     Examples::

>>> t = torch.tensor([[1., -1.], [1., -1.]])
>>> scale, zero_point, dtype = 1.0, 2, torch.qint8
>>> qm = Quantize(scale, zero_point, dtype)
>>> qt = qm(t)
>>> print(qt)
tensor([[ 1., -1.],
        [ 1., -1.]], size=(2, 2), dtype=torch.qint8, scale=1.0, zero_point=2)


class torch.nn.quantized.ReLU6(inplace=False) [source]

Applies the element-wise function: ReLU6(x)=min(max(x0,x),q(6))\text{ReLU6}(x) = \min(\max(x_0, x), q(6)) , where x0x_0  is the zero_point, and q(6)q(6)  is the quantized representation of number 6.  Parameters
inplace – can optionally do the operation in-place. Default: False    Shape:

 Input: (N,∗)(N, *)  where * means, any number of additional dimensions Output: (N,∗)(N, *) , same shape as the input     Examples: >>> m = nn.quantized.ReLU6()
>>> input = torch.randn(2)
>>> input = torch.quantize_per_tensor(input, 1.0, 0, dtype=torch.qint32)
>>> output = m(input)


class torch.nn.ReflectionPad1d(padding) [source]

Pads the input tensor using the reflection of the input boundary. For N-dimensional padding, use torch.nn.functional.pad().  Parameters
padding (int, tuple) – the size of the padding. If is int, uses the same padding in all boundaries. If a 2-tuple, uses (padding_left\text{padding\_left} , padding_right\text{padding\_right} )    Shape:

 Input: (N,C,Win)(N, C, W_{in})

Output: (N,C,Wout)(N, C, W_{out})  where Wout=Win+padding_left+padding_rightW_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}      Examples: >>> m = nn.ReflectionPad1d(2)
>>> input = torch.arange(8, dtype=torch.float).reshape(1, 2, 4)
>>> input
tensor([[[0., 1., 2., 3.],
         [4., 5., 6., 7.]]])
>>> m(input)
tensor([[[2., 1., 0., 1., 2., 3., 2., 1.],
         [6., 5., 4., 5., 6., 7., 6., 5.]]])
>>> # using different paddings for different sides
>>> m = nn.ReflectionPad1d((3, 1))
>>> m(input)
tensor([[[3., 2., 1., 0., 1., 2., 3., 2.],
         [7., 6., 5., 4., 5., 6., 7., 6.]]])