Returns a new tensor that is a narrowed version of input tensor. The dimension dim is input from start to start + length. The returned tensor and input tensor share the same underlying storage.
Example:


Computes input≠other\text{input} \neq \text{other}  element-wise.
The second argument can be a number or a tensor whose shape is broadcastable with the first argument.
out (Tensor, optional) – the output tensor.
A boolean tensor that is True where input is not equal to other and False elsewhere
Example:


Returns a new tensor with the negative of the elements of input.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.
Example:


Return the next floating-point value after input towards other, elementwise.
The shapes of input and other must be broadcastable.
out (Tensor, optional) – the output tensor.


These are the basic building block for graphs
torch.nn
A kind of Tensor that is to be considered a module parameter.
A parameter that is not initialized.
Base class for all neural network modules.
A sequential container.
Holds submodules in a list.
Holds submodules in a dictionary.
Holds parameters in a list.
Holds parameters in a dictionary.
Global Hooks For Module
Registers a forward pre-hook common to all modules.
Registers a global forward hook for all the modules
Registers a backward hook common to all the modules.
nn.Conv1d
Applies a 1D convolution over an input signal composed of several input planes.
nn.Conv2d
Applies a 2D convolution over an input signal composed of several input planes.
nn.Conv3d
Applies a 3D convolution over an input signal composed of several input planes.
nn.ConvTranspose1d
Applies a 1D transposed convolution operator over an input image composed of several input planes.
nn.ConvTranspose2d
Applies a 2D transposed convolution operator over an input image composed of several input planes.
nn.ConvTranspose3d
Applies a 3D transposed convolution operator over an input image composed of several input planes.
nn.LazyConv1d
A torch.nn.Conv1d module with lazy initialization of the in_channels argument of the Conv1d that is inferred from the input.size(1).
nn.LazyConv2d
A torch.nn.Conv2d module with lazy initialization of the in_channels argument of the Conv2d that is inferred from the input.size(1).
nn.LazyConv3d
A torch.nn.Conv3d module with lazy initialization of the in_channels argument of the Conv3d that is inferred from the input.size(1).
nn.LazyConvTranspose1d
A torch.nn.ConvTranspose1d module with lazy initialization of the in_channels argument of the ConvTranspose1d that is inferred from the input.size(1).
nn.LazyConvTranspose2d
A torch.nn.ConvTranspose2d module with lazy initialization of the in_channels argument of the ConvTranspose2d that is inferred from the input.size(1).
nn.LazyConvTranspose3d
A torch.nn.ConvTranspose3d module with lazy initialization of the in_channels argument of the ConvTranspose3d that is inferred from the input.size(1).
nn.Unfold
Extracts sliding local blocks from a batched input tensor.
nn.Fold
Combines an array of sliding local blocks into a large containing tensor.
nn.MaxPool1d
Applies a 1D max pooling over an input signal composed of several input planes.
nn.MaxPool2d
Applies a 2D max pooling over an input signal composed of several input planes.
nn.MaxPool3d
Applies a 3D max pooling over an input signal composed of several input planes.
nn.MaxUnpool1d
Computes a partial inverse of MaxPool1d.
nn.MaxUnpool2d
Computes a partial inverse of MaxPool2d.
nn.MaxUnpool3d
Computes a partial inverse of MaxPool3d.
nn.AvgPool1d
Applies a 1D average pooling over an input signal composed of several input planes.
nn.AvgPool2d
Applies a 2D average pooling over an input signal composed of several input planes.
nn.AvgPool3d
Applies a 3D average pooling over an input signal composed of several input planes.
nn.FractionalMaxPool2d
Applies a 2D fractional max pooling over an input signal composed of several input planes.
nn.LPPool1d
Applies a 1D power-average pooling over an input signal composed of several input planes.
nn.LPPool2d
Applies a 2D power-average pooling over an input signal composed of several input planes.
nn.AdaptiveMaxPool1d
Applies a 1D adaptive max pooling over an input signal composed of several input planes.
nn.AdaptiveMaxPool2d
Applies a 2D adaptive max pooling over an input signal composed of several input planes.
nn.AdaptiveMaxPool3d
Applies a 3D adaptive max pooling over an input signal composed of several input planes.
nn.AdaptiveAvgPool1d
Applies a 1D adaptive average pooling over an input signal composed of several input planes.
nn.AdaptiveAvgPool2d
Applies a 2D adaptive average pooling over an input signal composed of several input planes.
nn.AdaptiveAvgPool3d
Applies a 3D adaptive average pooling over an input signal composed of several input planes.
nn.ReflectionPad1d
Pads the input tensor using the reflection of the input boundary.
nn.ReflectionPad2d
Pads the input tensor using the reflection of the input boundary.
nn.ReplicationPad1d
Pads the input tensor using replication of the input boundary.
nn.ReplicationPad2d
Pads the input tensor using replication of the input boundary.
nn.ReplicationPad3d
Pads the input tensor using replication of the input boundary.
nn.ZeroPad2d
Pads the input tensor boundaries with zero.
nn.ConstantPad1d
Pads the input tensor boundaries with a constant value.
nn.ConstantPad2d
Pads the input tensor boundaries with a constant value.
nn.ConstantPad3d
Pads the input tensor boundaries with a constant value.
nn.ELU
Applies the element-wise function:
nn.Hardshrink
Applies the hard shrinkage function element-wise:
nn.Hardsigmoid
Applies the element-wise function:
nn.Hardtanh
Applies the HardTanh function element-wise
nn.Hardswish
Applies the hardswish function, element-wise, as described in the paper:
nn.LeakyReLU
Applies the element-wise function:
nn.LogSigmoid
Applies the element-wise function:
nn.MultiheadAttention
Allows the model to jointly attend to information from different representation subspaces.
nn.PReLU
Applies the element-wise function:
nn.ReLU
Applies the rectified linear unit function element-wise:
nn.ReLU6
Applies the element-wise function:
nn.RReLU
Applies the randomized leaky rectified liner unit function, element-wise, as described in the paper:
nn.SELU
Applied element-wise, as:
nn.CELU
Applies the element-wise function:
nn.GELU
Applies the Gaussian Error Linear Units function:
nn.Sigmoid
Applies the element-wise function:
nn.SiLU
Applies the silu function, element-wise.
nn.Softplus
Applies the element-wise function:
nn.Softshrink
Applies the soft shrinkage function elementwise:
nn.Softsign
Applies the element-wise function:
nn.Tanh
Applies the element-wise function:
nn.Tanhshrink
Applies the element-wise function:
nn.Threshold
Thresholds each element of the input Tensor.
nn.Softmin
Applies the Softmin function to an n-dimensional input Tensor rescaling them so that the elements of the n-dimensional output Tensor lie in the range [0, 1] and sum to 1.
nn.Softmax
Applies the Softmax function to an n-dimensional input Tensor rescaling them so that the elements of the n-dimensional output Tensor lie in the range [0,1] and sum to 1.
nn.Softmax2d
Applies SoftMax over features to each spatial location.
nn.LogSoftmax
Applies the log(Softmax(x))\log(\text{Softmax}(x))  function to an n-dimensional input Tensor.
nn.AdaptiveLogSoftmaxWithLoss
Efficient softmax approximation as described in Efficient softmax approximation for GPUs by Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou.
nn.BatchNorm1d
Applies Batch Normalization over a 2D or 3D input (a mini-batch of 1D inputs with optional additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
nn.BatchNorm2d
Applies Batch Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
nn.BatchNorm3d
Applies Batch Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
nn.GroupNorm
Applies Group Normalization over a mini-batch of inputs as described in the paper Group Normalization
nn.SyncBatchNorm
Applies Batch Normalization over a N-Dimensional input (a mini-batch of [N-2]D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
nn.InstanceNorm1d
Applies Instance Normalization over a 3D input (a mini-batch of 1D inputs with optional additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.
nn.InstanceNorm2d
Applies Instance Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.
nn.InstanceNorm3d
Applies Instance Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.
nn.LayerNorm
Applies Layer Normalization over a mini-batch of inputs as described in the paper Layer Normalization
nn.LocalResponseNorm
Applies local response normalization over an input signal composed of several input planes, where channels occupy the second dimension.
nn.RNNBase
nn.RNN
Applies a multi-layer Elman RNN with tanh\tanh  or ReLU\text{ReLU}  non-linearity to an input sequence.
nn.LSTM
Applies a multi-layer long short-term memory (LSTM) RNN to an input sequence.
nn.GRU
Applies a multi-layer gated recurrent unit (GRU) RNN to an input sequence.
nn.RNNCell
An Elman RNN cell with tanh or ReLU non-linearity.
nn.LSTMCell
A long short-term memory (LSTM) cell.
nn.GRUCell
A gated recurrent unit (GRU) cell
nn.Transformer
A transformer model.
nn.TransformerEncoder
TransformerEncoder is a stack of N encoder layers
nn.TransformerDecoder
TransformerDecoder is a stack of N decoder layers
nn.TransformerEncoderLayer
TransformerEncoderLayer is made up of self-attn and feedforward network.
nn.TransformerDecoderLayer
TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network.
nn.Identity
A placeholder identity operator that is argument-insensitive.
nn.Linear
Applies a linear transformation to the incoming data: y=xAT+by = xA^T + b
nn.Bilinear
Applies a bilinear transformation to the incoming data: y=x1TAx2+by = x_1^T A x_2 + b
nn.LazyLinear
A torch.nn.Linear module with lazy initialization.
nn.Dropout
During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution.
nn.Dropout2d
Randomly zero out entire channels (a channel is a 2D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 2D tensor input[i,j]\text{input}[i, j] ).
nn.Dropout3d
Randomly zero out entire channels (a channel is a 3D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 3D tensor input[i,j]\text{input}[i, j] ).
nn.AlphaDropout
Applies Alpha Dropout over the input.
nn.Embedding
A simple lookup table that stores embeddings of a fixed dictionary and size.
nn.EmbeddingBag
Computes sums or means of ‘bags’ of embeddings, without instantiating the intermediate embeddings.
nn.CosineSimilarity
Returns cosine similarity between x1x_1  and x2x_2 , computed along dim.
nn.PairwiseDistance
Computes the batchwise pairwise distance between vectors v1v_1 , v2v_2  using the p-norm:
nn.L1Loss
Creates a criterion that measures the mean absolute error (MAE) between each element in the input xx  and target yy .
nn.MSELoss
Creates a criterion that measures the mean squared error (squared L2 norm) between each element in the input xx  and target yy .
nn.CrossEntropyLoss
This criterion combines LogSoftmax and NLLLoss in one single class.
nn.CTCLoss
The Connectionist Temporal Classification loss.
nn.NLLLoss
The negative log likelihood loss.
nn.PoissonNLLLoss
Negative log likelihood loss with Poisson distribution of target.
nn.GaussianNLLLoss
Gaussian negative log likelihood loss.
nn.KLDivLoss
The Kullback-Leibler divergence loss measure
nn.BCELoss
Creates a criterion that measures the Binary Cross Entropy between the target and the output:
nn.BCEWithLogitsLoss
This loss combines a Sigmoid layer and the BCELoss in one single class.
nn.MarginRankingLoss
Creates a criterion that measures the loss given inputs x1x1 , x2x2 , two 1D mini-batch Tensors, and a label 1D mini-batch tensor yy  (containing 1 or -1).
nn.HingeEmbeddingLoss
Measures the loss given an input tensor xx  and a labels tensor yy  (containing 1 or -1).
nn.MultiLabelMarginLoss
Creates a criterion that optimizes a multi-class multi-classification hinge loss (margin-based loss) between input xx  (a 2D mini-batch Tensor) and output yy  (which is a 2D Tensor of target class indices).
nn.SmoothL1Loss
Creates a criterion that uses a squared term if the absolute element-wise error falls below beta and an L1 term otherwise.
nn.SoftMarginLoss
Creates a criterion that optimizes a two-class classification logistic loss between input tensor xx  and target tensor yy  (containing 1 or -1).
nn.MultiLabelSoftMarginLoss
Creates a criterion that optimizes a multi-label one-versus-all loss based on max-entropy, between input xx  and target yy  of size (N,C)(N, C) .
nn.CosineEmbeddingLoss
Creates a criterion that measures the loss given input tensors x1x_1 , x2x_2  and a Tensor label yy  with values 1 or -1.
nn.MultiMarginLoss
Creates a criterion that optimizes a multi-class classification hinge loss (margin-based loss) between input xx  (a 2D mini-batch Tensor) and output yy  (which is a 1D tensor of target class indices, 0≤y≤x.size(1)−10 \leq y \leq \text{x.size}(1)-1 ):
nn.TripletMarginLoss
Creates a criterion that measures the triplet loss given an input tensors x1x1 , x2x2 , x3x3  and a margin with a value greater than 00 .
nn.TripletMarginWithDistanceLoss
Creates a criterion that measures the triplet loss given input tensors aa , pp , and nn  (representing anchor, positive, and negative examples, respectively), and a nonnegative, real-valued function (“distance function”) used to compute the relationship between the anchor and positive example (“positive distance”) and the anchor and negative example (“negative distance”).
nn.PixelShuffle
Rearranges elements in a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W)  to a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r) , where r is an upscale factor.
nn.PixelUnshuffle
Reverses the PixelShuffle operation by rearranging elements in a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r)  to a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W) , where r is a downscale factor.
nn.Upsample
Upsamples a given multi-channel 1D (temporal), 2D (spatial) or 3D (volumetric) data.
nn.UpsamplingNearest2d
Applies a 2D nearest neighbor upsampling to an input signal composed of several input channels.
nn.UpsamplingBilinear2d
Applies a 2D bilinear upsampling to an input signal composed of several input channels.
nn.ChannelShuffle
Divide the channels in a tensor of shape (∗,C,H,W)(*, C , H, W)  into g groups and rearrange them as (∗,Cg,g,H,W)(*, C \frac g, g, H, W) , while keeping the original tensor shape.
nn.DataParallel
Implements data parallelism at the module level.
nn.parallel.DistributedDataParallel
Implements distributed data parallelism that is based on torch.distributed package at the module level.
From the torch.nn.utils module
Clips gradient norm of an iterable of parameters.
Clips gradient of an iterable of parameters at specified value.
Convert parameters to one vector
Convert one vector to the parameters
prune.BasePruningMethod
Abstract base class for creation of new pruning techniques.
prune.PruningContainer
Container holding a sequence of pruning methods for iterative pruning.
prune.Identity
Utility pruning method that does not prune any units but generates the pruning parametrization with a mask of ones.
prune.RandomUnstructured
Prune (currently unpruned) units in a tensor at random.
prune.L1Unstructured
Prune (currently unpruned) units in a tensor by zeroing out the ones with the lowest L1-norm.
prune.RandomStructured
Prune entire (currently unpruned) channels in a tensor at random.
prune.LnStructured
Prune entire (currently unpruned) channels in a tensor based on their Ln-norm.
prune.CustomFromMask
prune.identity
Applies pruning reparametrization to the tensor corresponding to the parameter called name in module without actually pruning any units.
prune.random_unstructured
Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) units selected at random.
prune.l1_unstructured
Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) units with the lowest L1-norm.
prune.random_structured
Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) channels along the specified dim selected at random.
prune.ln_structured
Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) channels along the specified dim with the lowest L``n``-norm.
prune.global_unstructured
Globally prunes tensors corresponding to all parameters in parameters by applying the specified pruning_method.
prune.custom_from_mask
Prunes tensor corresponding to parameter called name in module by applying the pre-computed mask in mask.
prune.remove
Removes the pruning reparameterization from a module and the pruning method from the forward hook.
prune.is_pruned
Check whether module is pruned by looking for forward_pre_hooks in its modules that inherit from the BasePruningMethod.
Applies weight normalization to a parameter in the given module.
Removes the weight normalization reparameterization from a module.
Applies spectral normalization to a parameter in the given module.
Removes the spectral normalization reparameterization from a module.
Utility functions in other modules
nn.utils.rnn.PackedSequence
Holds the data and list of batch_sizes of a packed sequence.
nn.utils.rnn.pack_padded_sequence
Packs a Tensor containing padded sequences of variable length.
nn.utils.rnn.pad_packed_sequence
Pads a packed batch of variable length sequences.
nn.utils.rnn.pad_sequence
Pad a list of variable length Tensors with padding_value
nn.utils.rnn.pack_sequence
Packs a list of variable length Tensors
nn.Flatten
Flattens a contiguous range of dims into a tensor.
nn.Unflatten
Unflattens a tensor dim expanding it to a desired shape.
Quantization refers to techniques for performing computations and storing tensors at lower bitwidths than floating point precision. PyTorch supports both per tensor and per channel asymmetric linear quantization. To learn more how to use quantized functions in PyTorch, please refer to the Quantization documentation.
nn.modules.lazy.LazyModuleMixin
A mixin for modules that lazily initialize parameters, also known as “lazy modules.”


Applies a 1D adaptive average pooling over an input signal composed of several input planes.
The output size is H, for any input size. The number of output features is equal to the number of input planes.
output_size – the target output size H


Applies a 2D adaptive average pooling over an input signal composed of several input planes.
The output is of size H x W, for any input size. The number of output features is equal to the number of input planes.
output_size – the target output size of the image of the form H x W. Can be a tuple (H, W) or a single H for a square image H x H. H and W can be either a int, or None which means the size will be the same as that of the input.


Applies a 3D adaptive average pooling over an input signal composed of several input planes.
The output is of size D x H x W, for any input size. The number of output features is equal to the number of input planes.
output_size – the target output size of the form D x H x W. Can be a tuple (D, H, W) or a single number D for a cube D x D x D. D, H and W can be either a int, or None which means the size will be the same as that of the input.


Efficient softmax approximation as described in Efficient softmax approximation for GPUs by Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou.
Adaptive softmax is an approximate strategy for training models with large output spaces. It is most effective when the label distribution is highly imbalanced, for example in natural language modelling, where the word frequency distribution approximately follows the Zipf’s law.
Adaptive softmax partitions the labels into several clusters, according to their frequency. These clusters may contain different number of targets each. Additionally, clusters containing less frequent labels assign lower dimensional embeddings to those labels, which speeds up the computation. For each minibatch, only clusters for which at least one target is present are evaluated.
The idea is that the clusters which are accessed frequently (like the first one, containing most frequent labels), should also be cheap to compute – that is, contain a small number of assigned labels.
We highly recommend taking a look at the original paper for more details.
Warning
Labels passed as inputs to this module should be sorted according to their frequency. This means that the most frequent label should be represented by the index 0, and the least frequent label should be represented by the index n_classes - 1.
Note
This module returns a NamedTuple with output and loss fields. See further documentation for details.
Note
To compute log-probabilities for all classes, the log_prob method can be used.
NamedTuple with output and loss fields
Computes log probabilities for all n_classes\texttt{n\_classes}
input (Tensor) – a minibatch of examples
log-probabilities of for each class cc  in range 0<=c<=n_classes0 <= c <= \texttt{n\_classes} , where n_classes\texttt{n\_classes}  is a parameter passed to AdaptiveLogSoftmaxWithLoss constructor.
This is equivalent to self.log_pob(input).argmax(dim=1), but is more efficient in some cases.
input (Tensor) – a minibatch of examples
a class with the highest probability for each example
output (Tensor)


Computes log probabilities for all n_classes\texttt{n\_classes}
input (Tensor) – a minibatch of examples
log-probabilities of for each class cc  in range 0<=c<=n_classes0 <= c <= \texttt{n\_classes} , where n_classes\texttt{n\_classes}  is a parameter passed to AdaptiveLogSoftmaxWithLoss constructor.


This is equivalent to self.log_pob(input).argmax(dim=1), but is more efficient in some cases.
input (Tensor) – a minibatch of examples
a class with the highest probability for each example
output (Tensor)


Applies a 1D adaptive max pooling over an input signal composed of several input planes.
The output size is H, for any input size. The number of output features is equal to the number of input planes.


Applies a 2D adaptive max pooling over an input signal composed of several input planes.
The output is of size H x W, for any input size. The number of output features is equal to the number of input planes.


Applies a 3D adaptive max pooling over an input signal composed of several input planes.
The output is of size D x H x W, for any input size. The number of output features is equal to the number of input planes.


Applies Alpha Dropout over the input.
Alpha Dropout is a type of Dropout that maintains the self-normalizing property. For an input with zero mean and unit standard deviation, the output of Alpha Dropout maintains the original mean and standard deviation of the input. Alpha Dropout goes hand-in-hand with SELU activation function, which ensures that the outputs have zero mean and unit standard deviation.
During training, it randomly masks some of the elements of the input tensor with probability p using samples from a bernoulli distribution. The elements to masked are randomized on every forward call, and scaled and shifted to maintain zero mean and unit standard deviation.
During evaluation the module simply computes an identity function.
More details can be found in the paper Self-Normalizing Neural Networks .
Examples:


Applies a 1D average pooling over an input signal composed of several input planes.
In the simplest case, the output value of the layer with input size (N,C,L)(N, C, L) , output (N,C,Lout)(N, C, L_{out})  and kernel_size kk  can be precisely described as:
If padding is non-zero, then the input is implicitly zero-padded on both sides for padding number of points.
Note
When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored.
The parameters kernel_size, stride, padding can each be an int or a one-element tuple.
Output: (N,C,Lout)(N, C, L_{out}) , where
Examples:


Applies a 2D average pooling over an input signal composed of several input planes.
In the simplest case, the output value of the layer with input size (N,C,H,W)(N, C, H, W) , output (N,C,Hout,Wout)(N, C, H_{out}, W_{out})  and kernel_size (kH,kW)(kH, kW)  can be precisely described as:
If padding is non-zero, then the input is implicitly zero-padded on both sides for padding number of points.
Note
When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored.
The parameters kernel_size, stride, padding can either be:
Output: (N,C,Hout,Wout)(N, C, H_{out}, W_{out}) , where
Examples:


Applies a 3D average pooling over an input signal composed of several input planes.
In the simplest case, the output value of the layer with input size (N,C,D,H,W)(N, C, D, H, W) , output (N,C,Dout,Hout,Wout)(N, C, D_{out}, H_{out}, W_{out})  and kernel_size (kD,kH,kW)(kD, kH, kW)  can be precisely described as:
If padding is non-zero, then the input is implicitly zero-padded on all three sides for padding number of points.
Note
When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored.
The parameters kernel_size, stride can either be:
Output: (N,C,Dout,Hout,Wout)(N, C, D_{out}, H_{out}, W_{out}) , where
Examples:


Applies Batch Normalization over a 2D or 3D input (a mini-batch of 1D inputs with optional additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
The mean and standard-deviation are calculated per-dimension over the mini-batches and γ\gamma  and β\beta  are learnable parameter vectors of size C (where C is the input size). By default, the elements of γ\gamma  are set to 1 and the elements of β\beta  are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).
Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.
If track_running_stats is set to False, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.
Note
This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x^new=(1−momentum)×x^+momentum×xt\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t , where x^\hat{x}  is the estimated statistic and xtx_t  is the new observed value.
Because the Batch Normalization is done over the C dimension, computing statistics on (N, L) slices, it’s common terminology to call this Temporal Batch Normalization.
Examples:


Applies Batch Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
The mean and standard-deviation are calculated per-dimension over the mini-batches and γ\gamma  and β\beta  are learnable parameter vectors of size C (where C is the input size). By default, the elements of γ\gamma  are set to 1 and the elements of β\beta  are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).
Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.
If track_running_stats is set to False, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.
Note
This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x^new=(1−momentum)×x^+momentum×xt\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t , where x^\hat{x}  is the estimated statistic and xtx_t  is the new observed value.
Because the Batch Normalization is done over the C dimension, computing statistics on (N, H, W) slices, it’s common terminology to call this Spatial Batch Normalization.
Examples:


Applies Batch Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
The mean and standard-deviation are calculated per-dimension over the mini-batches and γ\gamma  and β\beta  are learnable parameter vectors of size C (where C is the input size). By default, the elements of γ\gamma  are set to 1 and the elements of β\beta  are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).
Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.
If track_running_stats is set to False, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.
Note
This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x^new=(1−momentum)×x^+momentum×xt\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t , where x^\hat{x}  is the estimated statistic and xtx_t  is the new observed value.
Because the Batch Normalization is done over the C dimension, computing statistics on (N, D, H, W) slices, it’s common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization.
Examples:


Creates a criterion that measures the Binary Cross Entropy between the target and the output:
The unreduced (i.e. with reduction set to 'none') loss can be described as:
where NN  is the batch size. If reduction is not 'none' (default 'mean'), then
This is used for measuring the error of a reconstruction in for example an auto-encoder. Note that the targets yy  should be numbers between 0 and 1.
Notice that if xnx_n  is either 0 or 1, one of the log terms would be mathematically undefined in the above loss equation. PyTorch chooses to set log(0)=−∞\log (0) = -\infty , since limx→0log(x)=−∞\lim_{x\to 0} \log (x) = -\infty . However, an infinite term in the loss equation is not desirable for several reasons.
For one, if either yn=0y_n = 0  or (1−yn)=0(1 - y_n) = 0 , then we would be multiplying 0 with infinity. Secondly, if we have an infinite loss value, then we would also have an infinite term in our gradient, since limx→0ddxlog(x)=∞\lim_{x\to 0} \frac{d}{dx} \log (x) = \infty . This would make BCELoss’s backward method nonlinear with respect to xnx_n , and using it for things like linear regression would not be straight-forward.
Our solution is that BCELoss clamps its log function outputs to be greater than or equal to -100. This way, we can always have a finite loss value and a linear backward method.
Examples:


This loss combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.
The unreduced (i.e. with reduction set to 'none') loss can be described as:
where NN  is the batch size. If reduction is not 'none' (default 'mean'), then
This is used for measuring the error of a reconstruction in for example an auto-encoder. Note that the targets t[i] should be numbers between 0 and 1.
It’s possible to trade off recall and precision by adding weights to positive examples. In the case of multi-label classification the loss can be described as:
where cc  is the class number (c>1c > 1  for multi-label binary classification, c=1c = 1  for single-label binary classification), nn  is the number of the sample in the batch and pcp_c  is the weight of the positive answer for the class cc .
pc>1p_c > 1  increases the recall, pc<1p_c < 1  increases the precision.
For example, if a dataset contains 100 positive and 300 negative examples of a single class, then pos_weight for the class should be equal to 300100=3\frac{300}{100}=3 . The loss would act as if the dataset contains 3×100=3003\times 100=300  positive examples.
Examples:
Examples:


Applies a bilinear transformation to the incoming data: y=x1TAx2+by = x_1^T A x_2 + b
Examples:


Applies the element-wise function:
More details can be found in the paper Continuously Differentiable Exponential Linear Units .
Examples:


Divide the channels in a tensor of shape (∗,C,H,W)(*, C , H, W)  into g groups and rearrange them as (∗,Cg,g,H,W)(*, C \frac g, g, H, W) , while keeping the original tensor shape.
groups (int) – number of groups to divide channels in.
Examples:


Pads the input tensor boundaries with a constant value.
For N-dimensional padding, use torch.nn.functional.pad().
padding (int, tuple) – the size of the padding. If is int, uses the same padding in both boundaries. If a 2-tuple, uses (padding_left\text{padding\_left} , padding_right\text{padding\_right} )
Output: (N,C,Wout)(N, C, W_{out})  where
Wout=Win+padding_left+padding_rightW_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}
Examples:


Pads the input tensor boundaries with a constant value.
For N-dimensional padding, use torch.nn.functional.pad().
padding (int, tuple) – the size of the padding. If is int, uses the same padding in all boundaries. If a 4-tuple, uses (padding_left\text{padding\_left} , padding_right\text{padding\_right} , padding_top\text{padding\_top} , padding_bottom\text{padding\_bottom} )
Output: (N,C,Hout,Wout)(N, C, H_{out}, W_{out})  where
Hout=Hin+padding_top+padding_bottomH_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}
Wout=Win+padding_left+padding_rightW_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}
Examples:


Pads the input tensor boundaries with a constant value.
For N-dimensional padding, use torch.nn.functional.pad().
padding (int, tuple) – the size of the padding. If is int, uses the same padding in all boundaries. If a 6-tuple, uses (padding_left\text{padding\_left} , padding_right\text{padding\_right} , padding_top\text{padding\_top} , padding_bottom\text{padding\_bottom} , padding_front\text{padding\_front} , padding_back\text{padding\_back} )
Output: (N,C,Dout,Hout,Wout)(N, C, D_{out}, H_{out}, W_{out})  where
Dout=Din+padding_front+padding_backD_{out} = D_{in} + \text{padding\_front} + \text{padding\_back}
Hout=Hin+padding_top+padding_bottomH_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}
Wout=Win+padding_left+padding_rightW_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}
Examples:


Applies a 1D convolution over an input signal composed of several input planes.
In the simplest case, the output value of the layer with input size (N,Cin,L)(N, C_{\text{in}}, L)  and output (N,Cout,Lout)(N, C_{\text{out}}, L_{\text{out}})  can be precisely described as:
where ⋆\star  is the valid cross-correlation operator, NN  is a batch size, CC  denotes a number of channels, LL  is a length of signal sequence.
This module supports TensorFloat32.
groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,
Note
When groups == in_channels and out_channels == K * in_channels, where K is a positive integer, this operation is also known as a “depthwise convolution”.
In other words, for an input of size (N,Cin,Lin)(N, C_{in}, L_{in}) , a depthwise convolution with a depthwise multiplier K can be performed with the arguments (Cin=Cin,Cout=Cin×K,...,groups=Cin)(C_\text{in}=C_\text{in}, C_\text{out}=C_\text{in} \times \text{K}, ..., \text{groups}=C_\text{in}) .
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Output: (N,Cout,Lout)(N, C_{out}, L_{out})  where
Examples:


Applies a 2D convolution over an input signal composed of several input planes.
In the simplest case, the output value of the layer with input size (N,Cin,H,W)(N, C_{\text{in}}, H, W)  and output (N,Cout,Hout,Wout)(N, C_{\text{out}}, H_{\text{out}}, W_{\text{out}})  can be precisely described as:
where ⋆\star  is the valid 2D cross-correlation operator, NN  is a batch size, CC  denotes a number of channels, HH  is a height of input planes in pixels, and WW  is width in pixels.
This module supports TensorFloat32.
groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,
The parameters kernel_size, stride, padding, dilation can either be:
Note
When groups == in_channels and out_channels == K * in_channels, where K is a positive integer, this operation is also known as a “depthwise convolution”.
In other words, for an input of size (N,Cin,Lin)(N, C_{in}, L_{in}) , a depthwise convolution with a depthwise multiplier K can be performed with the arguments (Cin=Cin,Cout=Cin×K,...,groups=Cin)(C_\text{in}=C_\text{in}, C_\text{out}=C_\text{in} \times \text{K}, ..., \text{groups}=C_\text{in}) .
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Output: (N,Cout,Hout,Wout)(N, C_{out}, H_{out}, W_{out})  where


Applies a 3D convolution over an input signal composed of several input planes.
In the simplest case, the output value of the layer with input size (N,Cin,D,H,W)(N, C_{in}, D, H, W)  and output (N,Cout,Dout,Hout,Wout)(N, C_{out}, D_{out}, H_{out}, W_{out})  can be precisely described as:
where ⋆\star  is the valid 3D cross-correlation operator
This module supports TensorFloat32.
groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,
The parameters kernel_size, stride, padding, dilation can either be:
Note
When groups == in_channels and out_channels == K * in_channels, where K is a positive integer, this operation is also known as a “depthwise convolution”.
In other words, for an input of size (N,Cin,Lin)(N, C_{in}, L_{in}) , a depthwise convolution with a depthwise multiplier K can be performed with the arguments (Cin=Cin,Cout=Cin×K,...,groups=Cin)(C_\text{in}=C_\text{in}, C_\text{out}=C_\text{in} \times \text{K}, ..., \text{groups}=C_\text{in}) .
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Output: (N,Cout,Dout,Hout,Wout)(N, C_{out}, D_{out}, H_{out}, W_{out})  where
Examples:


Applies a 1D transposed convolution operator over an input image composed of several input planes.
This module can be seen as the gradient of Conv1d with respect to its input. It is also known as a fractionally-strided convolution or a deconvolution (although it is not an actual deconvolution operation).
This module supports TensorFloat32.
groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,
Note
The padding argument effectively adds dilation * (kernel_size - 1) - padding amount of zero padding to both sizes of the input. This is set so that when a Conv1d and a ConvTranspose1d are initialized with same parameters, they are inverses of each other in regard to the input and output shapes. However, when stride > 1, Conv1d maps multiple input shapes to the same output shape. output_padding is provided to resolve this ambiguity by effectively increasing the calculated output shape on one side. Note that output_padding is only used to find output shape, but does not actually add zero-padding to output.
Note
In some circumstances when using the CUDA backend with CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic =
True. Please see the notes on Reproducibility for background.
Output: (N,Cout,Lout)(N, C_{out}, L_{out})  where


Applies a 2D transposed convolution operator over an input image composed of several input planes.
This module can be seen as the gradient of Conv2d with respect to its input. It is also known as a fractionally-strided convolution or a deconvolution (although it is not an actual deconvolution operation).
This module supports TensorFloat32.
groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,
The parameters kernel_size, stride, padding, output_padding can either be:
Note
The padding argument effectively adds dilation * (kernel_size - 1) - padding amount of zero padding to both sizes of the input. This is set so that when a Conv2d and a ConvTranspose2d are initialized with same parameters, they are inverses of each other in regard to the input and output shapes. However, when stride > 1, Conv2d maps multiple input shapes to the same output shape. output_padding is provided to resolve this ambiguity by effectively increasing the calculated output shape on one side. Note that output_padding is only used to find output shape, but does not actually add zero-padding to output.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:


Applies a 3D transposed convolution operator over an input image composed of several input planes. The transposed convolution operator multiplies each input value element-wise by a learnable kernel, and sums over the outputs from all input feature planes.
This module can be seen as the gradient of Conv3d with respect to its input. It is also known as a fractionally-strided convolution or a deconvolution (although it is not an actual deconvolution operation).
This module supports TensorFloat32.
groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,
The parameters kernel_size, stride, padding, output_padding can either be:
Note
The padding argument effectively adds dilation * (kernel_size - 1) - padding amount of zero padding to both sizes of the input. This is set so that when a Conv3d and a ConvTranspose3d are initialized with same parameters, they are inverses of each other in regard to the input and output shapes. However, when stride > 1, Conv3d maps multiple input shapes to the same output shape. output_padding is provided to resolve this ambiguity by effectively increasing the calculated output shape on one side. Note that output_padding is only used to find output shape, but does not actually add zero-padding to output.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:


Creates a criterion that measures the loss given input tensors x1x_1 , x2x_2  and a Tensor label yy  with values 1 or -1. This is used for measuring whether two inputs are similar or dissimilar, using the cosine distance, and is typically used for learning nonlinear embeddings or semi-supervised learning.
The loss function for each sample is:


Returns cosine similarity between x1x_1  and x2x_2 , computed along dim.


This criterion combines LogSoftmax and NLLLoss in one single class.
It is useful when training a classification problem with C classes. If provided, the optional argument weight should be a 1D Tensor assigning weight to each of the classes. This is particularly useful when you have an unbalanced training set.
The input is expected to contain raw, unnormalized scores for each class.
input has to be a Tensor of size either (minibatch,C)(minibatch, C)  or (minibatch,C,d1,d2,...,dK)(minibatch, C, d_1, d_2, ..., d_K)  with K≥1K \geq 1  for the K-dimensional case (described later).
This criterion expects a class index in the range [0,C−1][0, C-1]  as the target for each value of a 1D tensor of size minibatch; if ignore_index is specified, this criterion also accepts this class index (this index may not necessarily be in the class range).
The loss can be described as:
or in the case of the weight argument being specified:
The losses are averaged across observations for each minibatch. If the weight argument is specified then this is a weighted average:
Can also be used for higher dimension inputs, such as 2D images, by providing an input of size (minibatch,C,d1,d2,...,dK)(minibatch, C, d_1, d_2, ..., d_K)  with K≥1K \geq 1 , where KK  is the number of dimensions, and a target of appropriate shape (see below).
Examples:


The Connectionist Temporal Classification loss.
Calculates loss between a continuous (unsegmented) time series and a target sequence. CTCLoss sums over the probability of possible alignments of input to target, producing a loss value which is differentiable with respect to each input node. The alignment of input to target is assumed to be “many-to-one”, which limits the length of the target sequence such that it must be ≤\leq  the input length.
Examples:
A. Graves et al.: Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks: https://www.cs.toronto.edu/~graves/icml_2006.pdf
Note
In order to use CuDNN, the following must be satisfied: targets must be in concatenated format, all input_lengths must be T. blank=0blank=0 , target_lengths ≤256\leq 256 , the integer arguments must be of dtype torch.int32.
The regular implementation uses the (more common in PyTorch) torch.long dtype.
Note
In some circumstances when using the CUDA backend with CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic =
True. Please see the notes on Reproducibility for background.


Implements data parallelism at the module level.
This container parallelizes the application of the given module by splitting the input across the specified devices by chunking in the batch dimension (other objects will be copied once per device). In the forward pass, the module is replicated on each device, and each replica handles a portion of the input. During the backwards pass, gradients from each replica are summed into the original module.
The batch size should be larger than the number of GPUs used.
Warning
It is recommended to use DistributedDataParallel, instead of this class, to do multi-GPU training, even if there is only a single node. See: Use nn.parallel.DistributedDataParallel instead of multiprocessing or nn.DataParallel and Distributed Data Parallel.
Arbitrary positional and keyword inputs are allowed to be passed into DataParallel but some types are specially handled. tensors will be scattered on dim specified (default 0). tuple, list and dict types will be shallow copied. The other types will be shared among different threads and can be corrupted if written to in the model’s forward pass.
The parallelized module must have its parameters and buffers on device_ids[0] before running this DataParallel module.
Warning
In each forward, module is replicated on each device, so any updates to the running module in forward will be lost. For example, if module has a counter attribute that is incremented in each forward, it will always stay at the initial value because the update is done on the replicas which are destroyed after forward. However, DataParallel guarantees that the replica on device[0] will have its parameters and buffers sharing storage with the base parallelized module. So in-place updates to the parameters or buffers on device[0] will be recorded. E.g., BatchNorm2d and spectral_norm() rely on this behavior to update the buffers.
Warning
Forward and backward hooks defined on module and its submodules will be invoked len(device_ids) times, each with inputs located on a particular device. Particularly, the hooks are only guaranteed to be executed in correct order with respect to operations on corresponding devices. For example, it is not guaranteed that hooks set via register_forward_pre_hook() be executed before all len(device_ids) forward() calls, but that each such hook be executed before the corresponding forward() call of that device.
Warning
When module returns a scalar (i.e., 0-dimensional tensor) in forward(), this wrapper will return a vector of length equal to number of devices used in data parallelism, containing the result from each device.
Note
There is a subtlety in using the pack sequence -> recurrent network -> unpack sequence pattern in a Module wrapped in DataParallel. See My recurrent network doesn’t work with data parallelism section in FAQ for details.
~DataParallel.module (Module) – the module to be parallelized
Example:


During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.
This has proven to be an effective technique for regularization and preventing the co-adaptation of neurons as described in the paper Improving neural networks by preventing co-adaptation of feature detectors .
Furthermore, the outputs are scaled by a factor of 11−p\frac{1}{1-p}  during training. This means that during evaluation the module simply computes an identity function.
Examples:


Randomly zero out entire channels (a channel is a 2D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 2D tensor input[i,j]\text{input}[i, j] ). Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.
Usually the input comes from nn.Conv2d modules.
As described in the paper Efficient Object Localization Using Convolutional Networks , if adjacent pixels within feature maps are strongly correlated (as is normally the case in early convolution layers) then i.i.d. dropout will not regularize the activations and will otherwise just result in an effective learning rate decrease.
In this case, nn.Dropout2d() will help promote independence between feature maps and should be used instead.
Examples:


Randomly zero out entire channels (a channel is a 3D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 3D tensor input[i,j]\text{input}[i, j] ). Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.
Usually the input comes from nn.Conv3d modules.
As described in the paper Efficient Object Localization Using Convolutional Networks , if adjacent pixels within feature maps are strongly correlated (as is normally the case in early convolution layers) then i.i.d. dropout will not regularize the activations and will otherwise just result in an effective learning rate decrease.
In this case, nn.Dropout3d() will help promote independence between feature maps and should be used instead.
Examples:


Applies the element-wise function:
Examples:


A simple lookup table that stores embeddings of a fixed dictionary and size.
This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.
~Embedding.weight (Tensor) – the learnable weights of the module of shape (num_embeddings, embedding_dim) initialized from N(0,1)\mathcal{N}(0, 1)
Note
Keep in mind that only a limited number of optimizers support sparse gradients: currently it’s optim.SGD (CUDA and CPU), optim.SparseAdam (CUDA and CPU) and optim.Adagrad (CPU)
Note
With padding_idx set, the embedding vector at padding_idx is initialized to all zeros. However, note that this vector can be modified afterwards, e.g., using a customized initialization method, and thus changing the vector used to pad the output. The gradient for this vector from Embedding is always zero.
Note
When max_norm is not None, Embedding’s forward method will modify the weight tensor in-place. Since tensors needed for gradient computations cannot be modified in-place, performing a differentiable operation on Embedding.weight before calling Embedding’s forward method requires cloning Embedding.weight when max_norm is not None. For example:
Examples:
Creates Embedding instance from given 2-dimensional FloatTensor.
Examples:


Creates Embedding instance from given 2-dimensional FloatTensor.
Examples:


Computes sums or means of ‘bags’ of embeddings, without instantiating the intermediate embeddings.
For bags of constant length and no per_sample_weights and 2D inputs, this class
However, EmbeddingBag is much more time and memory efficient than using a chain of these operations.
EmbeddingBag also supports per-sample weights as an argument to the forward pass. This scales the output of the Embedding before performing a weighted reduction as specified by mode. If per_sample_weights` is passed, the only supported mode is "sum", which computes a weighted sum according to per_sample_weights.
~EmbeddingBag.weight (Tensor) – the learnable weights of the module of shape (num_embeddings, embedding_dim) initialized from N(0,1)\mathcal{N}(0, 1) .
per_index_weights (Tensor, optional)
If input is 2D of shape (B, N),
it will be treated as B bags (sequences) each of fixed length N, and this will return B values aggregated in a way depending on the mode. offsets is ignored and required to be None in this case.
If input is 1D of shape (N),
it will be treated as a concatenation of multiple bags (sequences). offsets is required to be a 1D tensor containing the starting index positions of each bag in input. Therefore, for offsets of shape (B), input will be viewed as having B bags. Empty bags (i.e., having 0-length) will have returned vectors filled by zeros.
to indicate all weights should be taken to be 1. If specified, per_sample_weights must have exactly the same shape as input and is treated as having the same offsets, if those are not None. Only supported for mode='sum'.
Output shape: (B, embedding_dim)
Examples:
Creates EmbeddingBag instance from given 2-dimensional FloatTensor.
Examples:


Creates EmbeddingBag instance from given 2-dimensional FloatTensor.
Examples:


Flattens a contiguous range of dims into a tensor. For use with Sequential.
Adds a child module to the current module.
The module can be accessed as an attribute using the given name.
Applies fn recursively to every submodule (as returned by .children()) as well as self. Typical use includes initializing the parameters of a model (see also torch.nn.init).
fn (Module -> None) – function to be applied to each submodule
self
Module
Example:
Casts all floating point parameters and buffers to bfloat16 datatype.
self
Module
Returns an iterator over module buffers.
recurse (bool) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module.
torch.Tensor – module buffer
Example:
Returns an iterator over immediate children modules.
Module – a child module
Moves all model parameters and buffers to the CPU.
self
Module
Moves all model parameters and buffers to the GPU.
This also makes associated parameters and buffers different objects. So it should be called before constructing optimizer if the module will live on GPU while being optimized.
device (int, optional) – if specified, all parameters will be copied to that device
self
Module
Casts all floating point parameters and buffers to double datatype.
self
Module
Sets the module in evaluation mode.
This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.
This is equivalent with self.train(False).
self
Module
Casts all floating point parameters and buffers to float datatype.
self
Module
Casts all floating point parameters and buffers to half datatype.
self
Module
Copies parameters and buffers from state_dict into this module and its descendants. If strict is True, then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function.
NamedTuple with missing_keys and unexpected_keys fields
Returns an iterator over all modules in the network.
Module – a module in the network
Note
Duplicate modules are returned only once. In the following example, l will be returned only once.
Example:
Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
(string, torch.Tensor) – Tuple containing the name and buffer
Example:
Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
(string, Module) – Tuple containing a name and child module
Example:
Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
(string, Module) – Tuple of name and module
Note
Duplicate modules are returned only once. In the following example, l will be returned only once.
Example:
Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
(string, Parameter) – Tuple containing the name and parameter
Example:
Returns an iterator over module parameters.
This is typically passed to an optimizer.
recurse (bool) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module.
Parameter – module parameter
Example:
Registers a backward hook on the module.
This function is deprecated in favor of nn.Module.register_full_backward_hook() and the behavior of this function will change in future versions.
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle
Adds a buffer to the module.
This is typically used to register a buffer that should not to be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False. The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict.
Buffers can be accessed as attributes using given names.
Example:
Registers a forward hook on the module.
The hook will be called every time after forward() has computed an output. It should have the following signature:
The input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward. The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called.
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle
Registers a forward pre-hook on the module.
The hook will be called every time before forward() is invoked. It should have the following signature:
The input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward. The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned(unless that value is already a tuple).
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle
Registers a backward hook on the module.
The hook will be called every time the gradients with respect to module inputs are computed. The hook should have the following signature:
The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments.
Warning
Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error.
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle
Adds a parameter to the module.
The parameter can be accessed as an attribute using given name.
Change if autograd should record operations on parameters in this module.
This method sets the parameters’ requires_grad attributes in-place.
This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training).
requires_grad (bool) – whether autograd should record operations on parameters in this module. Default: True.
self
Module
Returns a dictionary containing a whole state of the module.
Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names.
a dictionary containing a whole state of the module
dict
Example:
Moves and/or casts the parameters and buffers.
This can be called as
Its signature is similar to torch.Tensor.to(), but only accepts floating point or complex dtype`s. In addition, this method will
only cast the floating point or complex parameters and buffers to :attr:`dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.
See below for examples.
Note
This method modifies the module in-place.
self
Module
Examples:
Sets the module in training mode.
This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.
mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.
self
Module
Casts all parameters and buffers to dst_type.
dst_type (type or string) – the desired type
self
Module
Moves all model parameters and buffers to the XPU.
This also makes associated parameters and buffers different objects. So it should be called before constructing optimizer if the module will live on XPU while being optimized.
device (int, optional) – if specified, all parameters will be copied to that device
self
Module
Sets gradients of all model parameters to zero. See similar function under torch.optim.Optimizer for more context.
set_to_none (bool) – instead of setting to zero, set the grads to None. See torch.optim.Optimizer.zero_grad() for details.


Adds a child module to the current module.
The module can be accessed as an attribute using the given name.


Applies fn recursively to every submodule (as returned by .children()) as well as self. Typical use includes initializing the parameters of a model (see also torch.nn.init).
fn (Module -> None) – function to be applied to each submodule
self
Module
Example:


Casts all floating point parameters and buffers to bfloat16 datatype.
self
Module


Returns an iterator over module buffers.
recurse (bool) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module.
torch.Tensor – module buffer
Example:


Returns an iterator over immediate children modules.
Module – a child module


Moves all model parameters and buffers to the CPU.
self
Module


Moves all model parameters and buffers to the GPU.
This also makes associated parameters and buffers different objects. So it should be called before constructing optimizer if the module will live on GPU while being optimized.
device (int, optional) – if specified, all parameters will be copied to that device
self
Module


Casts all floating point parameters and buffers to double datatype.
self
Module


Sets the module in evaluation mode.
This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.
This is equivalent with self.train(False).
self
Module


Casts all floating point parameters and buffers to float datatype.
self
Module


Casts all floating point parameters and buffers to half datatype.
self
Module


Copies parameters and buffers from state_dict into this module and its descendants. If strict is True, then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function.
NamedTuple with missing_keys and unexpected_keys fields


Returns an iterator over all modules in the network.
Module – a module in the network
Note
Duplicate modules are returned only once. In the following example, l will be returned only once.
Example:


Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
(string, torch.Tensor) – Tuple containing the name and buffer
Example:


Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
(string, Module) – Tuple containing a name and child module
Example:


Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
(string, Module) – Tuple of name and module
Note
Duplicate modules are returned only once. In the following example, l will be returned only once.
Example:


Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
(string, Parameter) – Tuple containing the name and parameter
Example:


Returns an iterator over module parameters.
This is typically passed to an optimizer.
recurse (bool) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module.
Parameter – module parameter
Example:


Registers a backward hook on the module.
This function is deprecated in favor of nn.Module.register_full_backward_hook() and the behavior of this function will change in future versions.
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle


Adds a buffer to the module.
This is typically used to register a buffer that should not to be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False. The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict.
Buffers can be accessed as attributes using given names.
Example:


Registers a forward hook on the module.
The hook will be called every time after forward() has computed an output. It should have the following signature:
The input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward. The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called.
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle


Registers a forward pre-hook on the module.
The hook will be called every time before forward() is invoked. It should have the following signature:
The input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward. The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned(unless that value is already a tuple).
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle


Registers a backward hook on the module.
The hook will be called every time the gradients with respect to module inputs are computed. The hook should have the following signature:
The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments.
Warning
Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error.
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle


Adds a parameter to the module.
The parameter can be accessed as an attribute using given name.


Change if autograd should record operations on parameters in this module.
This method sets the parameters’ requires_grad attributes in-place.
This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training).
requires_grad (bool) – whether autograd should record operations on parameters in this module. Default: True.
self
Module


Returns a dictionary containing a whole state of the module.
Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names.
a dictionary containing a whole state of the module
dict
Example:


Moves and/or casts the parameters and buffers.
This can be called as
Its signature is similar to torch.Tensor.to(), but only accepts floating point or complex dtype`s. In addition, this method will
only cast the floating point or complex parameters and buffers to :attr:`dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.
See below for examples.
Note
This method modifies the module in-place.
self
Module
Examples:


Sets the module in training mode.
This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.
mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.
self
Module


Casts all parameters and buffers to dst_type.
dst_type (type or string) – the desired type
self
Module


Moves all model parameters and buffers to the XPU.
This also makes associated parameters and buffers different objects. So it should be called before constructing optimizer if the module will live on XPU while being optimized.
device (int, optional) – if specified, all parameters will be copied to that device
self
Module


Sets gradients of all model parameters to zero. See similar function under torch.optim.Optimizer for more context.
set_to_none (bool) – instead of setting to zero, set the grads to None. See torch.optim.Optimizer.zero_grad() for details.


Combines an array of sliding local blocks into a large containing tensor.
Consider a batched input tensor containing sliding local blocks, e.g., patches of images, of shape (N,C×∏(kernel_size),L)(N, C \times \prod(\text{kernel\_size}), L) , where NN  is batch dimension, C×∏(kernel_size)C \times \prod(\text{kernel\_size})  is the number of values within a block (a block has ∏(kernel_size)\prod(\text{kernel\_size})  spatial locations each containing a CC -channeled vector), and LL  is the total number of blocks. (This is exactly the same specification as the output shape of Unfold.) This operation combines these local blocks into the large output tensor of shape (N,C,output_size[0],output_size[1],…)(N, C, \text{output\_size}[0], \text{output\_size}[1], \dots)  by summing the overlapping values. Similar to Unfold, the arguments must satisfy
where dd  is over all spatial dimensions.
The padding, stride and dilation arguments specify how the sliding blocks are retrieved.
Note
Fold calculates each combined value in the resulting large tensor by summing all values from all containing blocks. Unfold extracts the values in the local blocks by copying from the large tensor. So, if the blocks overlap, they are not inverses of each other.
In general, folding and unfolding operations are related as follows. Consider Fold and Unfold instances created with the same parameters:
Then for any (supported) input tensor the following equality holds:
where divisor is a tensor that depends only on the shape and dtype of the input:
When the divisor tensor contains no zero elements, then fold and unfold operations are inverses of each other (up to constant divisor).
Warning
Currently, only 4-D output tensors (batched image-like tensors) are supported.
Examples:


Applies a 2D fractional max pooling over an input signal composed of several input planes.
Fractional MaxPooling is described in detail in the paper Fractional MaxPooling by Ben Graham
The max-pooling operation is applied in kH×kWkH \times kW  regions by a stochastic step size determined by the target output size. The number of output features is equal to the number of input planes.