Applies a 1D convolution over an input signal composed of several input planes.
This operator supports TensorFloat32.
See Conv1d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:
Applies a 2D convolution over an input image composed of several input planes.
This operator supports TensorFloat32.
See Conv2d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:
Applies a 3D convolution over an input image composed of several input planes.
This operator supports TensorFloat32.
See Conv3d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:
Applies a 1D transposed convolution operator over an input signal composed of several input planes, sometimes also called “deconvolution”.
This operator supports TensorFloat32.
See ConvTranspose1d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:
Applies a 2D transposed convolution operator over an input image composed of several input planes, sometimes also called “deconvolution”.
This operator supports TensorFloat32.
See ConvTranspose2d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:
Applies a 3D transposed convolution operator over an input image composed of several input planes, sometimes also called “deconvolution”
This operator supports TensorFloat32.
See ConvTranspose3d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:
Extracts sliding local blocks from a batched input tensor.
Warning
Currently, only 4-D input tensors (batched image-like tensors) are supported.
Warning
More than one element of the unfolded tensor may refer to a single memory location. As a result, in-place operations (especially ones that are vectorized) may result in incorrect behavior. If you need to write to the tensor, please clone it first.
See torch.nn.Unfold for details
Combines an array of sliding local blocks into a large containing tensor.
Warning
Currently, only 3-D output tensors (unfolded batched image-like tensors) are supported.
See torch.nn.Fold for details
Applies a 1D average pooling over an input signal composed of several input planes.
See AvgPool1d for details and output shape.
Examples:
Applies 2D average-pooling operation in kH×kWkH \times kW  regions by step size sH×sWsH \times sW  steps. The number of output features is equal to the number of input planes.
See AvgPool2d for details and output shape.
Applies 3D average-pooling operation in kT×kH×kWkT \times kH \times kW  regions by step size sT×sH×sWsT \times sH \times sW  steps. The number of output features is equal to ⌊input planessT⌋\lfloor\frac{\text{input planes}}{sT}\rfloor .
See AvgPool3d for details and output shape.
Applies a 1D max pooling over an input signal composed of several input planes.
See MaxPool1d for details.
Applies a 2D max pooling over an input signal composed of several input planes.
See MaxPool2d for details.
Applies a 3D max pooling over an input signal composed of several input planes.
See MaxPool3d for details.
Computes a partial inverse of MaxPool1d.
See MaxUnpool1d for details.
Computes a partial inverse of MaxPool2d.
See MaxUnpool2d for details.
Computes a partial inverse of MaxPool3d.
See MaxUnpool3d for details.
Applies a 1D power-average pooling over an input signal composed of several input planes. If the sum of all inputs to the power of p is zero, the gradient is set to zero as well.
See LPPool1d for details.
Applies a 2D power-average pooling over an input signal composed of several input planes. If the sum of all inputs to the power of p is zero, the gradient is set to zero as well.
See LPPool2d for details.
Applies a 1D adaptive max pooling over an input signal composed of several input planes.
See AdaptiveMaxPool1d for details and output shape.
Applies a 2D adaptive max pooling over an input signal composed of several input planes.
See AdaptiveMaxPool2d for details and output shape.
Applies a 3D adaptive max pooling over an input signal composed of several input planes.
See AdaptiveMaxPool3d for details and output shape.
Applies a 1D adaptive average pooling over an input signal composed of several input planes.
See AdaptiveAvgPool1d for details and output shape.
output_size – the target output size (single integer)
Applies a 2D adaptive average pooling over an input signal composed of several input planes.
See AdaptiveAvgPool2d for details and output shape.
output_size – the target output size (single integer or double-integer tuple)
Applies a 3D adaptive average pooling over an input signal composed of several input planes.
See AdaptiveAvgPool3d for details and output shape.
output_size – the target output size (single integer or triple-integer tuple)
Thresholds each element of the input Tensor.
See Threshold for more details.
In-place version of threshold().
Applies the rectified linear unit function element-wise. See ReLU for more details.
In-place version of relu().
Applies the HardTanh function element-wise. See Hardtanh for more details.
In-place version of hardtanh().
Applies the hardswish function, element-wise, as described in the paper:
Searching for MobileNetV3.
See Hardswish for more details.
Applies the element-wise function ReLU6(x)=min(max(0,x),6)\text{ReLU6}(x) = \min(\max(0,x), 6) .
See ReLU6 for more details.
Applies element-wise, ELU(x)=max(0,x)+min(0,α∗(exp(x)−1))\text{ELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x) - 1)) .
See ELU for more details.
In-place version of elu().
Applies element-wise, SELU(x)=scale∗(max(0,x)+min(0,α∗(exp(x)−1)))\text{SELU}(x) = scale * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1))) , with α=1.6732632423543772848170429916717\alpha=1.6732632423543772848170429916717  and scale=1.0507009873554804934193349852946scale=1.0507009873554804934193349852946 .
See SELU for more details.
Applies element-wise, CELU(x)=max(0,x)+min(0,α∗(exp(x/α)−1))\text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1)) .
See CELU for more details.
Applies element-wise, LeakyReLU(x)=max(0,x)+negative_slope∗min(0,x)\text{LeakyReLU}(x) = \max(0, x) + \text{negative\_slope} * \min(0, x)
See LeakyReLU for more details.
In-place version of leaky_relu().
Applies element-wise the function PReLU(x)=max(0,x)+weight∗min(0,x)\text{PReLU}(x) = \max(0,x) + \text{weight} * \min(0,x)  where weight is a learnable parameter.
See PReLU for more details.
Randomized leaky ReLU.
See RReLU for more details.
In-place version of rrelu().
The gated linear unit. Computes:
where input is split in half along dim to form a and b, σ\sigma  is the sigmoid function and ⊗\otimes  is the element-wise product between matrices.
See Language Modeling with Gated Convolutional Networks.
Applies element-wise the function GELU(x)=x∗Φ(x)\text{GELU}(x) = x * \Phi(x)
where Φ(x)\Phi(x)  is the Cumulative Distribution Function for Gaussian Distribution.
See Gaussian Error Linear Units (GELUs).
Applies element-wise LogSigmoid(xi)=log(11+exp(−xi))\text{LogSigmoid}(x_i) = \log \left(\frac{1}{1 + \exp(-x_i)}\right)
See LogSigmoid for more details.
Applies the hard shrinkage function element-wise
See Hardshrink for more details.
Applies element-wise, Tanhshrink(x)=x−Tanh(x)\text{Tanhshrink}(x) = x - \text{Tanh}(x)
See Tanhshrink for more details.
Applies element-wise, the function SoftSign(x)=x1+∣x∣\text{SoftSign}(x) = \frac{x}{1 + |x|}
See Softsign for more details.
Applies element-wise, the function Softplus(x)=1β∗log(1+exp(β∗x))\text{Softplus}(x) = \frac{1}{\beta} * \log(1 + \exp(\beta * x)) .
For numerical stability the implementation reverts to the linear function when input×β>thresholdinput \times \beta > threshold .
See Softplus for more details.
Applies a softmin function.
Note that Softmin(x)=Softmax(−x)\text{Softmin}(x) = \text{Softmax}(-x) . See softmax definition for mathematical formula.
See Softmin for more details.
Applies a softmax function.
Softmax is defined as:
Softmax(xi)=exp(xi)∑jexp(xj)\text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}
It is applied to all slices along dim, and will re-scale them so that the elements lie in the range [0, 1] and sum to 1.
See Softmax for more details.
Note
This function doesn’t work directly with NLLLoss, which expects the Log to be computed between the Softmax and itself. Use log_softmax instead (it’s faster and has better numerical properties).
Applies the soft shrinkage function elementwise
See Softshrink for more details.
Samples from the Gumbel-Softmax distribution (Link 1 Link 2) and optionally discretizes.
Sampled tensor of same shape as logits from the Gumbel-Softmax distribution. If hard=True, the returned samples will be one-hot, otherwise they will be probability distributions that sum to 1 across dim.
Note
This function is here for legacy reasons, may be removed from nn.Functional in the future.
Note
The main trick for hard is to do y_hard - y_soft.detach() + y_soft
It achieves two things: - makes the output value exactly one-hot (since we add then subtract y_soft value) - makes the gradient equal to y_soft gradient (since we strip all other gradients)
Applies a softmax followed by a logarithm.
While mathematically equivalent to log(softmax(x)), doing these two operations separately is slower, and numerically unstable. This function uses an alternative formulation to compute the output and gradient correctly.
See LogSoftmax for more details.
Applies element-wise, Tanh(x)=tanh(x)=exp(x)−exp(−x)exp(x)+exp(−x)\text{Tanh}(x) = \tanh(x) = \frac{\exp(x) - \exp(-x)}{\exp(x) + \exp(-x)}
See Tanh for more details.
Applies the element-wise function Sigmoid(x)=11+exp(−x)\text{Sigmoid}(x) = \frac{1}{1 + \exp(-x)}
See Sigmoid for more details.
Applies the element-wise function
inplace – If set to True, will do this operation in-place. Default: False
See Hardsigmoid for more details.
Applies the silu function, element-wise.
Note
See Gaussian Error Linear Units (GELUs) where the SiLU (Sigmoid Linear Unit) was originally coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning and Swish: a Self-Gated Activation Function where the SiLU was experimented with later.
See SiLU for more details.
Applies Batch Normalization for each channel across a batch of data.
See BatchNorm1d, BatchNorm2d, BatchNorm3d for details.
Applies Instance Normalization for each channel in each data sample in a batch.
See InstanceNorm1d, InstanceNorm2d, InstanceNorm3d for details.
Applies Layer Normalization for last certain number of dimensions.
See LayerNorm for details.
Applies local response normalization over an input signal composed of several input planes, where channels occupy the second dimension. Applies normalization across channels.
See LocalResponseNorm for details.
Performs LpL_p  normalization of inputs over specified dimension.
For a tensor input of sizes (n0,...,ndim,...,nk)(n_0, ..., n_{dim}, ..., n_k) , each ndimn_{dim}  -element vector vv  along dimension dim is transformed as
With the default arguments it uses the Euclidean norm over vectors along dimension 11  for normalization.
Applies a linear transformation to the incoming data: y=xAT+by = xA^T + b .
This operator supports TensorFloat32.
Shape:
Applies a bilinear transformation to the incoming data: y=x1TAx2+by = x_1^T A x_2 + b
Shape:
During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution.
See Dropout for details.
Applies alpha dropout to the input.
See AlphaDropout for details.
Randomly masks out entire channels (a channel is a feature map, e.g. the jj -th channel of the ii -th sample in the batch input is a tensor input[i,j]\text{input}[i, j] ) of the input tensor). Instead of setting activations to zero, as in regular Dropout, the activations are set to the negative saturation value of the SELU activation function.
Each element will be masked independently on every forward call with probability p using samples from a Bernoulli distribution. The elements to be masked are randomized on every forward call, and scaled and shifted to maintain zero mean and unit variance.
See FeatureAlphaDropout for details.
Randomly zero out entire channels (a channel is a 2D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 2D tensor input[i,j]\text{input}[i, j] ) of the input tensor). Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.
See Dropout2d for details.
Randomly zero out entire channels (a channel is a 3D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 3D tensor input[i,j]\text{input}[i, j] ) of the input tensor). Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.
See Dropout3d for details.
A simple lookup table that looks up embeddings in a fixed dictionary and size.
This module is often used to retrieve word embeddings using indices. The input to the module is a list of indices, and the embedding matrix, and the output is the corresponding word embeddings.
See torch.nn.Embedding for more details.
where V = maximum index + 1 and embedding_dim = the embedding size
Examples:
Computes sums, means or maxes of bags of embeddings, without instantiating the intermediate embeddings.
See torch.nn.EmbeddingBag for more details.
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
Shape:
input (LongTensor) and offsets (LongTensor, optional)
If input is 2D of shape (B, N),
it will be treated as B bags (sequences) each of fixed length N, and this will return B values aggregated in a way depending on the mode. offsets is ignored and required to be None in this case.
If input is 1D of shape (N),
it will be treated as a concatenation of multiple bags (sequences). offsets is required to be a 1D tensor containing the starting index positions of each bag in input. Therefore, for offsets of shape (B), input will be viewed as having B bags. Empty bags (i.e., having 0-length) will have returned vectors filled by zeros.
Examples:
Takes LongTensor with index values of shape (*) and returns a tensor of shape (*, num_classes) that have zeros everywhere except where the index of last dimension matches the corresponding value of the input tensor, in which case it will be 1.
See also One-hot on Wikipedia .
LongTensor that has one more dimension with 1 values at the index of last dimension indicated by the input, and 0 everywhere else.
See torch.nn.PairwiseDistance for details
Returns cosine similarity between x1 and x2, computed along dim.
Example:
Computes the p-norm distance between every pair of row vectors in the input. This is identical to the upper triangular portion, excluding the diagonal, of torch.norm(input[:, None] - input, dim=2, p=p). This function will be faster if the rows are contiguous.
If input has shape N×MN \times M  then the output will have shape 12N(N−1)\frac{1}{2} N (N - 1) .
This function is equivalent to scipy.spatial.distance.pdist(input, ‘minkowski’, p=p) if p∈(0,∞)p \in (0, \infty) . When p=0p = 0  it is equivalent to scipy.spatial.distance.pdist(input, ‘hamming’) * M. When p=∞p = \infty , the closest scipy function is scipy.spatial.distance.pdist(xn, lambda x, y: np.abs(x - y).max()).
Function that measures the Binary Cross Entropy between the target and the output.
See BCELoss for details.
Examples:
Function that measures Binary Cross Entropy between target and output logits.
See BCEWithLogitsLoss for details.
Examples:
Poisson negative log likelihood loss.
See PoissonNLLLoss for details.
See CosineEmbeddingLoss for details.
This criterion combines log_softmax and nll_loss in a single function.
See CrossEntropyLoss for details.
Examples:
The Connectionist Temporal Classification loss.
See CTCLoss for details.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
Example:
See HingeEmbeddingLoss for details.
The Kullback-Leibler divergence Loss
See KLDivLoss for details.
Note
size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction.
Note
:attr:reduction = 'mean' doesn’t return the true kl divergence value, please use :attr:reduction = 'batchmean' which aligns with KL math definition. In the next major release, 'mean' will be changed to be the same as ‘batchmean’.
Function that takes the mean element-wise absolute value difference.
See L1Loss for details.
Measures the element-wise mean squared error.
See MSELoss for details.
See MarginRankingLoss for details.
See MultiLabelMarginLoss for details.
See MultiLabelSoftMarginLoss for details.
reduce=None, reduction=’mean’) -> Tensor
See MultiMarginLoss for details.
The negative log likelihood loss.
See NLLLoss for details.
Example:
Function that uses a squared term if the absolute element-wise error falls below beta and an L1 term otherwise.
See SmoothL1Loss for details.
See SoftMarginLoss for details.
See TripletMarginLoss for details
See TripletMarginWithDistanceLoss for details.
Rearranges elements in a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W)  to a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r) , where r is the upscale_factor.
See PixelShuffle for details.
Examples:
Reverses the PixelShuffle operation by rearranging elements in a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r)  to a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W) , where r is the downscale_factor.
See PixelUnshuffle for details.
Examples:
Pads tensor.
The padding size by which to pad some dimensions of input are described starting from the last dimension and moving forward. ⌊len(pad)2⌋\left\lfloor\frac{\text{len(pad)}}{2}\right\rfloor  dimensions of input will be padded. For example, to pad only the last dimension of the input tensor, then pad has the form (padding_left,padding_right)(\text{padding\_left}, \text{padding\_right}) ; to pad the last 2 dimensions of the input tensor, then use (padding_left,padding_right,(\text{padding\_left}, \text{padding\_right},  padding_top,padding_bottom)\text{padding\_top}, \text{padding\_bottom}) ; to pad the last 3 dimensions, use (padding_left,padding_right,(\text{padding\_left}, \text{padding\_right},  padding_top,padding_bottom\text{padding\_top}, \text{padding\_bottom}  padding_front,padding_back)\text{padding\_front}, \text{padding\_back}) .
See torch.nn.ConstantPad2d, torch.nn.ReflectionPad2d, and torch.nn.ReplicationPad2d for concrete examples on how each of the padding modes works. Constant padding is implemented for arbitrary dimensions. Replicate padding is implemented for padding the last 3 dimensions of 5D input tensor, or the last 2 dimensions of 4D input tensor, or the last dimension of 3D input tensor. Reflect padding is only implemented for padding the last 2 dimensions of 4D input tensor, or the last dimension of 3D input tensor.
Note
When using the CUDA backend, this operation may induce nondeterministic behaviour in its backward pass that is not easily switched off. Please see the notes on Reproducibility for background.
Examples:
Down/up samples the input to either the given size or the given scale_factor
The algorithm used for interpolation is determined by mode.
Currently temporal, spatial and volumetric sampling are supported, i.e. expected inputs are 3-D, 4-D or 5-D in shape.
The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width.
The modes available for resizing are: nearest, linear (3D-only), bilinear, bicubic (4D-only), trilinear (5D-only), area
Note
With mode='bicubic', it’s possible to cause overshoot, in other words it can produce negative values or values greater than 255 for images. Explicitly call result.clamp(min=0, max=255) if you want to reduce the overshoot when displaying the image.
Warning
With align_corners = True, the linearly interpolating modes (linear, bilinear, and trilinear) don’t proportionally align the output and input pixels, and thus the output values can depend on the input size. This was the default behavior for these modes up to version 0.3.1. Since then, the default behavior is align_corners = False. See Upsample for concrete examples on how this affects the outputs.
Warning
When scale_factor is specified, if recompute_scale_factor=True, scale_factor is used to compute the output_size which will then be used to infer new scales for the interpolation. The default behavior for recompute_scale_factor changed to False in 1.6.0, and scale_factor is used in the interpolation calculation.
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
Upsamples the input to either the given size or the given scale_factor
Warning
This function is deprecated in favor of torch.nn.functional.interpolate(). This is equivalent with nn.functional.interpolate(...).
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
The algorithm used for upsampling is determined by mode.
Currently temporal, spatial and volumetric upsampling are supported, i.e. expected inputs are 3-D, 4-D or 5-D in shape.
The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width.
The modes available for upsampling are: nearest, linear (3D-only), bilinear, bicubic (4D-only), trilinear (5D-only)
Note
With mode='bicubic', it’s possible to cause overshoot, in other words it can produce negative values or values greater than 255 for images. Explicitly call result.clamp(min=0, max=255) if you want to reduce the overshoot when displaying the image.
Warning
With align_corners = True, the linearly interpolating modes (linear, bilinear, and trilinear) don’t proportionally align the output and input pixels, and thus the output values can depend on the input size. This was the default behavior for these modes up to version 0.3.1. Since then, the default behavior is align_corners = False. See Upsample for concrete examples on how this affects the outputs.
Upsamples the input, using nearest neighbours’ pixel values.
Warning
This function is deprecated in favor of torch.nn.functional.interpolate(). This is equivalent with nn.functional.interpolate(..., mode='nearest').
Currently spatial and volumetric upsampling are supported (i.e. expected inputs are 4 or 5 dimensional).
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
Upsamples the input, using bilinear upsampling.
Warning
This function is deprecated in favor of torch.nn.functional.interpolate(). This is equivalent with nn.functional.interpolate(..., mode='bilinear', align_corners=True).
Expected inputs are spatial (4 dimensional). Use upsample_trilinear fo volumetric (5 dimensional) inputs.
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
Given an input and a flow-field grid, computes the output using input values and pixel locations from grid.
Currently, only spatial (4-D) and volumetric (5-D) input are supported.
In the spatial (4-D) case, for input with shape (N,C,Hin,Win)(N, C, H_\text{in}, W_\text{in})  and grid with shape (N,Hout,Wout,2)(N, H_\text{out}, W_\text{out}, 2) , the output will have shape (N,C,Hout,Wout)(N, C, H_\text{out}, W_\text{out}) .
For each output location output[n, :, h, w], the size-2 vector grid[n, h, w] specifies input pixel locations x and y, which are used to interpolate the output value output[n, :, h, w]. In the case of 5D inputs, grid[n, d, h, w] specifies the x, y, z pixel locations for interpolating output[n, :, d, h, w]. mode argument specifies nearest or bilinear interpolation method to sample the input pixels.
grid specifies the sampling pixel locations normalized by the input spatial dimensions. Therefore, it should have most values in the range of [-1, 1]. For example, values x = -1, y = -1 is the left-top pixel of input, and values x = 1, y = 1 is the right-bottom pixel of input.
If grid has values outside the range of [-1, 1], the corresponding outputs are handled as defined by padding_mode. Options are
Note
This function is often used in conjunction with affine_grid() to build Spatial Transformer Networks .
Note
When using the CUDA backend, this operation may induce nondeterministic behaviour in its backward pass that is not easily switched off. Please see the notes on Reproducibility for background.
Note
NaN values in grid would be interpreted as -1.
output Tensor
output (Tensor)
Warning
When align_corners = True, the grid positions depend on the pixel size relative to the input image size, and so the locations sampled by grid_sample() will differ for the same input given at different resolutions (that is, after being upsampled or downsampled). The default behavior up to version 1.2.0 was align_corners = True. Since then, the default behavior has been changed to align_corners = False, in order to bring it in line with the default for interpolate().
Note
mode='bicubic' is implemented using the cubic convolution algorithm with α=−0.75\alpha=-0.75 . The constant α\alpha  might be different from packages to packages. For example, PIL and OpenCV use -0.5 and -0.75 respectively. This algorithm may “overshoot” the range of values it’s interpolating. For example, it may produce negative values or values greater than 255 when interpolating input in [0, 255]. Clamp the results with :func: torch.clamp to ensure they are within the valid range.
Generates a 2D or 3D flow field (sampling grid), given a batch of affine matrices theta.
Note
This function is often used in conjunction with grid_sample() to build Spatial Transformer Networks .
output Tensor of size (N×H×W×2N \times H \times W \times 2 )
output (Tensor)
Warning
When align_corners = True, the grid positions depend on the pixel size relative to the input image size, and so the locations sampled by grid_sample() will differ for the same input given at different resolutions (that is, after being upsampled or downsampled). The default behavior up to version 1.2.0 was align_corners = True. Since then, the default behavior has been changed to align_corners = False, in order to bring it in line with the default for interpolate().
Warning
When align_corners = True, 2D affine transforms on 1D data and 3D affine transforms on 2D data (that is, when one of the spatial dimensions has unit size) are ill-defined, and not an intended use case. This is not a problem when align_corners = False. Up to version 1.2.0, all grid points along a unit dimension were considered arbitrarily to be at -1. From version 1.3.0, under align_corners = True all grid points along a unit dimension are considered to be at `0 (the center of the input image).
Evaluates module(input) in parallel across the GPUs given in device_ids.
This is the functional version of the DataParallel module.
a Tensor containing the result of module(input) located on output_device


Applies a 1D adaptive average pooling over an input signal composed of several input planes.
See AdaptiveAvgPool1d for details and output shape.
output_size – the target output size (single integer)


Applies a 2D adaptive average pooling over an input signal composed of several input planes.
See AdaptiveAvgPool2d for details and output shape.
output_size – the target output size (single integer or double-integer tuple)


Applies a 3D adaptive average pooling over an input signal composed of several input planes.
See AdaptiveAvgPool3d for details and output shape.
output_size – the target output size (single integer or triple-integer tuple)


Applies a 1D adaptive max pooling over an input signal composed of several input planes.
See AdaptiveMaxPool1d for details and output shape.


Applies a 2D adaptive max pooling over an input signal composed of several input planes.
See AdaptiveMaxPool2d for details and output shape.


Applies a 3D adaptive max pooling over an input signal composed of several input planes.
See AdaptiveMaxPool3d for details and output shape.


Generates a 2D or 3D flow field (sampling grid), given a batch of affine matrices theta.
Note
This function is often used in conjunction with grid_sample() to build Spatial Transformer Networks .
output Tensor of size (N×H×W×2N \times H \times W \times 2 )
output (Tensor)
Warning
When align_corners = True, the grid positions depend on the pixel size relative to the input image size, and so the locations sampled by grid_sample() will differ for the same input given at different resolutions (that is, after being upsampled or downsampled). The default behavior up to version 1.2.0 was align_corners = True. Since then, the default behavior has been changed to align_corners = False, in order to bring it in line with the default for interpolate().
Warning
When align_corners = True, 2D affine transforms on 1D data and 3D affine transforms on 2D data (that is, when one of the spatial dimensions has unit size) are ill-defined, and not an intended use case. This is not a problem when align_corners = False. Up to version 1.2.0, all grid points along a unit dimension were considered arbitrarily to be at -1. From version 1.3.0, under align_corners = True all grid points along a unit dimension are considered to be at `0 (the center of the input image).


Applies alpha dropout to the input.
See AlphaDropout for details.


Applies a 1D average pooling over an input signal composed of several input planes.
See AvgPool1d for details and output shape.
Examples:


Applies 2D average-pooling operation in kH×kWkH \times kW  regions by step size sH×sWsH \times sW  steps. The number of output features is equal to the number of input planes.
See AvgPool2d for details and output shape.


Applies 3D average-pooling operation in kT×kH×kWkT \times kH \times kW  regions by step size sT×sH×sWsT \times sH \times sW  steps. The number of output features is equal to ⌊input planessT⌋\lfloor\frac{\text{input planes}}{sT}\rfloor .
See AvgPool3d for details and output shape.


Applies Batch Normalization for each channel across a batch of data.
See BatchNorm1d, BatchNorm2d, BatchNorm3d for details.


Applies a bilinear transformation to the incoming data: y=x1TAx2+by = x_1^T A x_2 + b
Shape:


Function that measures the Binary Cross Entropy between the target and the output.
See BCELoss for details.
Examples:


Function that measures Binary Cross Entropy between target and output logits.
See BCEWithLogitsLoss for details.
Examples:


Applies element-wise, CELU(x)=max(0,x)+min(0,α∗(exp(x/α)−1))\text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1)) .
See CELU for more details.


Applies a 1D convolution over an input signal composed of several input planes.
This operator supports TensorFloat32.
See Conv1d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:


Applies a 2D convolution over an input image composed of several input planes.
This operator supports TensorFloat32.
See Conv2d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:


Applies a 3D convolution over an input image composed of several input planes.
This operator supports TensorFloat32.
See Conv3d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:


Applies a 1D transposed convolution operator over an input signal composed of several input planes, sometimes also called “deconvolution”.
This operator supports TensorFloat32.
See ConvTranspose1d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:


Applies a 2D transposed convolution operator over an input image composed of several input planes, sometimes also called “deconvolution”.
This operator supports TensorFloat32.
See ConvTranspose2d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:


Applies a 3D transposed convolution operator over an input image composed of several input planes, sometimes also called “deconvolution”
This operator supports TensorFloat32.
See ConvTranspose3d for details and output shape.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Examples:


Returns cosine similarity between x1 and x2, computed along dim.
Example:


This criterion combines log_softmax and nll_loss in a single function.
See CrossEntropyLoss for details.
Examples:


The Connectionist Temporal Classification loss.
See CTCLoss for details.
Note
In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
Example:


During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution.
See Dropout for details.


Randomly zero out entire channels (a channel is a 2D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 2D tensor input[i,j]\text{input}[i, j] ) of the input tensor). Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.
See Dropout2d for details.


Randomly zero out entire channels (a channel is a 3D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 3D tensor input[i,j]\text{input}[i, j] ) of the input tensor). Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.
See Dropout3d for details.


Applies element-wise, ELU(x)=max(0,x)+min(0,α∗(exp(x)−1))\text{ELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x) - 1)) .
See ELU for more details.


A simple lookup table that looks up embeddings in a fixed dictionary and size.
This module is often used to retrieve word embeddings using indices. The input to the module is a list of indices, and the embedding matrix, and the output is the corresponding word embeddings.
See torch.nn.Embedding for more details.
where V = maximum index + 1 and embedding_dim = the embedding size
Examples:


Computes sums, means or maxes of bags of embeddings, without instantiating the intermediate embeddings.
See torch.nn.EmbeddingBag for more details.
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
Shape:
input (LongTensor) and offsets (LongTensor, optional)
If input is 2D of shape (B, N),
it will be treated as B bags (sequences) each of fixed length N, and this will return B values aggregated in a way depending on the mode. offsets is ignored and required to be None in this case.
If input is 1D of shape (N),
it will be treated as a concatenation of multiple bags (sequences). offsets is required to be a 1D tensor containing the starting index positions of each bag in input. Therefore, for offsets of shape (B), input will be viewed as having B bags. Empty bags (i.e., having 0-length) will have returned vectors filled by zeros.
Examples:


Randomly masks out entire channels (a channel is a feature map, e.g. the jj -th channel of the ii -th sample in the batch input is a tensor input[i,j]\text{input}[i, j] ) of the input tensor). Instead of setting activations to zero, as in regular Dropout, the activations are set to the negative saturation value of the SELU activation function.
Each element will be masked independently on every forward call with probability p using samples from a Bernoulli distribution. The elements to be masked are randomized on every forward call, and scaled and shifted to maintain zero mean and unit variance.
See FeatureAlphaDropout for details.


Combines an array of sliding local blocks into a large containing tensor.
Warning
Currently, only 3-D output tensors (unfolded batched image-like tensors) are supported.
See torch.nn.Fold for details


Applies element-wise the function GELU(x)=x∗Φ(x)\text{GELU}(x) = x * \Phi(x)
where Φ(x)\Phi(x)  is the Cumulative Distribution Function for Gaussian Distribution.
See Gaussian Error Linear Units (GELUs).


The gated linear unit. Computes:
where input is split in half along dim to form a and b, σ\sigma  is the sigmoid function and ⊗\otimes  is the element-wise product between matrices.
See Language Modeling with Gated Convolutional Networks.


Given an input and a flow-field grid, computes the output using input values and pixel locations from grid.
Currently, only spatial (4-D) and volumetric (5-D) input are supported.
In the spatial (4-D) case, for input with shape (N,C,Hin,Win)(N, C, H_\text{in}, W_\text{in})  and grid with shape (N,Hout,Wout,2)(N, H_\text{out}, W_\text{out}, 2) , the output will have shape (N,C,Hout,Wout)(N, C, H_\text{out}, W_\text{out}) .
For each output location output[n, :, h, w], the size-2 vector grid[n, h, w] specifies input pixel locations x and y, which are used to interpolate the output value output[n, :, h, w]. In the case of 5D inputs, grid[n, d, h, w] specifies the x, y, z pixel locations for interpolating output[n, :, d, h, w]. mode argument specifies nearest or bilinear interpolation method to sample the input pixels.
grid specifies the sampling pixel locations normalized by the input spatial dimensions. Therefore, it should have most values in the range of [-1, 1]. For example, values x = -1, y = -1 is the left-top pixel of input, and values x = 1, y = 1 is the right-bottom pixel of input.
If grid has values outside the range of [-1, 1], the corresponding outputs are handled as defined by padding_mode. Options are
Note
This function is often used in conjunction with affine_grid() to build Spatial Transformer Networks .
Note
When using the CUDA backend, this operation may induce nondeterministic behaviour in its backward pass that is not easily switched off. Please see the notes on Reproducibility for background.
Note
NaN values in grid would be interpreted as -1.
output Tensor
output (Tensor)
Warning
When align_corners = True, the grid positions depend on the pixel size relative to the input image size, and so the locations sampled by grid_sample() will differ for the same input given at different resolutions (that is, after being upsampled or downsampled). The default behavior up to version 1.2.0 was align_corners = True. Since then, the default behavior has been changed to align_corners = False, in order to bring it in line with the default for interpolate().
Note
mode='bicubic' is implemented using the cubic convolution algorithm with α=−0.75\alpha=-0.75 . The constant α\alpha  might be different from packages to packages. For example, PIL and OpenCV use -0.5 and -0.75 respectively. This algorithm may “overshoot” the range of values it’s interpolating. For example, it may produce negative values or values greater than 255 when interpolating input in [0, 255]. Clamp the results with :func: torch.clamp to ensure they are within the valid range.


Samples from the Gumbel-Softmax distribution (Link 1 Link 2) and optionally discretizes.
Sampled tensor of same shape as logits from the Gumbel-Softmax distribution. If hard=True, the returned samples will be one-hot, otherwise they will be probability distributions that sum to 1 across dim.
Note
This function is here for legacy reasons, may be removed from nn.Functional in the future.
Note
The main trick for hard is to do y_hard - y_soft.detach() + y_soft
It achieves two things: - makes the output value exactly one-hot (since we add then subtract y_soft value) - makes the gradient equal to y_soft gradient (since we strip all other gradients)


Applies the hard shrinkage function element-wise
See Hardshrink for more details.


Applies the element-wise function
inplace – If set to True, will do this operation in-place. Default: False
See Hardsigmoid for more details.


Applies the hardswish function, element-wise, as described in the paper:
Searching for MobileNetV3.
See Hardswish for more details.


Applies the HardTanh function element-wise. See Hardtanh for more details.


Applies Instance Normalization for each channel in each data sample in a batch.
See InstanceNorm1d, InstanceNorm2d, InstanceNorm3d for details.


Down/up samples the input to either the given size or the given scale_factor
The algorithm used for interpolation is determined by mode.
Currently temporal, spatial and volumetric sampling are supported, i.e. expected inputs are 3-D, 4-D or 5-D in shape.
The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width.
The modes available for resizing are: nearest, linear (3D-only), bilinear, bicubic (4D-only), trilinear (5D-only), area
Note
With mode='bicubic', it’s possible to cause overshoot, in other words it can produce negative values or values greater than 255 for images. Explicitly call result.clamp(min=0, max=255) if you want to reduce the overshoot when displaying the image.
Warning
With align_corners = True, the linearly interpolating modes (linear, bilinear, and trilinear) don’t proportionally align the output and input pixels, and thus the output values can depend on the input size. This was the default behavior for these modes up to version 0.3.1. Since then, the default behavior is align_corners = False. See Upsample for concrete examples on how this affects the outputs.
Warning
When scale_factor is specified, if recompute_scale_factor=True, scale_factor is used to compute the output_size which will then be used to infer new scales for the interpolation. The default behavior for recompute_scale_factor changed to False in 1.6.0, and scale_factor is used in the interpolation calculation.
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.


The Kullback-Leibler divergence Loss
See KLDivLoss for details.
Note
size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction.
Note
:attr:reduction = 'mean' doesn’t return the true kl divergence value, please use :attr:reduction = 'batchmean' which aligns with KL math definition. In the next major release, 'mean' will be changed to be the same as ‘batchmean’.


Function that takes the mean element-wise absolute value difference.
See L1Loss for details.


Applies Layer Normalization for last certain number of dimensions.
See LayerNorm for details.


Applies element-wise, LeakyReLU(x)=max(0,x)+negative_slope∗min(0,x)\text{LeakyReLU}(x) = \max(0, x) + \text{negative\_slope} * \min(0, x)
See LeakyReLU for more details.


Applies a linear transformation to the incoming data: y=xAT+by = xA^T + b .
This operator supports TensorFloat32.
Shape:


Applies local response normalization over an input signal composed of several input planes, where channels occupy the second dimension. Applies normalization across channels.
See LocalResponseNorm for details.


Applies element-wise LogSigmoid(xi)=log(11+exp(−xi))\text{LogSigmoid}(x_i) = \log \left(\frac{1}{1 + \exp(-x_i)}\right)
See LogSigmoid for more details.


Applies a softmax followed by a logarithm.
While mathematically equivalent to log(softmax(x)), doing these two operations separately is slower, and numerically unstable. This function uses an alternative formulation to compute the output and gradient correctly.
See LogSoftmax for more details.


Applies a 1D power-average pooling over an input signal composed of several input planes. If the sum of all inputs to the power of p is zero, the gradient is set to zero as well.
See LPPool1d for details.


Applies a 2D power-average pooling over an input signal composed of several input planes. If the sum of all inputs to the power of p is zero, the gradient is set to zero as well.
See LPPool2d for details.


Applies a 1D max pooling over an input signal composed of several input planes.
See MaxPool1d for details.


Applies a 2D max pooling over an input signal composed of several input planes.
See MaxPool2d for details.


Applies a 3D max pooling over an input signal composed of several input planes.
See MaxPool3d for details.


Computes a partial inverse of MaxPool1d.
See MaxUnpool1d for details.


Computes a partial inverse of MaxPool2d.
See MaxUnpool2d for details.


Computes a partial inverse of MaxPool3d.
See MaxUnpool3d for details.


Measures the element-wise mean squared error.
See MSELoss for details.


See MultiLabelSoftMarginLoss for details.


reduce=None, reduction=’mean’) -> Tensor
See MultiMarginLoss for details.


The negative log likelihood loss.
See NLLLoss for details.
Example:


Performs LpL_p  normalization of inputs over specified dimension.
For a tensor input of sizes (n0,...,ndim,...,nk)(n_0, ..., n_{dim}, ..., n_k) , each ndimn_{dim}  -element vector vv  along dimension dim is transformed as
With the default arguments it uses the Euclidean norm over vectors along dimension 11  for normalization.


Takes LongTensor with index values of shape (*) and returns a tensor of shape (*, num_classes) that have zeros everywhere except where the index of last dimension matches the corresponding value of the input tensor, in which case it will be 1.
See also One-hot on Wikipedia .
LongTensor that has one more dimension with 1 values at the index of last dimension indicated by the input, and 0 everywhere else.


Pads tensor.
The padding size by which to pad some dimensions of input are described starting from the last dimension and moving forward. ⌊len(pad)2⌋\left\lfloor\frac{\text{len(pad)}}{2}\right\rfloor  dimensions of input will be padded. For example, to pad only the last dimension of the input tensor, then pad has the form (padding_left,padding_right)(\text{padding\_left}, \text{padding\_right}) ; to pad the last 2 dimensions of the input tensor, then use (padding_left,padding_right,(\text{padding\_left}, \text{padding\_right},  padding_top,padding_bottom)\text{padding\_top}, \text{padding\_bottom}) ; to pad the last 3 dimensions, use (padding_left,padding_right,(\text{padding\_left}, \text{padding\_right},  padding_top,padding_bottom\text{padding\_top}, \text{padding\_bottom}  padding_front,padding_back)\text{padding\_front}, \text{padding\_back}) .
See torch.nn.ConstantPad2d, torch.nn.ReflectionPad2d, and torch.nn.ReplicationPad2d for concrete examples on how each of the padding modes works. Constant padding is implemented for arbitrary dimensions. Replicate padding is implemented for padding the last 3 dimensions of 5D input tensor, or the last 2 dimensions of 4D input tensor, or the last dimension of 3D input tensor. Reflect padding is only implemented for padding the last 2 dimensions of 4D input tensor, or the last dimension of 3D input tensor.
Note
When using the CUDA backend, this operation may induce nondeterministic behaviour in its backward pass that is not easily switched off. Please see the notes on Reproducibility for background.
Examples:


See torch.nn.PairwiseDistance for details


Computes the p-norm distance between every pair of row vectors in the input. This is identical to the upper triangular portion, excluding the diagonal, of torch.norm(input[:, None] - input, dim=2, p=p). This function will be faster if the rows are contiguous.
If input has shape N×MN \times M  then the output will have shape 12N(N−1)\frac{1}{2} N (N - 1) .
This function is equivalent to scipy.spatial.distance.pdist(input, ‘minkowski’, p=p) if p∈(0,∞)p \in (0, \infty) . When p=0p = 0  it is equivalent to scipy.spatial.distance.pdist(input, ‘hamming’) * M. When p=∞p = \infty , the closest scipy function is scipy.spatial.distance.pdist(xn, lambda x, y: np.abs(x - y).max()).


Rearranges elements in a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W)  to a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r) , where r is the upscale_factor.
See PixelShuffle for details.
Examples:


Reverses the PixelShuffle operation by rearranging elements in a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r)  to a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W) , where r is the downscale_factor.
See PixelUnshuffle for details.
Examples:


Poisson negative log likelihood loss.
See PoissonNLLLoss for details.


Applies element-wise the function PReLU(x)=max(0,x)+weight∗min(0,x)\text{PReLU}(x) = \max(0,x) + \text{weight} * \min(0,x)  where weight is a learnable parameter.
See PReLU for more details.


Applies the rectified linear unit function element-wise. See ReLU for more details.


Applies the element-wise function ReLU6(x)=min(max(0,x),6)\text{ReLU6}(x) = \min(\max(0,x), 6) .
See ReLU6 for more details.


Randomized leaky ReLU.
See RReLU for more details.


Applies the element-wise function Sigmoid(x)=11+exp(−x)\text{Sigmoid}(x) = \frac{1}{1 + \exp(-x)}
See Sigmoid for more details.


Applies the silu function, element-wise.
Note
See Gaussian Error Linear Units (GELUs) where the SiLU (Sigmoid Linear Unit) was originally coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning and Swish: a Self-Gated Activation Function where the SiLU was experimented with later.
See SiLU for more details.


Function that uses a squared term if the absolute element-wise error falls below beta and an L1 term otherwise.
See SmoothL1Loss for details.


Applies a softmax function.
Softmax is defined as:
Softmax(xi)=exp(xi)∑jexp(xj)\text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}
It is applied to all slices along dim, and will re-scale them so that the elements lie in the range [0, 1] and sum to 1.
See Softmax for more details.
Note
This function doesn’t work directly with NLLLoss, which expects the Log to be computed between the Softmax and itself. Use log_softmax instead (it’s faster and has better numerical properties).


Applies a softmin function.
Note that Softmin(x)=Softmax(−x)\text{Softmin}(x) = \text{Softmax}(-x) . See softmax definition for mathematical formula.
See Softmin for more details.


Applies element-wise, the function Softplus(x)=1β∗log(1+exp(β∗x))\text{Softplus}(x) = \frac{1}{\beta} * \log(1 + \exp(\beta * x)) .
For numerical stability the implementation reverts to the linear function when input×β>thresholdinput \times \beta > threshold .
See Softplus for more details.


Applies the soft shrinkage function elementwise
See Softshrink for more details.


Applies element-wise, the function SoftSign(x)=x1+∣x∣\text{SoftSign}(x) = \frac{x}{1 + |x|}
See Softsign for more details.


Applies element-wise, Tanh(x)=tanh(x)=exp(x)−exp(−x)exp(x)+exp(−x)\text{Tanh}(x) = \tanh(x) = \frac{\exp(x) - \exp(-x)}{\exp(x) + \exp(-x)}
See Tanh for more details.


Applies element-wise, Tanhshrink(x)=x−Tanh(x)\text{Tanhshrink}(x) = x - \text{Tanh}(x)
See Tanhshrink for more details.


Thresholds each element of the input Tensor.
See Threshold for more details.


See TripletMarginWithDistanceLoss for details.


Extracts sliding local blocks from a batched input tensor.
Warning
Currently, only 4-D input tensors (batched image-like tensors) are supported.
Warning
More than one element of the unfolded tensor may refer to a single memory location. As a result, in-place operations (especially ones that are vectorized) may result in incorrect behavior. If you need to write to the tensor, please clone it first.
See torch.nn.Unfold for details


Upsamples the input to either the given size or the given scale_factor
Warning
This function is deprecated in favor of torch.nn.functional.interpolate(). This is equivalent with nn.functional.interpolate(...).
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
The algorithm used for upsampling is determined by mode.
Currently temporal, spatial and volumetric upsampling are supported, i.e. expected inputs are 3-D, 4-D or 5-D in shape.
The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width.
The modes available for upsampling are: nearest, linear (3D-only), bilinear, bicubic (4D-only), trilinear (5D-only)
Note
With mode='bicubic', it’s possible to cause overshoot, in other words it can produce negative values or values greater than 255 for images. Explicitly call result.clamp(min=0, max=255) if you want to reduce the overshoot when displaying the image.
Warning
With align_corners = True, the linearly interpolating modes (linear, bilinear, and trilinear) don’t proportionally align the output and input pixels, and thus the output values can depend on the input size. This was the default behavior for these modes up to version 0.3.1. Since then, the default behavior is align_corners = False. See Upsample for concrete examples on how this affects the outputs.


Upsamples the input, using bilinear upsampling.
Warning
This function is deprecated in favor of torch.nn.functional.interpolate(). This is equivalent with nn.functional.interpolate(..., mode='bilinear', align_corners=True).
Expected inputs are spatial (4 dimensional). Use upsample_trilinear fo volumetric (5 dimensional) inputs.
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.


Upsamples the input, using nearest neighbours’ pixel values.
Warning
This function is deprecated in favor of torch.nn.functional.interpolate(). This is equivalent with nn.functional.interpolate(..., mode='nearest').
Currently spatial and volumetric upsampling are supported (i.e. expected inputs are 4 or 5 dimensional).
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.


Gaussian negative log likelihood loss.
The targets are treated as samples from Gaussian distributions with expectations and variances predicted by the neural network. For a D-dimensional target tensor modelled as having heteroscedastic Gaussian distributions with a D-dimensional tensor of expectations input and a D-dimensional tensor of positive variances var the loss is:
where eps is used for stability. By default, the constant term of the loss function is omitted unless full is True. If var is a scalar (implying target tensor has homoscedastic Gaussian distributions) it is broadcasted to be the same size as the input.
Examples:
Note
The clamping of var is ignored with respect to autograd, and so the gradients are unaffected by it.
Nix, D. A. and Weigend, A. S., “Estimating the mean and variance of the target probability distribution”, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), Orlando, FL, USA, 1994, pp. 55-60 vol.1, doi: 10.1109/ICNN.1994.374138.


Applies the Gaussian Error Linear Units function:
where Φ(x)\Phi(x)  is the Cumulative Distribution Function for Gaussian Distribution.
Examples:


Applies Group Normalization over a mini-batch of inputs as described in the paper Group Normalization
The input channels are separated into num_groups groups, each containing num_channels / num_groups channels. The mean and standard-deviation are calculated separately over the each group. γ\gamma  and β\beta  are learnable per-channel affine transform parameter vectors of size num_channels if affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).
This layer uses statistics computed from input data in both training and evaluation modes.
Examples:


Applies a multi-layer gated recurrent unit (GRU) RNN to an input sequence.
For each element in the input sequence, each layer computes the following function:
where hth_t  is the hidden state at time t, xtx_t  is the input at time t, h(t−1)h_{(t-1)}  is the hidden state of the layer at time t-1 or the initial hidden state at time 0, and rtr_t , ztz_t , ntn_t  are the reset, update, and new gates, respectively. σ\sigma  is the sigmoid function, and ∗*  is the Hadamard product.
In a multilayer GRU, the input xt(l)x^{(l)}_t  of the ll  -th layer (l>=2l >= 2 ) is the hidden state ht(l−1)h^{(l-1)}_t  of the previous layer multiplied by dropout δt(l−1)\delta^{(l-1)}_t  where each δt(l−1)\delta^{(l-1)}_t  is a Bernoulli random variable which is 00  with probability dropout.
output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features h_t from the last layer of the GRU, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence. For the unpacked case, the directions can be separated using output.view(seq_len, batch, num_directions, hidden_size), with forward and backward being direction 0 and 1 respectively.
Similarly, the directions can be separated in the packed case.
h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len
Like output, the layers can be separated using h_n.view(num_layers, num_directions, batch, hidden_size).
Note
All the weights and biases are initialized from U(−k,k)\mathcal{U}(-\sqrt{k}, \sqrt{k})  where k=1hidden_sizek = \frac{1}{\text{hidden\_size}}
Note
If the following conditions are satisfied: 1) cudnn is enabled, 2) input data is on the GPU 3) input data has dtype torch.float16 4) V100 GPU is used, 5) input data is not in PackedSequence format persistent algorithm can be selected to improve performance.
Examples:


A gated recurrent unit (GRU) cell
where σ\sigma  is the sigmoid function, and ∗*  is the Hadamard product.
Note
All the weights and biases are initialized from U(−k,k)\mathcal{U}(-\sqrt{k}, \sqrt{k})  where k=1hidden_sizek = \frac{1}{\text{hidden\_size}}
Examples:


Applies the hard shrinkage function element-wise:
lambd – the λ\lambda  value for the Hardshrink formulation. Default: 0.5
Examples:


Applies the element-wise function:
inplace – can optionally do the operation in-place. Default: False
Examples:


Applies the hardswish function, element-wise, as described in the paper:
Searching for MobileNetV3.
inplace – can optionally do the operation in-place. Default: False
Examples:


Applies the HardTanh function element-wise
HardTanh is defined as:
The range of the linear region [−1,1][-1, 1]  can be adjusted using min_val and max_val.
Keyword arguments min_value and max_value have been deprecated in favor of min_val and max_val.
Examples:


Measures the loss given an input tensor xx  and a labels tensor yy  (containing 1 or -1). This is usually used for measuring whether two inputs are similar or dissimilar, e.g. using the L1 pairwise distance as xx , and is typically used for learning nonlinear embeddings or semi-supervised learning.
The loss function for nn -th sample in the mini-batch is
and the total loss functions is
where L={l1,…,lN}⊤L = \{l_1,\dots,l_N\}^\top .


A placeholder identity operator that is argument-insensitive.
Examples:


Return the recommended gain value for the given nonlinearity function. The values are as follows:
nonlinearity
gain
Linear / Identity
11
Conv{1,2,3}D
11
Sigmoid
11
Tanh
53\frac{5}{3}
ReLU
2\sqrt{2}
Leaky Relu
21+negative_slope2\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}
SELU
34\frac{3}{4}
Fills the input Tensor with values drawn from the uniform distribution U(a,b)\mathcal{U}(a, b) .
Fills the input Tensor with values drawn from the normal distribution N(mean,std2)\mathcal{N}(\text{mean}, \text{std}^2) .
Fills the input Tensor with the value val\text{val} .
Fills the input Tensor with the scalar value 1.
tensor – an n-dimensional torch.Tensor
Fills the input Tensor with the scalar value 0.
tensor – an n-dimensional torch.Tensor
Fills the 2-dimensional input Tensor with the identity matrix. Preserves the identity of the inputs in Linear layers, where as many inputs are preserved as possible.
tensor – a 2-dimensional torch.Tensor
Fills the {3, 4, 5}-dimensional input Tensor with the Dirac delta function. Preserves the identity of the inputs in Convolutional layers, where as many input channels are preserved as possible. In case of groups>1, each group of channels preserves identity
Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a uniform distribution. The resulting tensor will have values sampled from U(−a,a)\mathcal{U}(-a, a)  where
Also known as Glorot initialization.
Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a normal distribution. The resulting tensor will have values sampled from N(0,std2)\mathcal{N}(0, \text{std}^2)  where
Also known as Glorot initialization.
Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a uniform distribution. The resulting tensor will have values sampled from U(−bound,bound)\mathcal{U}(-\text{bound}, \text{bound})  where
Also known as He initialization.
Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a normal distribution. The resulting tensor will have values sampled from N(0,std2)\mathcal{N}(0, \text{std}^2)  where
Also known as He initialization.
Fills the input Tensor with a (semi) orthogonal matrix, as described in Exact solutions to the nonlinear dynamics of learning in deep linear neural networks - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened.
Fills the 2D input Tensor as a sparse matrix, where the non-zero elements will be drawn from the normal distribution N(0,0.01)\mathcal{N}(0, 0.01) , as described in Deep learning via Hessian-free optimization - Martens, J. (2010).


Return the recommended gain value for the given nonlinearity function. The values are as follows:
nonlinearity
gain
Linear / Identity
11
Conv{1,2,3}D
11
Sigmoid
11
Tanh
53\frac{5}{3}
ReLU
2\sqrt{2}
Leaky Relu
21+negative_slope2\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}
SELU
34\frac{3}{4}


Fills the input Tensor with the value val\text{val} .


Fills the {3, 4, 5}-dimensional input Tensor with the Dirac delta function. Preserves the identity of the inputs in Convolutional layers, where as many input channels are preserved as possible. In case of groups>1, each group of channels preserves identity


Fills the 2-dimensional input Tensor with the identity matrix. Preserves the identity of the inputs in Linear layers, where as many inputs are preserved as possible.
tensor – a 2-dimensional torch.Tensor


Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a normal distribution. The resulting tensor will have values sampled from N(0,std2)\mathcal{N}(0, \text{std}^2)  where
Also known as He initialization.


Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a uniform distribution. The resulting tensor will have values sampled from U(−bound,bound)\mathcal{U}(-\text{bound}, \text{bound})  where
Also known as He initialization.


Fills the input Tensor with values drawn from the normal distribution N(mean,std2)\mathcal{N}(\text{mean}, \text{std}^2) .


Fills the input Tensor with the scalar value 1.
tensor – an n-dimensional torch.Tensor


Fills the input Tensor with a (semi) orthogonal matrix, as described in Exact solutions to the nonlinear dynamics of learning in deep linear neural networks - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened.


Fills the 2D input Tensor as a sparse matrix, where the non-zero elements will be drawn from the normal distribution N(0,0.01)\mathcal{N}(0, 0.01) , as described in Deep learning via Hessian-free optimization - Martens, J. (2010).


Fills the input Tensor with values drawn from the uniform distribution U(a,b)\mathcal{U}(a, b) .


Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a normal distribution. The resulting tensor will have values sampled from N(0,std2)\mathcal{N}(0, \text{std}^2)  where
Also known as Glorot initialization.


Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a uniform distribution. The resulting tensor will have values sampled from U(−a,a)\mathcal{U}(-a, a)  where
Also known as Glorot initialization.


Fills the input Tensor with the scalar value 0.
tensor – an n-dimensional torch.Tensor


Applies Instance Normalization over a 3D input (a mini-batch of 1D inputs with optional additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.
The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. γ\gamma  and β\beta  are learnable parameter vectors of size C (where C is the input size) if affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).
By default, this layer uses instance statistics computed from input data in both training and evaluation modes.
If track_running_stats is set to True, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.
Note
This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x^new=(1−momentum)×x^+momentum×xt\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t , where x^\hat{x}  is the estimated statistic and xtx_t  is the new observed value.
Note
InstanceNorm1d and LayerNorm are very similar, but have some subtle differences. InstanceNorm1d is applied on each channel of channeled data like multidimensional time series, but LayerNorm is usually applied on entire sample and often in NLP tasks. Additionally, LayerNorm applies elementwise affine transform, while InstanceNorm1d usually don’t apply affine transform.
Examples:


Applies Instance Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.
The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. γ\gamma  and β\beta  are learnable parameter vectors of size C (where C is the input size) if affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).
By default, this layer uses instance statistics computed from input data in both training and evaluation modes.
If track_running_stats is set to True, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.
Note
This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x^new=(1−momentum)×x^+momentum×xt\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t , where x^\hat{x}  is the estimated statistic and xtx_t  is the new observed value.
Note
InstanceNorm2d and LayerNorm are very similar, but have some subtle differences. InstanceNorm2d is applied on each channel of channeled data like RGB images, but LayerNorm is usually applied on entire sample and often in NLP tasks. Additionally, LayerNorm applies elementwise affine transform, while InstanceNorm2d usually don’t apply affine transform.
Examples:


Applies Instance Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.
The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. γ\gamma  and β\beta  are learnable parameter vectors of size C (where C is the input size) if affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).
By default, this layer uses instance statistics computed from input data in both training and evaluation modes.
If track_running_stats is set to True, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.
Note
This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x^new=(1−momentum)×x^+momentum×xt\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t , where x^\hat{x}  is the estimated statistic and xtx_t  is the new observed value.
Note
InstanceNorm3d and LayerNorm are very similar, but have some subtle differences. InstanceNorm3d is applied on each channel of channeled data like 3D models with RGB color, but LayerNorm is usually applied on entire sample and often in NLP tasks. Additionally, LayerNorm applies elementwise affine transform, while InstanceNorm3d usually don’t apply affine transform.
Examples:


This is a sequential container which calls the Conv 1d and Batch Norm 1d modules. During quantization this will be replaced with the corresponding fused module.


This is a sequential container which calls the Conv 2d and Batch Norm 2d modules. During quantization this will be replaced with the corresponding fused module.


This is a sequential container which calls the Conv 1d, Batch Norm 1d, and ReLU modules. During quantization this will be replaced with the corresponding fused module.


This is a sequential container which calls the Conv 2d, Batch Norm 2d, and ReLU modules. During quantization this will be replaced with the corresponding fused module.


This is a sequential container which calls the Conv1d and ReLU modules. During quantization this will be replaced with the corresponding fused module.


This is a sequential container which calls the Conv2d and ReLU modules. During quantization this will be replaced with the corresponding fused module.


A ConvBn2d module is a module fused from Conv2d and BatchNorm2d, attached with FakeQuantize modules for weight, used in quantization aware training.
We combined the interface of torch.nn.Conv2d and torch.nn.BatchNorm2d.
Similar to torch.nn.Conv2d, with FakeQuantize modules initialized to default.


A ConvBnReLU2d module is a module fused from Conv2d, BatchNorm2d and ReLU, attached with FakeQuantize modules for weight, used in quantization aware training.
We combined the interface of torch.nn.Conv2d and torch.nn.BatchNorm2d and torch.nn.ReLU.
Similar to torch.nn.Conv2d, with FakeQuantize modules initialized to default.
~ConvBnReLU2d.weight_fake_quant – fake quant module for weight


A ConvReLU2d module is a fused module of Conv2d and ReLU, attached with FakeQuantize modules for weight for quantization aware training.
We combined the interface of Conv2d and BatchNorm2d.
~ConvReLU2d.weight_fake_quant – fake quant module for weight


A LinearReLU module fused from Linear and ReLU modules, attached with FakeQuantize modules for weight, used in quantization aware training.
We adopt the same interface as torch.nn.Linear.
Similar to torch.nn.intrinsic.LinearReLU, with FakeQuantize modules initialized to default.
~LinearReLU.weight – fake quant module for weight
Examples:


A ConvReLU2d module is a fused module of Conv2d and ReLU
We adopt the same interface as torch.nn.quantized.Conv2d.
as torch.nn.quantized.Conv2d (Same) –


A ConvReLU3d module is a fused module of Conv3d and ReLU
We adopt the same interface as torch.nn.quantized.Conv3d.
Attributes: Same as torch.nn.quantized.Conv3d


A LinearReLU module fused from Linear and ReLU modules
We adopt the same interface as torch.nn.quantized.Linear.
as torch.nn.quantized.Linear (Same) –
Examples:


The Kullback-Leibler divergence loss measure
Kullback-Leibler divergence is a useful distance measure for continuous distributions and is often useful when performing direct regression over the space of (discretely sampled) continuous output distributions.
As with NLLLoss, the input given is expected to contain log-probabilities and is not restricted to a 2D Tensor. The targets are interpreted as probabilities by default, but could be considered as log-probabilities with log_target set to True.
This criterion expects a target Tensor of the same size as the input Tensor.
The unreduced (i.e. with reduction set to 'none') loss can be described as:
where the index NN  spans all dimensions of input and LL  has the same shape as input. If reduction is not 'none' (default 'mean'), then:
In default reduction mode 'mean', the losses are averaged for each minibatch over observations as well as over dimensions. 'batchmean' mode gives the correct KL divergence where losses are averaged over batch dimension only. 'mean' mode’s behavior will be changed to the same as 'batchmean' in the next major release.
Note
size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction.
Note
reduction = 'mean' doesn’t return the true kl divergence value, please use reduction = 'batchmean' which aligns with KL math definition. In the next major release, 'mean' will be changed to be the same as 'batchmean'.


Creates a criterion that measures the mean absolute error (MAE) between each element in the input xx  and target yy .
The unreduced (i.e. with reduction set to 'none') loss can be described as:
where NN  is the batch size. If reduction is not 'none' (default 'mean'), then:
xx  and yy  are tensors of arbitrary shapes with a total of nn  elements each.
The sum operation still operates over all the elements, and divides by nn .
The division by nn  can be avoided if one sets reduction = 'sum'.
Supports real-valued and complex-valued inputs.
Examples:


Applies Layer Normalization over a mini-batch of inputs as described in the paper Layer Normalization
The mean and standard-deviation are calculated separately over the last certain number dimensions which have to be of the shape specified by normalized_shape. γ\gamma  and β\beta  are learnable affine transform parameters of normalized_shape if elementwise_affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).
Note
Unlike Batch Normalization and Instance Normalization, which applies scalar scale and bias for each entire channel/plane with the affine option, Layer Normalization applies per-element scale and bias with elementwise_affine.
This layer uses statistics computed from input data in both training and evaluation modes.
normalized_shape (int or list or torch.Size) –
input shape from an expected input of size
If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size.
Examples:


A torch.nn.Conv1d module with lazy initialization of the in_channels argument of the Conv1d that is inferred from the input.size(1).
See also
torch.nn.Conv1d and torch.nn.modules.lazy.LazyModuleMixin
alias of Conv1d


A torch.nn.Conv2d module with lazy initialization of the in_channels argument of the Conv2d that is inferred from the input.size(1).
See also
torch.nn.Conv2d and torch.nn.modules.lazy.LazyModuleMixin
alias of Conv2d


A torch.nn.Conv3d module with lazy initialization of the in_channels argument of the Conv3d that is inferred from the input.size(1).
See also
torch.nn.Conv3d and torch.nn.modules.lazy.LazyModuleMixin
alias of Conv3d


A torch.nn.ConvTranspose1d module with lazy initialization of the in_channels argument of the ConvTranspose1d that is inferred from the input.size(1).
See also
torch.nn.ConvTranspose1d and torch.nn.modules.lazy.LazyModuleMixin
alias of ConvTranspose1d


A torch.nn.ConvTranspose2d module with lazy initialization of the in_channels argument of the ConvTranspose2d that is inferred from the input.size(1).
See also
torch.nn.ConvTranspose2d and torch.nn.modules.lazy.LazyModuleMixin
alias of ConvTranspose2d


A torch.nn.ConvTranspose3d module with lazy initialization of the in_channels argument of the ConvTranspose3d that is inferred from the input.size(1).
See also
torch.nn.ConvTranspose3d and torch.nn.modules.lazy.LazyModuleMixin
alias of ConvTranspose3d


A torch.nn.Linear module with lazy initialization.
In this module, the weight and bias are of torch.nn.UninitializedParameter class. They will be initialized after the first call to forward is done and the module will become a regular torch.nn.Linear module.
Check the torch.nn.modules.lazy.LazyModuleMixin for further documentation on lazy modules and their limitations.
alias of Linear


Applies the element-wise function:
or
Examples:


Applies a linear transformation to the incoming data: y=xAT+by = xA^T + b
This module supports TensorFloat32.
Examples:


Applies local response normalization over an input signal composed of several input planes, where channels occupy the second dimension. Applies normalization across channels.
Examples:


Applies the element-wise function:
Examples:


Applies the log(Softmax(x))\log(\text{Softmax}(x))  function to an n-dimensional input Tensor. The LogSoftmax formulation can be simplified as:
dim (int) – A dimension along which LogSoftmax will be computed.
a Tensor of the same dimension and shape as the input with values in the range [-inf, 0)
Examples:


Applies a 1D power-average pooling over an input signal composed of several input planes.
On each window, the function computed is:
Note
If the sum to the power of p is zero, the gradient of this function is not defined. This implementation will set the gradient to zero in this case.
Output: (N,C,Lout)(N, C, L_{out}) , where


Applies a 2D power-average pooling over an input signal composed of several input planes.
On each window, the function computed is:
The parameters kernel_size, stride can either be:
Note
If the sum to the power of p is zero, the gradient of this function is not defined. This implementation will set the gradient to zero in this case.
Output: (N,C,Hout,Wout)(N, C, H_{out}, W_{out}) , where
Examples:


Applies a multi-layer long short-term memory (LSTM) RNN to an input sequence.
For each element in the input sequence, each layer computes the following function:
where hth_t  is the hidden state at time t, ctc_t  is the cell state at time t, xtx_t  is the input at time t, ht−1h_{t-1}  is the hidden state of the layer at time t-1 or the initial hidden state at time 0, and iti_t , ftf_t , gtg_t , oto_t  are the input, forget, cell, and output gates, respectively. σ\sigma  is the sigmoid function, and ⊙\odot  is the Hadamard product.
In a multilayer LSTM, the input xt(l)x^{(l)}_t  of the ll  -th layer (l>=2l >= 2 ) is the hidden state ht(l−1)h^{(l-1)}_t  of the previous layer multiplied by dropout δt(l−1)\delta^{(l-1)}_t  where each δt(l−1)\delta^{(l-1)}_t  is a Bernoulli random variable which is 00  with probability dropout.
If proj_size > 0 is specified, LSTM with projections will be used. This changes the LSTM cell in the following way. First, the dimension of hth_t  will be changed from hidden_size to proj_size (dimensions of WhiW_{hi}  will be changed accordingly). Second, the output hidden state of each layer will be multiplied by a learnable projection matrix: ht=Whrhth_t = W_{hr}h_t . Note that as a consequence of this, the output of LSTM network will be of different shape as well. See Inputs/Outputs sections below for exact dimensions of all variables. You can find more details in https://arxiv.org/abs/1402.1128.
c_0 of shape (num_layers * num_directions, batch, hidden_size): tensor containing the initial cell state for each element in the batch.
If (h_0, c_0) is not provided, both h_0 and c_0 default to zero.
output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the LSTM, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence. If proj_size > 0 was specified, output shape will be (seq_len, batch, num_directions * proj_size).
For the unpacked case, the directions can be separated using output.view(seq_len, batch, num_directions, hidden_size), with forward and backward being direction 0 and 1 respectively. Similarly, the directions can be separated in the packed case.
h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len. If proj_size > 0 was specified, h_n shape will be (num_layers * num_directions, batch, proj_size).
Like output, the layers can be separated using h_n.view(num_layers, num_directions, batch, hidden_size) and similarly for c_n.
Note
All the weights and biases are initialized from U(−k,k)\mathcal{U}(-\sqrt{k}, \sqrt{k})  where k=1hidden_sizek = \frac{1}{\text{hidden\_size}}
Warning
There are known non-determinism issues for RNN functions on some versions of cuDNN and CUDA. You can enforce deterministic behavior by setting the following environment variables:
On CUDA 10.1, set environment variable CUDA_LAUNCH_BLOCKING=1. This may affect performance.
On CUDA 10.2 or later, set environment variable (note the leading colon symbol) CUBLAS_WORKSPACE_CONFIG=:16:8 or CUBLAS_WORKSPACE_CONFIG=:4096:2.
See the cuDNN 8 Release Notes for more information.
Note
If the following conditions are satisfied: 1) cudnn is enabled, 2) input data is on the GPU 3) input data has dtype torch.float16 4) V100 GPU is used, 5) input data is not in PackedSequence format persistent algorithm can be selected to improve performance.
Examples:


A long short-term memory (LSTM) cell.
where σ\sigma  is the sigmoid function, and ∗*  is the Hadamard product.
c_0 of shape (batch, hidden_size): tensor containing the initial cell state for each element in the batch.
If (h_0, c_0) is not provided, both h_0 and c_0 default to zero.
Note
All the weights and biases are initialized from U(−k,k)\mathcal{U}(-\sqrt{k}, \sqrt{k})  where k=1hidden_sizek = \frac{1}{\text{hidden\_size}}
Examples:


Creates a criterion that measures the loss given inputs x1x1 , x2x2 , two 1D mini-batch Tensors, and a label 1D mini-batch tensor yy  (containing 1 or -1).
If y=1y = 1  then it assumed the first input should be ranked higher (have a larger value) than the second input, and vice-versa for y=−1y = -1 .
The loss function for each pair of samples in the mini-batch is:
Examples:


Applies a 1D max pooling over an input signal composed of several input planes.
In the simplest case, the output value of the layer with input size (N,C,L)(N, C, L)  and output (N,C,Lout)(N, C, L_{out})  can be precisely described as:
If padding is non-zero, then the input is implicitly padded with negative infinity on both sides for padding number of points. dilation is the stride between the elements within the sliding window. This link has a nice visualization of the pooling parameters.
Note
When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored.
Output: (N,C,Lout)(N, C, L_{out}) , where
Examples:


Applies a 2D max pooling over an input signal composed of several input planes.
In the simplest case, the output value of the layer with input size (N,C,H,W)(N, C, H, W) , output (N,C,Hout,Wout)(N, C, H_{out}, W_{out})  and kernel_size (kH,kW)(kH, kW)  can be precisely described as:
If padding is non-zero, then the input is implicitly zero-padded on both sides for padding number of points. dilation controls the spacing between the kernel points. It is harder to describe, but this link has a nice visualization of what dilation does.
Note
When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored.
The parameters kernel_size, stride, padding, dilation can either be:
Output: (N,C,Hout,Wout)(N, C, H_{out}, W_{out}) , where
Examples:


Applies a 3D max pooling over an input signal composed of several input planes.
In the simplest case, the output value of the layer with input size (N,C,D,H,W)(N, C, D, H, W) , output (N,C,Dout,Hout,Wout)(N, C, D_{out}, H_{out}, W_{out})  and kernel_size (kD,kH,kW)(kD, kH, kW)  can be precisely described as:
If padding is non-zero, then the input is implicitly zero-padded on both sides for padding number of points. dilation controls the spacing between the kernel points. It is harder to describe, but this link has a nice visualization of what dilation does.
Note
When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored.
The parameters kernel_size, stride, padding, dilation can either be:
Output: (N,C,Dout,Hout,Wout)(N, C, D_{out}, H_{out}, W_{out}) , where
Examples:


Computes a partial inverse of MaxPool1d.
MaxPool1d is not fully invertible, since the non-maximal values are lost.
MaxUnpool1d takes in as input the output of MaxPool1d including the indices of the maximal values and computes a partial inverse in which all non-maximal values are set to zero.
Note
MaxPool1d can map several input sizes to the same output sizes. Hence, the inversion process can get ambiguous. To accommodate this, you can provide the needed output size as an additional argument output_size in the forward call. See the Inputs and Example below.
Output: (N,C,Hout)(N, C, H_{out}) , where
or as given by output_size in the call operator
Example:


Computes a partial inverse of MaxPool2d.
MaxPool2d is not fully invertible, since the non-maximal values are lost.
MaxUnpool2d takes in as input the output of MaxPool2d including the indices of the maximal values and computes a partial inverse in which all non-maximal values are set to zero.
Note
MaxPool2d can map several input sizes to the same output sizes. Hence, the inversion process can get ambiguous. To accommodate this, you can provide the needed output size as an additional argument output_size in the forward call. See the Inputs and Example below.
Output: (N,C,Hout,Wout)(N, C, H_{out}, W_{out}) , where
or as given by output_size in the call operator
Example:


Computes a partial inverse of MaxPool3d.
MaxPool3d is not fully invertible, since the non-maximal values are lost. MaxUnpool3d takes in as input the output of MaxPool3d including the indices of the maximal values and computes a partial inverse in which all non-maximal values are set to zero.
Note
MaxPool3d can map several input sizes to the same output sizes. Hence, the inversion process can get ambiguous. To accommodate this, you can provide the needed output size as an additional argument output_size in the forward call. See the Inputs section below.
Output: (N,C,Dout,Hout,Wout)(N, C, D_{out}, H_{out}, W_{out}) , where
or as given by output_size in the call operator
Example: