torch.nn.functional.conv_transpose2d(input, weight, bias=None, stride=1, padding=0, output_padding=0, groups=1, dilation=1) → Tensor
Applies a 2D transposed convolution operator over an input image composed of several input planes, sometimes also called “deconvolution”. This operator supports TensorFloat32. See ConvTranspose2d for details and output shape.  Note In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.   Parameters

input – input tensor of shape (minibatch,in_channels,iH,iW)(\text{minibatch} , \text{in\_channels} , iH , iW)

weight – filters of shape (in_channels,out_channelsgroups,kH,kW)(\text{in\_channels} , \frac{\text{out\_channels}}{\text{groups}} , kH , kW)

bias – optional bias of shape (out_channels)(\text{out\_channels}) . Default: None
stride – the stride of the convolving kernel. Can be a single number or a tuple (sH, sW). Default: 1
padding – dilation * (kernel_size - 1) - padding zero-padding will be added to both sides of each dimension in the input. Can be a single number or a tuple (padH, padW). Default: 0
output_padding – additional size added to one side of each dimension in the output shape. Can be a single number or a tuple (out_padH, out_padW). Default: 0
groups – split input into groups, in_channels\text{in\_channels}  should be divisible by the number of groups. Default: 1
dilation – the spacing between kernel elements. Can be a single number or a tuple (dH, dW). Default: 1    Examples: >>> # With square kernels and equal stride
>>> inputs = torch.randn(1, 4, 5, 5)
>>> weights = torch.randn(4, 8, 3, 3)
>>> F.conv_transpose2d(inputs, weights, padding=1)


torch.nn.functional.conv_transpose3d(input, weight, bias=None, stride=1, padding=0, output_padding=0, groups=1, dilation=1) → Tensor
Applies a 3D transposed convolution operator over an input image composed of several input planes, sometimes also called “deconvolution” This operator supports TensorFloat32. See ConvTranspose3d for details and output shape.  Note In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.   Parameters

input – input tensor of shape (minibatch,in_channels,iT,iH,iW)(\text{minibatch} , \text{in\_channels} , iT , iH , iW)

weight – filters of shape (in_channels,out_channelsgroups,kT,kH,kW)(\text{in\_channels} , \frac{\text{out\_channels}}{\text{groups}} , kT , kH , kW)

bias – optional bias of shape (out_channels)(\text{out\_channels}) . Default: None
stride – the stride of the convolving kernel. Can be a single number or a tuple (sT, sH, sW). Default: 1
padding – dilation * (kernel_size - 1) - padding zero-padding will be added to both sides of each dimension in the input. Can be a single number or a tuple (padT, padH, padW). Default: 0
output_padding – additional size added to one side of each dimension in the output shape. Can be a single number or a tuple (out_padT, out_padH, out_padW). Default: 0
groups – split input into groups, in_channels\text{in\_channels}  should be divisible by the number of groups. Default: 1
dilation – the spacing between kernel elements. Can be a single number or a tuple (dT, dH, dW). Default: 1    Examples: >>> inputs = torch.randn(20, 16, 50, 10, 20)
>>> weights = torch.randn(16, 33, 3, 3, 3)
>>> F.conv_transpose3d(inputs, weights)


torch.nn.functional.cosine_embedding_loss(input1, input2, target, margin=0, size_average=None, reduce=None, reduction='mean') → Tensor [source]

See CosineEmbeddingLoss for details.


torch.nn.functional.cosine_similarity(x1, x2, dim=1, eps=1e-8) → Tensor
Returns cosine similarity between x1 and x2, computed along dim.  similarity=x1⋅x2max(∥x1∥2⋅∥x2∥2,ϵ)\text{similarity} = \dfrac{x_1 \cdot x_2}{\max(\Vert x_1 \Vert _2 \cdot \Vert x_2 \Vert _2, \epsilon)}
 Parameters

x1 (Tensor) – First input.
x2 (Tensor) – Second input (of size matching x1).
dim (int, optional) – Dimension of vectors. Default: 1
eps (float, optional) – Small value to avoid division by zero. Default: 1e-8     Shape:

 Input: (∗1,D,∗2)(\ast_1, D, \ast_2)  where D is at position dim. Output: (∗1,∗2)(\ast_1, \ast_2)  where 1 is at position dim.    Example: >>> input1 = torch.randn(100, 128)
>>> input2 = torch.randn(100, 128)
>>> output = F.cosine_similarity(input1, input2)
>>> print(output)


torch.nn.functional.cross_entropy(input, target, weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean') [source]

This criterion combines log_softmax and nll_loss in a single function. See CrossEntropyLoss for details.  Parameters

input (Tensor) – (N,C)(N, C)  where C = number of classes or (N,C,H,W)(N, C, H, W)  in case of 2D Loss, or (N,C,d1,d2,...,dK)(N, C, d_1, d_2, ..., d_K)  where K≥1K \geq 1  in the case of K-dimensional loss.
target (Tensor) – (N)(N)  where each value is 0≤targets[i]≤C−10 \leq \text{targets}[i] \leq C-1 , or (N,d1,d2,...,dK)(N, d_1, d_2, ..., d_K)  where K≥1K \geq 1  for K-dimensional loss.
weight (Tensor, optional) – a manual rescaling weight given to each class. If given, has to be a Tensor of size C

size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

ignore_index (int, optional) – Specifies a target value that is ignored and does not contribute to the input gradient. When size_average is True, the loss is averaged over non-ignored targets. Default: -100
reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'
    Examples: >>> input = torch.randn(3, 5, requires_grad=True)
>>> target = torch.randint(5, (3,), dtype=torch.int64)
>>> loss = F.cross_entropy(input, target)
>>> loss.backward()


torch.nn.functional.ctc_loss(log_probs, targets, input_lengths, target_lengths, blank=0, reduction='mean', zero_infinity=False) [source]

The Connectionist Temporal Classification loss. See CTCLoss for details.  Note In some circumstances when given tensors on a CUDA device and using CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. See Reproducibility for more information.   Note This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.   Parameters

log_probs – (T,N,C)(T, N, C)  where C = number of characters in alphabet including blank, T = input length, and N = batch size. The logarithmized probabilities of the outputs (e.g. obtained with torch.nn.functional.log_softmax()).
targets – (N,S)(N, S)  or (sum(target_lengths)). Targets cannot be blank. In the second form, the targets are assumed to be concatenated.
input_lengths – (N)(N) . Lengths of the inputs (must each be ≤T\leq T )
target_lengths – (N)(N) . Lengths of the targets
blank (int, optional) – Blank label. Default 00 .
reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the output losses will be divided by the target lengths and then the mean over the batch is taken, 'sum': the output will be summed. Default: 'mean'

zero_infinity (bool, optional) – Whether to zero infinite losses and the associated gradients. Default: False Infinite losses mainly occur when the inputs are too short to be aligned to the targets.    Example: >>> log_probs = torch.randn(50, 16, 20).log_softmax(2).detach().requires_grad_()
>>> targets = torch.randint(1, 20, (16, 30), dtype=torch.long)
>>> input_lengths = torch.full((16,), 50, dtype=torch.long)
>>> target_lengths = torch.randint(10,30,(16,), dtype=torch.long)
>>> loss = F.ctc_loss(log_probs, targets, input_lengths, target_lengths)
>>> loss.backward()


torch.nn.functional.dropout(input, p=0.5, training=True, inplace=False) [source]

During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. See Dropout for details.  Parameters

p – probability of an element to be zeroed. Default: 0.5
training – apply dropout if is True. Default: True

inplace – If set to True, will do this operation in-place. Default: False


torch.nn.functional.dropout2d(input, p=0.5, training=True, inplace=False) [source]

Randomly zero out entire channels (a channel is a 2D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 2D tensor input[i,j]\text{input}[i, j] ) of the input tensor). Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution. See Dropout2d for details.  Parameters

p – probability of a channel to be zeroed. Default: 0.5
training – apply dropout if is True. Default: True

inplace – If set to True, will do this operation in-place. Default: False


torch.nn.functional.dropout3d(input, p=0.5, training=True, inplace=False) [source]

Randomly zero out entire channels (a channel is a 3D feature map, e.g., the jj -th channel of the ii -th sample in the batched input is a 3D tensor input[i,j]\text{input}[i, j] ) of the input tensor). Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution. See Dropout3d for details.  Parameters

p – probability of a channel to be zeroed. Default: 0.5
training – apply dropout if is True. Default: True

inplace – If set to True, will do this operation in-place. Default: False


torch.nn.functional.elu(input, alpha=1.0, inplace=False) [source]

Applies element-wise, ELU(x)=max(0,x)+min(0,α∗(exp(x)−1))\text{ELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x) - 1)) . See ELU for more details.


torch.nn.functional.elu_(input, alpha=1.) → Tensor
In-place version of elu().


torch.nn.functional.embedding(input, weight, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False) [source]

A simple lookup table that looks up embeddings in a fixed dictionary and size. This module is often used to retrieve word embeddings using indices. The input to the module is a list of indices, and the embedding matrix, and the output is the corresponding word embeddings. See torch.nn.Embedding for more details.  Parameters

input (LongTensor) – Tensor containing indices into the embedding matrix
weight (Tensor) – The embedding matrix with number of rows equal to the maximum possible index + 1, and number of columns equal to the embedding size
padding_idx (int, optional) – If given, pads the output with the embedding vector at padding_idx (initialized to zeros) whenever it encounters the index.
max_norm (float, optional) – If given, each embedding vector with norm larger than max_norm is renormalized to have norm max_norm. Note: this will modify weight in-place.
norm_type (float, optional) – The p of the p-norm to compute for the max_norm option. Default 2.
scale_grad_by_freq (boolean, optional) – If given, this will scale gradients by the inverse of frequency of the words in the mini-batch. Default False.
sparse (bool, optional) – If True, gradient w.r.t. weight will be a sparse tensor. See Notes under torch.nn.Embedding for more details regarding sparse gradients.     Shape:

 Input: LongTensor of arbitrary shape containing the indices to extract

Weight: Embedding matrix of floating point type with shape (V, embedding_dim),

where V = maximum index + 1 and embedding_dim = the embedding size    Output: (*, embedding_dim), where * is the input shape    Examples: >>> # a batch of 2 samples of 4 indices each
>>> input = torch.tensor([[1,2,4,5],[4,3,2,9]])
>>> # an embedding matrix containing 10 tensors of size 3
>>> embedding_matrix = torch.rand(10, 3)
>>> F.embedding(input, embedding_matrix)
tensor([[[ 0.8490,  0.9625,  0.6753],
         [ 0.9666,  0.7761,  0.6108],
         [ 0.6246,  0.9751,  0.3618],
         [ 0.4161,  0.2419,  0.7383]],

        [[ 0.6246,  0.9751,  0.3618],
         [ 0.0237,  0.7794,  0.0528],
         [ 0.9666,  0.7761,  0.6108],
         [ 0.3385,  0.8612,  0.1867]]])

>>> # example with padding_idx
>>> weights = torch.rand(10, 3)
>>> weights[0, :].zero_()
>>> embedding_matrix = weights
>>> input = torch.tensor([[0,2,0,5]])
>>> F.embedding(input, embedding_matrix, padding_idx=0)
tensor([[[ 0.0000,  0.0000,  0.0000],
         [ 0.5609,  0.5384,  0.8720],
         [ 0.0000,  0.0000,  0.0000],
         [ 0.6262,  0.2438,  0.7471]]])


torch.nn.functional.embedding_bag(input, weight, offsets=None, max_norm=None, norm_type=2, scale_grad_by_freq=False, mode='mean', sparse=False, per_sample_weights=None, include_last_offset=False) [source]

Computes sums, means or maxes of bags of embeddings, without instantiating the intermediate embeddings. See torch.nn.EmbeddingBag for more details.  Note This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.   Parameters

input (LongTensor) – Tensor containing bags of indices into the embedding matrix
weight (Tensor) – The embedding matrix with number of rows equal to the maximum possible index + 1, and number of columns equal to the embedding size
offsets (LongTensor, optional) – Only used when input is 1D. offsets determines the starting index position of each bag (sequence) in input.
max_norm (float, optional) – If given, each embedding vector with norm larger than max_norm is renormalized to have norm max_norm. Note: this will modify weight in-place.
norm_type (float, optional) – The p in the p-norm to compute for the max_norm option. Default 2.
scale_grad_by_freq (boolean, optional) – if given, this will scale gradients by the inverse of frequency of the words in the mini-batch. Default False. Note: this option is not supported when mode="max".
mode (string, optional) – "sum", "mean" or "max". Specifies the way to reduce the bag. Default: "mean"

sparse (bool, optional) – if True, gradient w.r.t. weight will be a sparse tensor. See Notes under torch.nn.Embedding for more details regarding sparse gradients. Note: this option is not supported when mode="max".
per_sample_weights (Tensor, optional) – a tensor of float / double weights, or None to indicate all weights should be taken to be 1. If specified, per_sample_weights must have exactly the same shape as input and is treated as having the same offsets, if those are not None.
include_last_offset (bool, optional) – if True, the size of offsets is equal to the number of bags + 1.
last element is the size of the input, or the ending index position of the last bag (The) –     Shape:
input (LongTensor) and offsets (LongTensor, optional)
If input is 2D of shape (B, N), it will be treated as B bags (sequences) each of fixed length N, and this will return B values aggregated in a way depending on the mode. offsets is ignored and required to be None in this case.
If input is 1D of shape (N), it will be treated as a concatenation of multiple bags (sequences). offsets is required to be a 1D tensor containing the starting index positions of each bag in input. Therefore, for offsets of shape (B), input will be viewed as having B bags. Empty bags (i.e., having 0-length) will have returned vectors filled by zeros.
weight (Tensor): the learnable weights of the module of shape (num_embeddings, embedding_dim)

per_sample_weights (Tensor, optional). Has the same shape as input.
output: aggregated embedding values of shape (B, embedding_dim)
  Examples: >>> # an Embedding module containing 10 tensors of size 3
>>> embedding_matrix = torch.rand(10, 3)
>>> # a batch of 2 samples of 4 indices each
>>> input = torch.tensor([1,2,4,5,4,3,2,9])
>>> offsets = torch.tensor([0,4])
>>> F.embedding_bag(embedding_matrix, input, offsets)
tensor([[ 0.3397,  0.3552,  0.5545],
        [ 0.5893,  0.4386,  0.5882]])


torch.nn.functional.feature_alpha_dropout(input, p=0.5, training=False, inplace=False) [source]

Randomly masks out entire channels (a channel is a feature map, e.g. the jj -th channel of the ii -th sample in the batch input is a tensor input[i,j]\text{input}[i, j] ) of the input tensor). Instead of setting activations to zero, as in regular Dropout, the activations are set to the negative saturation value of the SELU activation function. Each element will be masked independently on every forward call with probability p using samples from a Bernoulli distribution. The elements to be masked are randomized on every forward call, and scaled and shifted to maintain zero mean and unit variance. See FeatureAlphaDropout for details.  Parameters

p – dropout probability of a channel to be zeroed. Default: 0.5
training – apply dropout if is True. Default: True

inplace – If set to True, will do this operation in-place. Default: False


torch.nn.functional.fold(input, output_size, kernel_size, dilation=1, padding=0, stride=1) [source]

Combines an array of sliding local blocks into a large containing tensor.  Warning Currently, only 3-D output tensors (unfolded batched image-like tensors) are supported.  See torch.nn.Fold for details


torch.nn.functional.gelu(input) → Tensor [source]

Applies element-wise the function GELU(x)=x∗Φ(x)\text{GELU}(x) = x * \Phi(x)  where Φ(x)\Phi(x)  is the Cumulative Distribution Function for Gaussian Distribution. See Gaussian Error Linear Units (GELUs).


torch.nn.functional.glu(input, dim=-1) → Tensor [source]

The gated linear unit. Computes:  GLU(a,b)=a⊗σ(b)\text{GLU}(a, b) = a \otimes \sigma(b)
where input is split in half along dim to form a and b, σ\sigma  is the sigmoid function and ⊗\otimes  is the element-wise product between matrices. See Language Modeling with Gated Convolutional Networks.  Parameters

input (Tensor) – input tensor
dim (int) – dimension on which to split the input. Default: -1


torch.nn.functional.grid_sample(input, grid, mode='bilinear', padding_mode='zeros', align_corners=None) [source]

Given an input and a flow-field grid, computes the output using input values and pixel locations from grid. Currently, only spatial (4-D) and volumetric (5-D) input are supported. In the spatial (4-D) case, for input with shape (N,C,Hin,Win)(N, C, H_\text{in}, W_\text{in})  and grid with shape (N,Hout,Wout,2)(N, H_\text{out}, W_\text{out}, 2) , the output will have shape (N,C,Hout,Wout)(N, C, H_\text{out}, W_\text{out}) . For each output location output[n, :, h, w], the size-2 vector grid[n, h, w] specifies input pixel locations x and y, which are used to interpolate the output value output[n, :, h, w]. In the case of 5D inputs, grid[n, d, h, w] specifies the x, y, z pixel locations for interpolating output[n, :, d, h, w]. mode argument specifies nearest or bilinear interpolation method to sample the input pixels. grid specifies the sampling pixel locations normalized by the input spatial dimensions. Therefore, it should have most values in the range of [-1, 1]. For example, values x = -1, y = -1 is the left-top pixel of input, and values x = 1, y = 1 is the right-bottom pixel of input. If grid has values outside the range of [-1, 1], the corresponding outputs are handled as defined by padding_mode. Options are
padding_mode="zeros": use 0 for out-of-bound grid locations,
padding_mode="border": use border values for out-of-bound grid locations,
padding_mode="reflection": use values at locations reflected by the border for out-of-bound grid locations. For location far away from the border, it will keep being reflected until becoming in bound, e.g., (normalized) pixel location x = -3.5 reflects by border -1 and becomes x' = 1.5, then reflects by border 1 and becomes x'' = -0.5.   Note This function is often used in conjunction with affine_grid() to build Spatial Transformer Networks .   Note When using the CUDA backend, this operation may induce nondeterministic behaviour in its backward pass that is not easily switched off. Please see the notes on Reproducibility for background.   Note NaN values in grid would be interpreted as -1.   Parameters

input (Tensor) – input of shape (N,C,Hin,Win)(N, C, H_\text{in}, W_\text{in})  (4-D case) or (N,C,Din,Hin,Win)(N, C, D_\text{in}, H_\text{in}, W_\text{in})  (5-D case)
grid (Tensor) – flow-field of shape (N,Hout,Wout,2)(N, H_\text{out}, W_\text{out}, 2)  (4-D case) or (N,Dout,Hout,Wout,3)(N, D_\text{out}, H_\text{out}, W_\text{out}, 3)  (5-D case)
mode (str) – interpolation mode to calculate output values 'bilinear' | 'nearest' | 'bicubic'. Default: 'bilinear' Note: mode='bicubic' supports only 4-D input. When mode='bilinear' and the input is 5-D, the interpolation mode used internally will actually be trilinear. However, when the input is 4-D, the interpolation mode will legitimately be bilinear.
padding_mode (str) – padding mode for outside grid values 'zeros' | 'border' | 'reflection'. Default: 'zeros'

align_corners (bool, optional) – Geometrically, we consider the pixels of the input as squares rather than points. If set to True, the extrema (-1 and 1) are considered as referring to the center points of the input’s corner pixels. If set to False, they are instead considered as referring to the corner points of the input’s corner pixels, making the sampling more resolution agnostic. This option parallels the align_corners option in interpolate(), and so whichever option is used here should also be used there to resize the input image before grid sampling. Default: False
   Returns
output Tensor  Return type
output (Tensor)    Warning When align_corners = True, the grid positions depend on the pixel size relative to the input image size, and so the locations sampled by grid_sample() will differ for the same input given at different resolutions (that is, after being upsampled or downsampled). The default behavior up to version 1.2.0 was align_corners = True. Since then, the default behavior has been changed to align_corners = False, in order to bring it in line with the default for interpolate().   Note mode='bicubic' is implemented using the cubic convolution algorithm with α=−0.75\alpha=-0.75 . The constant α\alpha  might be different from packages to packages. For example, PIL and OpenCV use -0.5 and -0.75 respectively. This algorithm may “overshoot” the range of values it’s interpolating. For example, it may produce negative values or values greater than 255 when interpolating input in [0, 255]. Clamp the results with :func: torch.clamp to ensure they are within the valid range.


torch.nn.functional.gumbel_softmax(logits, tau=1, hard=False, eps=1e-10, dim=-1) [source]

Samples from the Gumbel-Softmax distribution (Link 1 Link 2) and optionally discretizes.  Parameters

logits – […, num_features] unnormalized log probabilities
tau – non-negative scalar temperature
hard – if True, the returned samples will be discretized as one-hot vectors, but will be differentiated as if it is the soft sample in autograd
dim (int) – A dimension along which softmax will be computed. Default: -1.   Returns
Sampled tensor of same shape as logits from the Gumbel-Softmax distribution. If hard=True, the returned samples will be one-hot, otherwise they will be probability distributions that sum to 1 across dim.    Note This function is here for legacy reasons, may be removed from nn.Functional in the future.   Note The main trick for hard is to do y_hard - y_soft.detach() + y_soft It achieves two things: - makes the output value exactly one-hot (since we add then subtract y_soft value) - makes the gradient equal to y_soft gradient (since we strip all other gradients)   Examples::

>>> logits = torch.randn(20, 32)
>>> # Sample soft categorical using reparametrization trick:
>>> F.gumbel_softmax(logits, tau=1, hard=False)
>>> # Sample hard categorical using "Straight-through" trick:
>>> F.gumbel_softmax(logits, tau=1, hard=True)


torch.nn.functional.hardshrink(input, lambd=0.5) → Tensor [source]

Applies the hard shrinkage function element-wise See Hardshrink for more details.


torch.nn.functional.hardsigmoid(input) → Tensor [source]

Applies the element-wise function  Hardsigmoid(x)={0if x≤−3,1if x≥+3,x/6+1/2otherwise\text{Hardsigmoid}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ 1 & \text{if~} x \ge +3, \\ x / 6 + 1 / 2 & \text{otherwise} \end{cases}
 Parameters
inplace – If set to True, will do this operation in-place. Default: False   See Hardsigmoid for more details.


torch.nn.functional.hardswish(input, inplace=False) [source]

Applies the hardswish function, element-wise, as described in the paper: Searching for MobileNetV3.  Hardswish(x)={0if x≤−3,xif x≥+3,x⋅(x+3)/6otherwise\text{Hardswish}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ x & \text{if~} x \ge +3, \\ x \cdot (x + 3) /6 & \text{otherwise} \end{cases}
See Hardswish for more details.


torch.nn.functional.hardtanh(input, min_val=-1., max_val=1., inplace=False) → Tensor [source]

Applies the HardTanh function element-wise. See Hardtanh for more details.


torch.nn.functional.hardtanh_(input, min_val=-1., max_val=1.) → Tensor
In-place version of hardtanh().


torch.nn.functional.hinge_embedding_loss(input, target, margin=1.0, size_average=None, reduce=None, reduction='mean') → Tensor [source]

See HingeEmbeddingLoss for details.


torch.nn.functional.instance_norm(input, running_mean=None, running_var=None, weight=None, bias=None, use_input_stats=True, momentum=0.1, eps=1e-05) [source]

Applies Instance Normalization for each channel in each data sample in a batch. See InstanceNorm1d, InstanceNorm2d, InstanceNorm3d for details.


torch.nn.functional.interpolate(input, size=None, scale_factor=None, mode='nearest', align_corners=None, recompute_scale_factor=None) [source]

Down/up samples the input to either the given size or the given scale_factor The algorithm used for interpolation is determined by mode. Currently temporal, spatial and volumetric sampling are supported, i.e. expected inputs are 3-D, 4-D or 5-D in shape. The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width. The modes available for resizing are: nearest, linear (3D-only), bilinear, bicubic (4D-only), trilinear (5D-only), area  Parameters

input (Tensor) – the input tensor
size (int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int]) – output spatial size.
scale_factor (float or Tuple[float]) – multiplier for spatial size. Has to match input size if it is a tuple.
mode (str) – algorithm used for upsampling: 'nearest' | 'linear' | 'bilinear' | 'bicubic' | 'trilinear' | 'area'. Default: 'nearest'

align_corners (bool, optional) – Geometrically, we consider the pixels of the input and output as squares rather than points. If set to True, the input and output tensors are aligned by the center points of their corner pixels, preserving the values at the corner pixels. If set to False, the input and output tensors are aligned by the corner points of their corner pixels, and the interpolation uses edge value padding for out-of-boundary values, making this operation independent of input size when scale_factor is kept the same. This only has an effect when mode is 'linear', 'bilinear', 'bicubic' or 'trilinear'. Default: False

recompute_scale_factor (bool, optional) – recompute the scale_factor for use in the interpolation calculation. When scale_factor is passed as a parameter, it is used to compute the output_size. If recompute_scale_factor is False or not specified, the passed-in scale_factor will be used in the interpolation computation. Otherwise, a new scale_factor will be computed based on the output and input sizes for use in the interpolation computation (i.e. the computation will be identical to if the computed output_size were passed-in explicitly). Note that when scale_factor is floating-point, the recomputed scale_factor may differ from the one passed in due to rounding and precision issues.     Note With mode='bicubic', it’s possible to cause overshoot, in other words it can produce negative values or values greater than 255 for images. Explicitly call result.clamp(min=0, max=255) if you want to reduce the overshoot when displaying the image.   Warning With align_corners = True, the linearly interpolating modes (linear, bilinear, and trilinear) don’t proportionally align the output and input pixels, and thus the output values can depend on the input size. This was the default behavior for these modes up to version 0.3.1. Since then, the default behavior is align_corners = False. See Upsample for concrete examples on how this affects the outputs.   Warning When scale_factor is specified, if recompute_scale_factor=True, scale_factor is used to compute the output_size which will then be used to infer new scales for the interpolation. The default behavior for recompute_scale_factor changed to False in 1.6.0, and scale_factor is used in the interpolation calculation.   Note This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.


torch.nn.functional.kl_div(input, target, size_average=None, reduce=None, reduction='mean', log_target=False) [source]

The Kullback-Leibler divergence Loss See KLDivLoss for details.  Parameters

input – Tensor of arbitrary shape
target – Tensor of the same shape as input
size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'batchmean' | 'sum' | 'mean'. 'none': no reduction will be applied 'batchmean': the sum of the output will be divided by the batchsize 'sum': the output will be summed 'mean': the output will be divided by the number of elements in the output Default: 'mean'

log_target (bool) – A flag indicating whether target is passed in the log space. It is recommended to pass certain distributions (like softmax) in the log space to avoid numerical issues caused by explicit log. Default: False
     Note size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction.   Note :attr:reduction = 'mean' doesn’t return the true kl divergence value, please use :attr:reduction = 'batchmean' which aligns with KL math definition. In the next major release, 'mean' will be changed to be the same as ‘batchmean’.


torch.nn.functional.l1_loss(input, target, size_average=None, reduce=None, reduction='mean') → Tensor [source]

Function that takes the mean element-wise absolute value difference. See L1Loss for details.


torch.nn.functional.layer_norm(input, normalized_shape, weight=None, bias=None, eps=1e-05) [source]

Applies Layer Normalization for last certain number of dimensions. See LayerNorm for details.


torch.nn.functional.leaky_relu(input, negative_slope=0.01, inplace=False) → Tensor [source]

Applies element-wise, LeakyReLU(x)=max(0,x)+negative_slope∗min(0,x)\text{LeakyReLU}(x) = \max(0, x) + \text{negative\_slope} * \min(0, x)  See LeakyReLU for more details.


torch.nn.functional.leaky_relu_(input, negative_slope=0.01) → Tensor
In-place version of leaky_relu().


torch.nn.functional.linear(input, weight, bias=None) [source]

Applies a linear transformation to the incoming data: y=xAT+by = xA^T + b . This operator supports TensorFloat32. Shape:  Input: (N,∗,in_features)(N, *, in\_features)  N is the batch size, * means any number of additional dimensions Weight: (out_features,in_features)(out\_features, in\_features)
 Bias: (out_features)(out\_features)
 Output: (N,∗,out_features)(N, *, out\_features)


torch.nn.functional.local_response_norm(input, size, alpha=0.0001, beta=0.75, k=1.0) [source]

Applies local response normalization over an input signal composed of several input planes, where channels occupy the second dimension. Applies normalization across channels. See LocalResponseNorm for details.


torch.nn.functional.logsigmoid(input) → Tensor
Applies element-wise LogSigmoid(xi)=log(11+exp(−xi))\text{LogSigmoid}(x_i) = \log \left(\frac{1}{1 + \exp(-x_i)}\right)  See LogSigmoid for more details.


torch.nn.functional.log_softmax(input, dim=None, _stacklevel=3, dtype=None) [source]

Applies a softmax followed by a logarithm. While mathematically equivalent to log(softmax(x)), doing these two operations separately is slower, and numerically unstable. This function uses an alternative formulation to compute the output and gradient correctly. See LogSoftmax for more details.  Parameters

input (Tensor) – input
dim (int) – A dimension along which log_softmax will be computed.
dtype (torch.dtype, optional) – the desired data type of returned tensor. If specified, the input tensor is casted to dtype before the operation is performed. This is useful for preventing data type overflows. Default: None.


torch.nn.functional.lp_pool1d(input, norm_type, kernel_size, stride=None, ceil_mode=False) [source]

Applies a 1D power-average pooling over an input signal composed of several input planes. If the sum of all inputs to the power of p is zero, the gradient is set to zero as well. See LPPool1d for details.


torch.nn.functional.lp_pool2d(input, norm_type, kernel_size, stride=None, ceil_mode=False) [source]

Applies a 2D power-average pooling over an input signal composed of several input planes. If the sum of all inputs to the power of p is zero, the gradient is set to zero as well. See LPPool2d for details.


torch.nn.functional.margin_ranking_loss(input1, input2, target, margin=0, size_average=None, reduce=None, reduction='mean') → Tensor [source]

See MarginRankingLoss for details.


torch.nn.functional.max_pool1d(*args, **kwargs)
Applies a 1D max pooling over an input signal composed of several input planes. See MaxPool1d for details.


torch.nn.functional.max_pool2d(*args, **kwargs)
Applies a 2D max pooling over an input signal composed of several input planes. See MaxPool2d for details.


torch.nn.functional.max_pool3d(*args, **kwargs)
Applies a 3D max pooling over an input signal composed of several input planes. See MaxPool3d for details.


torch.nn.functional.max_unpool1d(input, indices, kernel_size, stride=None, padding=0, output_size=None) [source]

Computes a partial inverse of MaxPool1d. See MaxUnpool1d for details.


torch.nn.functional.max_unpool2d(input, indices, kernel_size, stride=None, padding=0, output_size=None) [source]

Computes a partial inverse of MaxPool2d. See MaxUnpool2d for details.


torch.nn.functional.max_unpool3d(input, indices, kernel_size, stride=None, padding=0, output_size=None) [source]

Computes a partial inverse of MaxPool3d. See MaxUnpool3d for details.


torch.nn.functional.mse_loss(input, target, size_average=None, reduce=None, reduction='mean') → Tensor [source]

Measures the element-wise mean squared error. See MSELoss for details.


torch.nn.functional.multilabel_margin_loss(input, target, size_average=None, reduce=None, reduction='mean') → Tensor [source]

See MultiLabelMarginLoss for details.


torch.nn.functional.multilabel_soft_margin_loss(input, target, weight=None, size_average=None) → Tensor [source]

See MultiLabelSoftMarginLoss for details.


torch.nn.functional.multi_margin_loss(input, target, p=1, margin=1.0, weight=None, size_average=None, reduce=None, reduction='mean') [source]

 multi_margin_loss(input, target, p=1, margin=1, weight=None, size_average=None,

reduce=None, reduction=’mean’) -> Tensor   See MultiMarginLoss for details.


torch.nn.functional.nll_loss(input, target, weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean') [source]

The negative log likelihood loss. See NLLLoss for details.  Parameters

input – (N,C)(N, C)  where C = number of classes or (N,C,H,W)(N, C, H, W)  in case of 2D Loss, or (N,C,d1,d2,...,dK)(N, C, d_1, d_2, ..., d_K)  where K≥1K \geq 1  in the case of K-dimensional loss.
target – (N)(N)  where each value is 0≤targets[i]≤C−10 \leq \text{targets}[i] \leq C-1 , or (N,d1,d2,...,dK)(N, d_1, d_2, ..., d_K)  where K≥1K \geq 1  for K-dimensional loss.
weight (Tensor, optional) – a manual rescaling weight given to each class. If given, has to be a Tensor of size C

size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

ignore_index (int, optional) – Specifies a target value that is ignored and does not contribute to the input gradient. When size_average is True, the loss is averaged over non-ignored targets. Default: -100
reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'
    Example: >>> # input is of size N x C = 3 x 5
>>> input = torch.randn(3, 5, requires_grad=True)
>>> # each element in target has to have 0 <= value < C
>>> target = torch.tensor([1, 0, 4])
>>> output = F.nll_loss(F.log_softmax(input), target)
>>> output.backward()


torch.nn.functional.normalize(input, p=2, dim=1, eps=1e-12, out=None) [source]

Performs LpL_p  normalization of inputs over specified dimension. For a tensor input of sizes (n0,...,ndim,...,nk)(n_0, ..., n_{dim}, ..., n_k) , each ndimn_{dim}  -element vector vv  along dimension dim is transformed as  v=vmax(∥v∥p,ϵ).v = \frac{v}{\max(\lVert v \rVert_p, \epsilon)}.
With the default arguments it uses the Euclidean norm over vectors along dimension 11  for normalization.  Parameters

input – input tensor of any shape
p (float) – the exponent value in the norm formulation. Default: 2
dim (int) – the dimension to reduce. Default: 1
eps (float) – small value to avoid division by zero. Default: 1e-12
out (Tensor, optional) – the output tensor. If out is used, this operation won’t be differentiable.


torch.nn.functional.one_hot(tensor, num_classes=-1) → LongTensor
Takes LongTensor with index values of shape (*) and returns a tensor of shape (*, num_classes) that have zeros everywhere except where the index of last dimension matches the corresponding value of the input tensor, in which case it will be 1. See also One-hot on Wikipedia .  Parameters

tensor (LongTensor) – class values of any shape.
num_classes (int) – Total number of classes. If set to -1, the number of classes will be inferred as one greater than the largest class value in the input tensor.   Returns
LongTensor that has one more dimension with 1 values at the index of last dimension indicated by the input, and 0 everywhere else.   Examples >>> F.one_hot(torch.arange(0, 5) % 3)
tensor([[1, 0, 0],
        [0, 1, 0],
        [0, 0, 1],
        [1, 0, 0],
        [0, 1, 0]])
>>> F.one_hot(torch.arange(0, 5) % 3, num_classes=5)
tensor([[1, 0, 0, 0, 0],
        [0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0],
        [1, 0, 0, 0, 0],
        [0, 1, 0, 0, 0]])
>>> F.one_hot(torch.arange(0, 6).view(3,2) % 3)
tensor([[[1, 0, 0],
         [0, 1, 0]],
        [[0, 0, 1],
         [1, 0, 0]],
        [[0, 1, 0],
         [0, 0, 1]]])


torch.nn.functional.pad(input, pad, mode='constant', value=0)
Pads tensor.  Padding size:

The padding size by which to pad some dimensions of input are described starting from the last dimension and moving forward. ⌊len(pad)2⌋\left\lfloor\frac{\text{len(pad)}}{2}\right\rfloor  dimensions of input will be padded. For example, to pad only the last dimension of the input tensor, then pad has the form (padding_left,padding_right)(\text{padding\_left}, \text{padding\_right}) ; to pad the last 2 dimensions of the input tensor, then use (padding_left,padding_right,(\text{padding\_left}, \text{padding\_right},  padding_top,padding_bottom)\text{padding\_top}, \text{padding\_bottom}) ; to pad the last 3 dimensions, use (padding_left,padding_right,(\text{padding\_left}, \text{padding\_right},  padding_top,padding_bottom\text{padding\_top}, \text{padding\_bottom}  padding_front,padding_back)\text{padding\_front}, \text{padding\_back}) .  Padding mode:

See torch.nn.ConstantPad2d, torch.nn.ReflectionPad2d, and torch.nn.ReplicationPad2d for concrete examples on how each of the padding modes works. Constant padding is implemented for arbitrary dimensions. Replicate padding is implemented for padding the last 3 dimensions of 5D input tensor, or the last 2 dimensions of 4D input tensor, or the last dimension of 3D input tensor. Reflect padding is only implemented for padding the last 2 dimensions of 4D input tensor, or the last dimension of 3D input tensor.    Note When using the CUDA backend, this operation may induce nondeterministic behaviour in its backward pass that is not easily switched off. Please see the notes on Reproducibility for background.   Parameters

input (Tensor) – N-dimensional tensor
pad (tuple) – m-elements tuple, where m2≤\frac{m}{2} \leq  input dimensions and mm  is even.
mode – 'constant', 'reflect', 'replicate' or 'circular'. Default: 'constant'

value – fill value for 'constant' padding. Default: 0
    Examples: >>> t4d = torch.empty(3, 3, 4, 2)
>>> p1d = (1, 1) # pad last dim by 1 on each side
>>> out = F.pad(t4d, p1d, "constant", 0)  # effectively zero padding
>>> print(out.size())
torch.Size([3, 3, 4, 4])
>>> p2d = (1, 1, 2, 2) # pad last dim by (1, 1) and 2nd to last by (2, 2)
>>> out = F.pad(t4d, p2d, "constant", 0)
>>> print(out.size())
torch.Size([3, 3, 8, 4])
>>> t4d = torch.empty(3, 3, 4, 2)
>>> p3d = (0, 1, 2, 1, 3, 3) # pad by (0, 1), (2, 1), and (3, 3)
>>> out = F.pad(t4d, p3d, "constant", 0)
>>> print(out.size())
torch.Size([3, 9, 7, 3])


torch.nn.functional.pairwise_distance(x1, x2, p=2.0, eps=1e-06, keepdim=False) [source]

See torch.nn.PairwiseDistance for details


torch.nn.functional.pdist(input, p=2) → Tensor
Computes the p-norm distance between every pair of row vectors in the input. This is identical to the upper triangular portion, excluding the diagonal, of torch.norm(input[:, None] - input, dim=2, p=p). This function will be faster if the rows are contiguous. If input has shape N×MN \times M  then the output will have shape 12N(N−1)\frac{1}{2} N (N - 1) . This function is equivalent to scipy.spatial.distance.pdist(input, ‘minkowski’, p=p) if p∈(0,∞)p \in (0, \infty) . When p=0p = 0  it is equivalent to scipy.spatial.distance.pdist(input, ‘hamming’) * M. When p=∞p = \infty , the closest scipy function is scipy.spatial.distance.pdist(xn, lambda x, y: np.abs(x - y).max()).  Parameters

input – input tensor of shape N×MN \times M .
p – p value for the p-norm distance to calculate between each vector pair ∈[0,∞]\in [0, \infty] .


torch.nn.functional.pixel_shuffle(input, upscale_factor) → Tensor
Rearranges elements in a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W)  to a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r) , where r is the upscale_factor. See PixelShuffle for details.  Parameters

input (Tensor) – the input tensor
upscale_factor (int) – factor to increase spatial resolution by    Examples: >>> input = torch.randn(1, 9, 4, 4)
>>> output = torch.nn.functional.pixel_shuffle(input, 3)
>>> print(output.size())
torch.Size([1, 1, 12, 12])


torch.nn.functional.pixel_unshuffle(input, downscale_factor) → Tensor
Reverses the PixelShuffle operation by rearranging elements in a tensor of shape (∗,C,H×r,W×r)(*, C, H \times r, W \times r)  to a tensor of shape (∗,C×r2,H,W)(*, C \times r^2, H, W) , where r is the downscale_factor. See PixelUnshuffle for details.  Parameters

input (Tensor) – the input tensor
downscale_factor (int) – factor to increase spatial resolution by    Examples: >>> input = torch.randn(1, 1, 12, 12)
>>> output = torch.nn.functional.pixel_unshuffle(input, 3)
>>> print(output.size())
torch.Size([1, 9, 4, 4])


torch.nn.functional.poisson_nll_loss(input, target, log_input=True, full=False, size_average=None, eps=1e-08, reduce=None, reduction='mean') [source]

Poisson negative log likelihood loss. See PoissonNLLLoss for details.  Parameters

input – expectation of underlying Poisson distribution.
target – random sample target∼Poisson(input)target \sim \text{Poisson}(input) .
log_input – if True the loss is computed as exp(input)−target∗input\exp(\text{input}) - \text{target} * \text{input} , if False then loss is input−target∗log(input+eps)\text{input} - \text{target} * \log(\text{input}+\text{eps}) . Default: True

full – whether to compute full loss, i. e. to add the Stirling approximation term. Default: False target∗log(target)−target+0.5∗log(2∗π∗target)\text{target} * \log(\text{target}) - \text{target} + 0.5 * \log(2 * \pi * \text{target}) .
size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

eps (float, optional) – Small value to avoid evaluation of log(0)\log(0)  when log_input`=``False`. Default: 1e-8
reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'


torch.nn.functional.prelu(input, weight) → Tensor [source]

Applies element-wise the function PReLU(x)=max(0,x)+weight∗min(0,x)\text{PReLU}(x) = \max(0,x) + \text{weight} * \min(0,x)  where weight is a learnable parameter. See PReLU for more details.


torch.nn.functional.relu(input, inplace=False) → Tensor [source]

Applies the rectified linear unit function element-wise. See ReLU for more details.


torch.nn.functional.relu6(input, inplace=False) → Tensor [source]

Applies the element-wise function ReLU6(x)=min(max(0,x),6)\text{ReLU6}(x) = \min(\max(0,x), 6) . See ReLU6 for more details.


torch.nn.functional.relu_(input) → Tensor
In-place version of relu().


torch.nn.functional.rrelu(input, lower=1./8, upper=1./3, training=False, inplace=False) → Tensor [source]

Randomized leaky ReLU. See RReLU for more details.


torch.nn.functional.rrelu_(input, lower=1./8, upper=1./3, training=False) → Tensor
In-place version of rrelu().


torch.nn.functional.sigmoid(input) → Tensor [source]

Applies the element-wise function Sigmoid(x)=11+exp(−x)\text{Sigmoid}(x) = \frac{1}{1 + \exp(-x)}  See Sigmoid for more details.


torch.nn.functional.silu(input, inplace=False) [source]

Applies the silu function, element-wise.  silu(x)=x∗σ(x),where σ(x) is the logistic sigmoid.\text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}
 Note See Gaussian Error Linear Units (GELUs) where the SiLU (Sigmoid Linear Unit) was originally coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning and Swish: a Self-Gated Activation Function where the SiLU was experimented with later.  See SiLU for more details.


torch.nn.functional.smooth_l1_loss(input, target, size_average=None, reduce=None, reduction='mean', beta=1.0) [source]

Function that uses a squared term if the absolute element-wise error falls below beta and an L1 term otherwise. See SmoothL1Loss for details.


torch.nn.functional.softmax(input, dim=None, _stacklevel=3, dtype=None) [source]

Applies a softmax function. Softmax is defined as: Softmax(xi)=exp(xi)∑jexp(xj)\text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}  It is applied to all slices along dim, and will re-scale them so that the elements lie in the range [0, 1] and sum to 1. See Softmax for more details.  Parameters

input (Tensor) – input
dim (int) – A dimension along which softmax will be computed.
dtype (torch.dtype, optional) – the desired data type of returned tensor. If specified, the input tensor is casted to dtype before the operation is performed. This is useful for preventing data type overflows. Default: None.     Note This function doesn’t work directly with NLLLoss, which expects the Log to be computed between the Softmax and itself. Use log_softmax instead (it’s faster and has better numerical properties).


torch.nn.functional.softmin(input, dim=None, _stacklevel=3, dtype=None) [source]

Applies a softmin function. Note that Softmin(x)=Softmax(−x)\text{Softmin}(x) = \text{Softmax}(-x) . See softmax definition for mathematical formula. See Softmin for more details.  Parameters

input (Tensor) – input
dim (int) – A dimension along which softmin will be computed (so every slice along dim will sum to 1).
dtype (torch.dtype, optional) – the desired data type of returned tensor. If specified, the input tensor is casted to dtype before the operation is performed. This is useful for preventing data type overflows. Default: None.


torch.nn.functional.softplus(input, beta=1, threshold=20) → Tensor
Applies element-wise, the function Softplus(x)=1β∗log(1+exp(β∗x))\text{Softplus}(x) = \frac{1}{\beta} * \log(1 + \exp(\beta * x)) . For numerical stability the implementation reverts to the linear function when input×β>thresholdinput \times \beta > threshold . See Softplus for more details.


torch.nn.functional.softshrink(input, lambd=0.5) → Tensor
Applies the soft shrinkage function elementwise See Softshrink for more details.


torch.nn.functional.softsign(input) → Tensor [source]

Applies element-wise, the function SoftSign(x)=x1+∣x∣\text{SoftSign}(x) = \frac{x}{1 + |x|}  See Softsign for more details.


torch.nn.functional.soft_margin_loss(input, target, size_average=None, reduce=None, reduction='mean') → Tensor [source]

See SoftMarginLoss for details.


torch.nn.functional.tanh(input) → Tensor [source]

Applies element-wise, Tanh(x)=tanh(x)=exp(x)−exp(−x)exp(x)+exp(−x)\text{Tanh}(x) = \tanh(x) = \frac{\exp(x) - \exp(-x)}{\exp(x) + \exp(-x)}  See Tanh for more details.


torch.nn.functional.tanhshrink(input) → Tensor [source]

Applies element-wise, Tanhshrink(x)=x−Tanh(x)\text{Tanhshrink}(x) = x - \text{Tanh}(x)  See Tanhshrink for more details.


torch.nn.functional.threshold(input, threshold, value, inplace=False)
Thresholds each element of the input Tensor. See Threshold for more details.


torch.nn.functional.threshold_(input, threshold, value) → Tensor
In-place version of threshold().


torch.nn.functional.triplet_margin_loss(anchor, positive, negative, margin=1.0, p=2, eps=1e-06, swap=False, size_average=None, reduce=None, reduction='mean') [source]

See TripletMarginLoss for details


torch.nn.functional.triplet_margin_with_distance_loss(anchor, positive, negative, *, distance_function=None, margin=1.0, swap=False, reduction='mean') [source]

See TripletMarginWithDistanceLoss for details.


torch.nn.functional.unfold(input, kernel_size, dilation=1, padding=0, stride=1) [source]

Extracts sliding local blocks from a batched input tensor.  Warning Currently, only 4-D input tensors (batched image-like tensors) are supported.   Warning More than one element of the unfolded tensor may refer to a single memory location. As a result, in-place operations (especially ones that are vectorized) may result in incorrect behavior. If you need to write to the tensor, please clone it first.  See torch.nn.Unfold for details


torch.nn.functional.upsample(input, size=None, scale_factor=None, mode='nearest', align_corners=None) [source]

Upsamples the input to either the given size or the given scale_factor  Warning This function is deprecated in favor of torch.nn.functional.interpolate(). This is equivalent with nn.functional.interpolate(...).   Note This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.  The algorithm used for upsampling is determined by mode. Currently temporal, spatial and volumetric upsampling are supported, i.e. expected inputs are 3-D, 4-D or 5-D in shape. The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width. The modes available for upsampling are: nearest, linear (3D-only), bilinear, bicubic (4D-only), trilinear (5D-only)  Parameters

input (Tensor) – the input tensor
size (int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int]) – output spatial size.
scale_factor (float or Tuple[float]) – multiplier for spatial size. Has to match input size if it is a tuple.
mode (string) – algorithm used for upsampling: 'nearest' | 'linear' | 'bilinear' | 'bicubic' | 'trilinear'. Default: 'nearest'

align_corners (bool, optional) – Geometrically, we consider the pixels of the input and output as squares rather than points. If set to True, the input and output tensors are aligned by the center points of their corner pixels, preserving the values at the corner pixels. If set to False, the input and output tensors are aligned by the corner points of their corner pixels, and the interpolation uses edge value padding for out-of-boundary values, making this operation independent of input size when scale_factor is kept the same. This only has an effect when mode is 'linear', 'bilinear', 'bicubic' or 'trilinear'. Default: False
     Note With mode='bicubic', it’s possible to cause overshoot, in other words it can produce negative values or values greater than 255 for images. Explicitly call result.clamp(min=0, max=255) if you want to reduce the overshoot when displaying the image.   Warning With align_corners = True, the linearly interpolating modes (linear, bilinear, and trilinear) don’t proportionally align the output and input pixels, and thus the output values can depend on the input size. This was the default behavior for these modes up to version 0.3.1. Since then, the default behavior is align_corners = False. See Upsample for concrete examples on how this affects the outputs.


torch.nn.functional.upsample_bilinear(input, size=None, scale_factor=None) [source]

Upsamples the input, using bilinear upsampling.  Warning This function is deprecated in favor of torch.nn.functional.interpolate(). This is equivalent with nn.functional.interpolate(..., mode='bilinear', align_corners=True).  Expected inputs are spatial (4 dimensional). Use upsample_trilinear fo volumetric (5 dimensional) inputs.  Parameters

input (Tensor) – input
size (int or Tuple[int, int]) – output spatial size.
scale_factor (int or Tuple[int, int]) – multiplier for spatial size     Note This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.


torch.nn.functional.upsample_nearest(input, size=None, scale_factor=None) [source]

Upsamples the input, using nearest neighbours’ pixel values.  Warning This function is deprecated in favor of torch.nn.functional.interpolate(). This is equivalent with nn.functional.interpolate(..., mode='nearest').  Currently spatial and volumetric upsampling are supported (i.e. expected inputs are 4 or 5 dimensional).  Parameters

input (Tensor) – input
size (int or Tuple[int, int] or Tuple[int, int, int]) – output spatia size.
scale_factor (int) – multiplier for spatial size. Has to be an integer.     Note This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.


class torch.nn.GaussianNLLLoss(*, full=False, eps=1e-06, reduction='mean') [source]

Gaussian negative log likelihood loss. The targets are treated as samples from Gaussian distributions with expectations and variances predicted by the neural network. For a D-dimensional target tensor modelled as having heteroscedastic Gaussian distributions with a D-dimensional tensor of expectations input and a D-dimensional tensor of positive variances var the loss is:  loss=12∑i=1D(log(max(var[i], eps))+(input[i]−target[i])2max(var[i], eps))+const.\text{loss} = \frac{1}{2}\sum_{i=1}^D \left(\log\left(\text{max}\left(\text{var}[i], \ \text{eps}\right)\right) + \frac{\left(\text{input}[i] - \text{target}[i]\right)^2} {\text{max}\left(\text{var}[i], \ \text{eps}\right)}\right) + \text{const.}
where eps is used for stability. By default, the constant term of the loss function is omitted unless full is True. If var is a scalar (implying target tensor has homoscedastic Gaussian distributions) it is broadcasted to be the same size as the input.  Parameters

full (bool, optional) – include the constant term in the loss calculation. Default: False.
eps (float, optional) – value used to clamp var (see note below), for stability. Default: 1e-6.
reduction (string, optional) – specifies the reduction to apply to the output:'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the output is the average of all batch member losses, 'sum': the output is the sum of all batch member losses. Default: 'mean'.     Shape:

 Input: (N,∗)(N, *)  where ∗*  means any number of additional dimensions Target: (N,∗)(N, *) , same shape as the input Var: (N,1)(N, 1)  or (N,∗)(N, *) , same shape as the input Output: scalar if reduction is 'mean' (default) or 'sum'. If reduction is 'none', then (N)(N)
    Examples: >>> loss = nn.GaussianNLLLoss()
>>> input = torch.randn(5, 2, requires_grad=True)
>>> target = torch.randn(5, 2)
>>> var = torch.ones(5, 2, requires_grad=True) #heteroscedastic
>>> output = loss(input, target, var)
>>> output.backward()

>>> loss = nn.GaussianNLLLoss()
>>> input = torch.randn(5, 2, requires_grad=True)
>>> target = torch.randn(5, 2)
>>> var = torch.ones(5, 1, requires_grad=True) #homoscedastic
>>> output = loss(input, target, var)
>>> output.backward()
  Note The clamping of var is ignored with respect to autograd, and so the gradients are unaffected by it.   Reference:

Nix, D. A. and Weigend, A. S., “Estimating the mean and variance of the target probability distribution”, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), Orlando, FL, USA, 1994, pp. 55-60 vol.1, doi: 10.1109/ICNN.1994.374138.


class torch.nn.GELU [source]

Applies the Gaussian Error Linear Units function:  GELU(x)=x∗Φ(x)\text{GELU}(x) = x * \Phi(x)
where Φ(x)\Phi(x)  is the Cumulative Distribution Function for Gaussian Distribution.  Shape:

 Input: (N,∗)(N, *)  where * means, any number of additional dimensions Output: (N,∗)(N, *) , same shape as the input     Examples: >>> m = nn.GELU()
>>> input = torch.randn(2)
>>> output = m(input)


class torch.nn.GroupNorm(num_groups, num_channels, eps=1e-05, affine=True) [source]

Applies Group Normalization over a mini-batch of inputs as described in the paper Group Normalization  y=x−E[x]Var[x]+ϵ∗γ+βy = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The input channels are separated into num_groups groups, each containing num_channels / num_groups channels. The mean and standard-deviation are calculated separately over the each group. γ\gamma  and β\beta  are learnable per-channel affine transform parameter vectors of size num_channels if affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False). This layer uses statistics computed from input data in both training and evaluation modes.  Parameters

num_groups (int) – number of groups to separate the channels into
num_channels (int) – number of channels expected in input
eps – a value added to the denominator for numerical stability. Default: 1e-5
affine – a boolean value that when set to True, this module has learnable per-channel affine parameters initialized to ones (for weights) and zeros (for biases). Default: True.     Shape:

 Input: (N,C,∗)(N, C, *)  where C=num_channelsC=\text{num\_channels}
 Output: (N,C,∗)(N, C, *)  (same shape as input)    Examples: >>> input = torch.randn(20, 6, 10, 10)
>>> # Separate 6 channels into 3 groups
>>> m = nn.GroupNorm(3, 6)
>>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
>>> m = nn.GroupNorm(6, 6)
>>> # Put all 6 channels into a single group (equivalent with LayerNorm)
>>> m = nn.GroupNorm(1, 6)
>>> # Activating the module
>>> output = m(input)


class torch.nn.GRU(*args, **kwargs) [source]

Applies a multi-layer gated recurrent unit (GRU) RNN to an input sequence. For each element in the input sequence, each layer computes the following function:  rt=σ(Wirxt+bir+Whrh(t−1)+bhr)zt=σ(Wizxt+biz+Whzh(t−1)+bhz)nt=tanh(Winxt+bin+rt∗(Whnh(t−1)+bhn))ht=(1−zt)∗nt+zt∗h(t−1)\begin{array}{ll} r_t = \sigma(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr}) \\ z_t = \sigma(W_{iz} x_t + b_{iz} + W_{hz} h_{(t-1)} + b_{hz}) \\ n_t = \tanh(W_{in} x_t + b_{in} + r_t * (W_{hn} h_{(t-1)}+ b_{hn})) \\ h_t = (1 - z_t) * n_t + z_t * h_{(t-1)} \end{array}
where hth_t  is the hidden state at time t, xtx_t  is the input at time t, h(t−1)h_{(t-1)}  is the hidden state of the layer at time t-1 or the initial hidden state at time 0, and rtr_t , ztz_t , ntn_t  are the reset, update, and new gates, respectively. σ\sigma  is the sigmoid function, and ∗*  is the Hadamard product. In a multilayer GRU, the input xt(l)x^{(l)}_t  of the ll  -th layer (l>=2l >= 2 ) is the hidden state ht(l−1)h^{(l-1)}_t  of the previous layer multiplied by dropout δt(l−1)\delta^{(l-1)}_t  where each δt(l−1)\delta^{(l-1)}_t  is a Bernoulli random variable which is 00  with probability dropout.  Parameters

input_size – The number of expected features in the input x

hidden_size – The number of features in the hidden state h

num_layers – Number of recurrent layers. E.g., setting num_layers=2 would mean stacking two GRUs together to form a stacked GRU, with the second GRU taking in outputs of the first GRU and computing the final results. Default: 1
bias – If False, then the layer does not use bias weights b_ih and b_hh. Default: True

batch_first – If True, then the input and output tensors are provided as (batch, seq, feature). Default: False

dropout – If non-zero, introduces a Dropout layer on the outputs of each GRU layer except the last layer, with dropout probability equal to dropout. Default: 0
bidirectional – If True, becomes a bidirectional GRU. Default: False
     Inputs: input, h_0

input of shape (seq_len, batch, input_size): tensor containing the features of the input sequence. The input can also be a packed variable length sequence. See torch.nn.utils.rnn.pack_padded_sequence() for details.
h_0 of shape (num_layers * num_directions, batch, hidden_size): tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. If the RNN is bidirectional, num_directions should be 2, else it should be 1.   Outputs: output, h_n

output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features h_t from the last layer of the GRU, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence. For the unpacked case, the directions can be separated using output.view(seq_len, batch, num_directions, hidden_size), with forward and backward being direction 0 and 1 respectively. Similarly, the directions can be separated in the packed case.
h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len Like output, the layers can be separated using h_n.view(num_layers, num_directions, batch, hidden_size).    Shape:

 Input1: (L,N,Hin)(L, N, H_{in})  tensor containing input features where Hin=input_sizeH_{in}=\text{input\_size}  and L represents a sequence length. Input2: (S,N,Hout)(S, N, H_{out})  tensor containing the initial hidden state for each element in the batch. Hout=hidden_sizeH_{out}=\text{hidden\_size}  Defaults to zero if not provided. where S=num_layers∗num_directionsS=\text{num\_layers} * \text{num\_directions}  If the RNN is bidirectional, num_directions should be 2, else it should be 1. Output1: (L,N,Hall)(L, N, H_{all})  where Hall=num_directions∗hidden_sizeH_{all}=\text{num\_directions} * \text{hidden\_size}
 Output2: (S,N,Hout)(S, N, H_{out})  tensor containing the next hidden state for each element in the batch     Variables

~GRU.weight_ih_l[k] – the learnable input-hidden weights of the kth\text{k}^{th}  layer (W_ir|W_iz|W_in), of shape (3*hidden_size, input_size) for k = 0. Otherwise, the shape is (3*hidden_size, num_directions * hidden_size)

~GRU.weight_hh_l[k] – the learnable hidden-hidden weights of the kth\text{k}^{th}  layer (W_hr|W_hz|W_hn), of shape (3*hidden_size, hidden_size)

~GRU.bias_ih_l[k] – the learnable input-hidden bias of the kth\text{k}^{th}  layer (b_ir|b_iz|b_in), of shape (3*hidden_size)

~GRU.bias_hh_l[k] – the learnable hidden-hidden bias of the kth\text{k}^{th}  layer (b_hr|b_hz|b_hn), of shape (3*hidden_size)
     Note All the weights and biases are initialized from U(−k,k)\mathcal{U}(-\sqrt{k}, \sqrt{k})  where k=1hidden_sizek = \frac{1}{\text{hidden\_size}}    Orphan   Note If the following conditions are satisfied: 1) cudnn is enabled, 2) input data is on the GPU 3) input data has dtype torch.float16 4) V100 GPU is used, 5) input data is not in PackedSequence format persistent algorithm can be selected to improve performance.  Examples: >>> rnn = nn.GRU(10, 20, 2)
>>> input = torch.randn(5, 3, 10)
>>> h0 = torch.randn(2, 3, 20)
>>> output, hn = rnn(input, h0)


class torch.nn.GRUCell(input_size, hidden_size, bias=True) [source]

A gated recurrent unit (GRU) cell  r=σ(Wirx+bir+Whrh+bhr)z=σ(Wizx+biz+Whzh+bhz)n=tanh(Winx+bin+r∗(Whnh+bhn))h′=(1−z)∗n+z∗h\begin{array}{ll} r = \sigma(W_{ir} x + b_{ir} + W_{hr} h + b_{hr}) \\ z = \sigma(W_{iz} x + b_{iz} + W_{hz} h + b_{hz}) \\ n = \tanh(W_{in} x + b_{in} + r * (W_{hn} h + b_{hn})) \\ h' = (1 - z) * n + z * h \end{array}
where σ\sigma  is the sigmoid function, and ∗*  is the Hadamard product.  Parameters

input_size – The number of expected features in the input x

hidden_size – The number of features in the hidden state h

bias – If False, then the layer does not use bias weights b_ih and b_hh. Default: True
     Inputs: input, hidden

input of shape (batch, input_size): tensor containing input features
hidden of shape (batch, hidden_size): tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided.   Outputs: h’

h’ of shape (batch, hidden_size): tensor containing the next hidden state for each element in the batch   Shape:

 Input1: (N,Hin)(N, H_{in})  tensor containing input features where HinH_{in}  = input_size
 Input2: (N,Hout)(N, H_{out})  tensor containing the initial hidden state for each element in the batch where HoutH_{out}  = hidden_size Defaults to zero if not provided. Output: (N,Hout)(N, H_{out})  tensor containing the next hidden state for each element in the batch     Variables

~GRUCell.weight_ih – the learnable input-hidden weights, of shape (3*hidden_size, input_size)

~GRUCell.weight_hh – the learnable hidden-hidden weights, of shape (3*hidden_size, hidden_size)

~GRUCell.bias_ih – the learnable input-hidden bias, of shape (3*hidden_size)

~GRUCell.bias_hh – the learnable hidden-hidden bias, of shape (3*hidden_size)
     Note All the weights and biases are initialized from U(−k,k)\mathcal{U}(-\sqrt{k}, \sqrt{k})  where k=1hidden_sizek = \frac{1}{\text{hidden\_size}}   Examples: >>> rnn = nn.GRUCell(10, 20)
>>> input = torch.randn(6, 3, 10)
>>> hx = torch.randn(3, 20)
>>> output = []
>>> for i in range(6):
        hx = rnn(input[i], hx)
        output.append(hx)


class torch.nn.Hardshrink(lambd=0.5) [source]

Applies the hard shrinkage function element-wise:  HardShrink(x)={x, if x>λx, if x<−λ0, otherwise \text{HardShrink}(x) = \begin{cases} x, & \text{ if } x > \lambda \\ x, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases}
 Parameters
lambd – the λ\lambda  value for the Hardshrink formulation. Default: 0.5    Shape:

 Input: (N,∗)(N, *)  where * means, any number of additional dimensions Output: (N,∗)(N, *) , same shape as the input     Examples: >>> m = nn.Hardshrink()
>>> input = torch.randn(2)
>>> output = m(input)


class torch.nn.Hardsigmoid(inplace=False) [source]

Applies the element-wise function:  Hardsigmoid(x)={0if x≤−3,1if x≥+3,x/6+1/2otherwise\text{Hardsigmoid}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ 1 & \text{if~} x \ge +3, \\ x / 6 + 1 / 2 & \text{otherwise} \end{cases}
 Parameters
inplace – can optionally do the operation in-place. Default: False    Shape:

 Input: (N,∗)(N, *)  where * means, any number of additional dimensions Output: (N,∗)(N, *) , same shape as the input    Examples: >>> m = nn.Hardsigmoid()
>>> input = torch.randn(2)
>>> output = m(input)


class torch.nn.Hardswish(inplace=False) [source]

Applies the hardswish function, element-wise, as described in the paper: Searching for MobileNetV3.  Hardswish(x)={0if x≤−3,xif x≥+3,x⋅(x+3)/6otherwise\text{Hardswish}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ x & \text{if~} x \ge +3, \\ x \cdot (x + 3) /6 & \text{otherwise} \end{cases}
 Parameters
inplace – can optionally do the operation in-place. Default: False    Shape:

 Input: (N,∗)(N, *)  where * means, any number of additional dimensions Output: (N,∗)(N, *) , same shape as the input    Examples: >>> m = nn.Hardswish()
>>> input = torch.randn(2)
>>> output = m(input)


class torch.nn.Hardtanh(min_val=-1.0, max_val=1.0, inplace=False, min_value=None, max_value=None) [source]

Applies the HardTanh function element-wise HardTanh is defined as:  HardTanh(x)={1 if x>1−1 if x<−1x otherwise \text{HardTanh}(x) = \begin{cases} 1 & \text{ if } x > 1 \\ -1 & \text{ if } x < -1 \\ x & \text{ otherwise } \\ \end{cases}
The range of the linear region [−1,1][-1, 1]  can be adjusted using min_val and max_val.  Parameters

min_val – minimum value of the linear region range. Default: -1
max_val – maximum value of the linear region range. Default: 1
inplace – can optionally do the operation in-place. Default: False
    Keyword arguments min_value and max_value have been deprecated in favor of min_val and max_val.  Shape:

 Input: (N,∗)(N, *)  where * means, any number of additional dimensions Output: (N,∗)(N, *) , same shape as the input     Examples: >>> m = nn.Hardtanh(-2, 2)
>>> input = torch.randn(2)
>>> output = m(input)


class torch.nn.HingeEmbeddingLoss(margin=1.0, size_average=None, reduce=None, reduction='mean') [source]

Measures the loss given an input tensor xx  and a labels tensor yy  (containing 1 or -1). This is usually used for measuring whether two inputs are similar or dissimilar, e.g. using the L1 pairwise distance as xx , and is typically used for learning nonlinear embeddings or semi-supervised learning. The loss function for nn -th sample in the mini-batch is  ln={xn,ifyn=1,max{0,Δ−xn},ifyn=−1,l_n = \begin{cases} x_n, & \text{if}\; y_n = 1,\\ \max \{0, \Delta - x_n\}, & \text{if}\; y_n = -1, \end{cases}
and the total loss functions is  ℓ(x,y)={mean(L),if reduction=‘mean’;sum(L),if reduction=‘sum’.\ell(x, y) = \begin{cases} \operatorname{mean}(L), & \text{if reduction} = \text{`mean';}\\ \operatorname{sum}(L), & \text{if reduction} = \text{`sum'.} \end{cases}
where L={l1,…,lN}⊤L = \{l_1,\dots,l_N\}^\top .  Parameters

margin (float, optional) – Has a default value of 1.
size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there are multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction (string, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'
     Shape:

 Input: (∗)(*)  where ∗*  means, any number of dimensions. The sum operation operates over all the elements. Target: (∗)(*) , same shape as the input Output: scalar. If reduction is 'none', then same shape as the input


class torch.nn.Identity(*args, **kwargs) [source]

A placeholder identity operator that is argument-insensitive.  Parameters

args – any argument (unused)
kwargs – any keyword argument (unused)    Examples: >>> m = nn.Identity(54, unused_argument1=0.1, unused_argument2=False)
>>> input = torch.randn(128, 20)
>>> output = m(input)
>>> print(output.size())
torch.Size([128, 20])


torch.nn.init
torch.nn.init.calculate_gain(nonlinearity, param=None) [source]

Return the recommended gain value for the given nonlinearity function. The values are as follows:
nonlinearity gain
Linear / Identity 11
Conv{1,2,3}D 11
Sigmoid 11
Tanh 53\frac{5}{3}
ReLU 2\sqrt{2}
Leaky Relu 21+negative_slope2\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}
SELU 34\frac{3}{4}     Parameters

nonlinearity – the non-linear function (nn.functional name)
param – optional parameter for the non-linear function    Examples >>> gain = nn.init.calculate_gain('leaky_relu', 0.2)  # leaky_relu with negative_slope=0.2

torch.nn.init.uniform_(tensor, a=0.0, b=1.0) [source]

Fills the input Tensor with values drawn from the uniform distribution U(a,b)\mathcal{U}(a, b) .  Parameters

tensor – an n-dimensional torch.Tensor

a – the lower bound of the uniform distribution
b – the upper bound of the uniform distribution    Examples >>> w = torch.empty(3, 5)
>>> nn.init.uniform_(w)

torch.nn.init.normal_(tensor, mean=0.0, std=1.0) [source]

Fills the input Tensor with values drawn from the normal distribution N(mean,std2)\mathcal{N}(\text{mean}, \text{std}^2) .  Parameters

tensor – an n-dimensional torch.Tensor

mean – the mean of the normal distribution
std – the standard deviation of the normal distribution    Examples >>> w = torch.empty(3, 5)
>>> nn.init.normal_(w)

torch.nn.init.constant_(tensor, val) [source]

Fills the input Tensor with the value val\text{val} .  Parameters

tensor – an n-dimensional torch.Tensor

val – the value to fill the tensor with    Examples >>> w = torch.empty(3, 5)
>>> nn.init.constant_(w, 0.3)

torch.nn.init.ones_(tensor) [source]

Fills the input Tensor with the scalar value 1.  Parameters
tensor – an n-dimensional torch.Tensor   Examples >>> w = torch.empty(3, 5)
>>> nn.init.ones_(w)

torch.nn.init.zeros_(tensor) [source]

Fills the input Tensor with the scalar value 0.  Parameters
tensor – an n-dimensional torch.Tensor   Examples >>> w = torch.empty(3, 5)
>>> nn.init.zeros_(w)

torch.nn.init.eye_(tensor) [source]

Fills the 2-dimensional input Tensor with the identity matrix. Preserves the identity of the inputs in Linear layers, where as many inputs are preserved as possible.  Parameters
tensor – a 2-dimensional torch.Tensor   Examples >>> w = torch.empty(3, 5)
>>> nn.init.eye_(w)

torch.nn.init.dirac_(tensor, groups=1) [source]

Fills the {3, 4, 5}-dimensional input Tensor with the Dirac delta function. Preserves the identity of the inputs in Convolutional layers, where as many input channels are preserved as possible. In case of groups>1, each group of channels preserves identity  Parameters

tensor – a {3, 4, 5}-dimensional torch.Tensor

groups (optional) – number of groups in the conv layer (default: 1)    Examples >>> w = torch.empty(3, 16, 5, 5)
>>> nn.init.dirac_(w)
>>> w = torch.empty(3, 24, 5, 5)
>>> nn.init.dirac_(w, 3)

torch.nn.init.xavier_uniform_(tensor, gain=1.0) [source]

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a uniform distribution. The resulting tensor will have values sampled from U(−a,a)\mathcal{U}(-a, a)  where  a=gain×6fan_in+fan_outa = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}}
Also known as Glorot initialization.  Parameters

tensor – an n-dimensional torch.Tensor

gain – an optional scaling factor    Examples >>> w = torch.empty(3, 5)
>>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu'))

torch.nn.init.xavier_normal_(tensor, gain=1.0) [source]

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a normal distribution. The resulting tensor will have values sampled from N(0,std2)\mathcal{N}(0, \text{std}^2)  where  std=gain×2fan_in+fan_out\text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}}
Also known as Glorot initialization.  Parameters

tensor – an n-dimensional torch.Tensor

gain – an optional scaling factor    Examples >>> w = torch.empty(3, 5)
>>> nn.init.xavier_normal_(w)

torch.nn.init.kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu') [source]

Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a uniform distribution. The resulting tensor will have values sampled from U(−bound,bound)\mathcal{U}(-\text{bound}, \text{bound})  where  bound=gain×3fan_mode\text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}}
Also known as He initialization.  Parameters

tensor – an n-dimensional torch.Tensor

a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')
mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.
nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).    Examples >>> w = torch.empty(3, 5)
>>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')

torch.nn.init.kaiming_normal_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu') [source]

Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a normal distribution. The resulting tensor will have values sampled from N(0,std2)\mathcal{N}(0, \text{std}^2)  where  std=gainfan_mode\text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}}
Also known as He initialization.  Parameters

tensor – an n-dimensional torch.Tensor

a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')
mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.
nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).    Examples >>> w = torch.empty(3, 5)
>>> nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu')

torch.nn.init.orthogonal_(tensor, gain=1) [source]

Fills the input Tensor with a (semi) orthogonal matrix, as described in Exact solutions to the nonlinear dynamics of learning in deep linear neural networks - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened.  Parameters

tensor – an n-dimensional torch.Tensor, where n≥2n \geq 2

gain – optional scaling factor    Examples >>> w = torch.empty(3, 5)
>>> nn.init.orthogonal_(w)

torch.nn.init.sparse_(tensor, sparsity, std=0.01) [source]

Fills the 2D input Tensor as a sparse matrix, where the non-zero elements will be drawn from the normal distribution N(0,0.01)\mathcal{N}(0, 0.01) , as described in Deep learning via Hessian-free optimization - Martens, J. (2010).  Parameters

tensor – an n-dimensional torch.Tensor

sparsity – The fraction of elements in each column to be set to zero
std – the standard deviation of the normal distribution used to generate the non-zero values    Examples >>> w = torch.empty(3, 5)
>>> nn.init.sparse_(w, sparsity=0.1)


torch.nn.init.calculate_gain(nonlinearity, param=None) [source]

Return the recommended gain value for the given nonlinearity function. The values are as follows:
nonlinearity gain
Linear / Identity 11
Conv{1,2,3}D 11
Sigmoid 11
Tanh 53\frac{5}{3}
ReLU 2\sqrt{2}
Leaky Relu 21+negative_slope2\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}
SELU 34\frac{3}{4}     Parameters

nonlinearity – the non-linear function (nn.functional name)
param – optional parameter for the non-linear function    Examples >>> gain = nn.init.calculate_gain('leaky_relu', 0.2)  # leaky_relu with negative_slope=0.2


torch.nn.init.constant_(tensor, val) [source]

Fills the input Tensor with the value val\text{val} .  Parameters

tensor – an n-dimensional torch.Tensor

val – the value to fill the tensor with    Examples >>> w = torch.empty(3, 5)
>>> nn.init.constant_(w, 0.3)


torch.nn.init.dirac_(tensor, groups=1) [source]

Fills the {3, 4, 5}-dimensional input Tensor with the Dirac delta function. Preserves the identity of the inputs in Convolutional layers, where as many input channels are preserved as possible. In case of groups>1, each group of channels preserves identity  Parameters

tensor – a {3, 4, 5}-dimensional torch.Tensor

groups (optional) – number of groups in the conv layer (default: 1)    Examples >>> w = torch.empty(3, 16, 5, 5)
>>> nn.init.dirac_(w)
>>> w = torch.empty(3, 24, 5, 5)
>>> nn.init.dirac_(w, 3)


torch.nn.init.eye_(tensor) [source]

Fills the 2-dimensional input Tensor with the identity matrix. Preserves the identity of the inputs in Linear layers, where as many inputs are preserved as possible.  Parameters
tensor – a 2-dimensional torch.Tensor   Examples >>> w = torch.empty(3, 5)
>>> nn.init.eye_(w)


torch.nn.init.kaiming_normal_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu') [source]

Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a normal distribution. The resulting tensor will have values sampled from N(0,std2)\mathcal{N}(0, \text{std}^2)  where  std=gainfan_mode\text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}}
Also known as He initialization.  Parameters

tensor – an n-dimensional torch.Tensor

a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')
mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.
nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).    Examples >>> w = torch.empty(3, 5)
>>> nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu')


torch.nn.init.kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu') [source]

Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a uniform distribution. The resulting tensor will have values sampled from U(−bound,bound)\mathcal{U}(-\text{bound}, \text{bound})  where  bound=gain×3fan_mode\text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}}
Also known as He initialization.  Parameters

tensor – an n-dimensional torch.Tensor

a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')
mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.
nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).    Examples >>> w = torch.empty(3, 5)
>>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')


torch.nn.init.normal_(tensor, mean=0.0, std=1.0) [source]

Fills the input Tensor with values drawn from the normal distribution N(mean,std2)\mathcal{N}(\text{mean}, \text{std}^2) .  Parameters

tensor – an n-dimensional torch.Tensor

mean – the mean of the normal distribution
std – the standard deviation of the normal distribution    Examples >>> w = torch.empty(3, 5)
>>> nn.init.normal_(w)


torch.nn.init.ones_(tensor) [source]

Fills the input Tensor with the scalar value 1.  Parameters
tensor – an n-dimensional torch.Tensor   Examples >>> w = torch.empty(3, 5)
>>> nn.init.ones_(w)


torch.nn.init.orthogonal_(tensor, gain=1) [source]

Fills the input Tensor with a (semi) orthogonal matrix, as described in Exact solutions to the nonlinear dynamics of learning in deep linear neural networks - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened.  Parameters

tensor – an n-dimensional torch.Tensor, where n≥2n \geq 2

gain – optional scaling factor    Examples >>> w = torch.empty(3, 5)
>>> nn.init.orthogonal_(w)


torch.nn.init.sparse_(tensor, sparsity, std=0.01) [source]

Fills the 2D input Tensor as a sparse matrix, where the non-zero elements will be drawn from the normal distribution N(0,0.01)\mathcal{N}(0, 0.01) , as described in Deep learning via Hessian-free optimization - Martens, J. (2010).  Parameters

tensor – an n-dimensional torch.Tensor

sparsity – The fraction of elements in each column to be set to zero
std – the standard deviation of the normal distribution used to generate the non-zero values    Examples >>> w = torch.empty(3, 5)
>>> nn.init.sparse_(w, sparsity=0.1)


torch.nn.init.uniform_(tensor, a=0.0, b=1.0) [source]

Fills the input Tensor with values drawn from the uniform distribution U(a,b)\mathcal{U}(a, b) .  Parameters

tensor – an n-dimensional torch.Tensor

a – the lower bound of the uniform distribution
b – the upper bound of the uniform distribution    Examples >>> w = torch.empty(3, 5)
>>> nn.init.uniform_(w)


torch.nn.init.xavier_normal_(tensor, gain=1.0) [source]

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a normal distribution. The resulting tensor will have values sampled from N(0,std2)\mathcal{N}(0, \text{std}^2)  where  std=gain×2fan_in+fan_out\text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}}
Also known as Glorot initialization.  Parameters

tensor – an n-dimensional torch.Tensor

gain – an optional scaling factor    Examples >>> w = torch.empty(3, 5)
>>> nn.init.xavier_normal_(w)


torch.nn.init.xavier_uniform_(tensor, gain=1.0) [source]

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a uniform distribution. The resulting tensor will have values sampled from U(−a,a)\mathcal{U}(-a, a)  where  a=gain×6fan_in+fan_outa = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}}
Also known as Glorot initialization.  Parameters

tensor – an n-dimensional torch.Tensor

gain – an optional scaling factor    Examples >>> w = torch.empty(3, 5)
>>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu'))


torch.nn.init.zeros_(tensor) [source]

Fills the input Tensor with the scalar value 0.  Parameters
tensor – an n-dimensional torch.Tensor   Examples >>> w = torch.empty(3, 5)
>>> nn.init.zeros_(w)


class torch.nn.InstanceNorm1d(num_features, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) [source]

Applies Instance Normalization over a 3D input (a mini-batch of 1D inputs with optional additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.  y=x−E[x]Var[x]+ϵ∗γ+βy = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. γ\gamma  and β\beta  are learnable parameter vectors of size C (where C is the input size) if affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False). By default, this layer uses instance statistics computed from input data in both training and evaluation modes. If track_running_stats is set to True, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.  Note This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x^new=(1−momentum)×x^+momentum×xt\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t , where x^\hat{x}  is the estimated statistic and xtx_t  is the new observed value.   Note InstanceNorm1d and LayerNorm are very similar, but have some subtle differences. InstanceNorm1d is applied on each channel of channeled data like multidimensional time series, but LayerNorm is usually applied on entire sample and often in NLP tasks. Additionally, LayerNorm applies elementwise affine transform, while InstanceNorm1d usually don’t apply affine transform.   Parameters

num_features – CC  from an expected input of size (N,C,L)(N, C, L)  or LL  from input of size (N,L)(N, L)

eps – a value added to the denominator for numerical stability. Default: 1e-5
momentum – the value used for the running_mean and running_var computation. Default: 0.1
affine – a boolean value that when set to True, this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: False.
track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: False
     Shape:

 Input: (N,C,L)(N, C, L)
 Output: (N,C,L)(N, C, L)  (same shape as input)    Examples: >>> # Without Learnable Parameters
>>> m = nn.InstanceNorm1d(100)
>>> # With Learnable Parameters
>>> m = nn.InstanceNorm1d(100, affine=True)
>>> input = torch.randn(20, 100, 40)
>>> output = m(input)


class torch.nn.InstanceNorm2d(num_features, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) [source]

Applies Instance Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.  y=x−E[x]Var[x]+ϵ∗γ+βy = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. γ\gamma  and β\beta  are learnable parameter vectors of size C (where C is the input size) if affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False). By default, this layer uses instance statistics computed from input data in both training and evaluation modes. If track_running_stats is set to True, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.  Note This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x^new=(1−momentum)×x^+momentum×xt\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t , where x^\hat{x}  is the estimated statistic and xtx_t  is the new observed value.   Note InstanceNorm2d and LayerNorm are very similar, but have some subtle differences. InstanceNorm2d is applied on each channel of channeled data like RGB images, but LayerNorm is usually applied on entire sample and often in NLP tasks. Additionally, LayerNorm applies elementwise affine transform, while InstanceNorm2d usually don’t apply affine transform.   Parameters

num_features – CC  from an expected input of size (N,C,H,W)(N, C, H, W)

eps – a value added to the denominator for numerical stability. Default: 1e-5
momentum – the value used for the running_mean and running_var computation. Default: 0.1
affine – a boolean value that when set to True, this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: False.
track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: False
     Shape:

 Input: (N,C,H,W)(N, C, H, W)
 Output: (N,C,H,W)(N, C, H, W)  (same shape as input)    Examples: >>> # Without Learnable Parameters
>>> m = nn.InstanceNorm2d(100)
>>> # With Learnable Parameters
>>> m = nn.InstanceNorm2d(100, affine=True)
>>> input = torch.randn(20, 100, 35, 45)
>>> output = m(input)


class torch.nn.InstanceNorm3d(num_features, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) [source]

Applies Instance Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.  y=x−E[x]Var[x]+ϵ∗γ+βy = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. γ\gamma  and β\beta  are learnable parameter vectors of size C (where C is the input size) if affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False). By default, this layer uses instance statistics computed from input data in both training and evaluation modes. If track_running_stats is set to True, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.  Note This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x^new=(1−momentum)×x^+momentum×xt\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t , where x^\hat{x}  is the estimated statistic and xtx_t  is the new observed value.   Note InstanceNorm3d and LayerNorm are very similar, but have some subtle differences. InstanceNorm3d is applied on each channel of channeled data like 3D models with RGB color, but LayerNorm is usually applied on entire sample and often in NLP tasks. Additionally, LayerNorm applies elementwise affine transform, while InstanceNorm3d usually don’t apply affine transform.   Parameters

num_features – CC  from an expected input of size (N,C,D,H,W)(N, C, D, H, W)

eps – a value added to the denominator for numerical stability. Default: 1e-5
momentum – the value used for the running_mean and running_var computation. Default: 0.1
affine – a boolean value that when set to True, this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: False.
track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: False
     Shape:

 Input: (N,C,D,H,W)(N, C, D, H, W)
 Output: (N,C,D,H,W)(N, C, D, H, W)  (same shape as input)    Examples: >>> # Without Learnable Parameters
>>> m = nn.InstanceNorm3d(100)
>>> # With Learnable Parameters
>>> m = nn.InstanceNorm3d(100, affine=True)
>>> input = torch.randn(20, 100, 35, 45, 10)
>>> output = m(input)


class torch.nn.intrinsic.ConvBn1d(conv, bn) [source]

This is a sequential container which calls the Conv 1d and Batch Norm 1d modules. During quantization this will be replaced with the corresponding fused module.


class torch.nn.intrinsic.ConvBn2d(conv, bn) [source]

This is a sequential container which calls the Conv 2d and Batch Norm 2d modules. During quantization this will be replaced with the corresponding fused module.