Registers a forward pre-hook on the module.
The hook will be called every time before forward() is invoked. It should have the following signature:
The input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward. The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned(unless that value is already a tuple).
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle


Registers a backward hook on the module.
The hook will be called every time the gradients with respect to module inputs are computed. The hook should have the following signature:
The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments.
Warning
Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error.
a handle that can be used to remove the added hook by calling handle.remove()
torch.utils.hooks.RemovableHandle


Adds a parameter to the module.
The parameter can be accessed as an attribute using given name.


Change if autograd should record operations on parameters in this module.
This method sets the parameters’ requires_grad attributes in-place.
This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training).
requires_grad (bool) – whether autograd should record operations on parameters in this module. Default: True.
self
Module


Returns a dictionary containing a whole state of the module.
Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names.
a dictionary containing a whole state of the module
dict
Example:


Moves and/or casts the parameters and buffers.
This can be called as
Its signature is similar to torch.Tensor.to(), but only accepts floating point or complex dtype`s. In addition, this method will
only cast the floating point or complex parameters and buffers to :attr:`dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.
See below for examples.
Note
This method modifies the module in-place.
self
Module
Examples:


Sets the module in training mode.
This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.
mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.
self
Module


Casts all parameters and buffers to dst_type.
dst_type (type or string) – the desired type
self
Module


Moves all model parameters and buffers to the XPU.
This also makes associated parameters and buffers different objects. So it should be called before constructing optimizer if the module will live on XPU while being optimized.
device (int, optional) – if specified, all parameters will be copied to that device
self
Module


Sets gradients of all model parameters to zero. See similar function under torch.optim.Optimizer for more context.
set_to_none (bool) – instead of setting to zero, set the grads to None. See torch.optim.Optimizer.zero_grad() for details.


Extracts sliding local blocks from a batched input tensor.
Consider a batched input tensor of shape (N,C,∗)(N, C, *) , where NN  is the batch dimension, CC  is the channel dimension, and ∗*  represent arbitrary spatial dimensions. This operation flattens each sliding kernel_size-sized block within the spatial dimensions of input into a column (i.e., last dimension) of a 3-D output tensor of shape (N,C×∏(kernel_size),L)(N, C \times \prod(\text{kernel\_size}), L) , where C×∏(kernel_size)C \times \prod(\text{kernel\_size})  is the total number of values within each block (a block has ∏(kernel_size)\prod(\text{kernel\_size})  spatial locations each containing a CC -channeled vector), and LL  is the total number of such blocks:
where spatial_size\text{spatial\_size}  is formed by the spatial dimensions of input (∗*  above), and dd  is over all spatial dimensions.
Therefore, indexing output at the last dimension (column dimension) gives all values within a certain block.
The padding, stride and dilation arguments specify how the sliding blocks are retrieved.
Note
Fold calculates each combined value in the resulting large tensor by summing all values from all containing blocks. Unfold extracts the values in the local blocks by copying from the large tensor. So, if the blocks overlap, they are not inverses of each other.
In general, folding and unfolding operations are related as follows. Consider Fold and Unfold instances created with the same parameters:
Then for any (supported) input tensor the following equality holds:
where divisor is a tensor that depends only on the shape and dtype of the input:
When the divisor tensor contains no zero elements, then fold and unfold operations are inverses of each other (up to constant divisor).
Warning
Currently, only 4-D input tensors (batched image-like tensors) are supported.
Examples:


Upsamples a given multi-channel 1D (temporal), 2D (spatial) or 3D (volumetric) data.
The input data is assumed to be of the form minibatch x channels x [optional depth] x [optional height] x width. Hence, for spatial inputs, we expect a 4D Tensor and for volumetric inputs, we expect a 5D Tensor.
The algorithms available for upsampling are nearest neighbor and linear, bilinear, bicubic and trilinear for 3D, 4D and 5D input Tensor, respectively.
One can either give a scale_factor or the target output size to calculate the output size. (You cannot give both, as it is ambiguous)
Warning
With align_corners = True, the linearly interpolating modes (linear, bilinear, bicubic, and trilinear) don’t proportionally align the output and input pixels, and thus the output values can depend on the input size. This was the default behavior for these modes up to version 0.3.1. Since then, the default behavior is align_corners = False. See below for concrete examples on how this affects the outputs.
Note
If you want downsampling/general resizing, you should use interpolate().
Examples:


Applies a 2D bilinear upsampling to an input signal composed of several input channels.
To specify the scale, it takes either the size or the scale_factor as it’s constructor argument.
When size is given, it is the output size of the image (h, w).
Warning
This class is deprecated in favor of interpolate(). It is equivalent to nn.functional.interpolate(..., mode='bilinear', align_corners=True).
Examples:


Applies a 2D nearest neighbor upsampling to an input signal composed of several input channels.
To specify the scale, it takes either the size or the scale_factor as it’s constructor argument.
When size is given, it is the output size of the image (h, w).
Warning
This class is deprecated in favor of interpolate().
Examples:


Clips gradient norm of an iterable of parameters.
The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place.
Total norm of the parameters (viewed as a single vector).


Clips gradient of an iterable of parameters at specified value.
Gradients are modified in-place.


Convert parameters to one vector
parameters (Iterable[Tensor]) – an iterator of Tensors that are the parameters of a model.
The parameters represented by a single vector


Abstract base class for creation of new pruning techniques.
Provides a skeleton for customization requiring the overriding of methods such as compute_mask() and apply().
Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)
Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the default_mask according to the specific pruning method recipe.
mask to apply to t, of same dims as t
mask (torch.Tensor)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.


Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)


Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the default_mask according to the specific pruning method recipe.
mask to apply to t, of same dims as t
mask (torch.Tensor)


Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.


Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Prunes tensor corresponding to parameter called name in module by applying the pre-computed mask in mask. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.
modified (i.e. pruned) version of the input module
module (nn.Module)


Globally prunes tensors corresponding to all parameters in parameters by applying the specified pruning_method. Modifies modules in place by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.
TypeError – if PRUNING_TYPE != 'unstructured'
Note
Since global structured pruning doesn’t make much sense unless the norm is normalized by the size of the parameter, we now limit the scope of global pruning to unstructured methods.


Utility pruning method that does not prune any units but generates the pruning parametrization with a mask of ones.
Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Check whether module is pruned by looking for forward_pre_hooks in its modules that inherit from the BasePruningMethod.
module (nn.Module) – object that is either pruned or unpruned
binary answer to whether module is pruned.


Prune (currently unpruned) units in a tensor by zeroing out the ones with the lowest L1-norm.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) units with the lowest L1-norm. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.
modified (i.e. pruned) version of the input module
module (nn.Module)


Prune entire (currently unpruned) channels in a tensor based on their Ln-norm.
Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)
Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a mask to apply on top of the default_mask by zeroing out the channels along the specified dim with the lowest Ln-norm.
mask to apply to t, of same dims as t
mask (torch.Tensor)
IndexError – if self.dim >= len(t.shape)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a mask to apply on top of the default_mask by zeroing out the channels along the specified dim with the lowest Ln-norm.
mask to apply to t, of same dims as t
mask (torch.Tensor)
IndexError – if self.dim >= len(t.shape)


Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) channels along the specified dim with the lowest L``n``-norm. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.
modified (i.e. pruned) version of the input module
module (nn.Module)


Container holding a sequence of pruning methods for iterative pruning. Keeps track of the order in which pruning methods are applied and handles combining successive pruning calls.
Accepts as argument an instance of a BasePruningMethod or an iterable of them.
Adds a child pruning method to the container.
method (subclass of BasePruningMethod) – child pruning method to be added to the container.
Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)
Applies the latest method by computing the new partial masks and returning its combination with the default_mask. The new partial mask should be computed on the entries or channels that were not zeroed out by the default_mask. Which portions of the tensor t the new mask will be calculated from depends on the PRUNING_TYPE (handled by the type handler):
new mask that combines the effects of the default_mask and the new mask from the current pruning method (of same dimensions as default_mask and t).
mask (torch.Tensor)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Adds a child pruning method to the container.
method (subclass of BasePruningMethod) – child pruning method to be added to the container.


Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.


Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)


Applies the latest method by computing the new partial masks and returning its combination with the default_mask. The new partial mask should be computed on the entries or channels that were not zeroed out by the default_mask. Which portions of the tensor t the new mask will be calculated from depends on the PRUNING_TYPE (handled by the type handler):
new mask that combines the effects of the default_mask and the new mask from the current pruning method (of same dimensions as default_mask and t).
mask (torch.Tensor)


Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.


Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Prune entire (currently unpruned) channels in a tensor at random.
Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)
Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the default_mask by randomly zeroing out channels along the specified dim of the tensor.
mask to apply to t, of same dims as t
mask (torch.Tensor)
IndexError – if self.dim >= len(t.shape)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the default_mask by randomly zeroing out channels along the specified dim of the tensor.
mask to apply to t, of same dims as t
mask (torch.Tensor)
IndexError – if self.dim >= len(t.shape)


Prune (currently unpruned) units in a tensor at random.
Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.
module (nn.Module) – module containing the tensor to prune
pruned version of the input tensor
pruned_tensor (torch.Tensor)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().
pruned version of tensor t.
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) channels along the specified dim selected at random. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.
modified (i.e. pruned) version of the input module
module (nn.Module)


Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) units selected at random. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.
modified (i.e. pruned) version of the input module
module (nn.Module)


Removes the pruning reparameterization from a module and the pruning method from the forward hook. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.
Note
Pruning itself is NOT undone or reversed!


Removes the spectral normalization reparameterization from a module.


Removes the weight normalization reparameterization from a module.


Holds the data and list of batch_sizes of a packed sequence.
All RNN modules accept packed sequences as inputs.
Note
Instances of this class should never be created manually. They are meant to be instantiated by functions like pack_padded_sequence().
Batch sizes represent the number elements at each sequence step in the batch, not the varying sequence lengths passed to pack_padded_sequence(). For instance, given data abc and x the PackedSequence would contain data axbc with batch_sizes=[2,1,1].
Note
data can be on arbitrary device and of arbitrary dtype. sorted_indices and unsorted_indices must be torch.int64 tensors on the same device as data.
However, batch_sizes should always be a CPU torch.int64 tensor.
This invariant is maintained throughout PackedSequence class, and all functions that construct a :class:PackedSequence in PyTorch (i.e., they only pass in tensors conforming to this constraint).
Alias for field number 1
Return number of occurrences of value.
Alias for field number 0
Return first index of value.
Raises ValueError if the value is not present.
Returns true if self.data stored on a gpu
Returns true if self.data stored on in pinned memory
Alias for field number 2
Performs dtype and/or device conversion on self.data.
It has similar signature as torch.Tensor.to(), except optional arguments like non_blocking and copy should be passed as kwargs, not args, or they will not apply to the index tensors.
Note
If the self.data Tensor already has the correct torch.dtype and torch.device, then self is returned. Otherwise, returns a copy with the desired configuration.
Alias for field number 3


Return first index of value.
Raises ValueError if the value is not present.


Returns true if self.data stored on a gpu


Returns true if self.data stored on in pinned memory


Performs dtype and/or device conversion on self.data.
It has similar signature as torch.Tensor.to(), except optional arguments like non_blocking and copy should be passed as kwargs, not args, or they will not apply to the index tensors.
Note
If the self.data Tensor already has the correct torch.dtype and torch.device, then self is returned. Otherwise, returns a copy with the desired configuration.


Packs a Tensor containing padded sequences of variable length.
input can be of size T x B x * where T is the length of the longest sequence (equal to lengths[0]), B is the batch size, and * is any number of dimensions (including 0). If batch_first is True, B x T x * input is expected.
For unsorted sequences, use enforce_sorted = False. If enforce_sorted is True, the sequences should be sorted by length in a decreasing order, i.e. input[:,0] should be the longest sequence, and input[:,B-1] the shortest one. enforce_sorted = True is only necessary for ONNX export.
Note
This function accepts any input that has at least two dimensions. You can apply it to pack the labels, and use the output of the RNN with them to compute the loss directly. A Tensor can be retrieved from a PackedSequence object by accessing its .data attribute.
a PackedSequence object


Packs a list of variable length Tensors
sequences should be a list of Tensors of size L x *, where L is the length of a sequence and * is any number of trailing dimensions, including zero.
For unsorted sequences, use enforce_sorted = False. If enforce_sorted is True, the sequences should be sorted in the order of decreasing length. enforce_sorted = True is only necessary for ONNX export.
a PackedSequence object


Pads a packed batch of variable length sequences.
It is an inverse operation to pack_padded_sequence().
The returned Tensor’s data will be of size T x B x *, where T is the length of the longest sequence and B is the batch size. If batch_first is True, the data will be transposed into B x T x * format.
Note
total_length is useful to implement the pack sequence -> recurrent network -> unpack sequence pattern in a Module wrapped in DataParallel. See this FAQ section for details.
Tuple of Tensor containing the padded sequence, and a Tensor containing the list of lengths of each sequence in the batch. Batch elements will be re-ordered as they were ordered originally when the batch was passed to pack_padded_sequence or pack_sequence.


Pad a list of variable length Tensors with padding_value
pad_sequence stacks a list of Tensors along a new dimension, and pads them to equal length. For example, if the input is list of sequences with size L x * and if batch_first is False, and T x B x * otherwise.
B is batch size. It is equal to the number of elements in sequences. T is length of the longest sequence. L is length of the sequence. * is any number of trailing dimensions, including none.
Note
This function returns a Tensor of size T x B x * or B x T x * where T is the length of the longest sequence. This function assumes trailing dimensions and type of all the Tensors in sequences are same.
Tensor of size T x B x * if batch_first is False. Tensor of size B x T x * otherwise


Applies spectral normalization to a parameter in the given module.
Spectral normalization stabilizes the training of discriminators (critics) in Generative Adversarial Networks (GANs) by rescaling the weight tensor with spectral norm σ\sigma  of the weight matrix calculated using power iteration method. If the dimension of the weight tensor is greater than 2, it is reshaped to 2D in power iteration method to get spectral norm. This is implemented via a hook that calculates spectral norm and rescales weight before every forward() call.
See Spectral Normalization for Generative Adversarial Networks .
The original module with the spectral norm hook
Example:


Applies weight normalization to a parameter in the given module.
Weight normalization is a reparameterization that decouples the magnitude of a weight tensor from its direction. This replaces the parameter specified by name (e.g. 'weight') with two parameters: one specifying the magnitude (e.g. 'weight_g') and one specifying the direction (e.g. 'weight_v'). Weight normalization is implemented via a hook that recomputes the weight tensor from the magnitude and direction before every forward() call.
By default, with dim=0, the norm is computed independently per output channel/plane. To compute a norm over the entire weight tensor, use dim=None.
See https://arxiv.org/abs/1602.07868
The original module with the weight norm hook
Example:


Pads the input tensor boundaries with zero.
For N-dimensional padding, use torch.nn.functional.pad().
padding (int, tuple) – the size of the padding. If is int, uses the same padding in all boundaries. If a 4-tuple, uses (padding_left\text{padding\_left} , padding_right\text{padding\_right} , padding_top\text{padding\_top} , padding_bottom\text{padding\_bottom} )
Output: (N,C,Hout,Wout)(N, C, H_{out}, W_{out})  where
Hout=Hin+padding_top+padding_bottomH_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}
Wout=Win+padding_left+padding_rightW_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}
Examples:


Note
torch.nonzero(..., as_tuple=False) (default) returns a 2-D tensor where each row is the index for a nonzero value.
torch.nonzero(..., as_tuple=True) returns a tuple of 1-D index tensors, allowing for advanced indexing, so x[x.nonzero(as_tuple=True)] gives all nonzero values of tensor x. Of the returned tuple, each index tensor contains nonzero indices for a certain dimension.
See below for more details on the two behaviors.
When input is on CUDA, torch.nonzero() causes host-device synchronization.
When as_tuple is ``False`` (default):
Returns a tensor containing the indices of all non-zero elements of input. Each row in the result contains the indices of a non-zero element in input. The result is sorted lexicographically, with the last index changing the fastest (C-style).
If input has nn  dimensions, then the resulting indices tensor out is of size (z×n)(z \times n) , where zz  is the total number of non-zero elements in the input tensor.
When as_tuple is ``True``:
Returns a tuple of 1-D tensors, one for each dimension in input, each containing the indices (in that dimension) of all non-zero elements of input .
If input has nn  dimensions, then the resulting tuple contains nn  tensors of size zz , where zz  is the total number of non-zero elements in the input tensor.
As a special case, when input has zero dimensions and a nonzero scalar value, it is treated as a one-dimensional tensor with one element.
input (Tensor) – the input tensor.
out (LongTensor, optional) – the output tensor containing indices
If as_tuple is False, the output tensor containing indices. If as_tuple is True, one 1-D tensor for each dimension, containing the indices of each nonzero element along that dimension.
LongTensor or tuple of LongTensor
Example:


Returns the matrix norm or vector norm of a given tensor.
Warning
torch.norm is deprecated and may be removed in a future PyTorch release. Use torch.linalg.norm() instead, but note that torch.linalg.norm() has a different signature and slightly different behavior that is more consistent with NumPy’s numpy.linalg.norm.
p (int, float, inf, -inf, 'fro', 'nuc', optional) –
the order of norm. Default: 'fro' The following norms can be calculated:
ord
matrix norm
vector norm
’fro’
Frobenius norm
–
‘nuc’
nuclear norm
–
Number
–
sum(abs(x)**ord)**(1./ord)
The vector norm can be calculated across any number of dimensions. The corresponding dimensions of input are flattened into one dimension, and the norm is calculated on the flattened dimension.
Frobenius norm produces the same result as p=2 in all cases except when dim is a list of three or more dims, in which case Frobenius norm throws an error.
Nuclear norm can only be calculated across exactly two dimensions.
Note
Even though p='fro' supports any number of dimensions, the true mathematical definition of Frobenius norm only applies to tensors with exactly two dimensions. torch.linalg.norm() with ord='fro' aligns with the mathematical definition, since it can only be applied across exactly two dimensions.
Example:


Returns a tensor of random numbers drawn from separate normal distributions whose mean and standard deviation are given.
The mean is a tensor with the mean of each output element’s normal distribution
The std is a tensor with the standard deviation of each output element’s normal distribution
The shapes of mean and std don’t need to match, but the total number of elements in each tensor need to be the same.
Note
When the shapes do not match, the shape of mean is used as the shape for the returned output tensor
Example:
Similar to the function above, but the means are shared among all drawn elements.
out (Tensor, optional) – the output tensor.
Example:
Similar to the function above, but the standard-deviations are shared among all drawn elements.
out (Tensor, optional) – the output tensor
Example:
Similar to the function above, but the means and standard deviations are shared among all drawn elements. The resulting tensor has size given by size.
out (Tensor, optional) – the output tensor.
Example:


Context-manager that disabled gradient calculation.
Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True.
In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True.
This context manager is thread local; it will not affect computation in other threads.
Also functions as a decorator. (Make sure to instantiate with parenthesis.)
Example:


Returns the total number of elements in the input tensor.
input (Tensor) – the input tensor.
Example:


Returns a tensor filled with the scalar value 1, with the shape defined by the variable argument size.
size (int...) – a sequence of integers defining the shape of the output tensor. Can be a variable number of arguments or a collection like a list or tuple.
Example:


Returns a tensor filled with the scalar value 1, with the same size as input. torch.ones_like(input) is equivalent to torch.ones(input.size(), dtype=input.dtype, layout=input.layout, device=input.device).
Warning
As of 0.4, this function does not support an out keyword. As an alternative, the old torch.ones_like(input, out=output) is equivalent to torch.ones(input.size(), out=output).
input (Tensor) – the size of input will determine size of the output tensor.
Example:


Indexing
Adding support for operators
Operator Export Type
Here is a simple script which exports a pretrained AlexNet as defined in torchvision into ONNX. It runs a single round of inference and then saves the resulting traced model to alexnet.onnx:
The resulting alexnet.onnx is a binary protobuf file which contains both the network structure and parameters of the model you exported (in this case, AlexNet). The keyword argument verbose=True causes the exporter to print out a human-readable representation of the network:
You can also verify the protobuf using the ONNX library. You can install ONNX with conda:
Then, you can run:
To run the exported script with caffe2, you will need to install caffe2: If you don’t have one already, Please follow the install instructions.
Once these are installed, you can use the backend for Caffe2:
You can also run the exported model with ONNX Runtime, you will need to install ONNX Runtime: please follow these instructions.
Once these are installed, you can use the backend for ONNX Runtime:
Here is another tutorial of exporting the SuperResolution model to ONNX..
In the future, there will be backends for other frameworks as well.
The ONNX exporter can be both trace-based and script-based exporter.
We allow mixing tracing and scripting. You can compose tracing and scripting to suit the particular requirements of a part of a model. Checkout this example:
With trace-based exporter, we get the result ONNX graph which unrolls the for loop:
To utilize script-based exporter for capturing the dynamic loop, we can write the loop in script, and call it from the regular nn.Module:
Now the exported ONNX graph becomes:
The dynamic control flow is captured correctly. We can verify in backends with different loop range.
To avoid exporting a variable scalar tensor as a fixed value constant as part of the ONNX model, please avoid use of torch.Tensor.item(). Torch supports implicit cast of single-element tensors to numbers. E.g.:
PyTorch models can be written using numpy manipulations, but this is not proper when we convert to the ONNX model. For the trace-based exporter, tracing treats the numpy values as the constant node, therefore it calculates the wrong result if we change the input. So the PyTorch model need implement using torch operators. For example, do not use numpy operators on numpy tensors:
do not convert to numpy types:
Always use torch tensors and torch operators: torch.concat, etc. In addition, Dropout layer need defined in init function so that inferencing can handle it properly, i.e.,
There are two ways to handle models which consist of named parameters or keyword arguments as inputs:
For example, in the model:
There are two ways of exporting the model:
Not using a dictionary for the keyword arguments and passing all the inputs in the same order as required by the model
Using a dictionary to represent the keyword arguments. This dictionary is always passed in addition to the non-keyword arguments and is always the last argument in the args tuple.
For cases in which there are no keyword arguments, models can be exported with either an empty or no dictionary. For example,
An exception to this rule are cases in which the last input is also of a dictionary type. In these cases it is mandatory to have an empty dictionary as the last argument in the args tuple. For example,
Without the presence of the empty dictionary, the export call assumes that the ‘x’ input is intended to represent the optional dictionary consisting of named arguments. In order to prevent this from being an issue a constraint is placed to provide an empty dictionary as the last input in the tuple args in such cases. The new call would look like this.
Tensor indexing in PyTorch is very flexible and complicated. There are two categories of indexing. Both are largely supported in exporting today. If you are experiencing issues exporting indexing that belongs to the supported patterns below, please double check that you are exporting with the latest opset (opset_version=12).
This type of indexing occurs on the RHS. Export is supported for ONNX opset version >= 9. E.g.:
Below is the list of supported patterns for RHS indexing.
And below is the list of unsupported patterns for RHS indexing.
In code, this type of indexing occurs on the LHS. Export is supported for ONNX opset version >= 11. E.g.:
Below is the list of supported patterns for LHS indexing.
And below is the list of unsupported patterns for LHS indexing.
If you are experiencing issues exporting indexing that belongs to the above supported patterns, please double check that you are exporting with the latest opset (opset_version=12).
All TorchVision models, except for quantized versions, are exportable to ONNX. More details can be found in TorchVision.
The following operators are supported:
The operator set above is sufficient to export the following models:
Adding export support for operators is an advance usage.
To achieve this, developers need to touch the source code of PyTorch. Please follow the instructions for installing PyTorch from source. If the wanted operator is standardized in ONNX, it should be easy to add support for exporting such operator (adding a symbolic function for the operator). To confirm whether the operator is standardized or not, please check the ONNX operator list.
If the operator is an ATen operator, which means you can find the declaration of the function in torch/csrc/autograd/generated/VariableType.h (available in generated code in PyTorch install dir), you should add the symbolic function in torch/onnx/symbolic_opset<version>.py and follow the instructions listed as below:
If the operator is a non-ATen operator, the symbolic function has to be added in the corresponding PyTorch Function class. Please read the following instructions:
Symbolic functions should be implemented in Python. All of these functions interact with Python methods which are implemented via C++-Python bindings, but intuitively the interface they provide looks like this:
The ONNX graph C++ definition is in torch/csrc/jit/ir/ir.h.
Here is an example of handling missing symbolic function for elu operator. We try to export the model and see the error message as below:
The export fails because PyTorch does not support exporting elu operator. We find virtual Tensor elu(const Tensor & input, Scalar alpha, bool inplace) const override; in VariableType.h. This means elu is an ATen operator. We check the ONNX operator list, and confirm that Elu is standardized in ONNX. We add the following lines to symbolic_opset9.py:
Now PyTorch is able to export elu operator.
There are more examples in symbolic_opset9.py, symbolic_opset10.py.
The interface for specifying operator definitions is experimental; adventurous users should note that the APIs will probably change in a future interface.
Following this tutorial Extending TorchScript with Custom C++ Operators, you can create and register your own custom ops implementation in PyTorch. Here’s how to export such model to ONNX.:
Depending on the custom operator, you can export it as one or a combination of existing ONNX ops. You can also export it as a custom op in ONNX as well. In that case, you can specify the custom domain and version (custom opset) using the custom_opsets dictionary at export. If not explicitly specified, the custom opset version is set to 1 by default. Using custom ONNX ops, you will need to extend the backend of your choice with matching custom ops implementation, e.g. Caffe2 custom ops, ONNX Runtime custom ops.
Exporting models with unsupported ONNX operators can be achieved using the operator_export_type flag in export API. This flag is useful when users try to export ATen and non-ATen operators that are not registered and supported in ONNX.
This mode is used to export all operators as regular ONNX operators. This is the default operator_export_type mode.
This mode is used to export all operators as ATen ops, and avoid conversion to ONNX.
To fallback on unsupported ATen operators in ONNX. Supported operators are exported to ONNX regularly. In the following example, aten::triu is not supported in ONNX. Exporter falls back on this operator.
To export a raw ir.
This mode can be used to export any operator (ATen or non-ATen) that is not registered and supported in ONNX. Exported falls through and exports the operator as is, as custom op. Exporting custom operators enables users to register and implement the operator as part of their runtime backend.
Q: I have exported my lstm model, but its input size seems to be fixed?
The tracer records the example inputs shape in the graph. In case the model should accept inputs of dynamic shape, you can utilize the parameter dynamic_axes in export api.
Q: How to export models with loops in it?
Please checkout Tracing vs Scripting.
Q: Does ONNX support implicit scalar datatype casting?
No, but the exporter will try to handle that part. Scalars are converted to constant tensors in ONNX. The exporter will try to figure out the right datatype for scalars. However for cases that it failed to do so, you will need to manually provide the datatype information. This often happens with scripted models, where the datatypes are not recorded. We are trying to improve the datatype propagation in the exporter such that manual changes are not required in the future.
Q: Is tensor in-place indexed assignment like data[index] = new_data supported?
Yes, this is supported for ONNX opset version >= 11. Please checkout Indexing.
Q: Is tensor list exportable to ONNX?
Yes, this is supported now for ONNX opset version >= 11. ONNX introduced the concept of Sequence in opset 11. Similar to list, Sequence is a data type that contains arbitrary number of Tensors. Associated operators are also introduced in ONNX, such as SequenceInsert, SequenceAt, etc. However, in-place list append within loops is not exportable to ONNX. To implement this, please use inplace add operator. E.g.:
use_external_data_format argument in export API enables export of models in ONNX external data format. With this option enabled, the exporter stores some model parameters in external binary files, rather than the ONNX file itself. These external binary files are stored in the same location as the ONNX file. Argument ‘f’ must be a string specifying the location of the model.
This argument enables export of large models to ONNX. Models larger than 2GB cannot be exported in one file because of the protobuf size limit. Users should set use_external_data_format to True to successfully export such models.
Training argument in export API allows users to export models in a training-friendly mode. TrainingMode.TRAINING exports model in a training-friendly mode that avoids certain model optimizations which might interfere with model parameter training. TrainingMode.PRESERVE exports the model in inference mode if model.training is False. Otherwise, it exports the model in a training-friendly mode. The default mode for this argument is TrainingMode.EVAL which exports the model in inference mode.
Export a model into ONNX format. This exporter runs your model once in order to get a trace of its execution to be exported; at the moment, it supports a limited set of dynamic models (e.g., RNNs.)
args (tuple of arguments or torch.Tensor, a dictionary consisting of named arguments (optional)) –
a dictionary to specify the input to the corresponding named parameter: - KEY: str, named parameter - VALUE: corresponding input args can be structured either as:
ONLY A TUPLE OF ARGUMENTS or torch.Tensor:
The inputs to the model, e.g., such that model(*args) is a valid invocation of the model. Any non-Tensor arguments will be hard-coded into the exported model; any Tensor arguments will become inputs of the exported model, in the order they occur in args. If args is a Tensor, this is equivalent to having called it with a 1-ary tuple of that Tensor.
A TUPLE OF ARGUEMENTS WITH A DICTIONARY OF NAMED PARAMETERS:
The inputs to the model are structured as a tuple consisting of non-keyword arguments and the last value of this tuple being a dictionary consisting of named parameters and the corresponding inputs as key-value pairs. If certain named argument is not present in the dictionary, it is assigned the default value, or None if default value is not provided.
Cases in which an dictionary input is the last input of the args tuple would cause a conflict when a dictionary of named parameters is used. The model below provides such an example.
… return x
m = Model() k = torch.randn(2, 3) x = {torch.tensor(1.): torch.randn(2, 3)}
In the previous iteration, the call to export API would look like
torch.onnx.export(model, (k, x), ‘test.onnx’)
This would work as intended. However, the export function would now assume that the ‘x’ input is intended to represent the optional dictionary consisting of named arguments. In order to prevent this from being an issue a constraint is placed to provide an empty dictionary as the last input in the tuple args in such cases. The new call would look like this.
torch.onnx.export(model, (k, x, {}), ‘test.onnx’)
operator_export_type (enum, default OperatorExportTypes.ONNX) –
OperatorExportTypes.ONNX: All ops are exported as regular ONNX ops (with ONNX namespace). OperatorExportTypes.ONNX_ATEN: All ops are exported as ATen ops (with aten namespace). OperatorExportTypes.ONNX_ATEN_FALLBACK: If an ATen op is not supported in ONNX or its symbolic is missing, fall back on ATen op. Registered ops are exported to ONNX regularly. Example graph:
is exported as:
In the above example, aten::triu is not supported in ONNX, hence exporter falls back on this op. OperatorExportTypes.RAW: Export raw ir. OperatorExportTypes.ONNX_FALLTHROUGH: If an op is not supported in ONNX, fall through and export the operator as is, as a custom ONNX op. Using this mode, the op can be exported and implemented by the user for their runtime backend. Example graph:
is exported as:
In the above example, prim::ListConstruct is not supported, hence exporter falls through.
dynamic_axes (dict<string, dict<python:int, string>> or dict<string, list(int)>, default empty dict) –
a dictionary to specify dynamic axes of input/output, such that: - KEY: input and/or output names - VALUE: index of dynamic axes for given key and potentially the name to be used for exported dynamic axes. In general the value is defined according to one of the following ways or a combination of both: (1). A list of integers specifying the dynamic axes of provided input. In this scenario automated names will be generated and applied to dynamic axes of provided input/output during export. OR (2). An inner dictionary that specifies a mapping FROM the index of dynamic axis in corresponding input/output TO the name that is desired to be applied on such axis of such input/output during export.
Example. if we have the following shape for inputs and outputs:
Then dynamic axes can be defined either as:
ONLY INDICES:
INDICES WITH CORRESPONDING NAMES:
MIXED MODE OF (1) and (2):
keep_initializers_as_inputs (bool, default None) –
If True, all the initializers (typically corresponding to parameters) in the exported graph will also be added as inputs to the graph. If False, then initializers are not added as inputs to the graph, and only the non-parameter inputs are added as inputs.
This may allow for better optimizations (such as constant folding etc.) by backends/runtimes that execute these graphs. If unspecified (default None), then the behavior is chosen automatically as follows. If operator_export_type is OperatorExportTypes.ONNX, the behavior is equivalent to setting this argument to False. For other values of operator_export_type, the behavior is equivalent to setting this argument to True. Note that for ONNX opset version < 9, initializers MUST be part of graph inputs. Therefore, if opset_version argument is set to a 8 or lower, this argument will be ignored.
A context manager to temporarily set the training mode of ‘model’ to ‘mode’, resetting it when we exit the with-block. A no-op if mode is None.
In version 1.6 changed to this from set_training
Check whether it’s in the middle of the ONNX export. This function returns True in the middle of torch.onnx.export(). torch.onnx.export should be executed with single thread.


Export a model into ONNX format. This exporter runs your model once in order to get a trace of its execution to be exported; at the moment, it supports a limited set of dynamic models (e.g., RNNs.)
args (tuple of arguments or torch.Tensor, a dictionary consisting of named arguments (optional)) –
a dictionary to specify the input to the corresponding named parameter: - KEY: str, named parameter - VALUE: corresponding input args can be structured either as:
ONLY A TUPLE OF ARGUMENTS or torch.Tensor:
The inputs to the model, e.g., such that model(*args) is a valid invocation of the model. Any non-Tensor arguments will be hard-coded into the exported model; any Tensor arguments will become inputs of the exported model, in the order they occur in args. If args is a Tensor, this is equivalent to having called it with a 1-ary tuple of that Tensor.
A TUPLE OF ARGUEMENTS WITH A DICTIONARY OF NAMED PARAMETERS:
The inputs to the model are structured as a tuple consisting of non-keyword arguments and the last value of this tuple being a dictionary consisting of named parameters and the corresponding inputs as key-value pairs. If certain named argument is not present in the dictionary, it is assigned the default value, or None if default value is not provided.
Cases in which an dictionary input is the last input of the args tuple would cause a conflict when a dictionary of named parameters is used. The model below provides such an example.
… return x
m = Model() k = torch.randn(2, 3) x = {torch.tensor(1.): torch.randn(2, 3)}
In the previous iteration, the call to export API would look like
torch.onnx.export(model, (k, x), ‘test.onnx’)
This would work as intended. However, the export function would now assume that the ‘x’ input is intended to represent the optional dictionary consisting of named arguments. In order to prevent this from being an issue a constraint is placed to provide an empty dictionary as the last input in the tuple args in such cases. The new call would look like this.
torch.onnx.export(model, (k, x, {}), ‘test.onnx’)
operator_export_type (enum, default OperatorExportTypes.ONNX) –
OperatorExportTypes.ONNX: All ops are exported as regular ONNX ops (with ONNX namespace). OperatorExportTypes.ONNX_ATEN: All ops are exported as ATen ops (with aten namespace). OperatorExportTypes.ONNX_ATEN_FALLBACK: If an ATen op is not supported in ONNX or its symbolic is missing, fall back on ATen op. Registered ops are exported to ONNX regularly. Example graph:
is exported as:
In the above example, aten::triu is not supported in ONNX, hence exporter falls back on this op. OperatorExportTypes.RAW: Export raw ir. OperatorExportTypes.ONNX_FALLTHROUGH: If an op is not supported in ONNX, fall through and export the operator as is, as a custom ONNX op. Using this mode, the op can be exported and implemented by the user for their runtime backend. Example graph:
is exported as:
In the above example, prim::ListConstruct is not supported, hence exporter falls through.
dynamic_axes (dict<string, dict<python:int, string>> or dict<string, list(int)>, default empty dict) –
a dictionary to specify dynamic axes of input/output, such that: - KEY: input and/or output names - VALUE: index of dynamic axes for given key and potentially the name to be used for exported dynamic axes. In general the value is defined according to one of the following ways or a combination of both: (1). A list of integers specifying the dynamic axes of provided input. In this scenario automated names will be generated and applied to dynamic axes of provided input/output during export. OR (2). An inner dictionary that specifies a mapping FROM the index of dynamic axis in corresponding input/output TO the name that is desired to be applied on such axis of such input/output during export.
Example. if we have the following shape for inputs and outputs:
Then dynamic axes can be defined either as:
ONLY INDICES:
INDICES WITH CORRESPONDING NAMES:
MIXED MODE OF (1) and (2):
keep_initializers_as_inputs (bool, default None) –
If True, all the initializers (typically corresponding to parameters) in the exported graph will also be added as inputs to the graph. If False, then initializers are not added as inputs to the graph, and only the non-parameter inputs are added as inputs.
This may allow for better optimizations (such as constant folding etc.) by backends/runtimes that execute these graphs. If unspecified (default None), then the behavior is chosen automatically as follows. If operator_export_type is OperatorExportTypes.ONNX, the behavior is equivalent to setting this argument to False. For other values of operator_export_type, the behavior is equivalent to setting this argument to True. Note that for ONNX opset version < 9, initializers MUST be part of graph inputs. Therefore, if opset_version argument is set to a 8 or lower, this argument will be ignored.


Check whether it’s in the middle of the ONNX export. This function returns True in the middle of torch.onnx.export(). torch.onnx.export should be executed with single thread.


A context manager to temporarily set the training mode of ‘model’ to ‘mode’, resetting it when we exit the with-block. A no-op if mode is None.
In version 1.6 changed to this from set_training


torch.optim is a package implementing various optimization algorithms. Most commonly used methods are already supported, and the interface is general enough, so that more sophisticated ones can be also easily integrated in the future.
To use torch.optim you have to construct an optimizer object, that will hold the current state and will update the parameters based on the computed gradients.
To construct an Optimizer you have to give it an iterable containing the parameters (all should be Variable s) to optimize. Then, you can specify optimizer-specific options such as the learning rate, weight decay, etc.
Note
If you need to move a model to GPU via .cuda(), please do so before constructing optimizers for it. Parameters of a model after .cuda() will be different objects with those before the call.
In general, you should make sure that optimized parameters live in consistent locations when optimizers are constructed and used.
Example:
Optimizer s also support specifying per-parameter options. To do this, instead of passing an iterable of Variable s, pass in an iterable of dict s. Each of them will define a separate parameter group, and should contain a params key, containing a list of parameters belonging to it. Other keys should match the keyword arguments accepted by the optimizers, and will be used as optimization options for this group.
Note
You can still pass options as keyword arguments. They will be used as defaults, in the groups that didn’t override them. This is useful when you only want to vary a single option, while keeping all others consistent between parameter groups.
For example, this is very useful when one wants to specify per-layer learning rates:
This means that model.base’s parameters will use the default learning rate of 1e-2, model.classifier’s parameters will use a learning rate of 1e-3, and a momentum of 0.9 will be used for all parameters.
All optimizers implement a step() method, that updates the parameters. It can be used in two ways:
This is a simplified version supported by most optimizers. The function can be called once the gradients are computed using e.g. backward().
Example:
Some optimization algorithms such as Conjugate Gradient and LBFGS need to reevaluate the function multiple times, so you have to pass in a closure that allows them to recompute your model. The closure should clear the gradients, compute the loss, and return it.
Example:
Base class for all optimizers.
Warning
Parameters need to be specified as collections that have a deterministic ordering that is consistent between runs. Examples of objects that don’t satisfy those properties are sets and iterators over values of dictionaries.
Add a param group to the Optimizer s param_groups.
This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the Optimizer as training progresses.
Loads the optimizer state.
state_dict (dict) – optimizer state. Should be an object returned from a call to state_dict().
Returns the state of the optimizer as a dict.
It contains two entries:
differs between optimizer classes.
Performs a single optimization step (parameter update).
closure (callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.
Note
Unless otherwise specified, this function should not modify the .grad field of the parameters.
Sets the gradients of all optimized torch.Tensor s to zero.
set_to_none (bool) – instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests zero_grad(set_to_none=True) followed by a backward pass, .grads are guaranteed to be None for params that did not receive a gradient. 3. torch.optim optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).
Implements Adadelta algorithm.
It has been proposed in ADADELTA: An Adaptive Learning Rate Method.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
Implements Adagrad algorithm.
It has been proposed in Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
Implements Adam algorithm.
It has been proposed in Adam: A Method for Stochastic Optimization. The implementation of the L2 penalty follows changes proposed in Decoupled Weight Decay Regularization.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
Implements AdamW algorithm.
The original Adam algorithm was proposed in Adam: A Method for Stochastic Optimization. The AdamW variant was proposed in Decoupled Weight Decay Regularization.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
Implements lazy version of Adam algorithm suitable for sparse tensors.
In this variant, only moments that show up in the gradient get updated, and only those portions of the gradient get applied to the parameters.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
Implements Adamax algorithm (a variant of Adam based on infinity norm).
It has been proposed in Adam: A Method for Stochastic Optimization.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
Implements Averaged Stochastic Gradient Descent.
It has been proposed in Acceleration of stochastic approximation by averaging.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
Implements L-BFGS algorithm, heavily inspired by minFunc <https://www.cs.ubc.ca/~schmidtm/Software/minFunc.html>.
Warning
This optimizer doesn’t support per-parameter options and parameter groups (there can be only one).
Warning
Right now all parameters have to be on a single device. This will be improved in the future.
Note
This is a very memory intensive optimizer (it requires additional param_bytes * (history_size + 1) bytes). If it doesn’t fit in memory try reducing the history size, or use a different algorithm.
Performs a single optimization step.
closure (callable) – A closure that reevaluates the model and returns the loss.
Implements RMSprop algorithm.
Proposed by G. Hinton in his course.
The centered version first appears in Generating Sequences With Recurrent Neural Networks.
The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus α/(v+ϵ)\alpha/(\sqrt{v} + \epsilon)  where α\alpha  is the scheduled learning rate and vv  is the weighted moving average of the squared gradient.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
Implements the resilient backpropagation algorithm.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
Implements stochastic gradient descent (optionally with momentum).
Nesterov momentum is based on the formula from On the importance of initialization and momentum in deep learning.
Note
The implementation of SGD with Momentum/Nesterov subtly differs from Sutskever et. al. and implementations in some other frameworks.
Considering the specific case of Momentum, the update can be written as
where pp , gg , vv  and μ\mu  denote the parameters, gradient, velocity, and momentum respectively.
This is in contrast to Sutskever et. al. and other frameworks which employ an update of the form
The Nesterov version is analogously modified.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
torch.optim.lr_scheduler provides several methods to adjust the learning rate based on the number of epochs. torch.optim.lr_scheduler.ReduceLROnPlateau allows dynamic learning rate reducing based on some validation measurements.
Learning rate scheduling should be applied after optimizer’s update; e.g., you should write your code this way:
Warning
Prior to PyTorch 1.1.0, the learning rate scheduler was expected to be called before the optimizer’s update; 1.1.0 changed this behavior in a BC-breaking way. If you use the learning rate scheduler (calling scheduler.step()) before the optimizer’s update (calling optimizer.step()), this will skip the first value of the learning rate schedule. If you are unable to reproduce results after upgrading to PyTorch 1.1.0, please check if you are calling scheduler.step() at the wrong time.
Sets the learning rate of each parameter group to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr.
Loads the schedulers state.
When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().
Returns the state of the scheduler as a dict.
It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas.
When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.
Multiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, sets initial lr as lr.
Loads the schedulers state.
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().
Returns the state of the scheduler as a dict.
It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas.
Decays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.
Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.
Decays the learning rate of each parameter group by gamma every epoch. When last_epoch=-1, sets initial lr as lr.
Set the learning rate of each parameter group using a cosine annealing schedule, where ηmax\eta_{max}  is set to the initial lr and TcurT_{cur}  is the number of epochs since the last restart in SGDR:
When last_epoch=-1, sets initial lr as lr. Notice that because the schedule is defined recursively, the learning rate can be simultaneously modified outside this scheduler by other operators. If the learning rate is set solely by this scheduler, the learning rate at each step becomes:
It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts. Note that this only implements the cosine annealing part of SGDR, and not the restarts.
Reduce learning rate when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This scheduler reads a metrics quantity and if no improvement is seen for a ‘patience’ number of epochs, the learning rate is reduced.
Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). The policy cycles the learning rate between two boundaries with a constant frequency, as detailed in the paper Cyclical Learning Rates for Training Neural Networks. The distance between the two boundaries can be scaled on a per-iteration or per-cycle basis.
Cyclical learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training.
This class has three built-in policies, as put forth in the paper:
This implementation was adapted from the github repo: bckenstler/CLR
Calculates the learning rate at batch index. This function treats self.last_epoch as the last batch index.
If self.cycle_momentum is True, this function has a side effect of updating the optimizer’s momentum.
Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.
The 1cycle learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training.
This scheduler is not chainable.
Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence):
You must either provide a value for total_steps or provide a value for both epochs and steps_per_epoch.
The default behaviour of this scheduler follows the fastai implementation of 1cycle, which claims that “unpublished work has shown even better results by using only two phases”. To mimic the behaviour of the original paper instead, set three_phase=True.
Set the learning rate of each parameter group using a cosine annealing schedule, where ηmax\eta_{max}  is set to the initial lr, TcurT_{cur}  is the number of epochs since the last restart and TiT_{i}  is the number of epochs between two warm restarts in SGDR:
When Tcur=TiT_{cur}=T_{i} , set ηt=ηmin\eta_t = \eta_{min} . When Tcur=0T_{cur}=0  after restart, set ηt=ηmax\eta_t=\eta_{max} .
It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts.
Step could be called after every batch update
This function can be called in an interleaved way.
torch.optim.swa_utils implements Stochastic Weight Averaging (SWA). In particular, torch.optim.swa_utils.AveragedModel class implements SWA models, torch.optim.swa_utils.SWALR implements the SWA learning rate scheduler and torch.optim.swa_utils.update_bn() is a utility function used to update SWA batch normalization statistics at the end of training.
SWA has been proposed in Averaging Weights Leads to Wider Optima and Better Generalization.
AveragedModel class serves to compute the weights of the SWA model. You can create an averaged model by running:
Here the model model can be an arbitrary torch.nn.Module object. swa_model will keep track of the running averages of the parameters of the model. To update these averages, you can use the update_parameters() function:
Typically, in SWA the learning rate is set to a high constant value. SWALR is a learning rate scheduler that anneals the learning rate to a fixed value, and then keeps it constant. For example, the following code creates a scheduler that linearly anneals the learning rate from its initial value to 0.05 in 5 epochs within each parameter group:
You can also use cosine annealing to a fixed value instead of linear annealing by setting anneal_strategy="cos".
update_bn() is a utility function that allows to compute the batchnorm statistics for the SWA model on a given dataloader loader at the end of training:
update_bn() applies the swa_model to every element in the dataloader and computes the activation statistics for each batch normalization layer in the model.
Warning
update_bn() assumes that each batch in the dataloader loader is either a tensors or a list of tensors where the first element is the tensor that the network swa_model should be applied to. If your dataloader has a different structure, you can update the batch normalization statistics of the swa_model by doing a forward pass with the swa_model on each element of the dataset.
By default, torch.optim.swa_utils.AveragedModel computes a running equal average of the parameters that you provide, but you can also use custom averaging functions with the avg_fn parameter. In the following example ema_model computes an exponential moving average.
Example:
In the example below, swa_model is the SWA model that accumulates the averages of the weights. We train the model for a total of 300 epochs and we switch to the SWA learning rate schedule and start to collect SWA averages of the parameters at epoch 160:


Implements Adadelta algorithm.
It has been proposed in ADADELTA: An Adaptive Learning Rate Method.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Implements Adagrad algorithm.
It has been proposed in Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Implements Adam algorithm.
It has been proposed in Adam: A Method for Stochastic Optimization. The implementation of the L2 penalty follows changes proposed in Decoupled Weight Decay Regularization.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Implements Adamax algorithm (a variant of Adam based on infinity norm).
It has been proposed in Adam: A Method for Stochastic Optimization.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Implements AdamW algorithm.
The original Adam algorithm was proposed in Adam: A Method for Stochastic Optimization. The AdamW variant was proposed in Decoupled Weight Decay Regularization.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Implements Averaged Stochastic Gradient Descent.
It has been proposed in Acceleration of stochastic approximation by averaging.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Implements L-BFGS algorithm, heavily inspired by minFunc <https://www.cs.ubc.ca/~schmidtm/Software/minFunc.html>.
Warning
This optimizer doesn’t support per-parameter options and parameter groups (there can be only one).
Warning
Right now all parameters have to be on a single device. This will be improved in the future.
Note
This is a very memory intensive optimizer (it requires additional param_bytes * (history_size + 1) bytes). If it doesn’t fit in memory try reducing the history size, or use a different algorithm.
Performs a single optimization step.
closure (callable) – A closure that reevaluates the model and returns the loss.


Performs a single optimization step.
closure (callable) – A closure that reevaluates the model and returns the loss.


Set the learning rate of each parameter group using a cosine annealing schedule, where ηmax\eta_{max}  is set to the initial lr and TcurT_{cur}  is the number of epochs since the last restart in SGDR:
When last_epoch=-1, sets initial lr as lr. Notice that because the schedule is defined recursively, the learning rate can be simultaneously modified outside this scheduler by other operators. If the learning rate is set solely by this scheduler, the learning rate at each step becomes:
It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts. Note that this only implements the cosine annealing part of SGDR, and not the restarts.


Set the learning rate of each parameter group using a cosine annealing schedule, where ηmax\eta_{max}  is set to the initial lr, TcurT_{cur}  is the number of epochs since the last restart and TiT_{i}  is the number of epochs between two warm restarts in SGDR:
When Tcur=TiT_{cur}=T_{i} , set ηt=ηmin\eta_t = \eta_{min} . When Tcur=0T_{cur}=0  after restart, set ηt=ηmax\eta_t=\eta_{max} .
It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts.
Step could be called after every batch update
This function can be called in an interleaved way.


Step could be called after every batch update
This function can be called in an interleaved way.


Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). The policy cycles the learning rate between two boundaries with a constant frequency, as detailed in the paper Cyclical Learning Rates for Training Neural Networks. The distance between the two boundaries can be scaled on a per-iteration or per-cycle basis.
Cyclical learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training.
This class has three built-in policies, as put forth in the paper:
This implementation was adapted from the github repo: bckenstler/CLR
Calculates the learning rate at batch index. This function treats self.last_epoch as the last batch index.
If self.cycle_momentum is True, this function has a side effect of updating the optimizer’s momentum.


Calculates the learning rate at batch index. This function treats self.last_epoch as the last batch index.
If self.cycle_momentum is True, this function has a side effect of updating the optimizer’s momentum.


Decays the learning rate of each parameter group by gamma every epoch. When last_epoch=-1, sets initial lr as lr.


Sets the learning rate of each parameter group to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr.
Loads the schedulers state.
When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().
Returns the state of the scheduler as a dict.
It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas.
When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.


Loads the schedulers state.
When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().


Returns the state of the scheduler as a dict.
It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas.
When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.


Multiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, sets initial lr as lr.
Loads the schedulers state.
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().
Returns the state of the scheduler as a dict.
It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas.


Loads the schedulers state.
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().


Returns the state of the scheduler as a dict.
It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas.


Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.


Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.
The 1cycle learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training.
This scheduler is not chainable.
Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence):
You must either provide a value for total_steps or provide a value for both epochs and steps_per_epoch.
The default behaviour of this scheduler follows the fastai implementation of 1cycle, which claims that “unpublished work has shown even better results by using only two phases”. To mimic the behaviour of the original paper instead, set three_phase=True.


Reduce learning rate when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This scheduler reads a metrics quantity and if no improvement is seen for a ‘patience’ number of epochs, the learning rate is reduced.


Decays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.


Base class for all optimizers.
Warning
Parameters need to be specified as collections that have a deterministic ordering that is consistent between runs. Examples of objects that don’t satisfy those properties are sets and iterators over values of dictionaries.
Add a param group to the Optimizer s param_groups.
This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the Optimizer as training progresses.
Loads the optimizer state.
state_dict (dict) – optimizer state. Should be an object returned from a call to state_dict().
Returns the state of the optimizer as a dict.
It contains two entries:
differs between optimizer classes.
Performs a single optimization step (parameter update).
closure (callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.
Note
Unless otherwise specified, this function should not modify the .grad field of the parameters.
Sets the gradients of all optimized torch.Tensor s to zero.
set_to_none (bool) – instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests zero_grad(set_to_none=True) followed by a backward pass, .grads are guaranteed to be None for params that did not receive a gradient. 3. torch.optim optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).


Add a param group to the Optimizer s param_groups.
This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the Optimizer as training progresses.


Loads the optimizer state.
state_dict (dict) – optimizer state. Should be an object returned from a call to state_dict().


Returns the state of the optimizer as a dict.
It contains two entries:
differs between optimizer classes.


Performs a single optimization step (parameter update).
closure (callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.
Note
Unless otherwise specified, this function should not modify the .grad field of the parameters.


Sets the gradients of all optimized torch.Tensor s to zero.
set_to_none (bool) – instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests zero_grad(set_to_none=True) followed by a backward pass, .grads are guaranteed to be None for params that did not receive a gradient. 3. torch.optim optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).


Implements RMSprop algorithm.
Proposed by G. Hinton in his course.
The centered version first appears in Generating Sequences With Recurrent Neural Networks.
The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus α/(v+ϵ)\alpha/(\sqrt{v} + \epsilon)  where α\alpha  is the scheduled learning rate and vv  is the weighted moving average of the squared gradient.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Implements the resilient backpropagation algorithm.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Implements stochastic gradient descent (optionally with momentum).
Nesterov momentum is based on the formula from On the importance of initialization and momentum in deep learning.
Note
The implementation of SGD with Momentum/Nesterov subtly differs from Sutskever et. al. and implementations in some other frameworks.
Considering the specific case of Momentum, the update can be written as
where pp , gg , vv  and μ\mu  denote the parameters, gradient, velocity, and momentum respectively.
This is in contrast to Sutskever et. al. and other frameworks which employ an update of the form
The Nesterov version is analogously modified.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Implements lazy version of Adam algorithm suitable for sparse tensors.
In this variant, only moments that show up in the gradient get updated, and only those portions of the gradient get applied to the parameters.
Performs a single optimization step.
closure (callable, optional) – A closure that reevaluates the model and returns the loss.


Computes the orthogonal matrix Q of a QR factorization, from the (input, input2) tuple returned by torch.geqrf().
This directly calls the underlying LAPACK function ?orgqr. See LAPACK documentation for orgqr for further details.


Multiplies mat (given by input3) by the orthogonal Q matrix of the QR factorization formed by torch.geqrf() that is represented by (a, tau) (given by (input, input2)).
This directly calls the underlying LAPACK function ?ormqr. See LAPACK documentation for ormqr for further details.


Outer product of input and vec2. If input is a vector of size nn  and vec2 is a vector of size mm , then out must be a matrix of size (n×m)(n \times m) .
Note
This function does not broadcast.
out (Tensor, optional) – optional output matrix
Example:


This module exposes various helper functions for the __torch_function__ protocol. See Extending torch for more detail on the __torch_function__ protocol.
Return public functions that cannot be overridden by __torch_function__.
A tuple of functions that are publicly available in the torch API but cannot be overridden with __torch_function__. Mostly this is because none of the arguments of these functions are tensors or tensor-likes.
Set[Callable]
List functions that are overridable via __torch_function__
A dictionary that maps namespaces that contain overridable functions to functions in that namespace that can be overridden.
Dict[Any, List[Callable]]
Return a dict containing dummy overrides for all overridable functions
A dictionary that maps overridable functions in the PyTorch API to lambda functions that have the same signature as the real function and unconditionally return -1. These lambda functions are useful for testing API coverage for a type that defines __torch_function__.
Dict[Callable, Callable]
Implement a function with checks for __torch_function__ overrides.
See torch::autograd::handle_torch_function for the equivalent of this function in the C++ implementation.
Result from calling implementation or an __torch_function__ method, as appropriate.
object
:raises TypeError : if no implementation is found.:
Check for __torch_function__ implementations in the elements of an iterable. Considers exact Tensor s and Parameter s non-dispatchable. :param relevant_args: Iterable or aguments to check for __torch_function__ methods. :type relevant_args: iterable
True if any of the elements of relevant_args have __torch_function__ implementations, False otherwise.
bool
See also
Checks if something is a Tensor-like, including an exact Tensor.
Returns True if the passed-in input is a Tensor-like.
Currently, this occurs whenever there’s a __torch_function__ attribute on the type of the input.
A subclass of tensor is generally a Tensor-like.
Built-in or user types aren’t usually Tensor-like.
But, they can be made Tensor-like by implementing __torch_function__.
Returns True if the function passed in is a handler for a method or property belonging to torch.Tensor, as passed into __torch_function__.
Note
For properties, their __get__ method must be passed in.
This may be needed, in particular, for the following reasons:
Wraps a given function with __torch_function__ -related functionality.
dispatcher (Callable) – A callable that returns an iterable of Tensor-likes passed into the function.
Note
This decorator may reduce the performance of your code. Generally, it’s enough to express your code as a series of functions that, themselves, support __torch_function__. If you find yourself in the rare situation where this is not the case, e.g. if you’re wrapping a low-level library and you also need it to work for Tensor-likes, then this function is available.


Return public functions that cannot be overridden by __torch_function__.
A tuple of functions that are publicly available in the torch API but cannot be overridden with __torch_function__. Mostly this is because none of the arguments of these functions are tensors or tensor-likes.
Set[Callable]


List functions that are overridable via __torch_function__
A dictionary that maps namespaces that contain overridable functions to functions in that namespace that can be overridden.
Dict[Any, List[Callable]]


Return a dict containing dummy overrides for all overridable functions
A dictionary that maps overridable functions in the PyTorch API to lambda functions that have the same signature as the real function and unconditionally return -1. These lambda functions are useful for testing API coverage for a type that defines __torch_function__.
Dict[Callable, Callable]


Implement a function with checks for __torch_function__ overrides.
See torch::autograd::handle_torch_function for the equivalent of this function in the C++ implementation.
Result from calling implementation or an __torch_function__ method, as appropriate.
object
:raises TypeError : if no implementation is found.:


Check for __torch_function__ implementations in the elements of an iterable. Considers exact Tensor s and Parameter s non-dispatchable. :param relevant_args: Iterable or aguments to check for __torch_function__ methods. :type relevant_args: iterable
True if any of the elements of relevant_args have __torch_function__ implementations, False otherwise.
bool
See also
Checks if something is a Tensor-like, including an exact Tensor.


Returns True if the passed-in input is a Tensor-like.
Currently, this occurs whenever there’s a __torch_function__ attribute on the type of the input.
A subclass of tensor is generally a Tensor-like.
Built-in or user types aren’t usually Tensor-like.
But, they can be made Tensor-like by implementing __torch_function__.


Returns True if the function passed in is a handler for a method or property belonging to torch.Tensor, as passed into __torch_function__.
Note
For properties, their __get__ method must be passed in.
This may be needed, in particular, for the following reasons:


Wraps a given function with __torch_function__ -related functionality.
dispatcher (Callable) – A callable that returns an iterable of Tensor-likes passed into the function.
Note
This decorator may reduce the performance of your code. Generally, it’s enough to express your code as a series of functions that, themselves, support __torch_function__. If you find yourself in the rare situation where this is not the case, e.g. if you’re wrapping a low-level library and you also need it to work for Tensor-likes, then this function is available.


Performs linear Principal Component Analysis (PCA) on a low-rank matrix, batches of such matrices, or sparse matrix.
This function returns a namedtuple (U, S, V) which is the nearly optimal approximation of a singular value decomposition of a centered matrix AA  such that A=Udiag(S)VTA = U diag(S) V^T .
Note
The relation of (U, S, V) to PCA is as follows:
Note
Different from the standard SVD, the size of returned matrices depend on the specified rank and q values as follows:
Note
To obtain repeatable results, reset the seed for the pseudorandom number generator
References:


Calculates the pseudo-inverse (also known as the Moore-Penrose inverse) of a 2D tensor. Please look at Moore-Penrose inverse for more details
Note
torch.pinverse() is deprecated. Please use torch.linalg.pinv() instead which includes new parameters hermitian and out.
Note
This method is implemented using the Singular Value Decomposition.
Note
The pseudo-inverse is not necessarily a continuous function in the elements of the matrix [1]. Therefore, derivatives are not always existent, and exist for a constant rank only [2]. However, this method is backprop-able due to the implementation by using SVD results, and could be unstable. Double-backward will also be unstable due to the usage of SVD internally. See svd() for more details.
Note
Supports real and complex inputs. Batched version for complex inputs is only supported on the CPU.
The pseudo-inverse of input of dimensions (∗,n,m)(*, n, m)
Example:


Returns a tensor of the same size as input with each element sampled from a Poisson distribution with rate parameter given by the corresponding element in input i.e.,
input (Tensor) – the input tensor containing the rates of the Poisson distribution
generator (torch.Generator, optional) – a pseudorandom number generator for sampling
Example: