torch.nn.utils.parameters_to_vector(parameters) [source]

Convert parameters to one vector  Parameters
parameters (Iterable[Tensor]) – an iterator of Tensors that are the parameters of a model.  Returns
The parameters represented by a single vector


class torch.nn.utils.prune.BasePruningMethod [source]

Abstract base class for creation of new pruning techniques. Provides a skeleton for customization requiring the overriding of methods such as compute_mask() and apply().
classmethod apply(module, name, *args, importance_scores=None, **kwargs) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
args – arguments passed on to a subclass of BasePruningMethod

importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the parameter will be used in its place.
kwargs – keyword arguments passed on to a subclass of a BasePruningMethod

apply_mask(module) [source]

Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)

abstract compute_mask(t, default_mask) [source]

Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the default_mask according to the specific pruning method recipe.  Parameters

t (torch.Tensor) – tensor representing the importance scores of the
to prune. (parameter) –
default_mask (torch.Tensor) – Base mask from previous pruning
that need to be respected after the new mask is (iterations,) –
Same dims as t. (applied.) –    Returns
mask to apply to t, of same dims as t  Return type
mask (torch.Tensor)

prune(t, default_mask=None, importance_scores=None) [source]

Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.

remove(module) [source]

Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


classmethod apply(module, name, *args, importance_scores=None, **kwargs) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
args – arguments passed on to a subclass of BasePruningMethod

importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the parameter will be used in its place.
kwargs – keyword arguments passed on to a subclass of a BasePruningMethod


apply_mask(module) [source]

Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)


abstract compute_mask(t, default_mask) [source]

Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the default_mask according to the specific pruning method recipe.  Parameters

t (torch.Tensor) – tensor representing the importance scores of the
to prune. (parameter) –
default_mask (torch.Tensor) – Base mask from previous pruning
that need to be respected after the new mask is (iterations,) –
Same dims as t. (applied.) –    Returns
mask to apply to t, of same dims as t  Return type
mask (torch.Tensor)


prune(t, default_mask=None, importance_scores=None) [source]

Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.


remove(module) [source]

Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


class torch.nn.utils.prune.CustomFromMask(mask) [source]

classmethod apply(module, name, mask) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.

apply_mask(module)
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)

prune(t, default_mask=None, importance_scores=None)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.

remove(module)
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


classmethod apply(module, name, mask) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.


apply_mask(module)
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)


prune(t, default_mask=None, importance_scores=None)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.


remove(module)
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


torch.nn.utils.prune.custom_from_mask(module, name, mask) [source]

Prunes tensor corresponding to parameter called name in module by applying the pre-computed mask in mask. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
mask (Tensor) – binary mask to be applied to the parameter.   Returns
modified (i.e. pruned) version of the input module  Return type
module (nn.Module)   Examples >>> m = prune.custom_from_mask(
        nn.Linear(5, 3), name='bias', mask=torch.Tensor([0, 1, 0])
    )
>>> print(m.bias_mask)
tensor([0., 1., 0.])


torch.nn.utils.prune.global_unstructured(parameters, pruning_method, importance_scores=None, **kwargs) [source]

Globally prunes tensors corresponding to all parameters in parameters by applying the specified pruning_method. Modifies modules in place by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.  Parameters

parameters (Iterable of (module, name) tuples) – parameters of the model to prune in a global fashion, i.e. by aggregating all weights prior to deciding which ones to prune. module must be of type nn.Module, and name must be a string.
pruning_method (function) – a valid pruning function from this module, or a custom one implemented by the user that satisfies the implementation guidelines and has PRUNING_TYPE='unstructured'.
importance_scores (dict) – a dictionary mapping (module, name) tuples to the corresponding parameter’s importance scores tensor. The tensor should be the same shape as the parameter, and is used for computing mask for pruning. If unspecified or None, the parameter will be used in place of its importance scores.
kwargs – other keyword arguments such as: amount (int or float): quantity of parameters to prune across the specified parameters. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.   Raises
TypeError – if PRUNING_TYPE != 'unstructured'    Note Since global structured pruning doesn’t make much sense unless the norm is normalized by the size of the parameter, we now limit the scope of global pruning to unstructured methods.  Examples >>> net = nn.Sequential(OrderedDict([
        ('first', nn.Linear(10, 4)),
        ('second', nn.Linear(4, 1)),
    ]))
>>> parameters_to_prune = (
        (net.first, 'weight'),
        (net.second, 'weight'),
    )
>>> prune.global_unstructured(
        parameters_to_prune,
        pruning_method=prune.L1Unstructured,
        amount=10,
    )
>>> print(sum(torch.nn.utils.parameters_to_vector(net.buffers()) == 0))
tensor(10, dtype=torch.uint8)


class torch.nn.utils.prune.Identity [source]

Utility pruning method that does not prune any units but generates the pruning parametrization with a mask of ones.
classmethod apply(module, name) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.

apply_mask(module)
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)

prune(t, default_mask=None, importance_scores=None)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.

remove(module)
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


classmethod apply(module, name) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.


torch.nn.utils.prune.is_pruned(module) [source]

Check whether module is pruned by looking for forward_pre_hooks in its modules that inherit from the BasePruningMethod.  Parameters
module (nn.Module) – object that is either pruned or unpruned  Returns
binary answer to whether module is pruned.   Examples >>> m = nn.Linear(5, 7)
>>> print(prune.is_pruned(m))
False
>>> prune.random_unstructured(m, name='weight', amount=0.2)
>>> print(prune.is_pruned(m))
True


class torch.nn.utils.prune.L1Unstructured(amount) [source]

Prune (currently unpruned) units in a tensor by zeroing out the ones with the lowest L1-norm.  Parameters
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
classmethod apply(module, name, amount, importance_scores=None) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place.

apply_mask(module)
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)

prune(t, default_mask=None, importance_scores=None)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.

remove(module)
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


classmethod apply(module, name, amount, importance_scores=None) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place.


torch.nn.utils.prune.l1_unstructured(module, name, amount, importance_scores=None) [source]

Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) units with the lowest L1-norm. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place.   Returns
modified (i.e. pruned) version of the input module  Return type
module (nn.Module)   Examples >>> m = prune.l1_unstructured(nn.Linear(2, 3), 'weight', amount=0.2)
>>> m.state_dict().keys()
odict_keys(['bias', 'weight_orig', 'weight_mask'])


class torch.nn.utils.prune.LnStructured(amount, n, dim=-1) [source]

Prune entire (currently unpruned) channels in a tensor based on their Ln-norm.  Parameters

amount (int or float) – quantity of channels to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
n (int, float, inf, -inf, 'fro', 'nuc') – See documentation of valid entries for argument p in torch.norm().
dim (int, optional) – index of the dim along which we define channels to prune. Default: -1.
classmethod apply(module, name, amount, n, dim, importance_scores=None) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
n (int, float, inf, -inf, 'fro', 'nuc') – See documentation of valid entries for argument p in torch.norm().
dim (int) – index of the dim along which we define channels to prune.
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place.

apply_mask(module)
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)

compute_mask(t, default_mask) [source]

Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a mask to apply on top of the default_mask by zeroing out the channels along the specified dim with the lowest Ln-norm.  Parameters

t (torch.Tensor) – tensor representing the parameter to prune
default_mask (torch.Tensor) – Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as t.   Returns
mask to apply to t, of same dims as t  Return type
mask (torch.Tensor)  Raises
IndexError – if self.dim >= len(t.shape)

prune(t, default_mask=None, importance_scores=None)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.

remove(module)
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


classmethod apply(module, name, amount, n, dim, importance_scores=None) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
n (int, float, inf, -inf, 'fro', 'nuc') – See documentation of valid entries for argument p in torch.norm().
dim (int) – index of the dim along which we define channels to prune.
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place.


compute_mask(t, default_mask) [source]

Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a mask to apply on top of the default_mask by zeroing out the channels along the specified dim with the lowest Ln-norm.  Parameters

t (torch.Tensor) – tensor representing the parameter to prune
default_mask (torch.Tensor) – Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as t.   Returns
mask to apply to t, of same dims as t  Return type
mask (torch.Tensor)  Raises
IndexError – if self.dim >= len(t.shape)


torch.nn.utils.prune.ln_structured(module, name, amount, n, dim, importance_scores=None) [source]

Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) channels along the specified dim with the lowest L``n``-norm. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
n (int, float, inf, -inf, 'fro', 'nuc') – See documentation of valid entries for argument p in torch.norm().
dim (int) – index of the dim along which we define channels to prune.
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place.   Returns
modified (i.e. pruned) version of the input module  Return type
module (nn.Module)   Examples >>> m = prune.ln_structured(
       nn.Conv2d(5, 3, 2), 'weight', amount=0.3, dim=1, n=float('-inf')
    )


class torch.nn.utils.prune.PruningContainer(*args) [source]

Container holding a sequence of pruning methods for iterative pruning. Keeps track of the order in which pruning methods are applied and handles combining successive pruning calls. Accepts as argument an instance of a BasePruningMethod or an iterable of them.
add_pruning_method(method) [source]

Adds a child pruning method to the container.  Parameters
method (subclass of BasePruningMethod) – child pruning method to be added to the container.

classmethod apply(module, name, *args, importance_scores=None, **kwargs)
Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
args – arguments passed on to a subclass of BasePruningMethod

importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the parameter will be used in its place.
kwargs – keyword arguments passed on to a subclass of a BasePruningMethod

apply_mask(module)
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)

compute_mask(t, default_mask) [source]

Applies the latest method by computing the new partial masks and returning its combination with the default_mask. The new partial mask should be computed on the entries or channels that were not zeroed out by the default_mask. Which portions of the tensor t the new mask will be calculated from depends on the PRUNING_TYPE (handled by the type handler):  for ‘unstructured’, the mask will be computed from the raveled list of nonmasked entries; for ‘structured’, the mask will be computed from the nonmasked channels in the tensor; for ‘global’, the mask will be computed across all entries.   Parameters

t (torch.Tensor) – tensor representing the parameter to prune (of same dimensions as default_mask).
default_mask (torch.Tensor) – mask from previous pruning iteration.   Returns
new mask that combines the effects of the default_mask and the new mask from the current pruning method (of same dimensions as default_mask and t).  Return type
mask (torch.Tensor)

prune(t, default_mask=None, importance_scores=None)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.

remove(module)
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


add_pruning_method(method) [source]

Adds a child pruning method to the container.  Parameters
method (subclass of BasePruningMethod) – child pruning method to be added to the container.


classmethod apply(module, name, *args, importance_scores=None, **kwargs)
Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
args – arguments passed on to a subclass of BasePruningMethod

importance_scores (torch.Tensor) – tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the parameter will be used in its place.
kwargs – keyword arguments passed on to a subclass of a BasePruningMethod


apply_mask(module)
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)


compute_mask(t, default_mask) [source]

Applies the latest method by computing the new partial masks and returning its combination with the default_mask. The new partial mask should be computed on the entries or channels that were not zeroed out by the default_mask. Which portions of the tensor t the new mask will be calculated from depends on the PRUNING_TYPE (handled by the type handler):  for ‘unstructured’, the mask will be computed from the raveled list of nonmasked entries; for ‘structured’, the mask will be computed from the nonmasked channels in the tensor; for ‘global’, the mask will be computed across all entries.   Parameters

t (torch.Tensor) – tensor representing the parameter to prune (of same dimensions as default_mask).
default_mask (torch.Tensor) – mask from previous pruning iteration.   Returns
new mask that combines the effects of the default_mask and the new mask from the current pruning method (of same dimensions as default_mask and t).  Return type
mask (torch.Tensor)


prune(t, default_mask=None, importance_scores=None)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.


remove(module)
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


class torch.nn.utils.prune.RandomStructured(amount, dim=-1) [source]

Prune entire (currently unpruned) channels in a tensor at random.  Parameters

amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
dim (int, optional) – index of the dim along which we define channels to prune. Default: -1.
classmethod apply(module, name, amount, dim=-1) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
dim (int, optional) – index of the dim along which we define channels to prune. Default: -1.

apply_mask(module)
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)

compute_mask(t, default_mask) [source]

Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the default_mask by randomly zeroing out channels along the specified dim of the tensor.  Parameters

t (torch.Tensor) – tensor representing the parameter to prune
default_mask (torch.Tensor) – Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as t.   Returns
mask to apply to t, of same dims as t  Return type
mask (torch.Tensor)  Raises
IndexError – if self.dim >= len(t.shape)

prune(t, default_mask=None, importance_scores=None)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.

remove(module)
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


classmethod apply(module, name, amount, dim=-1) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
dim (int, optional) – index of the dim along which we define channels to prune. Default: -1.


compute_mask(t, default_mask) [source]

Computes and returns a mask for the input tensor t. Starting from a base default_mask (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the default_mask by randomly zeroing out channels along the specified dim of the tensor.  Parameters

t (torch.Tensor) – tensor representing the parameter to prune
default_mask (torch.Tensor) – Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as t.   Returns
mask to apply to t, of same dims as t  Return type
mask (torch.Tensor)  Raises
IndexError – if self.dim >= len(t.shape)


class torch.nn.utils.prune.RandomUnstructured(amount) [source]

Prune (currently unpruned) units in a tensor at random.  Parameters

name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
classmethod apply(module, name, amount) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.

apply_mask(module)
Simply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor.  Parameters
module (nn.Module) – module containing the tensor to prune  Returns
pruned version of the input tensor  Return type
pruned_tensor (torch.Tensor)

prune(t, default_mask=None, importance_scores=None)
Computes and returns a pruned version of input tensor t according to the pruning rule specified in compute_mask().  Parameters

t (torch.Tensor) – tensor to prune (of same dimensions as default_mask).
importance_scores (torch.Tensor) – tensor of importance scores (of same shape as t) used to compute mask for pruning t. The values in this tensor indicate the importance of the corresponding elements in the t that is being pruned. If unspecified or None, the tensor t will be used in its place.
default_mask (torch.Tensor, optional) – mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones.   Returns
pruned version of tensor t.

remove(module)
Removes the pruning reparameterization from a module. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!


classmethod apply(module, name, amount) [source]

Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.


torch.nn.utils.prune.random_structured(module, name, amount, dim) [source]

Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) channels along the specified dim selected at random. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.
dim (int) – index of the dim along which we define channels to prune.   Returns
modified (i.e. pruned) version of the input module  Return type
module (nn.Module)   Examples >>> m = prune.random_structured(
        nn.Linear(5, 3), 'weight', amount=3, dim=1
    )
>>> columns_pruned = int(sum(torch.sum(m.weight, dim=0) == 0))
>>> print(columns_pruned)
3


torch.nn.utils.prune.random_unstructured(module, name, amount) [source]

Prunes tensor corresponding to parameter called name in module by removing the specified amount of (currently unpruned) units selected at random. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method. 2) replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named name+'_orig'.  Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.
amount (int or float) – quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune.   Returns
modified (i.e. pruned) version of the input module  Return type
module (nn.Module)   Examples >>> m = prune.random_unstructured(nn.Linear(2, 3), 'weight', amount=1)
>>> torch.sum(m.weight_mask == 0)
tensor(1)


torch.nn.utils.prune.remove(module, name) [source]

Removes the pruning reparameterization from a module and the pruning method from the forward hook. The pruned parameter named name remains permanently pruned, and the parameter named name+'_orig' is removed from the parameter list. Similarly, the buffer named name+'_mask' is removed from the buffers.  Note Pruning itself is NOT undone or reversed!   Parameters

module (nn.Module) – module containing the tensor to prune
name (str) – parameter name within module on which pruning will act.    Examples >>> m = random_unstructured(nn.Linear(5, 7), name='weight', amount=0.2)
>>> m = remove(m, name='weight')


torch.nn.utils.remove_spectral_norm(module, name='weight') [source]

Removes the spectral normalization reparameterization from a module.  Parameters

module (Module) – containing module
name (str, optional) – name of weight parameter    Example >>> m = spectral_norm(nn.Linear(40, 10))
>>> remove_spectral_norm(m)


torch.nn.utils.remove_weight_norm(module, name='weight') [source]

Removes the weight normalization reparameterization from a module.  Parameters

module (Module) – containing module
name (str, optional) – name of weight parameter    Example >>> m = weight_norm(nn.Linear(20, 40))
>>> remove_weight_norm(m)


class torch.nn.utils.rnn.PackedSequence [source]

Holds the data and list of batch_sizes of a packed sequence. All RNN modules accept packed sequences as inputs.  Note Instances of this class should never be created manually. They are meant to be instantiated by functions like pack_padded_sequence(). Batch sizes represent the number elements at each sequence step in the batch, not the varying sequence lengths passed to pack_padded_sequence(). For instance, given data abc and x the PackedSequence would contain data axbc with batch_sizes=[2,1,1].   Variables

~PackedSequence.data (Tensor) – Tensor containing packed sequence
~PackedSequence.batch_sizes (Tensor) – Tensor of integers holding information about the batch size at each sequence step
~PackedSequence.sorted_indices (Tensor, optional) – Tensor of integers holding how this PackedSequence is constructed from sequences.
~PackedSequence.unsorted_indices (Tensor, optional) – Tensor of integers holding how this to recover the original sequences with correct order.     Note data can be on arbitrary device and of arbitrary dtype. sorted_indices and unsorted_indices must be torch.int64 tensors on the same device as data. However, batch_sizes should always be a CPU torch.int64 tensor. This invariant is maintained throughout PackedSequence class, and all functions that construct a :class:PackedSequence in PyTorch (i.e., they only pass in tensors conforming to this constraint).
property batch_sizes
Alias for field number 1

count()
Return number of occurrences of value.

property data
Alias for field number 0

index()
Return first index of value. Raises ValueError if the value is not present.

property is_cuda
Returns true if self.data stored on a gpu

is_pinned() [source]

Returns true if self.data stored on in pinned memory

property sorted_indices
Alias for field number 2

to(*args, **kwargs) [source]

Performs dtype and/or device conversion on self.data. It has similar signature as torch.Tensor.to(), except optional arguments like non_blocking and copy should be passed as kwargs, not args, or they will not apply to the index tensors.  Note If the self.data Tensor already has the correct torch.dtype and torch.device, then self is returned. Otherwise, returns a copy with the desired configuration.

property unsorted_indices
Alias for field number 3


property batch_sizes
Alias for field number 1


count()
Return number of occurrences of value.


index()
Return first index of value. Raises ValueError if the value is not present.


property is_cuda
Returns true if self.data stored on a gpu


is_pinned() [source]

Returns true if self.data stored on in pinned memory


property sorted_indices
Alias for field number 2


to(*args, **kwargs) [source]

Performs dtype and/or device conversion on self.data. It has similar signature as torch.Tensor.to(), except optional arguments like non_blocking and copy should be passed as kwargs, not args, or they will not apply to the index tensors.  Note If the self.data Tensor already has the correct torch.dtype and torch.device, then self is returned. Otherwise, returns a copy with the desired configuration.


property unsorted_indices
Alias for field number 3


torch.nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first=False, enforce_sorted=True) [source]

Packs a Tensor containing padded sequences of variable length. input can be of size T x B x * where T is the length of the longest sequence (equal to lengths[0]), B is the batch size, and * is any number of dimensions (including 0). If batch_first is True, B x T x * input is expected. For unsorted sequences, use enforce_sorted = False. If enforce_sorted is True, the sequences should be sorted by length in a decreasing order, i.e. input[:,0] should be the longest sequence, and input[:,B-1] the shortest one. enforce_sorted = True is only necessary for ONNX export.  Note This function accepts any input that has at least two dimensions. You can apply it to pack the labels, and use the output of the RNN with them to compute the loss directly. A Tensor can be retrieved from a PackedSequence object by accessing its .data attribute.   Parameters

input (Tensor) – padded batch of variable length sequences.
lengths (Tensor or list(int)) – list of sequence lengths of each batch element (must be on the CPU if provided as a tensor).
batch_first (bool, optional) – if True, the input is expected in B x T x * format.
enforce_sorted (bool, optional) – if True, the input is expected to contain sequences sorted by length in a decreasing order. If False, the input will get sorted unconditionally. Default: True.   Returns
a PackedSequence object


torch.nn.utils.rnn.pack_sequence(sequences, enforce_sorted=True) [source]

Packs a list of variable length Tensors sequences should be a list of Tensors of size L x *, where L is the length of a sequence and * is any number of trailing dimensions, including zero. For unsorted sequences, use enforce_sorted = False. If enforce_sorted is True, the sequences should be sorted in the order of decreasing length. enforce_sorted = True is only necessary for ONNX export. Example >>> from torch.nn.utils.rnn import pack_sequence
>>> a = torch.tensor([1,2,3])
>>> b = torch.tensor([4,5])
>>> c = torch.tensor([6])
>>> pack_sequence([a, b, c])
PackedSequence(data=tensor([ 1,  4,  6,  2,  5,  3]), batch_sizes=tensor([ 3,  2,  1]))
  Parameters

sequences (list[Tensor]) – A list of sequences of decreasing length.
enforce_sorted (bool, optional) – if True, checks that the input contains sequences sorted by length in a decreasing order. If False, this condition is not checked. Default: True.   Returns
a PackedSequence object


torch.nn.utils.rnn.pad_packed_sequence(sequence, batch_first=False, padding_value=0.0, total_length=None) [source]

Pads a packed batch of variable length sequences. It is an inverse operation to pack_padded_sequence(). The returned Tensor’s data will be of size T x B x *, where T is the length of the longest sequence and B is the batch size. If batch_first is True, the data will be transposed into B x T x * format. Example >>> from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
>>> seq = torch.tensor([[1,2,0], [3,0,0], [4,5,6]])
>>> lens = [2, 1, 3]
>>> packed = pack_padded_sequence(seq, lens, batch_first=True, enforce_sorted=False)
>>> packed
PackedSequence(data=tensor([4, 1, 3, 5, 2, 6]), batch_sizes=tensor([3, 2, 1]),
               sorted_indices=tensor([2, 0, 1]), unsorted_indices=tensor([1, 2, 0]))
>>> seq_unpacked, lens_unpacked = pad_packed_sequence(packed, batch_first=True)
>>> seq_unpacked
tensor([[1, 2, 0],
        [3, 0, 0],
        [4, 5, 6]])
>>> lens_unpacked
tensor([2, 1, 3])
  Note total_length is useful to implement the pack sequence -> recurrent network -> unpack sequence pattern in a Module wrapped in DataParallel. See this FAQ section for details.   Parameters

sequence (PackedSequence) – batch to pad
batch_first (bool, optional) – if True, the output will be in B x T x * format.
padding_value (float, optional) – values for padded elements.
total_length (int, optional) – if not None, the output will be padded to have length total_length. This method will throw ValueError if total_length is less than the max sequence length in sequence.   Returns
Tuple of Tensor containing the padded sequence, and a Tensor containing the list of lengths of each sequence in the batch. Batch elements will be re-ordered as they were ordered originally when the batch was passed to pack_padded_sequence or pack_sequence.


torch.nn.utils.rnn.pad_sequence(sequences, batch_first=False, padding_value=0.0) [source]

Pad a list of variable length Tensors with padding_value pad_sequence stacks a list of Tensors along a new dimension, and pads them to equal length. For example, if the input is list of sequences with size L x * and if batch_first is False, and T x B x * otherwise. B is batch size. It is equal to the number of elements in sequences. T is length of the longest sequence. L is length of the sequence. * is any number of trailing dimensions, including none. Example >>> from torch.nn.utils.rnn import pad_sequence
>>> a = torch.ones(25, 300)
>>> b = torch.ones(22, 300)
>>> c = torch.ones(15, 300)
>>> pad_sequence([a, b, c]).size()
torch.Size([25, 3, 300])
  Note This function returns a Tensor of size T x B x * or B x T x * where T is the length of the longest sequence. This function assumes trailing dimensions and type of all the Tensors in sequences are same.   Parameters

sequences (list[Tensor]) – list of variable length sequences.
batch_first (bool, optional) – output will be in B x T x * if True, or in T x B x * otherwise
padding_value (float, optional) – value for padded elements. Default: 0.   Returns
Tensor of size T x B x * if batch_first is False. Tensor of size B x T x * otherwise


torch.nn.utils.spectral_norm(module, name='weight', n_power_iterations=1, eps=1e-12, dim=None) [source]

Applies spectral normalization to a parameter in the given module.  WSN=Wσ(W),σ(W)=maxh:h≠0∥Wh∥2∥h∥2\mathbf{W}_{SN} = \dfrac{\mathbf{W}}{\sigma(\mathbf{W})}, \sigma(\mathbf{W}) = \max_{\mathbf{h}: \mathbf{h} \ne 0} \dfrac{\|\mathbf{W} \mathbf{h}\|_2}{\|\mathbf{h}\|_2}
Spectral normalization stabilizes the training of discriminators (critics) in Generative Adversarial Networks (GANs) by rescaling the weight tensor with spectral norm σ\sigma  of the weight matrix calculated using power iteration method. If the dimension of the weight tensor is greater than 2, it is reshaped to 2D in power iteration method to get spectral norm. This is implemented via a hook that calculates spectral norm and rescales weight before every forward() call. See Spectral Normalization for Generative Adversarial Networks .  Parameters

module (nn.Module) – containing module
name (str, optional) – name of weight parameter
n_power_iterations (int, optional) – number of power iterations to calculate spectral norm
eps (float, optional) – epsilon for numerical stability in calculating norms
dim (int, optional) – dimension corresponding to number of outputs, the default is 0, except for modules that are instances of ConvTranspose{1,2,3}d, when it is 1
   Returns
The original module with the spectral norm hook   Example: >>> m = spectral_norm(nn.Linear(20, 40))
>>> m
Linear(in_features=20, out_features=40, bias=True)
>>> m.weight_u.size()
torch.Size([40])


torch.nn.utils.vector_to_parameters(vec, parameters) [source]

Convert one vector to the parameters  Parameters

vec (Tensor) – a single vector represents the parameters of a model.
parameters (Iterable[Tensor]) – an iterator of Tensors that are the parameters of a model.


torch.nn.utils.weight_norm(module, name='weight', dim=0) [source]

Applies weight normalization to a parameter in the given module.  w=gv∥v∥\mathbf{w} = g \dfrac{\mathbf{v}}{\|\mathbf{v}\|}
Weight normalization is a reparameterization that decouples the magnitude of a weight tensor from its direction. This replaces the parameter specified by name (e.g. 'weight') with two parameters: one specifying the magnitude (e.g. 'weight_g') and one specifying the direction (e.g. 'weight_v'). Weight normalization is implemented via a hook that recomputes the weight tensor from the magnitude and direction before every forward() call. By default, with dim=0, the norm is computed independently per output channel/plane. To compute a norm over the entire weight tensor, use dim=None. See https://arxiv.org/abs/1602.07868  Parameters

module (Module) – containing module
name (str, optional) – name of weight parameter
dim (int, optional) – dimension over which to compute the norm   Returns
The original module with the weight norm hook   Example: >>> m = weight_norm(nn.Linear(20, 40), name='weight')
>>> m
Linear(in_features=20, out_features=40, bias=True)
>>> m.weight_g.size()
torch.Size([40, 1])
>>> m.weight_v.size()
torch.Size([40, 20])


torch.nonzero(input, *, out=None, as_tuple=False) → LongTensor or tuple of LongTensors
 Note torch.nonzero(..., as_tuple=False) (default) returns a 2-D tensor where each row is the index for a nonzero value. torch.nonzero(..., as_tuple=True) returns a tuple of 1-D index tensors, allowing for advanced indexing, so x[x.nonzero(as_tuple=True)] gives all nonzero values of tensor x. Of the returned tuple, each index tensor contains nonzero indices for a certain dimension. See below for more details on the two behaviors. When input is on CUDA, torch.nonzero() causes host-device synchronization.  When as_tuple is ``False`` (default): Returns a tensor containing the indices of all non-zero elements of input. Each row in the result contains the indices of a non-zero element in input. The result is sorted lexicographically, with the last index changing the fastest (C-style). If input has nn  dimensions, then the resulting indices tensor out is of size (z×n)(z \times n) , where zz  is the total number of non-zero elements in the input tensor. When as_tuple is ``True``: Returns a tuple of 1-D tensors, one for each dimension in input, each containing the indices (in that dimension) of all non-zero elements of input . If input has nn  dimensions, then the resulting tuple contains nn  tensors of size zz , where zz  is the total number of non-zero elements in the input tensor. As a special case, when input has zero dimensions and a nonzero scalar value, it is treated as a one-dimensional tensor with one element.  Parameters
input (Tensor) – the input tensor.  Keyword Arguments
out (LongTensor, optional) – the output tensor containing indices  Returns
If as_tuple is False, the output tensor containing indices. If as_tuple is True, one 1-D tensor for each dimension, containing the indices of each nonzero element along that dimension.  Return type
LongTensor or tuple of LongTensor   Example: >>> torch.nonzero(torch.tensor([1, 1, 1, 0, 1]))
tensor([[ 0],
        [ 1],
        [ 2],
        [ 4]])
>>> torch.nonzero(torch.tensor([[0.6, 0.0, 0.0, 0.0],
...                             [0.0, 0.4, 0.0, 0.0],
...                             [0.0, 0.0, 1.2, 0.0],
...                             [0.0, 0.0, 0.0,-0.4]]))
tensor([[ 0,  0],
        [ 1,  1],
        [ 2,  2],
        [ 3,  3]])
>>> torch.nonzero(torch.tensor([1, 1, 1, 0, 1]), as_tuple=True)
(tensor([0, 1, 2, 4]),)
>>> torch.nonzero(torch.tensor([[0.6, 0.0, 0.0, 0.0],
...                             [0.0, 0.4, 0.0, 0.0],
...                             [0.0, 0.0, 1.2, 0.0],
...                             [0.0, 0.0, 0.0,-0.4]]), as_tuple=True)
(tensor([0, 1, 2, 3]), tensor([0, 1, 2, 3]))
>>> torch.nonzero(torch.tensor(5), as_tuple=True)
(tensor([0]),)


torch.norm(input, p='fro', dim=None, keepdim=False, out=None, dtype=None) [source]

Returns the matrix norm or vector norm of a given tensor.  Warning torch.norm is deprecated and may be removed in a future PyTorch release. Use torch.linalg.norm() instead, but note that torch.linalg.norm() has a different signature and slightly different behavior that is more consistent with NumPy’s numpy.linalg.norm.   Parameters

input (Tensor) – The input tensor. Its data type must be either a floating point or complex type. For complex inputs, the norm is calculated using the absolute value of each element. If the input is complex and neither dtype nor out is specified, the result’s data type will be the corresponding floating point type (e.g. float if input is complexfloat).
p (int, float, inf, -inf, 'fro', 'nuc', optional) –
the order of norm. Default: 'fro' The following norms can be calculated:
ord matrix norm vector norm
’fro’ Frobenius norm –
‘nuc’ nuclear norm –
Number – sum(abs(x)**ord)**(1./ord)   The vector norm can be calculated across any number of dimensions. The corresponding dimensions of input are flattened into one dimension, and the norm is calculated on the flattened dimension. Frobenius norm produces the same result as p=2 in all cases except when dim is a list of three or more dims, in which case Frobenius norm throws an error. Nuclear norm can only be calculated across exactly two dimensions.
dim (int, tuple of python:ints, list of python:ints, optional) – Specifies which dimension or dimensions of input to calculate the norm across. If dim is None, the norm will be calculated across all dimensions of input. If the norm type indicated by p does not support the specified number of dimensions, an error will occur.
keepdim (bool, optional) – whether the output tensors have dim retained or not. Ignored if dim = None and out = None. Default: False

out (Tensor, optional) – the output tensor. Ignored if dim = None and out = None.
dtype (torch.dtype, optional) – the desired data type of returned tensor. If specified, the input tensor is casted to :attr:’dtype’ while performing the operation. Default: None.     Note Even though p='fro' supports any number of dimensions, the true mathematical definition of Frobenius norm only applies to tensors with exactly two dimensions. torch.linalg.norm() with ord='fro' aligns with the mathematical definition, since it can only be applied across exactly two dimensions.  Example: >>> import torch
>>> a = torch.arange(9, dtype= torch.float) - 4
>>> b = a.reshape((3, 3))
>>> torch.norm(a)
tensor(7.7460)
>>> torch.norm(b)
tensor(7.7460)
>>> torch.norm(a, float('inf'))
tensor(4.)
>>> torch.norm(b, float('inf'))
tensor(4.)
>>> c = torch.tensor([[ 1, 2, 3],[-1, 1, 4]] , dtype= torch.float)
>>> torch.norm(c, dim=0)
tensor([1.4142, 2.2361, 5.0000])
>>> torch.norm(c, dim=1)
tensor([3.7417, 4.2426])
>>> torch.norm(c, p=1, dim=1)
tensor([6., 6.])
>>> d = torch.arange(8, dtype= torch.float).reshape(2,2,2)
>>> torch.norm(d, dim=(1,2))
tensor([ 3.7417, 11.2250])
>>> torch.norm(d[0, :, :]), torch.norm(d[1, :, :])
(tensor(3.7417), tensor(11.2250))


torch.normal(mean, std, *, generator=None, out=None) → Tensor
Returns a tensor of random numbers drawn from separate normal distributions whose mean and standard deviation are given. The mean is a tensor with the mean of each output element’s normal distribution The std is a tensor with the standard deviation of each output element’s normal distribution The shapes of mean and std don’t need to match, but the total number of elements in each tensor need to be the same.  Note When the shapes do not match, the shape of mean is used as the shape for the returned output tensor   Parameters

mean (Tensor) – the tensor of per-element means
std (Tensor) – the tensor of per-element standard deviations   Keyword Arguments

generator (torch.Generator, optional) – a pseudorandom number generator for sampling
out (Tensor, optional) – the output tensor.    Example: >>> torch.normal(mean=torch.arange(1., 11.), std=torch.arange(1, 0, -0.1))
tensor([  1.0425,   3.5672,   2.7969,   4.2925,   4.7229,   6.2134,
          8.0505,   8.1408,   9.0563,  10.0566])

torch.normal(mean=0.0, std, *, out=None) → Tensor
 Similar to the function above, but the means are shared among all drawn elements.  Parameters

mean (float, optional) – the mean for all distributions
std (Tensor) – the tensor of per-element standard deviations   Keyword Arguments
out (Tensor, optional) – the output tensor.   Example: >>> torch.normal(mean=0.5, std=torch.arange(1., 6.))
tensor([-1.2793, -1.0732, -2.0687,  5.1177, -1.2303])

torch.normal(mean, std=1.0, *, out=None) → Tensor
 Similar to the function above, but the standard-deviations are shared among all drawn elements.  Parameters

mean (Tensor) – the tensor of per-element means
std (float, optional) – the standard deviation for all distributions   Keyword Arguments
out (Tensor, optional) – the output tensor   Example: >>> torch.normal(mean=torch.arange(1., 6.))
tensor([ 1.1552,  2.6148,  2.6535,  5.8318,  4.2361])

torch.normal(mean, std, size, *, out=None) → Tensor
 Similar to the function above, but the means and standard deviations are shared among all drawn elements. The resulting tensor has size given by size.  Parameters

mean (float) – the mean for all distributions
std (float) – the standard deviation for all distributions
size (int...) – a sequence of integers defining the shape of the output tensor.   Keyword Arguments
out (Tensor, optional) – the output tensor.   Example: >>> torch.normal(2, 3, size=(1, 4))
tensor([[-1.3987, -1.9544,  3.6048,  0.7909]])


torch.not_equal(input, other, *, out=None) → Tensor
Alias for torch.ne().


class torch.no_grad [source]

Context-manager that disabled gradient calculation. Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True. In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True. This context manager is thread local; it will not affect computation in other threads. Also functions as a decorator. (Make sure to instantiate with parenthesis.) Example: >>> x = torch.tensor([1], requires_grad=True)
>>> with torch.no_grad():
...   y = x * 2
>>> y.requires_grad
False
>>> @torch.no_grad()
... def doubler(x):
...     return x * 2
>>> z = doubler(x)
>>> z.requires_grad
False


torch.numel(input) → int
Returns the total number of elements in the input tensor.  Parameters
input (Tensor) – the input tensor.   Example: >>> a = torch.randn(1, 2, 3, 4, 5)
>>> torch.numel(a)
120
>>> a = torch.zeros(4,4)
>>> torch.numel(a)
16


torch.ones(*size, *, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) → Tensor
Returns a tensor filled with the scalar value 1, with the shape defined by the variable argument size.  Parameters
size (int...) – a sequence of integers defining the shape of the output tensor. Can be a variable number of arguments or a collection like a list or tuple.  Keyword Arguments

out (Tensor, optional) – the output tensor.
dtype (torch.dtype, optional) – the desired data type of returned tensor. Default: if None, uses a global default (see torch.set_default_tensor_type()).
layout (torch.layout, optional) – the desired layout of returned Tensor. Default: torch.strided.
device (torch.device, optional) – the desired device of returned tensor. Default: if None, uses the current device for the default tensor type (see torch.set_default_tensor_type()). device will be the CPU for CPU tensor types and the current CUDA device for CUDA tensor types.
requires_grad (bool, optional) – If autograd should record operations on the returned tensor. Default: False.    Example: >>> torch.ones(2, 3)
tensor([[ 1.,  1.,  1.],
        [ 1.,  1.,  1.]])

>>> torch.ones(5)
tensor([ 1.,  1.,  1.,  1.,  1.])


torch.ones_like(input, *, dtype=None, layout=None, device=None, requires_grad=False, memory_format=torch.preserve_format) → Tensor
Returns a tensor filled with the scalar value 1, with the same size as input. torch.ones_like(input) is equivalent to torch.ones(input.size(), dtype=input.dtype, layout=input.layout, device=input.device).  Warning As of 0.4, this function does not support an out keyword. As an alternative, the old torch.ones_like(input, out=output) is equivalent to torch.ones(input.size(), out=output).   Parameters
input (Tensor) – the size of input will determine size of the output tensor.  Keyword Arguments

dtype (torch.dtype, optional) – the desired data type of returned Tensor. Default: if None, defaults to the dtype of input.
layout (torch.layout, optional) – the desired layout of returned tensor. Default: if None, defaults to the layout of input.
device (torch.device, optional) – the desired device of returned tensor. Default: if None, defaults to the device of input.
requires_grad (bool, optional) – If autograd should record operations on the returned tensor. Default: False.
memory_format (torch.memory_format, optional) – the desired memory format of returned Tensor. Default: torch.preserve_format.    Example: >>> input = torch.empty(2, 3)
>>> torch.ones_like(input)
tensor([[ 1.,  1.,  1.],
        [ 1.,  1.,  1.]])