neulab/docprompting-conala huggingface dataset:
class torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones, gamma=0.1, last_epoch=-1, verbose=False) [source]
 
Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.  Parameters 
 
optimizer (Optimizer) – Wrapped optimizer. 
milestones (list) – List of epoch indices. Must be increasing. 
gamma (float) – Multiplicative factor of learning rate decay. Default: 0.1. 
last_epoch (int) – The index of last epoch. Default: -1. 
verbose (bool) – If True, prints a message to stdout for each update. Default: False.    Example >>> # Assuming optimizer uses lr = 0.05 for all groups
>>> # lr = 0.05     if epoch < 30
>>> # lr = 0.005    if 30 <= epoch < 80
>>> # lr = 0.0005   if epoch >= 80
>>> scheduler = MultiStepLR(optimizer, milestones=[30,80], gamma=0.1)
>>> for epoch in range(100):
>>>     train(...)
>>>     validate(...)
>>>     scheduler.step()
class torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08, verbose=False) [source]
 
Reduce learning rate when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This scheduler reads a metrics quantity and if no improvement is seen for a ‘patience’ number of epochs, the learning rate is reduced.  Parameters 
 
optimizer (Optimizer) – Wrapped optimizer. 
mode (str) – One of min, max. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; in max mode it will be reduced when the quantity monitored has stopped increasing. Default: ‘min’. 
factor (float) – Factor by which the learning rate will be reduced. new_lr = lr * factor. Default: 0.1. 
patience (int) – Number of epochs with no improvement after which learning rate will be reduced. For example, if patience = 2, then we will ignore the first 2 epochs with no improvement, and will only decrease the LR after the 3rd epoch if the loss still hasn’t improved then. Default: 10. 
threshold (float) – Threshold for measuring the new optimum, to only focus on significant changes. Default: 1e-4. 
threshold_mode (str) – One of rel, abs. In rel mode, dynamic_threshold = best * ( 1 + threshold ) in ‘max’ mode or best * ( 1 - threshold ) in min mode. In abs mode, dynamic_threshold = best + threshold in max mode or best - threshold in min mode. Default: ‘rel’. 
cooldown (int) – Number of epochs to wait before resuming normal operation after lr has been reduced. Default: 0. 
min_lr (float or list) – A scalar or a list of scalars. A lower bound on the learning rate of all param groups or each group respectively. Default: 0. 
eps (float) – Minimal decay applied to lr. If the difference between new and old lr is smaller than eps, the update is ignored. Default: 1e-8. 
verbose (bool) – If True, prints a message to stdout for each update. Default: False.    Example >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
>>> scheduler = ReduceLROnPlateau(optimizer, 'min')
>>> for epoch in range(10):
>>>     train(...)
>>>     val_loss = validate(...)
>>>     # Note that step should be called after validate()
>>>     scheduler.step(val_loss)
class torch.optim.lr_scheduler.StepLR(optimizer, step_size, gamma=0.1, last_epoch=-1, verbose=False) [source]
 
Decays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.  Parameters 
 
optimizer (Optimizer) – Wrapped optimizer. 
step_size (int) – Period of learning rate decay. 
gamma (float) – Multiplicative factor of learning rate decay. Default: 0.1. 
last_epoch (int) – The index of last epoch. Default: -1. 
verbose (bool) – If True, prints a message to stdout for each update. Default: False.    Example >>> # Assuming optimizer uses lr = 0.05 for all groups
>>> # lr = 0.05     if epoch < 30
>>> # lr = 0.005    if 30 <= epoch < 60
>>> # lr = 0.0005   if 60 <= epoch < 90
>>> # ...
>>> scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
>>> for epoch in range(100):
>>>     train(...)
>>>     validate(...)
>>>     scheduler.step()
class torch.optim.Optimizer(params, defaults) [source]
 
Base class for all optimizers.  Warning Parameters need to be specified as collections that have a deterministic ordering that is consistent between runs. Examples of objects that don’t satisfy those properties are sets and iterators over values of dictionaries.   Parameters 
 
params (iterable) – an iterable of torch.Tensor s or dict s. Specifies what Tensors should be optimized. 
defaults – (dict): a dict containing default values of optimization options (used when a parameter group doesn’t specify them).     
add_param_group(param_group) [source]
 
Add a param group to the Optimizer s param_groups. This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the Optimizer as training progresses.  Parameters 
 
param_group (dict) – Specifies what Tensors should be optimized along with group 
optimization options. (specific) –     
  
load_state_dict(state_dict) [source]
 
Loads the optimizer state.  Parameters 
state_dict (dict) – optimizer state. Should be an object returned from a call to state_dict().   
  
state_dict() [source]
 
Returns the state of the optimizer as a dict. It contains two entries:  
 state - a dict holding current optimization state. Its content

differs between optimizer classes.    param_groups - a dict containing all parameter groups  
  
step(closure) [source]
 
Performs a single optimization step (parameter update).  Parameters 
closure (callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.    Note Unless otherwise specified, this function should not modify the .grad field of the parameters.  
  
zero_grad(set_to_none=False) [source]
 
Sets the gradients of all optimized torch.Tensor s to zero.  Parameters 
set_to_none (bool) – instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests zero_grad(set_to_none=True) followed by a backward pass, .grads are guaranteed to be None for params that did not receive a gradient. 3. torch.optim optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).
add_param_group(param_group) [source]
 
Add a param group to the Optimizer s param_groups. This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the Optimizer as training progresses.  Parameters 
 
param_group (dict) – Specifies what Tensors should be optimized along with group 
optimization options. (specific) –
load_state_dict(state_dict) [source]
 
Loads the optimizer state.  Parameters 
state_dict (dict) – optimizer state. Should be an object returned from a call to state_dict().
state_dict() [source]
 
Returns the state of the optimizer as a dict. It contains two entries:  
 state - a dict holding current optimization state. Its content

differs between optimizer classes.    param_groups - a dict containing all parameter groups
step(closure) [source]
 
Performs a single optimization step (parameter update).  Parameters 
closure (callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.    Note Unless otherwise specified, this function should not modify the .grad field of the parameters.
zero_grad(set_to_none=False) [source]
 
Sets the gradients of all optimized torch.Tensor s to zero.  Parameters 
set_to_none (bool) – instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests zero_grad(set_to_none=True) followed by a backward pass, .grads are guaranteed to be None for params that did not receive a gradient. 3. torch.optim optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).
class torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, centered=False) [source]
 
Implements RMSprop algorithm. Proposed by G. Hinton in his course. The centered version first appears in Generating Sequences With Recurrent Neural Networks. The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus α/(v+ϵ)\alpha/(\sqrt{v} + \epsilon)  where α\alpha  is the scheduled learning rate and vv  is the weighted moving average of the squared gradient.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
lr (float, optional) – learning rate (default: 1e-2) 
momentum (float, optional) – momentum factor (default: 0) 
alpha (float, optional) – smoothing constant (default: 0.99) 
eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8) 
centered (bool, optional) – if True, compute the centered RMSProp, the gradient is normalized by an estimation of its variance 
weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)     
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
class torch.optim.Rprop(params, lr=0.01, etas=(0.5, 1.2), step_sizes=(1e-06, 50)) [source]
 
Implements the resilient backpropagation algorithm.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
lr (float, optional) – learning rate (default: 1e-2) 
etas (Tuple[float, float], optional) – pair of (etaminus, etaplis), that are multiplicative increase and decrease factors (default: (0.5, 1.2)) 
step_sizes (Tuple[float, float], optional) – a pair of minimal and maximal allowed step sizes (default: (1e-6, 50))     
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
class torch.optim.SGD(params, lr=<required parameter>, momentum=0, dampening=0, weight_decay=0, nesterov=False) [source]
 
Implements stochastic gradient descent (optionally with momentum). Nesterov momentum is based on the formula from On the importance of initialization and momentum in deep learning.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
lr (float) – learning rate 
momentum (float, optional) – momentum factor (default: 0) 
weight_decay (float, optional) – weight decay (L2 penalty) (default: 0) 
dampening (float, optional) – dampening for momentum (default: 0) 
nesterov (bool, optional) – enables Nesterov momentum (default: False)    Example >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
>>> optimizer.zero_grad()
>>> loss_fn(model(input), target).backward()
>>> optimizer.step()
  Note The implementation of SGD with Momentum/Nesterov subtly differs from Sutskever et. al. and implementations in some other frameworks. Considering the specific case of Momentum, the update can be written as  vt+1=μ∗vt+gt+1,pt+1=pt−lr∗vt+1,\begin{aligned} v_{t+1} & = \mu * v_{t} + g_{t+1}, \\ p_{t+1} & = p_{t} - \text{lr} * v_{t+1}, \end{aligned}  
where pp , gg , vv  and μ\mu  denote the parameters, gradient, velocity, and momentum respectively. This is in contrast to Sutskever et. al. and other frameworks which employ an update of the form  vt+1=μ∗vt+lr∗gt+1,pt+1=pt−vt+1.\begin{aligned} v_{t+1} & = \mu * v_{t} + \text{lr} * g_{t+1}, \\ p_{t+1} & = p_{t} - v_{t+1}. \end{aligned}  
The Nesterov version is analogously modified.   
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
class torch.optim.SparseAdam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08) [source]
 
Implements lazy version of Adam algorithm suitable for sparse tensors. In this variant, only moments that show up in the gradient get updated, and only those portions of the gradient get applied to the parameters.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
lr (float, optional) – learning rate (default: 1e-3) 
betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) 
eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)     
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
torch.orgqr(input, input2) → Tensor  
Computes the orthogonal matrix Q of a QR factorization, from the (input, input2) tuple returned by torch.geqrf(). This directly calls the underlying LAPACK function ?orgqr. See LAPACK documentation for orgqr for further details.  Parameters 
 
input (Tensor) – the a from torch.geqrf(). 
input2 (Tensor) – the tau from torch.geqrf().
torch.ormqr(input, input2, input3, left=True, transpose=False) → Tensor  
Multiplies mat (given by input3) by the orthogonal Q matrix of the QR factorization formed by torch.geqrf() that is represented by (a, tau) (given by (input, input2)). This directly calls the underlying LAPACK function ?ormqr. See LAPACK documentation for ormqr for further details.  Parameters 
 
input (Tensor) – the a from torch.geqrf(). 
input2 (Tensor) – the tau from torch.geqrf(). 
input3 (Tensor) – the matrix to be multiplied.
torch.outer(input, vec2, *, out=None) → Tensor  
Outer product of input and vec2. If input is a vector of size nn  and vec2 is a vector of size mm , then out must be a matrix of size (n×m)(n \times m) .  Note This function does not broadcast.   Parameters 
 
input (Tensor) – 1-D input vector 
vec2 (Tensor) – 1-D input vector   Keyword Arguments 
out (Tensor, optional) – optional output matrix   Example: >>> v1 = torch.arange(1., 5.)
>>> v2 = torch.arange(1., 4.)
>>> torch.outer(v1, v2)
tensor([[  1.,   2.,   3.],
        [  2.,   4.,   6.],
        [  3.,   6.,   9.],
        [  4.,   8.,  12.]])
torch.overrides.get_ignored_functions() [source]
 
Return public functions that cannot be overridden by __torch_function__.  Returns 
A tuple of functions that are publicly available in the torch API but cannot be overridden with __torch_function__. Mostly this is because none of the arguments of these functions are tensors or tensor-likes.  Return type 
Set[Callable]   Examples >>> torch.Tensor.as_subclass in torch.overrides.get_ignored_functions()
True
>>> torch.add in torch.overrides.get_ignored_functions()
False
torch.overrides.get_overridable_functions() [source]
 
List functions that are overridable via __torch_function__  Returns 
A dictionary that maps namespaces that contain overridable functions to functions in that namespace that can be overridden.  Return type 
Dict[Any, List[Callable]]
torch.overrides.get_testing_overrides() [source]
 
Return a dict containing dummy overrides for all overridable functions  Returns 
A dictionary that maps overridable functions in the PyTorch API to lambda functions that have the same signature as the real function and unconditionally return -1. These lambda functions are useful for testing API coverage for a type that defines __torch_function__.  Return type 
Dict[Callable, Callable]   Examples >>> import inspect
>>> my_add = torch.overrides.get_testing_overrides()[torch.add]
>>> inspect.signature(my_add)
<Signature (input, other, out=None)>
torch.overrides.handle_torch_function(public_api, relevant_args, *args, **kwargs) [source]
 
Implement a function with checks for __torch_function__ overrides. See torch::autograd::handle_torch_function for the equivalent of this function in the C++ implementation.  Parameters 
 
public_api (function) – Function exposed by the public torch API originally called like public_api(*args, **kwargs) on which arguments are now being checked. 
relevant_args (iterable) – Iterable of arguments to check for __torch_function__ methods. 
args (tuple) – Arbitrary positional arguments originally passed into public_api. 
kwargs (tuple) – Arbitrary keyword arguments originally passed into public_api.   Returns 
Result from calling implementation or an __torch_function__ method, as appropriate.  Return type 
object   :raises TypeError : if no implementation is found.: Example >>> def func(a):
...     if type(a) is not torch.Tensor:  # This will make func dispatchable by __torch_function__
...         return handle_torch_function(func, (a,), a)
...     return a + 0
torch.overrides.has_torch_function()  
Check for __torch_function__ implementations in the elements of an iterable. Considers exact Tensor s and Parameter s non-dispatchable. :param relevant_args: Iterable or aguments to check for __torch_function__ methods. :type relevant_args: iterable  Returns 
True if any of the elements of relevant_args have __torch_function__ implementations, False otherwise.  Return type 
bool    See also  
torch.is_tensor_like() 

Checks if something is a Tensor-like, including an exact Tensor.
torch.overrides.is_tensor_like(inp) [source]
 
Returns True if the passed-in input is a Tensor-like. Currently, this occurs whenever there’s a __torch_function__ attribute on the type of the input. Examples A subclass of tensor is generally a Tensor-like. >>> class SubTensor(torch.Tensor): ...
>>> is_tensor_like(SubTensor([0]))
True
 Built-in or user types aren’t usually Tensor-like. >>> is_tensor_like(6)
False
>>> is_tensor_like(None)
False
>>> class NotATensor: ...
>>> is_tensor_like(NotATensor())
False
 But, they can be made Tensor-like by implementing __torch_function__. >>> class TensorLike:
...     def __torch_function__(self, func, types, args, kwargs):
...         return -1
>>> is_tensor_like(TensorLike())
True
torch.overrides.is_tensor_method_or_property(func) [source]
 
Returns True if the function passed in is a handler for a method or property belonging to torch.Tensor, as passed into __torch_function__.  Note For properties, their __get__ method must be passed in.  This may be needed, in particular, for the following reasons:  Methods/properties sometimes don’t contain a __module__ slot. They require that the first passed-in argument is an instance of torch.Tensor.  Examples >>> is_tensor_method_or_property(torch.Tensor.add)
True
>>> is_tensor_method_or_property(torch.add)
False
torch.overrides.wrap_torch_function(dispatcher) [source]
 
Wraps a given function with __torch_function__ -related functionality.  Parameters 
dispatcher (Callable) – A callable that returns an iterable of Tensor-likes passed into the function.    Note This decorator may reduce the performance of your code. Generally, it’s enough to express your code as a series of functions that, themselves, support __torch_function__. If you find yourself in the rare situation where this is not the case, e.g. if you’re wrapping a low-level library and you also need it to work for Tensor-likes, then this function is available.  Examples >>> def dispatcher(a): # Must have the same signature as func
...     return (a,)
>>> @torch.overrides.wrap_torch_function(dispatcher)
>>> def func(a): # This will make func dispatchable by __torch_function__
...     return a + 0
torch.pca_lowrank(A, q=None, center=True, niter=2) [source]
 
Performs linear Principal Component Analysis (PCA) on a low-rank matrix, batches of such matrices, or sparse matrix. This function returns a namedtuple (U, S, V) which is the nearly optimal approximation of a singular value decomposition of a centered matrix AA  such that A=Udiag(S)VTA = U diag(S) V^T .  Note The relation of (U, S, V) to PCA is as follows:  
AA  is a data matrix with m samples and n features the VV  columns represent the principal directions 
S∗∗2/(m−1)S ** 2 / (m - 1)  contains the eigenvalues of ATA/(m−1)A^T A / (m - 1)  which is the covariance of A when center=True is provided. 
matmul(A, V[:, :k]) projects data to the first k principal components    Note Different from the standard SVD, the size of returned matrices depend on the specified rank and q values as follows:  
UU  is m x q matrix 
SS  is q-vector 
VV  is n x q matrix    Note To obtain repeatable results, reset the seed for the pseudorandom number generator   Parameters 
 
A (Tensor) – the input tensor of size (∗,m,n)(*, m, n) 
 
q (int, optional) – a slightly overestimated rank of AA . By default, q = min(6, m,
n). 
center (bool, optional) – if True, center the input tensor, otherwise, assume that the input is centered. 
niter (int, optional) – the number of subspace iterations to conduct; niter must be a nonnegative integer, and defaults to 2.    References: - Nathan Halko, Per-Gunnar Martinsson, and Joel Tropp, Finding
  structure with randomness: probabilistic algorithms for
  constructing approximate matrix decompositions,
  arXiv:0909.4061 [math.NA; math.PR], 2009 (available at
  `arXiv <http://arxiv.org/abs/0909.4061>`_).
torch.pinverse(input, rcond=1e-15) → Tensor  
Calculates the pseudo-inverse (also known as the Moore-Penrose inverse) of a 2D tensor. Please look at Moore-Penrose inverse for more details  Note torch.pinverse() is deprecated. Please use torch.linalg.pinv() instead which includes new parameters hermitian and out.   Note This method is implemented using the Singular Value Decomposition.   Note The pseudo-inverse is not necessarily a continuous function in the elements of the matrix [1]. Therefore, derivatives are not always existent, and exist for a constant rank only [2]. However, this method is backprop-able due to the implementation by using SVD results, and could be unstable. Double-backward will also be unstable due to the usage of SVD internally. See svd() for more details.   Note Supports real and complex inputs. Batched version for complex inputs is only supported on the CPU.   Parameters 
 
input (Tensor) – The input tensor of size (∗,m,n)(*, m, n)  where ∗*  is zero or more batch dimensions. 
rcond (float, optional) – A floating point value to determine the cutoff for small singular values. Default: 1e-15.   Returns 
The pseudo-inverse of input of dimensions (∗,n,m)(*, n, m)    Example: >>> input = torch.randn(3, 5)
>>> input
tensor([[ 0.5495,  0.0979, -1.4092, -0.1128,  0.4132],
        [-1.1143, -0.3662,  0.3042,  1.6374, -0.9294],
        [-0.3269, -0.5745, -0.0382, -0.5922, -0.6759]])
>>> torch.pinverse(input)
tensor([[ 0.0600, -0.1933, -0.2090],
        [-0.0903, -0.0817, -0.4752],
        [-0.7124, -0.1631, -0.2272],
        [ 0.1356,  0.3933, -0.5023],
        [-0.0308, -0.1725, -0.5216]])
>>> # Batched pinverse example
>>> a = torch.randn(2,6,3)
>>> b = torch.pinverse(a)
>>> torch.matmul(b, a)
tensor([[[ 1.0000e+00,  1.6391e-07, -1.1548e-07],
        [ 8.3121e-08,  1.0000e+00, -2.7567e-07],
        [ 3.5390e-08,  1.4901e-08,  1.0000e+00]],

        [[ 1.0000e+00, -8.9407e-08,  2.9802e-08],
        [-2.2352e-07,  1.0000e+00,  1.1921e-07],
        [ 0.0000e+00,  8.9407e-08,  1.0000e+00]]])