neulab/docprompting-conala huggingface dataset:
torch.normal(mean, std, *, generator=None, out=None) → Tensor  
Returns a tensor of random numbers drawn from separate normal distributions whose mean and standard deviation are given. The mean is a tensor with the mean of each output element’s normal distribution The std is a tensor with the standard deviation of each output element’s normal distribution The shapes of mean and std don’t need to match, but the total number of elements in each tensor need to be the same.  Note When the shapes do not match, the shape of mean is used as the shape for the returned output tensor   Parameters 
 
mean (Tensor) – the tensor of per-element means 
std (Tensor) – the tensor of per-element standard deviations   Keyword Arguments 
 
generator (torch.Generator, optional) – a pseudorandom number generator for sampling 
out (Tensor, optional) – the output tensor.    Example: >>> torch.normal(mean=torch.arange(1., 11.), std=torch.arange(1, 0, -0.1))
tensor([  1.0425,   3.5672,   2.7969,   4.2925,   4.7229,   6.2134,
          8.0505,   8.1408,   9.0563,  10.0566])
  
torch.normal(mean=0.0, std, *, out=None) → Tensor 
 Similar to the function above, but the means are shared among all drawn elements.  Parameters 
 
mean (float, optional) – the mean for all distributions 
std (Tensor) – the tensor of per-element standard deviations   Keyword Arguments 
out (Tensor, optional) – the output tensor.   Example: >>> torch.normal(mean=0.5, std=torch.arange(1., 6.))
tensor([-1.2793, -1.0732, -2.0687,  5.1177, -1.2303])
  
torch.normal(mean, std=1.0, *, out=None) → Tensor 
 Similar to the function above, but the standard-deviations are shared among all drawn elements.  Parameters 
 
mean (Tensor) – the tensor of per-element means 
std (float, optional) – the standard deviation for all distributions   Keyword Arguments 
out (Tensor, optional) – the output tensor   Example: >>> torch.normal(mean=torch.arange(1., 6.))
tensor([ 1.1552,  2.6148,  2.6535,  5.8318,  4.2361])
  
torch.normal(mean, std, size, *, out=None) → Tensor 
 Similar to the function above, but the means and standard deviations are shared among all drawn elements. The resulting tensor has size given by size.  Parameters 
 
mean (float) – the mean for all distributions 
std (float) – the standard deviation for all distributions 
size (int...) – a sequence of integers defining the shape of the output tensor.   Keyword Arguments 
out (Tensor, optional) – the output tensor.   Example: >>> torch.normal(2, 3, size=(1, 4))
tensor([[-1.3987, -1.9544,  3.6048,  0.7909]])
torch.not_equal(input, other, *, out=None) → Tensor  
Alias for torch.ne().
class torch.no_grad [source]
 
Context-manager that disabled gradient calculation. Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True. In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True. This context manager is thread local; it will not affect computation in other threads. Also functions as a decorator. (Make sure to instantiate with parenthesis.) Example: >>> x = torch.tensor([1], requires_grad=True)
>>> with torch.no_grad():
...   y = x * 2
>>> y.requires_grad
False
>>> @torch.no_grad()
... def doubler(x):
...     return x * 2
>>> z = doubler(x)
>>> z.requires_grad
False
torch.numel(input) → int  
Returns the total number of elements in the input tensor.  Parameters 
input (Tensor) – the input tensor.   Example: >>> a = torch.randn(1, 2, 3, 4, 5)
>>> torch.numel(a)
120
>>> a = torch.zeros(4,4)
>>> torch.numel(a)
16
torch.ones(*size, *, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) → Tensor  
Returns a tensor filled with the scalar value 1, with the shape defined by the variable argument size.  Parameters 
size (int...) – a sequence of integers defining the shape of the output tensor. Can be a variable number of arguments or a collection like a list or tuple.  Keyword Arguments 
 
out (Tensor, optional) – the output tensor. 
dtype (torch.dtype, optional) – the desired data type of returned tensor. Default: if None, uses a global default (see torch.set_default_tensor_type()). 
layout (torch.layout, optional) – the desired layout of returned Tensor. Default: torch.strided. 
device (torch.device, optional) – the desired device of returned tensor. Default: if None, uses the current device for the default tensor type (see torch.set_default_tensor_type()). device will be the CPU for CPU tensor types and the current CUDA device for CUDA tensor types. 
requires_grad (bool, optional) – If autograd should record operations on the returned tensor. Default: False.    Example: >>> torch.ones(2, 3)
tensor([[ 1.,  1.,  1.],
        [ 1.,  1.,  1.]])

>>> torch.ones(5)
tensor([ 1.,  1.,  1.,  1.,  1.])
torch.ones_like(input, *, dtype=None, layout=None, device=None, requires_grad=False, memory_format=torch.preserve_format) → Tensor  
Returns a tensor filled with the scalar value 1, with the same size as input. torch.ones_like(input) is equivalent to torch.ones(input.size(), dtype=input.dtype, layout=input.layout, device=input.device).  Warning As of 0.4, this function does not support an out keyword. As an alternative, the old torch.ones_like(input, out=output) is equivalent to torch.ones(input.size(), out=output).   Parameters 
input (Tensor) – the size of input will determine size of the output tensor.  Keyword Arguments 
 
dtype (torch.dtype, optional) – the desired data type of returned Tensor. Default: if None, defaults to the dtype of input. 
layout (torch.layout, optional) – the desired layout of returned tensor. Default: if None, defaults to the layout of input. 
device (torch.device, optional) – the desired device of returned tensor. Default: if None, defaults to the device of input. 
requires_grad (bool, optional) – If autograd should record operations on the returned tensor. Default: False. 
memory_format (torch.memory_format, optional) – the desired memory format of returned Tensor. Default: torch.preserve_format.    Example: >>> input = torch.empty(2, 3)
>>> torch.ones_like(input)
tensor([[ 1.,  1.,  1.],
        [ 1.,  1.,  1.]])
torch.onnx.export_to_pretty_string(*args, **kwargs) [source]
torch.onnx.is_in_onnx_export() [source]
 
Check whether it’s in the middle of the ONNX export. This function returns True in the middle of torch.onnx.export(). torch.onnx.export should be executed with single thread.
torch.onnx.register_custom_op_symbolic(symbolic_name, symbolic_fn, opset_version) [source]
torch.onnx.select_model_mode_for_export(model, mode) [source]
 
A context manager to temporarily set the training mode of ‘model’ to ‘mode’, resetting it when we exit the with-block. A no-op if mode is None. In version 1.6 changed to this from set_training
class torch.optim.Adadelta(params, lr=1.0, rho=0.9, eps=1e-06, weight_decay=0) [source]
 
Implements Adadelta algorithm. It has been proposed in ADADELTA: An Adaptive Learning Rate Method.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
rho (float, optional) – coefficient used for computing a running average of squared gradients (default: 0.9) 
eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-6) 
lr (float, optional) – coefficient that scale delta before it is applied to the parameters (default: 1.0) 
weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)     
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
class torch.optim.Adagrad(params, lr=0.01, lr_decay=0, weight_decay=0, initial_accumulator_value=0, eps=1e-10) [source]
 
Implements Adagrad algorithm. It has been proposed in Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
lr (float, optional) – learning rate (default: 1e-2) 
lr_decay (float, optional) – learning rate decay (default: 0) 
weight_decay (float, optional) – weight decay (L2 penalty) (default: 0) 
eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-10)     
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
class torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False) [source]
 
Implements Adam algorithm. It has been proposed in Adam: A Method for Stochastic Optimization. The implementation of the L2 penalty follows changes proposed in Decoupled Weight Decay Regularization.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
lr (float, optional) – learning rate (default: 1e-3) 
betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) 
eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8) 
weight_decay (float, optional) – weight decay (L2 penalty) (default: 0) 
amsgrad (boolean, optional) – whether to use the AMSGrad variant of this algorithm from the paper On the Convergence of Adam and Beyond (default: False)     
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
class torch.optim.Adamax(params, lr=0.002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0) [source]
 
Implements Adamax algorithm (a variant of Adam based on infinity norm). It has been proposed in Adam: A Method for Stochastic Optimization.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
lr (float, optional) – learning rate (default: 2e-3) 
betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square 
eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8) 
weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)     
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
class torch.optim.AdamW(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01, amsgrad=False) [source]
 
Implements AdamW algorithm. The original Adam algorithm was proposed in Adam: A Method for Stochastic Optimization. The AdamW variant was proposed in Decoupled Weight Decay Regularization.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
lr (float, optional) – learning rate (default: 1e-3) 
betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) 
eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8) 
weight_decay (float, optional) – weight decay coefficient (default: 1e-2) 
amsgrad (boolean, optional) – whether to use the AMSGrad variant of this algorithm from the paper On the Convergence of Adam and Beyond (default: False)     
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
class torch.optim.ASGD(params, lr=0.01, lambd=0.0001, alpha=0.75, t0=1000000.0, weight_decay=0) [source]
 
Implements Averaged Stochastic Gradient Descent. It has been proposed in Acceleration of stochastic approximation by averaging.  Parameters 
 
params (iterable) – iterable of parameters to optimize or dicts defining parameter groups 
lr (float, optional) – learning rate (default: 1e-2) 
lambd (float, optional) – decay term (default: 1e-4) 
alpha (float, optional) – power for eta update (default: 0.75) 
t0 (float, optional) – point at which to start averaging (default: 1e6) 
weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)     
step(closure=None) [source]
 
Performs a single optimization step.  Parameters 
closure (callable, optional) – A closure that reevaluates the model and returns the loss.
class torch.optim.LBFGS(params, lr=1, max_iter=20, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn=None) [source]
 
Implements L-BFGS algorithm, heavily inspired by minFunc <https://www.cs.ubc.ca/~schmidtm/Software/minFunc.html>.  Warning This optimizer doesn’t support per-parameter options and parameter groups (there can be only one).   Warning Right now all parameters have to be on a single device. This will be improved in the future.   Note This is a very memory intensive optimizer (it requires additional param_bytes * (history_size + 1) bytes). If it doesn’t fit in memory try reducing the history size, or use a different algorithm.   Parameters 
 
lr (float) – learning rate (default: 1) 
max_iter (int) – maximal number of iterations per optimization step (default: 20) 
max_eval (int) – maximal number of function evaluations per optimization step (default: max_iter * 1.25). 
tolerance_grad (float) – termination tolerance on first order optimality (default: 1e-5). 
tolerance_change (float) – termination tolerance on function value/parameter changes (default: 1e-9). 
history_size (int) – update history size (default: 100). 
line_search_fn (str) – either ‘strong_wolfe’ or None (default: None).     
step(closure) [source]
 
Performs a single optimization step.  Parameters 
closure (callable) – A closure that reevaluates the model and returns the loss.
step(closure) [source]
 
Performs a single optimization step.  Parameters 
closure (callable) – A closure that reevaluates the model and returns the loss.
class torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max, eta_min=0, last_epoch=-1, verbose=False) [source]
 
Set the learning rate of each parameter group using a cosine annealing schedule, where ηmax\eta_{max}  is set to the initial lr and TcurT_{cur}  is the number of epochs since the last restart in SGDR:  ηt=ηmin+12(ηmax−ηmin)(1+cos⁡(TcurTmaxπ)),Tcur≠(2k+1)Tmax;ηt+1=ηt+12(ηmax−ηmin)(1−cos⁡(1Tmaxπ)),Tcur=(2k+1)Tmax.\begin{aligned} \eta_t & = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right), & T_{cur} \neq (2k+1)T_{max}; \\ \eta_{t+1} & = \eta_{t} + \frac{1}{2}(\eta_{max} - \eta_{min}) \left(1 - \cos\left(\frac{1}{T_{max}}\pi\right)\right), & T_{cur} = (2k+1)T_{max}. \end{aligned}  
When last_epoch=-1, sets initial lr as lr. Notice that because the schedule is defined recursively, the learning rate can be simultaneously modified outside this scheduler by other operators. If the learning rate is set solely by this scheduler, the learning rate at each step becomes:  ηt=ηmin+12(ηmax−ηmin)(1+cos⁡(TcurTmaxπ))\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right)  
It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts. Note that this only implements the cosine annealing part of SGDR, and not the restarts.  Parameters 
 
optimizer (Optimizer) – Wrapped optimizer. 
T_max (int) – Maximum number of iterations. 
eta_min (float) – Minimum learning rate. Default: 0. 
last_epoch (int) – The index of last epoch. Default: -1. 
verbose (bool) – If True, prints a message to stdout for each update. Default: False.
class torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0, T_mult=1, eta_min=0, last_epoch=-1, verbose=False) [source]
 
Set the learning rate of each parameter group using a cosine annealing schedule, where ηmax\eta_{max}  is set to the initial lr, TcurT_{cur}  is the number of epochs since the last restart and TiT_{i}  is the number of epochs between two warm restarts in SGDR:  ηt=ηmin+12(ηmax−ηmin)(1+cos⁡(TcurTiπ))\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{i}}\pi\right)\right)  
When Tcur=TiT_{cur}=T_{i} , set ηt=ηmin\eta_t = \eta_{min} . When Tcur=0T_{cur}=0  after restart, set ηt=ηmax\eta_t=\eta_{max} . It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts.  Parameters 
 
optimizer (Optimizer) – Wrapped optimizer. 
T_0 (int) – Number of iterations for the first restart. 
T_mult (int, optional) – A factor increases TiT_{i}  after a restart. Default: 1. 
eta_min (float, optional) – Minimum learning rate. Default: 0. 
last_epoch (int, optional) – The index of last epoch. Default: -1. 
verbose (bool) – If True, prints a message to stdout for each update. Default: False.     
step(epoch=None) [source]
 
Step could be called after every batch update Example >>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult)
>>> iters = len(dataloader)
>>> for epoch in range(20):
>>>     for i, sample in enumerate(dataloader):
>>>         inputs, labels = sample['inputs'], sample['labels']
>>>         optimizer.zero_grad()
>>>         outputs = net(inputs)
>>>         loss = criterion(outputs, labels)
>>>         loss.backward()
>>>         optimizer.step()
>>>         scheduler.step(epoch + i / iters)
 This function can be called in an interleaved way. Example >>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult)
>>> for epoch in range(20):
>>>     scheduler.step()
>>> scheduler.step(26)
>>> scheduler.step() # scheduler.step(27), instead of scheduler(20)
step(epoch=None) [source]
 
Step could be called after every batch update Example >>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult)
>>> iters = len(dataloader)
>>> for epoch in range(20):
>>>     for i, sample in enumerate(dataloader):
>>>         inputs, labels = sample['inputs'], sample['labels']
>>>         optimizer.zero_grad()
>>>         outputs = net(inputs)
>>>         loss = criterion(outputs, labels)
>>>         loss.backward()
>>>         optimizer.step()
>>>         scheduler.step(epoch + i / iters)
 This function can be called in an interleaved way. Example >>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult)
>>> for epoch in range(20):
>>>     scheduler.step()
>>> scheduler.step(26)
>>> scheduler.step() # scheduler.step(27), instead of scheduler(20)
get_lr() [source]
 
Calculates the learning rate at batch index. This function treats self.last_epoch as the last batch index. If self.cycle_momentum is True, this function has a side effect of updating the optimizer’s momentum.
class torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma, last_epoch=-1, verbose=False) [source]
 
Decays the learning rate of each parameter group by gamma every epoch. When last_epoch=-1, sets initial lr as lr.  Parameters 
 
optimizer (Optimizer) – Wrapped optimizer. 
gamma (float) – Multiplicative factor of learning rate decay. 
last_epoch (int) – The index of last epoch. Default: -1. 
verbose (bool) – If True, prints a message to stdout for each update. Default: False.
class torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda, last_epoch=-1, verbose=False) [source]
 
Sets the learning rate of each parameter group to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr.  Parameters 
 
optimizer (Optimizer) – Wrapped optimizer. 
lr_lambda (function or list) – A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups. 
last_epoch (int) – The index of last epoch. Default: -1. 
verbose (bool) – If True, prints a message to stdout for each update. Default: False.    Example >>> # Assuming optimizer has two groups.
>>> lambda1 = lambda epoch: epoch // 30
>>> lambda2 = lambda epoch: 0.95 ** epoch
>>> scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
>>> for epoch in range(100):
>>>     train(...)
>>>     validate(...)
>>>     scheduler.step()
  
load_state_dict(state_dict) [source]
 
Loads the schedulers state. When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.  Parameters 
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().   
  
state_dict() [source]
 
Returns the state of the scheduler as a dict. It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas. When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.
load_state_dict(state_dict) [source]
 
Loads the schedulers state. When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.  Parameters 
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().
state_dict() [source]
 
Returns the state of the scheduler as a dict. It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas. When saving or loading the scheduler, please make sure to also save or load the state of the optimizer.
class torch.optim.lr_scheduler.MultiplicativeLR(optimizer, lr_lambda, last_epoch=-1, verbose=False) [source]
 
Multiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, sets initial lr as lr.  Parameters 
 
optimizer (Optimizer) – Wrapped optimizer. 
lr_lambda (function or list) – A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups. 
last_epoch (int) – The index of last epoch. Default: -1. 
verbose (bool) – If True, prints a message to stdout for each update. Default: False.    Example >>> lmbda = lambda epoch: 0.95
>>> scheduler = MultiplicativeLR(optimizer, lr_lambda=lmbda)
>>> for epoch in range(100):
>>>     train(...)
>>>     validate(...)
>>>     scheduler.step()
  
load_state_dict(state_dict) [source]
 
Loads the schedulers state.  Parameters 
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().   
  
state_dict() [source]
 
Returns the state of the scheduler as a dict. It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas.
load_state_dict(state_dict) [source]
 
Loads the schedulers state.  Parameters 
state_dict (dict) – scheduler state. Should be an object returned from a call to state_dict().
state_dict() [source]
 
Returns the state of the scheduler as a dict. It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas.