Returns True if the global deterministic flag is turned on. Refer to torch.use_deterministic_algorithms() documentation for more details.


Returns the indices of the maximum value of all elements in the input tensor.
This is the second value returned by torch.max(). See its documentation for the exact semantics of this method.
Note
If there are multiple minimal values then the indices of the first minimal value are returned.
input (Tensor) – the input tensor.
Example:
Returns the indices of the maximum values of a tensor across a dimension.
This is the second value returned by torch.max(). See its documentation for the exact semantics of this method.
Example:


Returns the indices of the minimum value(s) of the flattened tensor or along a dimension
This is the second value returned by torch.min(). See its documentation for the exact semantics of this method.
Note
If there are multiple minimal values then the indices of the first minimal value are returned.
Example:


Returns the indices that sort a tensor along a given dimension in ascending order by value.
This is the second value returned by torch.sort(). See its documentation for the exact semantics of this method.
Example:


Returns a new tensor with the arcsine of the elements of input.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.
Example:


Returns a new tensor with the inverse hyperbolic sine of the elements of input.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.
Example:


Create a view of an existing torch.Tensor input with specified size, stride and storage_offset.
Warning
More than one element of a created tensor may refer to a single memory location. As a result, in-place operations (especially ones that are vectorized) may result in incorrect behavior. If you need to write to the tensors, please clone them first.
Many PyTorch functions, which return a view of a tensor, are internally implemented with this function. Those functions, like torch.Tensor.expand(), are easier to read and are therefore more advisable to use.
Example:


Convert the data into a torch.Tensor. If the data is already a Tensor with the same dtype and device, no copy will be performed, otherwise a new Tensor will be returned with computational graph retained if data Tensor has requires_grad=True. Similarly, if the data is an ndarray of the corresponding dtype and the device is the cpu, no copy will be performed.
Example:


Returns a new tensor with the arctangent of the elements of input.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.
Example:


Element-wise arctangent of inputi/otheri\text{input}_{i} / \text{other}_{i}  with consideration of the quadrant. Returns a new tensor with the signed angles in radians between vector (otheri,inputi)(\text{other}_{i}, \text{input}_{i})  and vector (1,0)(1, 0) . (Note that otheri\text{other}_{i} , the second parameter, is the x-coordinate, while inputi\text{input}_{i} , the first parameter, is the y-coordinate.)
The shapes of input and other must be broadcastable.
out (Tensor, optional) – the output tensor.
Example:


Returns a new tensor with the inverse hyperbolic tangent of the elements of input.
Note
The domain of the inverse hyperbolic tangent is (-1, 1) and values outside this range will be mapped to NaN, except for the values 1 and -1 for which the output is mapped to +/-INF respectively.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.
Example:


Returns a 1-dimensional view of each input tensor with zero dimensions. Input tensors with one or more dimensions are returned as-is.
input (Tensor or list of Tensors) –
output (Tensor or tuple of Tensors)


Returns a 2-dimensional view of each input tensor with zero dimensions. Input tensors with two or more dimensions are returned as-is. :param input: :type input: Tensor or list of Tensors
output (Tensor or tuple of Tensors)


Returns a 3-dimensional view of each input tensor with zero dimensions. Input tensors with three or more dimensions are returned as-is. :param input: :type input: Tensor or list of Tensors
output (Tensor or tuple of Tensors)


torch.autograd provides classes and functions implementing automatic differentiation of arbitrary scalar valued functions. It requires minimal changes to the existing code - you only need to declare Tensor s for which gradients should be computed with the requires_grad=True keyword. As of now, we only support autograd for floating point Tensor types ( half, float, double and bfloat16) and complex Tensor types (cfloat, cdouble).
Computes the sum of gradients of given tensors w.r.t. graph leaves.
The graph is differentiated using the chain rule. If any of tensors are non-scalar (i.e. their data has more than one element) and require gradient, then the Jacobian-vector product would be computed, in this case the function additionally requires specifying grad_tensors. It should be a sequence of matching length, that contains the “vector” in the Jacobian-vector product, usually the gradient of the differentiated function w.r.t. corresponding tensors (None is an acceptable value for all tensors that don’t need gradient tensors).
This function accumulates gradients in the leaves - you might need to zero .grad attributes or set them to None before calling it. See Default gradient layouts for details on the memory layout of accumulated gradients.
Note
Using this method with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak.
Note
If you run any forward ops, create grad_tensors, and/or call backward in a user-specified CUDA stream context, see Stream semantics of backward passes.
Computes and returns the sum of gradients of outputs w.r.t. the inputs.
grad_outputs should be a sequence of length matching output containing the “vector” in Jacobian-vector product, usually the pre-computed gradients w.r.t. each of the outputs. If an output doesn’t require_grad, then the gradient can be None).
If only_inputs is True, the function will only return a list of gradients w.r.t the specified inputs. If it’s False, then gradient w.r.t. all remaining leaves will still be computed, and will be accumulated into their .grad attribute.
Note
If you run any forward ops, create grad_outputs, and/or call grad in a user-specified CUDA stream context, see Stream semantics of backward passes.
Warning
This API is in beta. Even though the function signatures are very unlikely to change, major improvements to performances are planned before we consider this stable.
This section contains the higher level API for the autograd that builds on the basic API above and allows you to compute jacobians, hessians, etc.
This API works with user-provided functions that take only Tensors as input and return only Tensors. If your function takes other arguments that are not Tensors or Tensors that don’t have requires_grad set, you can use a lambda to capture them. For example, for a function f that takes three inputs, a Tensor for which we want the jacobian, another tensor that should be considered constant and a boolean flag as f(input, constant, flag=flag) you can use it as functional.jacobian(lambda x: f(x, constant, flag=flag), input).
Function that computes the Jacobian of a given function.
if there is a single input and output, this will be a single Tensor containing the Jacobian for the linearized inputs and output. If one of the two is a tuple, then the Jacobian will be a tuple of Tensors. If both of them are tuples, then the Jacobian will be a tuple of tuple of Tensors where Jacobian[i][j] will contain the Jacobian of the ith output and jth input and will have as size the concatenation of the sizes of the corresponding output and the corresponding input and will have same dtype and device as the corresponding input.
Jacobian (Tensor or nested tuple of Tensors)
Function that computes the Hessian of a given scalar function.
if there is a single input, this will be a single Tensor containing the Hessian for the input. If it is a tuple, then the Hessian will be a tuple of tuples where Hessian[i][j] will contain the Hessian of the ith input and jth input with size the sum of the size of the ith input plus the size of the jth input. Hessian[i][j] will have the same dtype and device as the corresponding ith input.
Hessian (Tensor or a tuple of tuple of Tensors)
Function that computes the dot product between a vector v and the Jacobian of the given function at the point given by the inputs.
func_output (tuple of Tensors or Tensor): output of func(inputs)
vjp (tuple of Tensors or Tensor): result of the dot product with the same shape as the inputs.
output (tuple)
Function that computes the dot product between the Jacobian of the given function at the point given by the inputs and a vector v.
func_output (tuple of Tensors or Tensor): output of func(inputs)
jvp (tuple of Tensors or Tensor): result of the dot product with the same shape as the output.
output (tuple)
Note
The jvp is currently computed by using the backward of the backward (sometimes called the double backwards trick) as we don’t have support for forward mode AD in PyTorch at the moment.
Function that computes the dot product between a vector v and the Hessian of a given scalar function at the point given by the inputs.
func_output (tuple of Tensors or Tensor): output of func(inputs)
vhp (tuple of Tensors or Tensor): result of the dot product with the same shape as the inputs.
output (tuple)
Function that computes the dot product between the Hessian of a given scalar function and a vector v at the point given by the inputs.
func_output (tuple of Tensors or Tensor): output of func(inputs)
hvp (tuple of Tensors or Tensor): result of the dot product with the same shape as the inputs.
output (tuple)
Note
This function is significantly slower than vhp due to backward mode AD constraints. If your functions is twice continuously differentiable, then hvp = vhp.t(). So if you know that your function satisfies this condition, you should use vhp instead that is much faster with the current implementation.
Context-manager that disabled gradient calculation.
Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True.
In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True.
This context manager is thread local; it will not affect computation in other threads.
Also functions as a decorator. (Make sure to instantiate with parenthesis.)
Example:
Context-manager that enables gradient calculation.
Enables gradient calculation, if it has been disabled via no_grad or set_grad_enabled.
This context manager is thread local; it will not affect computation in other threads.
Also functions as a decorator. (Make sure to instantiate with parenthesis.)
Example:
Context-manager that sets gradient calculation to on or off.
set_grad_enabled will enable or disable grads based on its argument mode. It can be used as a context-manager or as a function.
This context manager is thread local; it will not affect computation in other threads.
mode (bool) – Flag whether to enable grad (True), or disable (False). This can be used to conditionally enable gradients.
Example:
When a non-sparse param receives a non-sparse gradient during torch.autograd.backward() or torch.Tensor.backward() param.grad is accumulated as follows.
If param.grad is initially None:
If param already has a non-sparse .grad attribute:
The default behavior (letting .grads be None before the first backward(), such that their layout is created according to 1 or 2, and retained over time according to 3 or 4) is recommended for best performance. Calls to model.zero_grad() or optimizer.zero_grad() will not affect .grad layouts.
In fact, resetting all .grads to None before each accumulation phase, e.g.:
such that they’re recreated according to 1 or 2 every time, is a valid alternative to model.zero_grad() or optimizer.zero_grad() that may improve performance for some networks.
If you need manual control over .grad’s strides, assign param.grad = a zeroed tensor with desired strides before the first backward(), and never reset it to None. 3 guarantees your layout is preserved as long as create_graph=False. 4 indicates your layout is likely preserved even if create_graph=True.
Supporting in-place operations in autograd is a hard matter, and we discourage their use in most cases. Autograd’s aggressive buffer freeing and reuse makes it very efficient and there are very few occasions when in-place operations actually lower memory usage by any significant amount. Unless you’re operating under heavy memory pressure, you might never need to use them.
All Tensor s keep track of in-place operations applied to them, and if the implementation detects that a tensor was saved for backward in one of the functions, but it was modified in-place afterwards, an error will be raised once backward pass is started. This ensures that if you’re using in-place functions and not seeing any errors, you can be sure that the computed gradients are correct.
Warning
The Variable API has been deprecated: Variables are no longer necessary to use autograd with tensors. Autograd automatically supports Tensors with requires_grad set to True. Below please find a quick guide on what has changed:
In addition, one can now create tensors with requires_grad=True using factory methods such as torch.randn(), torch.zeros(), torch.ones(), and others like the following:
autograd_tensor = torch.randn((2, 3, 4), requires_grad=True)
This attribute is None by default and becomes a Tensor the first time a call to backward() computes gradients for self. The attribute will then contain the gradients computed and future calls to backward() will accumulate (add) gradients into it.
Is True if gradients need to be computed for this Tensor, False otherwise.
Note
The fact that gradients need to be computed for a Tensor do not mean that the grad attribute will be populated, see is_leaf for more details.
All Tensors that have requires_grad which is False will be leaf Tensors by convention.
For Tensors that have requires_grad which is True, they will be leaf Tensors if they were created by the user. This means that they are not the result of an operation and so grad_fn is None.
Only leaf Tensors will have their grad populated during a call to backward(). To get grad populated for non-leaf Tensors, you can use retain_grad().
Example:
Computes the gradient of current tensor w.r.t. graph leaves.
The graph is differentiated using the chain rule. If the tensor is non-scalar (i.e. its data has more than one element) and requires gradient, the function additionally requires specifying gradient. It should be a tensor of matching type and location, that contains the gradient of the differentiated function w.r.t. self.
This function accumulates gradients in the leaves - you might need to zero .grad attributes or set them to None before calling it. See Default gradient layouts for details on the memory layout of accumulated gradients.
Note
If you run any forward ops, create gradient, and/or call backward in a user-specified CUDA stream context, see Stream semantics of backward passes.
Returns a new Tensor, detached from the current graph.
The result will never require gradient.
Note
Returned Tensor shares the same storage with the original one. In-place modifications on either of them will be seen, and may trigger errors in correctness checks. IMPORTANT NOTE: Previously, in-place size / stride / storage changes (such as resize_ / resize_as_ / set_ / transpose_) to the returned tensor also update the original tensor. Now, these in-place changes will not update the original tensor anymore, and will instead trigger an error. For sparse tensors: In-place indices / values changes (such as zero_ / copy_ / add_) to the returned tensor will not update the original tensor anymore, and will instead trigger an error.
Detaches the Tensor from the graph that created it, making it a leaf. Views cannot be detached in-place.
Registers a backward hook.
The hook will be called every time a gradient with respect to the Tensor is computed. The hook should have the following signature:
The hook should not modify its argument, but it can optionally return a new gradient which will be used in place of grad.
This function returns a handle with a method handle.remove() that removes the hook from the module.
Example:
Enables .grad attribute for non-leaf Tensors.
Records operation history and defines formulas for differentiating ops.
See the Note on extending the autograd engine for more details on how to use this class: https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd
Every operation performed on Tensor s creates a new function object, that performs the computation, and records that it happened. The history is retained in the form of a DAG of functions, with edges denoting data dependencies (input <- output). Then, when backward is called, the graph is processed in the topological ordering, by calling backward() methods of each Function object, and passing returned gradients on to next Function s.
Normally, the only way users interact with functions is by creating subclasses and defining new operations. This is a recommended way of extending torch.autograd.
Examples:
Defines a formula for differentiating the operation.
This function is to be overridden by all subclasses.
It must accept a context ctx as the first argument, followed by as many outputs did forward() return, and it should return as many tensors, as there were inputs to forward(). Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input.
The context can be used to retrieve tensors saved during the forward pass. It also has an attribute ctx.needs_input_grad as a tuple of booleans representing whether each input needs gradient. E.g., backward() will have ctx.needs_input_grad[0] = True if the first input to forward() needs gradient computated w.r.t. the output.
Performs the operation.
This function is to be overridden by all subclasses.
It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).
The context can be used to store tensors that can be then retrieved during the backward pass.
When creating a new Function, the following methods are available to ctx.
Marks given tensors as modified in an in-place operation.
This should be called at most once, only from inside the forward() method, and all arguments should be inputs.
Every tensor that’s been modified in-place in a call to forward() should be given to this function, to ensure correctness of our checks. It doesn’t matter whether the function is called before or after modification.
Marks outputs as non-differentiable.
This should be called at most once, only from inside the forward() method, and all arguments should be outputs.
This will mark outputs as not requiring gradients, increasing the efficiency of backward computation. You still need to accept a gradient for each output in backward(), but it’s always going to be a zero tensor with the same shape as the shape of a corresponding output.
This is used e.g. for indices returned from a max Function.
Saves given tensors for a future call to backward().
This should be called at most once, and only from inside the forward() method.
Later, saved tensors can be accessed through the saved_tensors attribute. Before returning them to the user, a check is made to ensure they weren’t used in any in-place operation that modified their content.
Arguments can also be None.
Sets whether to materialize output grad tensors. Default is true.
This should be called only from inside the forward() method
If true, undefined output grad tensors will be expanded to tensors full of zeros prior to calling the backward() method.
Check gradients computed via small finite differences against analytical gradients w.r.t. tensors in inputs that are of floating point or complex type and with requires_grad=True.
The check between numerical and analytical gradients uses allclose().
For complex functions, no notion of Jacobian exists. Gradcheck verifies if the numerical and analytical values of Wirtinger and Conjugate Wirtinger derivative are consistent. The gradient computation is done under the assumption that the overall function has a real valued output. For functions with complex output, gradcheck compares the numerical and analytical gradients for two values of grad_output: 1 and 1j. For more details, check out Autograd for Complex Numbers.
Note
The default values are designed for input of double precision. This check will likely fail if input is of less precision, e.g., FloatTensor.
Warning
If any checked tensor in input has overlapping memory, i.e., different indices pointing to the same memory address (e.g., from torch.expand()), this check will likely fail because the numerical gradients computed by point perturbation at such indices will change values at all other indices that share the same memory address.
True if all differences satisfy allclose condition
Check gradients of gradients computed via small finite differences against analytical gradients w.r.t. tensors in inputs and grad_outputs that are of floating point or complex type and with requires_grad=True.
This function checks that backpropagating through the gradients computed to the given grad_outputs are correct.
The check between numerical and analytical gradients uses allclose().
Note
The default values are designed for input and grad_outputs of double precision. This check will likely fail if they are of less precision, e.g., FloatTensor.
Warning
If any checked tensor in input and grad_outputs has overlapping memory, i.e., different indices pointing to the same memory address (e.g., from torch.expand()), this check will likely fail because the numerical gradients computed by point perturbation at such indices will change values at all other indices that share the same memory address.
True if all differences satisfy allclose condition
Autograd includes a profiler that lets you inspect the cost of different operators inside your model - both on the CPU and GPU. There are two modes implemented at the moment - CPU-only using profile. and nvprof based (registers both CPU and GPU activity) using emit_nvtx.
Context manager that manages autograd profiler state and holds a summary of results. Under the hood it just records events of functions being executed in C++ and exposes those events to Python. You can wrap any code into it and it will only report runtime of PyTorch functions. Note: profiler is thread local and is automatically propagated into the async tasks
Exports an EventList as a Chrome tracing tools file.
The checkpoint can be later loaded and inspected under chrome://tracing URL.
path (str) – Path where the trace will be written.
Averages all function events over their keys.
An EventList containing FunctionEventAvg objects.
Returns total time spent on CPU obtained as a sum of all self times across all the events.
Prints an EventList as a nicely formatted table.
A string containing the table.
Averages all events.
A FunctionEventAvg object.
Context manager that makes every autograd operation emit an NVTX range.
It is useful when running the program under nvprof:
Unfortunately, there’s no way to force nvprof to flush the data it collected to disk, so for CUDA profiling one has to use this context manager to annotate nvprof traces and wait for the process to exit before inspecting them. Then, either NVIDIA Visual Profiler (nvvp) can be used to visualize the timeline, or torch.autograd.profiler.load_nvprof() can load the results for inspection e.g. in Python REPL.
Forward-backward correlation
When viewing a profile created using emit_nvtx in the Nvidia Visual Profiler, correlating each backward-pass op with the corresponding forward-pass op can be difficult. To ease this task, emit_nvtx appends sequence number information to the ranges it generates.
During the forward pass, each function range is decorated with seq=<N>. seq is a running counter, incremented each time a new backward Function object is created and stashed for backward. Thus, the seq=<N> annotation associated with each forward function range tells you that if a backward Function object is created by this forward function, the backward object will receive sequence number N. During the backward pass, the top-level range wrapping each C++ backward Function’s apply() call is decorated with stashed seq=<M>. M is the sequence number that the backward object was created with. By comparing stashed seq numbers in backward with seq numbers in forward, you can track down which forward op created each backward Function.
Any functions executed during the backward pass are also decorated with seq=<N>. During default backward (with create_graph=False) this information is irrelevant, and in fact, N may simply be 0 for all such functions. Only the top-level ranges associated with backward Function objects’ apply() methods are useful, as a way to correlate these Function objects with the earlier forward pass.
Double-backward
If, on the other hand, a backward pass with create_graph=True is underway (in other words, if you are setting up for a double-backward), each function’s execution during backward is given a nonzero, useful seq=<N>. Those functions may themselves create Function objects to be executed later during double-backward, just as the original functions in the forward pass did. The relationship between backward and double-backward is conceptually the same as the relationship between forward and backward: The functions still emit current-sequence-number-tagged ranges, the Function objects they create still stash those sequence numbers, and during the eventual double-backward, the Function objects’ apply() ranges are still tagged with stashed seq numbers, which can be compared to seq numbers from the backward pass.
Opens an nvprof trace file and parses autograd annotations.
path (str) – path to nvprof trace
Context-manager that enable anomaly detection for the autograd engine.
This does two things:
Warning
This mode should be enabled only for debugging as the different tests will slow down your program execution.
Context-manager that sets the anomaly detection for the autograd engine on or off.
set_detect_anomaly will enable or disable the autograd anomaly detection based on its argument mode. It can be used as a context-manager or as a function.
See detect_anomaly above for details of the anomaly detection behaviour.
mode (bool) – Flag whether to enable anomaly detection (True), or disable (False).


Computes the sum of gradients of given tensors w.r.t. graph leaves.
The graph is differentiated using the chain rule. If any of tensors are non-scalar (i.e. their data has more than one element) and require gradient, then the Jacobian-vector product would be computed, in this case the function additionally requires specifying grad_tensors. It should be a sequence of matching length, that contains the “vector” in the Jacobian-vector product, usually the gradient of the differentiated function w.r.t. corresponding tensors (None is an acceptable value for all tensors that don’t need gradient tensors).
This function accumulates gradients in the leaves - you might need to zero .grad attributes or set them to None before calling it. See Default gradient layouts for details on the memory layout of accumulated gradients.
Note
Using this method with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak.
Note
If you run any forward ops, create grad_tensors, and/or call backward in a user-specified CUDA stream context, see Stream semantics of backward passes.


Context-manager that enable anomaly detection for the autograd engine.
This does two things:
Warning
This mode should be enabled only for debugging as the different tests will slow down your program execution.


Context-manager that enables gradient calculation.
Enables gradient calculation, if it has been disabled via no_grad or set_grad_enabled.
This context manager is thread local; it will not affect computation in other threads.
Also functions as a decorator. (Make sure to instantiate with parenthesis.)
Example:


Records operation history and defines formulas for differentiating ops.
See the Note on extending the autograd engine for more details on how to use this class: https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd
Every operation performed on Tensor s creates a new function object, that performs the computation, and records that it happened. The history is retained in the form of a DAG of functions, with edges denoting data dependencies (input <- output). Then, when backward is called, the graph is processed in the topological ordering, by calling backward() methods of each Function object, and passing returned gradients on to next Function s.
Normally, the only way users interact with functions is by creating subclasses and defining new operations. This is a recommended way of extending torch.autograd.
Examples:
Defines a formula for differentiating the operation.
This function is to be overridden by all subclasses.
It must accept a context ctx as the first argument, followed by as many outputs did forward() return, and it should return as many tensors, as there were inputs to forward(). Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input.
The context can be used to retrieve tensors saved during the forward pass. It also has an attribute ctx.needs_input_grad as a tuple of booleans representing whether each input needs gradient. E.g., backward() will have ctx.needs_input_grad[0] = True if the first input to forward() needs gradient computated w.r.t. the output.
Performs the operation.
This function is to be overridden by all subclasses.
It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).
The context can be used to store tensors that can be then retrieved during the backward pass.


Defines a formula for differentiating the operation.
This function is to be overridden by all subclasses.
It must accept a context ctx as the first argument, followed by as many outputs did forward() return, and it should return as many tensors, as there were inputs to forward(). Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input.
The context can be used to retrieve tensors saved during the forward pass. It also has an attribute ctx.needs_input_grad as a tuple of booleans representing whether each input needs gradient. E.g., backward() will have ctx.needs_input_grad[0] = True if the first input to forward() needs gradient computated w.r.t. the output.


Performs the operation.
This function is to be overridden by all subclasses.
It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).
The context can be used to store tensors that can be then retrieved during the backward pass.


Marks given tensors as modified in an in-place operation.
This should be called at most once, only from inside the forward() method, and all arguments should be inputs.
Every tensor that’s been modified in-place in a call to forward() should be given to this function, to ensure correctness of our checks. It doesn’t matter whether the function is called before or after modification.
Marks outputs as non-differentiable.
This should be called at most once, only from inside the forward() method, and all arguments should be outputs.
This will mark outputs as not requiring gradients, increasing the efficiency of backward computation. You still need to accept a gradient for each output in backward(), but it’s always going to be a zero tensor with the same shape as the shape of a corresponding output.
This is used e.g. for indices returned from a max Function.
Saves given tensors for a future call to backward().
This should be called at most once, and only from inside the forward() method.
Later, saved tensors can be accessed through the saved_tensors attribute. Before returning them to the user, a check is made to ensure they weren’t used in any in-place operation that modified their content.
Arguments can also be None.
Sets whether to materialize output grad tensors. Default is true.
This should be called only from inside the forward() method
If true, undefined output grad tensors will be expanded to tensors full of zeros prior to calling the backward() method.


Marks given tensors as modified in an in-place operation.
This should be called at most once, only from inside the forward() method, and all arguments should be inputs.
Every tensor that’s been modified in-place in a call to forward() should be given to this function, to ensure correctness of our checks. It doesn’t matter whether the function is called before or after modification.


Marks outputs as non-differentiable.
This should be called at most once, only from inside the forward() method, and all arguments should be outputs.
This will mark outputs as not requiring gradients, increasing the efficiency of backward computation. You still need to accept a gradient for each output in backward(), but it’s always going to be a zero tensor with the same shape as the shape of a corresponding output.
This is used e.g. for indices returned from a max Function.


Saves given tensors for a future call to backward().
This should be called at most once, and only from inside the forward() method.
Later, saved tensors can be accessed through the saved_tensors attribute. Before returning them to the user, a check is made to ensure they weren’t used in any in-place operation that modified their content.
Arguments can also be None.


Sets whether to materialize output grad tensors. Default is true.
This should be called only from inside the forward() method
If true, undefined output grad tensors will be expanded to tensors full of zeros prior to calling the backward() method.


Function that computes the Hessian of a given scalar function.
if there is a single input, this will be a single Tensor containing the Hessian for the input. If it is a tuple, then the Hessian will be a tuple of tuples where Hessian[i][j] will contain the Hessian of the ith input and jth input with size the sum of the size of the ith input plus the size of the jth input. Hessian[i][j] will have the same dtype and device as the corresponding ith input.
Hessian (Tensor or a tuple of tuple of Tensors)


Function that computes the dot product between the Hessian of a given scalar function and a vector v at the point given by the inputs.
func_output (tuple of Tensors or Tensor): output of func(inputs)
hvp (tuple of Tensors or Tensor): result of the dot product with the same shape as the inputs.
output (tuple)
Note
This function is significantly slower than vhp due to backward mode AD constraints. If your functions is twice continuously differentiable, then hvp = vhp.t(). So if you know that your function satisfies this condition, you should use vhp instead that is much faster with the current implementation.


Function that computes the Jacobian of a given function.
if there is a single input and output, this will be a single Tensor containing the Jacobian for the linearized inputs and output. If one of the two is a tuple, then the Jacobian will be a tuple of Tensors. If both of them are tuples, then the Jacobian will be a tuple of tuple of Tensors where Jacobian[i][j] will contain the Jacobian of the ith output and jth input and will have as size the concatenation of the sizes of the corresponding output and the corresponding input and will have same dtype and device as the corresponding input.
Jacobian (Tensor or nested tuple of Tensors)


Function that computes the dot product between the Jacobian of the given function at the point given by the inputs and a vector v.
func_output (tuple of Tensors or Tensor): output of func(inputs)
jvp (tuple of Tensors or Tensor): result of the dot product with the same shape as the output.
output (tuple)
Note
The jvp is currently computed by using the backward of the backward (sometimes called the double backwards trick) as we don’t have support for forward mode AD in PyTorch at the moment.


Function that computes the dot product between a vector v and the Hessian of a given scalar function at the point given by the inputs.
func_output (tuple of Tensors or Tensor): output of func(inputs)
vhp (tuple of Tensors or Tensor): result of the dot product with the same shape as the inputs.
output (tuple)


Function that computes the dot product between a vector v and the Jacobian of the given function at the point given by the inputs.
func_output (tuple of Tensors or Tensor): output of func(inputs)
vjp (tuple of Tensors or Tensor): result of the dot product with the same shape as the inputs.
output (tuple)


Computes and returns the sum of gradients of outputs w.r.t. the inputs.
grad_outputs should be a sequence of length matching output containing the “vector” in Jacobian-vector product, usually the pre-computed gradients w.r.t. each of the outputs. If an output doesn’t require_grad, then the gradient can be None).
If only_inputs is True, the function will only return a list of gradients w.r.t the specified inputs. If it’s False, then gradient w.r.t. all remaining leaves will still be computed, and will be accumulated into their .grad attribute.
Note
If you run any forward ops, create grad_outputs, and/or call grad in a user-specified CUDA stream context, see Stream semantics of backward passes.


Check gradients computed via small finite differences against analytical gradients w.r.t. tensors in inputs that are of floating point or complex type and with requires_grad=True.
The check between numerical and analytical gradients uses allclose().
For complex functions, no notion of Jacobian exists. Gradcheck verifies if the numerical and analytical values of Wirtinger and Conjugate Wirtinger derivative are consistent. The gradient computation is done under the assumption that the overall function has a real valued output. For functions with complex output, gradcheck compares the numerical and analytical gradients for two values of grad_output: 1 and 1j. For more details, check out Autograd for Complex Numbers.
Note
The default values are designed for input of double precision. This check will likely fail if input is of less precision, e.g., FloatTensor.
Warning
If any checked tensor in input has overlapping memory, i.e., different indices pointing to the same memory address (e.g., from torch.expand()), this check will likely fail because the numerical gradients computed by point perturbation at such indices will change values at all other indices that share the same memory address.
True if all differences satisfy allclose condition


Check gradients of gradients computed via small finite differences against analytical gradients w.r.t. tensors in inputs and grad_outputs that are of floating point or complex type and with requires_grad=True.
This function checks that backpropagating through the gradients computed to the given grad_outputs are correct.
The check between numerical and analytical gradients uses allclose().
Note
The default values are designed for input and grad_outputs of double precision. This check will likely fail if they are of less precision, e.g., FloatTensor.
Warning
If any checked tensor in input and grad_outputs has overlapping memory, i.e., different indices pointing to the same memory address (e.g., from torch.expand()), this check will likely fail because the numerical gradients computed by point perturbation at such indices will change values at all other indices that share the same memory address.
True if all differences satisfy allclose condition


Context-manager that disabled gradient calculation.
Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True.
In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True.
This context manager is thread local; it will not affect computation in other threads.
Also functions as a decorator. (Make sure to instantiate with parenthesis.)
Example:


Context manager that makes every autograd operation emit an NVTX range.
It is useful when running the program under nvprof:
Unfortunately, there’s no way to force nvprof to flush the data it collected to disk, so for CUDA profiling one has to use this context manager to annotate nvprof traces and wait for the process to exit before inspecting them. Then, either NVIDIA Visual Profiler (nvvp) can be used to visualize the timeline, or torch.autograd.profiler.load_nvprof() can load the results for inspection e.g. in Python REPL.
Forward-backward correlation
When viewing a profile created using emit_nvtx in the Nvidia Visual Profiler, correlating each backward-pass op with the corresponding forward-pass op can be difficult. To ease this task, emit_nvtx appends sequence number information to the ranges it generates.
During the forward pass, each function range is decorated with seq=<N>. seq is a running counter, incremented each time a new backward Function object is created and stashed for backward. Thus, the seq=<N> annotation associated with each forward function range tells you that if a backward Function object is created by this forward function, the backward object will receive sequence number N. During the backward pass, the top-level range wrapping each C++ backward Function’s apply() call is decorated with stashed seq=<M>. M is the sequence number that the backward object was created with. By comparing stashed seq numbers in backward with seq numbers in forward, you can track down which forward op created each backward Function.
Any functions executed during the backward pass are also decorated with seq=<N>. During default backward (with create_graph=False) this information is irrelevant, and in fact, N may simply be 0 for all such functions. Only the top-level ranges associated with backward Function objects’ apply() methods are useful, as a way to correlate these Function objects with the earlier forward pass.
Double-backward
If, on the other hand, a backward pass with create_graph=True is underway (in other words, if you are setting up for a double-backward), each function’s execution during backward is given a nonzero, useful seq=<N>. Those functions may themselves create Function objects to be executed later during double-backward, just as the original functions in the forward pass did. The relationship between backward and double-backward is conceptually the same as the relationship between forward and backward: The functions still emit current-sequence-number-tagged ranges, the Function objects they create still stash those sequence numbers, and during the eventual double-backward, the Function objects’ apply() ranges are still tagged with stashed seq numbers, which can be compared to seq numbers from the backward pass.


Opens an nvprof trace file and parses autograd annotations.
path (str) – path to nvprof trace


Context manager that manages autograd profiler state and holds a summary of results. Under the hood it just records events of functions being executed in C++ and exposes those events to Python. You can wrap any code into it and it will only report runtime of PyTorch functions. Note: profiler is thread local and is automatically propagated into the async tasks
Exports an EventList as a Chrome tracing tools file.
The checkpoint can be later loaded and inspected under chrome://tracing URL.
path (str) – Path where the trace will be written.
Averages all function events over their keys.
An EventList containing FunctionEventAvg objects.
Returns total time spent on CPU obtained as a sum of all self times across all the events.
Prints an EventList as a nicely formatted table.
A string containing the table.
Averages all events.
A FunctionEventAvg object.


Exports an EventList as a Chrome tracing tools file.
The checkpoint can be later loaded and inspected under chrome://tracing URL.
path (str) – Path where the trace will be written.


Averages all function events over their keys.
An EventList containing FunctionEventAvg objects.


Returns total time spent on CPU obtained as a sum of all self times across all the events.


Prints an EventList as a nicely formatted table.
A string containing the table.


Averages all events.
A FunctionEventAvg object.


Context-manager that sets the anomaly detection for the autograd engine on or off.
set_detect_anomaly will enable or disable the autograd anomaly detection based on its argument mode. It can be used as a context-manager or as a function.
See detect_anomaly above for details of the anomaly detection behaviour.
mode (bool) – Flag whether to enable anomaly detection (True), or disable (False).


Context-manager that sets gradient calculation to on or off.
set_grad_enabled will enable or disable grads based on its argument mode. It can be used as a context-manager or as a function.
This context manager is thread local; it will not affect computation in other threads.
mode (bool) – Flag whether to enable grad (True), or disable (False). This can be used to conditionally enable gradients.
Example:


torch.backends controls the behavior of various backends that PyTorch supports.
These backends include:
Returns whether PyTorch is built with CUDA support. Note that this doesn’t necessarily mean CUDA is available; just that if this PyTorch binary were run a machine with working CUDA drivers and devices, we would be able to use it.
A bool that controls whether TensorFloat-32 tensor cores may be used in matrix multiplications on Ampere or newer GPUs. See TensorFloat-32(TF32) on Ampere devices.
cufft_plan_cache caches the cuFFT plans
A readonly int that shows the number of plans currently in the cuFFT plan cache.
A int that controls cache capacity of cuFFT plan.
Clears the cuFFT plan cache.
Returns the version of cuDNN
Returns a bool indicating if CUDNN is currently available.
A bool that controls whether cuDNN is enabled.
A bool that controls where TensorFloat-32 tensor cores may be used in cuDNN convolutions on Ampere or newer GPUs. See TensorFloat-32(TF32) on Ampere devices.
A bool that, if True, causes cuDNN to only use deterministic convolution algorithms. See also torch.are_deterministic_algorithms_enabled() and torch.use_deterministic_algorithms().
A bool that, if True, causes cuDNN to benchmark multiple convolution algorithms and select the fastest.
Returns whether PyTorch is built with MKL support.
Returns whether PyTorch is built with MKL-DNN support.
Returns whether PyTorch is built with OpenMP support.


cufft_plan_cache caches the cuFFT plans
A readonly int that shows the number of plans currently in the cuFFT plan cache.
A int that controls cache capacity of cuFFT plan.
Clears the cuFFT plan cache.


Returns whether PyTorch is built with CUDA support. Note that this doesn’t necessarily mean CUDA is available; just that if this PyTorch binary were run a machine with working CUDA drivers and devices, we would be able to use it.


A bool that controls whether TensorFloat-32 tensor cores may be used in matrix multiplications on Ampere or newer GPUs. See TensorFloat-32(TF32) on Ampere devices.


A readonly int that shows the number of plans currently in the cuFFT plan cache.


A bool that controls where TensorFloat-32 tensor cores may be used in cuDNN convolutions on Ampere or newer GPUs. See TensorFloat-32(TF32) on Ampere devices.


A bool that, if True, causes cuDNN to benchmark multiple convolution algorithms and select the fastest.


A bool that, if True, causes cuDNN to only use deterministic convolution algorithms. See also torch.are_deterministic_algorithms_enabled() and torch.use_deterministic_algorithms().


A bool that controls whether cuDNN is enabled.


Returns a bool indicating if CUDNN is currently available.


Returns whether PyTorch is built with MKL support.


Returns whether PyTorch is built with MKL-DNN support.


Returns whether PyTorch is built with OpenMP support.


Performs a batch matrix-matrix product of matrices in batch1 and batch2. input is added to the final result.
batch1 and batch2 must be 3-D tensors each containing the same number of matrices.
If batch1 is a (b×n×m)(b \times n \times m)  tensor, batch2 is a (b×m×p)(b \times m \times p)  tensor, then input must be broadcastable with a (b×n×p)(b \times n \times p)  tensor and out will be a (b×n×p)(b \times n \times p)  tensor. Both alpha and beta mean the same as the scaling factors used in torch.addbmm().
If beta is 0, then input will be ignored, and nan and inf in it will not be propagated.
For inputs of type FloatTensor or DoubleTensor, arguments beta and alpha must be real numbers, otherwise they should be integers.
This operator supports TensorFloat32.
Example:


Bartlett window function.
where NN  is the full window size.
The input window_length is a positive integer controlling the returned window size. periodic flag determines whether the returned window trims off the last duplicate value from the symmetric window and is ready to be used as a periodic window with functions like torch.stft(). Therefore, if periodic is true, the NN  in above formula is in fact window_length+1\text{window\_length} + 1 . Also, we always have torch.bartlett_window(L, periodic=True) equal to torch.bartlett_window(L + 1, periodic=False)[:-1]).
Note
If window_length =1=1 , the returned window contains a single value 1.
A 1-D tensor of size (window_length,)(\text{window\_length},)  containing the window
Tensor


Draws binary random numbers (0 or 1) from a Bernoulli distribution.
The input tensor should be a tensor containing probabilities to be used for drawing the binary random number. Hence, all values in input have to be in the range: 0≤inputi≤10 \leq \text{input}_i \leq 1 .
The ith\text{i}^{th}  element of the output tensor will draw a value 11  according to the ith\text{i}^{th}  probability value given in input.
The returned out tensor only has values 0 or 1 and is of the same shape as input.
out can have integral dtype, but input must have floating point dtype.
input (Tensor) – the input tensor of probability values for the Bernoulli distribution
Example:


Count the frequency of each value in an array of non-negative ints.
The number of bins (size 1) is one larger than the largest value in input unless input is empty, in which case the result is a tensor of size 0. If minlength is specified, the number of bins is at least minlength and if input is empty, then the result is tensor of size minlength filled with zeros. If n is the value at position i, out[n] += weights[i] if weights is specified else out[n] += 1.
Note
This operation may produce nondeterministic gradients when given tensors on a CUDA device. See Reproducibility for more information.
a tensor of shape Size([max(input) + 1]) if input is non-empty, else Size(0)
output (Tensor)
Example:


Computes the bitwise AND of input and other. The input tensor must be of integral or Boolean types. For bool tensors, it computes the logical AND.
out (Tensor, optional) – the output tensor.


Computes the bitwise NOT of the given input tensor. The input tensor must be of integral or Boolean types. For bool tensors, it computes the logical NOT.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.


Computes the bitwise OR of input and other. The input tensor must be of integral or Boolean types. For bool tensors, it computes the logical OR.
out (Tensor, optional) – the output tensor.


Computes the bitwise XOR of input and other. The input tensor must be of integral or Boolean types. For bool tensors, it computes the logical XOR.
out (Tensor, optional) – the output tensor.


Blackman window function.
where NN  is the full window size.
The input window_length is a positive integer controlling the returned window size. periodic flag determines whether the returned window trims off the last duplicate value from the symmetric window and is ready to be used as a periodic window with functions like torch.stft(). Therefore, if periodic is true, the NN  in above formula is in fact window_length+1\text{window\_length} + 1 . Also, we always have torch.blackman_window(L, periodic=True) equal to torch.blackman_window(L + 1, periodic=False)[:-1]).
Note
If window_length =1=1 , the returned window contains a single value 1.
A 1-D tensor of size (window_length,)(\text{window\_length},)  containing the window
Tensor


Create a block diagonal matrix from provided tensors.
*tensors – One or more tensors with 0, 1, or 2 dimensions.
order such that their upper left and lower right corners are diagonally adjacent. All other elements are set to 0.
Tensor
Example:


Performs a batch matrix-matrix product of matrices stored in input and mat2.
input and mat2 must be 3-D tensors each containing the same number of matrices.
If input is a (b×n×m)(b \times n \times m)  tensor, mat2 is a (b×m×p)(b \times m \times p)  tensor, out will be a (b×n×p)(b \times n \times p)  tensor.
This operator supports TensorFloat32.
Note
This function does not broadcast. For broadcasting matrix products, see torch.matmul().
Example:


Similar to broadcast_tensors() but for shapes.
This is equivalent to torch.broadcast_tensors(*map(torch.empty, shapes))[0].shape but avoids the need create to intermediate tensors. This is useful for broadcasting tensors of common batch shape but different rightmost shape, e.g. to broadcast mean vectors with covariance matrices.
Example:
*shapes (torch.Size) – Shapes of tensors.
A shape compatible with all input shapes.
shape (torch.Size)
RuntimeError – If shapes are incompatible.


Broadcasts the given tensors according to Broadcasting semantics.
*tensors – any number of tensors of the same type
Warning
More than one element of a broadcasted tensor may refer to a single memory location. As a result, in-place operations (especially ones that are vectorized) may result in incorrect behavior. If you need to write to the tensors, please clone them first.
Example:


Broadcasts input to the shape shape. Equivalent to calling input.expand(shape). See expand() for details.
Example:


Returns the indices of the buckets to which each value in the input belongs, where the boundaries of the buckets are set by boundaries. Return a new tensor with the same size as input. If right is False (default), then the left boundary is closed. More formally, the returned index satisfies the following rules:
right
returned index satisfies
False
boundaries[i-1] < input[m][n]...[l][x] <= boundaries[i]
True
boundaries[i-1] <= input[m][n]...[l][x] < boundaries[i]
Example:


Determines if a type conversion is allowed under PyTorch casting rules described in the type promotion documentation.
Example:


Do cartesian product of the given sequence of tensors. The behavior is similar to python’s itertools.product.
*tensors – any number of 1 dimensional tensors.
do itertools.product on these lists, and finally convert the resulting list into tensor.
Tensor
Example:


Concatenates the given sequence of seq tensors in the given dimension. All tensors must either have the same shape (except in the concatenating dimension) or be empty.
torch.cat() can be seen as an inverse operation for torch.split() and torch.chunk().
torch.cat() can be best understood via examples.
out (Tensor, optional) – the output tensor.
Example:


Computes batched the p-norm distance between each pair of the two collections of row vectors.
If x1 has shape B×P×MB \times P \times M  and x2 has shape B×R×MB \times R \times M  then the output will have shape B×P×RB \times P \times R .
This function is equivalent to scipy.spatial.distance.cdist(input,’minkowski’, p=p) if p∈(0,∞)p \in (0, \infty) . When p=0p = 0  it is equivalent to scipy.spatial.distance.cdist(input, ‘hamming’) * M. When p=∞p = \infty , the closest scipy function is scipy.spatial.distance.cdist(xn, lambda x, y: np.abs(x - y).max()).


Returns a new tensor with the ceil of the elements of input, the smallest integer greater than or equal to each element.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.
Example:


Returns the matrix product of the NN  2-D tensors. This product is efficiently computed using the matrix chain order algorithm which selects the order in which incurs the lowest cost in terms of arithmetic operations ([CLRS]). Note that since this is a function to compute the product, NN  needs to be greater than or equal to 2; if equal to 2 then a trivial matrix-matrix product is returned. If NN  is 1, then this is a no-op - the original matrix is returned as is.
matrices (Tensors...) – a sequence of 2 or more 2-D tensors whose product is to be determined.
if the ithi^{th}  tensor was of dimensions pi×pi+1p_{i} \times p_{i + 1} , then the product would be of dimensions p1×pN+1p_{1} \times p_{N + 1} .
Tensor
Example:


Computes the Cholesky decomposition of a symmetric positive-definite matrix AA  or for batches of symmetric positive-definite matrices.
If upper is True, the returned matrix U is upper-triangular, and the decomposition has the form:
If upper is False, the returned matrix L is lower-triangular, and the decomposition has the form:
If upper is True, and AA  is a batch of symmetric positive-definite matrices, then the returned tensor will be composed of upper-triangular Cholesky factors of each of the individual matrices. Similarly, when upper is False, the returned tensor will be composed of lower-triangular Cholesky factors of each of the individual matrices.
Note
torch.linalg.cholesky() should be used over torch.cholesky when possible. Note however that torch.linalg.cholesky() does not yet support the upper parameter and instead always returns the lower triangular matrix.
out (Tensor, optional) – the output matrix
Example:


Computes the inverse of a symmetric positive-definite matrix AA  using its Cholesky factor uu : returns matrix inv. The inverse is computed using LAPACK routines dpotri and spotri (and the corresponding MAGMA routines).
If upper is False, uu  is lower triangular such that the returned tensor is
If upper is True or not provided, uu  is upper triangular such that the returned tensor is
out (Tensor, optional) – the output tensor for inv
Example:


Solves a linear system of equations with a positive semidefinite matrix to be inverted given its Cholesky factor matrix uu .
If upper is False, uu  is and lower triangular and c is returned such that:
If upper is True or not provided, uu  is upper triangular and c is returned such that:
torch.cholesky_solve(b, u) can take in 2D inputs b, u or inputs that are batches of 2D matrices. If the inputs are batches, then returns batched outputs c
Supports real-valued and complex-valued inputs. For the complex-valued inputs the transpose operator above is the conjugate transpose.
out (Tensor, optional) – the output tensor for c
Example:


Splits a tensor into a specific number of chunks. Each chunk is a view of the input tensor.
Last chunk will be smaller if the tensor size along the given dimension dim is not divisible by chunks.


Clamp all elements in input into the range [ min, max ]. Let min_value and max_value be min and max, respectively, this returns:
out (Tensor, optional) – the output tensor.
Example:
Clamps all elements in input to be larger or equal min.
input (Tensor) – the input tensor.
Example:
Clamps all elements in input to be smaller or equal max.
input (Tensor) – the input tensor.
Example:


Returns a copy of input.
Note
This function is differentiable, so gradients will flow back from the result of this operation to input. To create a tensor without an autograd relationship to input see detach().
input (Tensor) – the input tensor.
memory_format (torch.memory_format, optional) – the desired memory format of returned tensor. Default: torch.preserve_format.


Creates a new tensor by horizontally stacking the tensors in tensors.
Equivalent to torch.hstack(tensors), except each zero or one dimensional tensor t in tensors is first reshaped into a (t.numel(), 1) column before being stacked horizontally.
tensors (sequence of Tensors) – sequence of tensors to concatenate
out (Tensor, optional) – the output tensor.
Example:


Compute combinations of length rr  of the given tensor. The behavior is similar to python’s itertools.combinations when with_replacement is set to False, and itertools.combinations_with_replacement when with_replacement is set to True.
A tensor equivalent to converting all the input tensors into lists, do itertools.combinations or itertools.combinations_with_replacement on these lists, and finally convert the resulting list into tensor.
Tensor
Example:


Returns whether PyTorch was built with _GLIBCXX_USE_CXX11_ABI=1


Constructs a complex tensor with its real part equal to real and its imaginary part equal to imag.
out (Tensor) – If the inputs are torch.float32, must be torch.complex64. If the inputs are torch.float64, must be torch.complex128.


Computes the element-wise conjugate of the given input tensor. If :attr`input` has a non-complex dtype, this function just returns input.
Warning
In the future, torch.conj() may return a non-writeable view for an input of non-complex dtype. It’s recommended that programs not modify the tensor returned by torch.conj() when input is of non-complex dtype to be compatible with this change.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.
Example:


Create a new floating-point tensor with the magnitude of input and the sign of other, elementwise.
Supports broadcasting to a common shape, and integer and float inputs.
out (Tensor, optional) – the output tensor.
Example:


Returns a new tensor with the cosine of the elements of input.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.
Example:


Returns a new tensor with the hyperbolic cosine of the elements of input.
input (Tensor) – the input tensor.
out (Tensor, optional) – the output tensor.
Example:
Note
When input is on the CPU, the implementation of torch.cosh may use the Sleef library, which rounds very large results to infinity or negative infinity. See here for details.


Counts the number of non-zero values in the tensor input along the given dim. If no dim is specified then all non-zeros in the tensor are counted.
Example:


Returns the cross product of vectors in dimension dim of input and other.
input and other must have the same size, and the size of their dim dimension should be 3.
If dim is not given, it defaults to the first dimension found with the size 3. Note that this might be unexpected.
out (Tensor, optional) – the output tensor.
Example:


This package adds support for CUDA tensor types, that implement the same function as CPU tensors, but they utilize GPUs for computation.
It is lazily initialized, so you can always import it, and use is_available() to determine if your system supports CUDA.
CUDA semantics has more details about working with CUDA.
Checks if peer access between two devices is possible.
Returns cublasHandle_t pointer to current cuBLAS handle
Returns the index of a currently selected device.
Returns the currently selected Stream for a given device.
device (torch.device or int, optional) – selected device. Returns the currently selected Stream for the current device, given by current_device(), if device is None (default).
Returns the default Stream for a given device.
device (torch.device or int, optional) – selected device. Returns the default Stream for the current device, given by current_device(), if device is None (default).
Context-manager that changes the selected device.
device (torch.device or int) – device index to select. It’s a no-op if this argument is a negative integer or None.
Returns the number of GPUs available.
Context-manager that changes the current device to that of given object.
You can use both tensors and storages as arguments. If a given object is not allocated on a GPU, this is a no-op.
obj (Tensor or Storage) – object allocated on the selected device.
Returns list CUDA architectures this library was compiled for.
Gets the cuda capability of a device.
device (torch.device or int, optional) – device for which to return the device capability. This function is a no-op if this argument is a negative integer. It uses the current device, given by current_device(), if device is None (default).
the major and minor cuda capability of the device
tuple(int, int)
Gets the name of a device.
device (torch.device or int, optional) – device for which to return the name. This function is a no-op if this argument is a negative integer. It uses the current device, given by current_device(), if device is None (default).
the name of the device
str
Gets the properties of a device.
device (torch.device or int or str) – device for which to return the properties of the device.
the properties of the device
_CudaDeviceProperties
Returns NVCC gencode flags this library were compiled with.
Initialize PyTorch’s CUDA state. You may need to call this explicitly if you are interacting with PyTorch via its C API, as Python bindings for CUDA functionality will not be available until this initialization takes place. Ordinary users should not need this, as all of PyTorch’s CUDA methods automatically initialize CUDA state on-demand.
Does nothing if the CUDA state is already initialized.
Force collects GPU memory after it has been released by CUDA IPC.
Note
Checks if any sent CUDA tensors could be cleaned from the memory. Force closes shared memory file used for reference counting if there is no active counters. Useful when the producer process stopped actively sending tensors and want to release unused memory.
Returns a bool indicating if CUDA is currently available.
Returns whether PyTorch’s CUDA state has been initialized.
Sets the current device.
Usage of this function is discouraged in favor of device. In most cases it’s better to use CUDA_VISIBLE_DEVICES environmental variable.
device (torch.device or int) – selected device. This function is a no-op if this argument is negative.
Context-manager that selects a given stream.
All CUDA kernels queued within its context will be enqueued on a selected stream.
stream (Stream) – selected stream. This manager is a no-op if it’s None.
Note
Streams are per-device. If the selected stream is not on the current device, this function will also change the current device to match the stream.
Waits for all kernels in all streams on a CUDA device to complete.
device (torch.device or int, optional) – device for which to synchronize. It uses the current device, given by current_device(), if device is None (default).
Returns the random number generator state of the specified GPU as a ByteTensor.
device (torch.device or int, optional) – The device to return the RNG state of. Default: 'cuda' (i.e., torch.device('cuda'), the current CUDA device).
Warning
This function eagerly initializes CUDA.
Returns a list of ByteTensor representing the random number states of all devices.
Sets the random number generator state of the specified GPU.
Sets the random number generator state of all devices.
new_states (Iterable of torch.ByteTensor) – The desired state for each device
Sets the seed for generating random numbers for the current GPU. It’s safe to call this function if CUDA is not available; in that case, it is silently ignored.
seed (int) – The desired seed.
Warning
If you are working with a multi-GPU model, this function is insufficient to get determinism. To seed all GPUs, use manual_seed_all().
Sets the seed for generating random numbers on all GPUs. It’s safe to call this function if CUDA is not available; in that case, it is silently ignored.
seed (int) – The desired seed.
Sets the seed for generating random numbers to a random number for the current GPU. It’s safe to call this function if CUDA is not available; in that case, it is silently ignored.
Warning
If you are working with a multi-GPU model, this function will only initialize the seed on one GPU. To initialize all GPUs, use seed_all().
Sets the seed for generating random numbers to a random number on all GPUs. It’s safe to call this function if CUDA is not available; in that case, it is silently ignored.
Returns the current random seed of the current GPU.
Warning
This function eagerly initializes CUDA.
Broadcasts a tensor to specified GPU devices.
Note
Exactly one of devices and out must be specified.
a tuple containing copies of tensor, placed on devices.
a tuple containing out tensors, each containing a copy of tensor.
Broadcasts a sequence tensors to the specified GPUs. Small tensors are first coalesced into a buffer to reduce the number of synchronizations.
A tuple containing copies of tensor, placed on devices.
Sums tensors from multiple GPUs.
All inputs should have matching shapes, dtype, and layout. The output tensor will be of the same shape, dtype, and layout.
A tensor containing an elementwise sum of all inputs, placed on the destination device.
Scatters tensor across multiple GPUs.
Note
Exactly one of devices and out must be specified. When out is specified, chunk_sizes must not be specified and will be inferred from sizes of out.
a tuple containing chunks of tensor, placed on devices.
a tuple containing out tensors, each containing a chunk of tensor.
Gathers tensors from multiple GPU devices.
Note
destination must not be specified when out is specified.
a tensor located on destination device, that is a result of concatenating tensors along dim.
the out tensor, now containing results of concatenating tensors along dim.
Wrapper around a CUDA stream.
A CUDA stream is a linear sequence of execution that belongs to a specific device, independent from other streams. See CUDA semantics for details.
Note
Although CUDA versions >= 11 support more than two levels of priorities, in PyTorch, we only support two levels of priorities.
Checks if all the work submitted has been completed.
A boolean indicating if all kernels in this stream are completed.
Records an event.
event (Event, optional) – event to record. If not given, a new one will be allocated.
Recorded event.
Wait for all the kernels in this stream to complete.
Note
This is a wrapper around cudaStreamSynchronize(): see CUDA Stream documentation for more info.
Makes all future work submitted to the stream wait for an event.
event (Event) – an event to wait for.
Note
This is a wrapper around cudaStreamWaitEvent(): see CUDA Stream documentation for more info.
This function returns without waiting for event: only future operations are affected.
Synchronizes with another stream.
All future work submitted to this stream will wait until all kernels submitted to a given stream at the time of call complete.
stream (Stream) – a stream to synchronize.
Note
This function returns without waiting for currently enqueued kernels in stream: only future operations are affected.
Wrapper around a CUDA event.
CUDA events are synchronization markers that can be used to monitor the device’s progress, to accurately measure timing, and to synchronize CUDA streams.
The underlying CUDA events are lazily initialized when the event is first recorded or exported to another process. After creation, only streams on the same device may record the event. However, streams on any device can wait on the event.
Returns the time elapsed in milliseconds after the event was recorded and before the end_event was recorded.
Reconstruct an event from an IPC handle on the given device.
Returns an IPC handle of this event. If not recorded yet, the event will use the current device.
Checks if all work currently captured by event has completed.
A boolean indicating if all work currently captured by event has completed.
Records the event in a given stream.
Uses torch.cuda.current_stream() if no stream is specified. The stream’s device must match the event’s device.
Waits for the event to complete.
Waits until the completion of all work currently captured in this event. This prevents the CPU thread from proceeding until the event completes.
Note
This is a wrapper around cudaEventSynchronize(): see CUDA Event documentation for more info.
Makes all future work submitted to the given stream wait for this event.
Use torch.cuda.current_stream() if no stream is specified.
Releases all unoccupied cached memory currently held by the caching allocator so that those can be used in other GPU application and visible in nvidia-smi.
Note
empty_cache() doesn’t increase the amount of GPU memory available for PyTorch. However, it may help reduce fragmentation of GPU memory in certain cases. See Memory management for more details about GPU memory management.
Returns a human-readable printout of the running processes and their GPU memory use for a given device.
This can be useful to display periodically during training, or when handling out-of-memory exceptions.
device (torch.device or int, optional) – selected device. Returns printout for the current device, given by current_device(), if device is None (default).
Returns a dictionary of CUDA memory allocator statistics for a given device.
The return value of this function is a dictionary of statistics, each of which is a non-negative integer.
Core statistics:
For these core statistics, values are broken down as follows.
Pool type:
Metric type:
In addition to the core statistics, we also provide some simple event counters:
device (torch.device or int, optional) – selected device. Returns statistics for the current device, given by current_device(), if device is None (default).
Note
See Memory management for more details about GPU memory management.
Returns a human-readable printout of the current memory allocator statistics for a given device.
This can be useful to display periodically during training, or when handling out-of-memory exceptions.
Note
See Memory management for more details about GPU memory management.
Returns a snapshot of the CUDA memory allocator state across all devices.
Interpreting the output of this function requires familiarity with the memory allocator internals.
Note
See Memory management for more details about GPU memory management.
Returns the current GPU memory occupied by tensors in bytes for a given device.
device (torch.device or int, optional) – selected device. Returns statistic for the current device, given by current_device(), if device is None (default).
Note
This is likely less than the amount shown in nvidia-smi since some unused memory can be held by the caching allocator and some context needs to be created on GPU. See Memory management for more details about GPU memory management.
Returns the maximum GPU memory occupied by tensors in bytes for a given device.
By default, this returns the peak allocated memory since the beginning of this program. reset_peak_stats() can be used to reset the starting point in tracking this metric. For example, these two functions can measure the peak allocated memory usage of each iteration in a training loop.
device (torch.device or int, optional) – selected device. Returns statistic for the current device, given by current_device(), if device is None (default).
Note
See Memory management for more details about GPU memory management.
Resets the starting point in tracking maximum GPU memory occupied by tensors for a given device.
See max_memory_allocated() for details.
device (torch.device or int, optional) – selected device. Returns statistic for the current device, given by current_device(), if device is None (default).
Warning
This function now calls reset_peak_memory_stats(), which resets /all/ peak memory stats.
Note
See Memory management for more details about GPU memory management.
Returns the current GPU memory managed by the caching allocator in bytes for a given device.
device (torch.device or int, optional) – selected device. Returns statistic for the current device, given by current_device(), if device is None (default).
Note
See Memory management for more details about GPU memory management.
Returns the maximum GPU memory managed by the caching allocator in bytes for a given device.
By default, this returns the peak cached memory since the beginning of this program. reset_peak_stats() can be used to reset the starting point in tracking this metric. For example, these two functions can measure the peak cached memory amount of each iteration in a training loop.
device (torch.device or int, optional) – selected device. Returns statistic for the current device, given by current_device(), if device is None (default).
Note
See Memory management for more details about GPU memory management.
Set memory fraction for a process. The fraction is used to limit an caching allocator to allocated memory on a CUDA device. The allowed value equals the total visible memory multiplied fraction. If trying to allocate more than the allowed value in a process, will raise an out of memory error in allocator.
Note
In general, the total available free memory is less than the total capacity.
Deprecated; see memory_reserved().
Deprecated; see max_memory_reserved().
Resets the starting point in tracking maximum GPU memory managed by the caching allocator for a given device.
See max_memory_cached() for details.
device (torch.device or int, optional) – selected device. Returns statistic for the current device, given by current_device(), if device is None (default).
Warning
This function now calls reset_peak_memory_stats(), which resets /all/ peak memory stats.
Note
See Memory management for more details about GPU memory management.
Describe an instantaneous event that occurred at some point.
msg (string) – ASCII message to associate with the event.
Pushes a range onto a stack of nested range span. Returns zero-based depth of the range that is started.
msg (string) – ASCII message to associate with range
Pops a range off of a stack of nested range spans. Returns the zero-based depth of the range that is ended.


torch.cuda.amp provides convenience methods for mixed precision, where some operations use the torch.float32 (float) datatype and other operations use torch.float16 (half). Some ops, like linear layers and convolutions, are much faster in float16. Other ops, like reductions, often require the dynamic range of float32. Mixed precision tries to match each op to its appropriate datatype.
Ordinarily, “automatic mixed precision training” uses torch.cuda.amp.autocast and torch.cuda.amp.GradScaler together, as shown in the Automatic Mixed Precision examples and Automatic Mixed Precision recipe. However, autocast and GradScaler are modular, and may be used separately if desired.
Autocast Op Reference
Op-Specific Behavior
Instances of autocast serve as context managers or decorators that allow regions of your script to run in mixed precision.
In these regions, CUDA ops run in an op-specific dtype chosen by autocast to improve performance while maintaining accuracy. See the Autocast Op Reference for details.
When entering an autocast-enabled region, Tensors may be any type. You should not call .half() on your model(s) or inputs when using autocasting.
autocast should wrap only the forward pass(es) of your network, including the loss computation(s). Backward passes under autocast are not recommended. Backward ops run in the same type that autocast used for corresponding forward ops.
Example:
See the Automatic Mixed Precision examples for usage (along with gradient scaling) in more complex scenarios (e.g., gradient penalty, multiple models/losses, custom autograd functions).
autocast can also be used as a decorator, e.g., on the forward method of your model:
Floating-point Tensors produced in an autocast-enabled region may be float16. After returning to an autocast-disabled region, using them with floating-point Tensors of different dtypes may cause type mismatch errors. If so, cast the Tensor(s) produced in the autocast region back to float32 (or other dtype if desired). If a Tensor from the autocast region is already float32, the cast is a no-op, and incurs no additional overhead. Example:
Type mismatch errors in an autocast-enabled region are a bug; if this is what you observe, please file an issue.
autocast(enabled=False) subregions can be nested in autocast-enabled regions. Locally disabling autocast can be useful, for example, if you want to force a subregion to run in a particular dtype. Disabling autocast gives you explicit control over the execution type. In the subregion, inputs from the surrounding region should be cast to dtype before use:
The autocast state is thread-local. If you want it enabled in a new thread, the context manager or decorator must be invoked in that thread. This affects torch.nn.DataParallel and torch.nn.parallel.DistributedDataParallel when used with more than one GPU per process (see Working with Multiple GPUs).
enabled (bool, optional, default=True) – Whether autocasting should be enabled in the region.
Helper decorator for forward methods of custom autograd functions (subclasses of torch.autograd.Function). See the example page for more detail.
cast_inputs (torch.dtype or None, optional, default=None) – If not None, when forward runs in an autocast-enabled region, casts incoming floating-point CUDA Tensors to the target dtype (non-floating-point Tensors are not affected), then executes forward with autocast disabled. If None, forward’s internal ops execute with the current autocast state.
Note
If the decorated forward is called outside an autocast-enabled region, custom_fwd is a no-op and cast_inputs has no effect.
Helper decorator for backward methods of custom autograd functions (subclasses of torch.autograd.Function). Ensures that backward executes with the same autocast state as forward. See the example page for more detail.
If the forward pass for a particular op has float16 inputs, the backward pass for that op will produce float16 gradients. Gradient values with small magnitudes may not be representable in float16. These values will flush to zero (“underflow”), so the update for the corresponding parameters will be lost.
To prevent underflow, “gradient scaling” multiplies the network’s loss(es) by a scale factor and invokes a backward pass on the scaled loss(es). Gradients flowing backward through the network are then scaled by the same factor. In other words, gradient values have a larger magnitude, so they don’t flush to zero.
Each parameter’s gradient (.grad attribute) should be unscaled before the optimizer updates the parameters, so the scale factor does not interfere with the learning rate.
Returns a Python float containing the scale backoff factor.
Returns a Python float containing the scale growth factor.
Returns a Python int containing the growth interval.
Returns a Python float containing the current scale, or 1.0 if scaling is disabled.
Warning
get_scale() incurs a CPU-GPU sync.
Returns a bool indicating whether this instance is enabled.
Loads the scaler state. If this instance is disabled, load_state_dict() is a no-op.
state_dict (dict) – scaler state. Should be an object returned from a call to state_dict().
Multiplies (‘scales’) a tensor or list of tensors by the scale factor.
Returns scaled outputs. If this instance of GradScaler is not enabled, outputs are returned unmodified.
outputs (Tensor or iterable of Tensors) – Outputs to scale.
new_scale (float) – Value to use as the new scale backoff factor.
new_scale (float) – Value to use as the new scale growth factor.
new_interval (int) – Value to use as the new growth interval.
Returns the state of the scaler as a dict. It contains five entries:
If this instance is not enabled, returns an empty dict.
Note
If you wish to checkpoint the scaler’s state after a particular iteration, state_dict() should be called after update().
step() carries out the following two operations:
*args and **kwargs are forwarded to optimizer.step().
Returns the return value of optimizer.step(*args, **kwargs).
Warning
Closure use is not currently supported.
Divides (“unscales”) the optimizer’s gradient tensors by the scale factor.
unscale_() is optional, serving cases where you need to modify or inspect gradients between the backward pass(es) and step(). If unscale_() is not called explicitly, gradients will be unscaled automatically during step().
Simple example, using unscale_() to enable clipping of unscaled gradients:
optimizer (torch.optim.Optimizer) – Optimizer that owns the gradients to be unscaled.
Note
unscale_() does not incur a CPU-GPU sync.
Warning
unscale_() should only be called once per optimizer per step() call, and only after all gradients for that optimizer’s assigned parameters have been accumulated. Calling unscale_() twice for a given optimizer between each step() triggers a RuntimeError.
Warning
unscale_() may unscale sparse gradients out of place, replacing the .grad attribute.
Updates the scale factor.
If any optimizer steps were skipped the scale is multiplied by backoff_factor to reduce it. If growth_interval unskipped iterations occurred consecutively, the scale is multiplied by growth_factor to increase it.
Passing new_scale sets the scale directly.
new_scale (float or torch.cuda.FloatTensor, optional, default=None) – New scale factor.
Warning
update() should only be called at the end of the iteration, after scaler.step(optimizer) has been invoked for all optimizers used this iteration.
Only CUDA ops are eligible for autocasting.
Ops that run in float64 or non-floating-point dtypes are not eligible, and will run in these types whether or not autocast is enabled.
Only out-of-place ops and Tensor methods are eligible. In-place variants and calls that explicitly supply an out=... Tensor are allowed in autocast-enabled regions, but won’t go through autocasting. For example, in an autocast-enabled region a.addmm(b, c) can autocast, but a.addmm_(b, c) and a.addmm(b, c, out=d) cannot. For best performance and stability, prefer out-of-place ops in autocast-enabled regions.
Ops called with an explicit dtype=... argument are not eligible, and will produce output that respects the dtype argument.
The following lists describe the behavior of eligible ops in autocast-enabled regions. These ops always go through autocasting whether they are invoked as part of a torch.nn.Module, as a function, or as a torch.Tensor method. If functions are exposed in multiple namespaces, they go through autocasting regardless of the namespace.
Ops not listed below do not go through autocasting. They run in the type defined by their inputs. However, autocasting may still change the type in which unlisted ops run if they’re downstream from autocasted ops.
If an op is unlisted, we assume it’s numerically stable in float16. If you believe an unlisted op is numerically unstable in float16, please file an issue.
__matmul__, addbmm, addmm, addmv, addr, baddbmm, bmm, chain_matmul, conv1d, conv2d, conv3d, conv_transpose1d, conv_transpose2d, conv_transpose3d, GRUCell, linear, LSTMCell, matmul, mm, mv, prelu, RNNCell
__pow__, __rdiv__, __rpow__, __rtruediv__, acos, asin, binary_cross_entropy_with_logits, cosh, cosine_embedding_loss, cdist, cosine_similarity, cross_entropy, cumprod, cumsum, dist, erfinv, exp, expm1, gelu, group_norm, hinge_embedding_loss, kl_div, l1_loss, layer_norm, log, log_softmax, log10, log1p, log2, margin_ranking_loss, mse_loss, multilabel_margin_loss, multi_margin_loss, nll_loss, norm, normalize, pdist, poisson_nll_loss, pow, prod, reciprocal, rsqrt, sinh, smooth_l1_loss, soft_margin_loss, softmax, softmin, softplus, sum, renorm, tan, triplet_margin_loss
These ops don’t require a particular dtype for stability, but take multiple inputs and require that the inputs’ dtypes match. If all of the inputs are float16, the op runs in float16. If any of the inputs is float32, autocast casts all inputs to float32 and runs the op in float32.
addcdiv, addcmul, atan2, bilinear, cat, cross, dot, equal, index_put, stack, tensordot
Some ops not listed here (e.g., binary ops like add) natively promote inputs without autocasting’s intervention. If inputs are a mixture of float16 and float32, these ops run in float32 and produce float32 output, regardless of whether autocast is enabled.
The backward passes of torch.nn.functional.binary_cross_entropy() (and torch.nn.BCELoss, which wraps it) can produce gradients that aren’t representable in float16. In autocast-enabled regions, the forward input may be float16, which means the backward gradient must be representable in float16 (autocasting float16 forward inputs to float32 doesn’t help, because that cast must be reversed in backward). Therefore, binary_cross_entropy and BCELoss raise an error in autocast-enabled regions.
Many models use a sigmoid layer right before the binary cross entropy layer. In this case, combine the two layers using torch.nn.functional.binary_cross_entropy_with_logits() or torch.nn.BCEWithLogitsLoss. binary_cross_entropy_with_logits and BCEWithLogits are safe to autocast.


Instances of autocast serve as context managers or decorators that allow regions of your script to run in mixed precision.
In these regions, CUDA ops run in an op-specific dtype chosen by autocast to improve performance while maintaining accuracy. See the Autocast Op Reference for details.
When entering an autocast-enabled region, Tensors may be any type. You should not call .half() on your model(s) or inputs when using autocasting.
autocast should wrap only the forward pass(es) of your network, including the loss computation(s). Backward passes under autocast are not recommended. Backward ops run in the same type that autocast used for corresponding forward ops.
Example:
See the Automatic Mixed Precision examples for usage (along with gradient scaling) in more complex scenarios (e.g., gradient penalty, multiple models/losses, custom autograd functions).
autocast can also be used as a decorator, e.g., on the forward method of your model:
Floating-point Tensors produced in an autocast-enabled region may be float16. After returning to an autocast-disabled region, using them with floating-point Tensors of different dtypes may cause type mismatch errors. If so, cast the Tensor(s) produced in the autocast region back to float32 (or other dtype if desired). If a Tensor from the autocast region is already float32, the cast is a no-op, and incurs no additional overhead. Example:
Type mismatch errors in an autocast-enabled region are a bug; if this is what you observe, please file an issue.
autocast(enabled=False) subregions can be nested in autocast-enabled regions. Locally disabling autocast can be useful, for example, if you want to force a subregion to run in a particular dtype. Disabling autocast gives you explicit control over the execution type. In the subregion, inputs from the surrounding region should be cast to dtype before use:
The autocast state is thread-local. If you want it enabled in a new thread, the context manager or decorator must be invoked in that thread. This affects torch.nn.DataParallel and torch.nn.parallel.DistributedDataParallel when used with more than one GPU per process (see Working with Multiple GPUs).
enabled (bool, optional, default=True) – Whether autocasting should be enabled in the region.


Helper decorator for backward methods of custom autograd functions (subclasses of torch.autograd.Function). Ensures that backward executes with the same autocast state as forward. See the example page for more detail.


Helper decorator for forward methods of custom autograd functions (subclasses of torch.autograd.Function). See the example page for more detail.
cast_inputs (torch.dtype or None, optional, default=None) – If not None, when forward runs in an autocast-enabled region, casts incoming floating-point CUDA Tensors to the target dtype (non-floating-point Tensors are not affected), then executes forward with autocast disabled. If None, forward’s internal ops execute with the current autocast state.
Note
If the decorated forward is called outside an autocast-enabled region, custom_fwd is a no-op and cast_inputs has no effect.


Returns a Python float containing the scale backoff factor.
Returns a Python float containing the scale growth factor.
Returns a Python int containing the growth interval.
Returns a Python float containing the current scale, or 1.0 if scaling is disabled.
Warning
get_scale() incurs a CPU-GPU sync.
Returns a bool indicating whether this instance is enabled.
Loads the scaler state. If this instance is disabled, load_state_dict() is a no-op.
state_dict (dict) – scaler state. Should be an object returned from a call to state_dict().
Multiplies (‘scales’) a tensor or list of tensors by the scale factor.
Returns scaled outputs. If this instance of GradScaler is not enabled, outputs are returned unmodified.
outputs (Tensor or iterable of Tensors) – Outputs to scale.
new_scale (float) – Value to use as the new scale backoff factor.
new_scale (float) – Value to use as the new scale growth factor.
new_interval (int) – Value to use as the new growth interval.
Returns the state of the scaler as a dict. It contains five entries:
If this instance is not enabled, returns an empty dict.
Note
If you wish to checkpoint the scaler’s state after a particular iteration, state_dict() should be called after update().
step() carries out the following two operations:
*args and **kwargs are forwarded to optimizer.step().
Returns the return value of optimizer.step(*args, **kwargs).
Warning
Closure use is not currently supported.
Divides (“unscales”) the optimizer’s gradient tensors by the scale factor.
unscale_() is optional, serving cases where you need to modify or inspect gradients between the backward pass(es) and step(). If unscale_() is not called explicitly, gradients will be unscaled automatically during step().
Simple example, using unscale_() to enable clipping of unscaled gradients:
optimizer (torch.optim.Optimizer) – Optimizer that owns the gradients to be unscaled.
Note
unscale_() does not incur a CPU-GPU sync.
Warning
unscale_() should only be called once per optimizer per step() call, and only after all gradients for that optimizer’s assigned parameters have been accumulated. Calling unscale_() twice for a given optimizer between each step() triggers a RuntimeError.
Warning
unscale_() may unscale sparse gradients out of place, replacing the .grad attribute.
Updates the scale factor.
If any optimizer steps were skipped the scale is multiplied by backoff_factor to reduce it. If growth_interval unskipped iterations occurred consecutively, the scale is multiplied by growth_factor to increase it.
Passing new_scale sets the scale directly.
new_scale (float or torch.cuda.FloatTensor, optional, default=None) – New scale factor.
Warning
update() should only be called at the end of the iteration, after scaler.step(optimizer) has been invoked for all optimizers used this iteration.


Returns a Python float containing the scale backoff factor.


Returns a Python float containing the scale growth factor.


Returns a Python int containing the growth interval.


Returns a Python float containing the current scale, or 1.0 if scaling is disabled.
Warning
get_scale() incurs a CPU-GPU sync.


Returns a bool indicating whether this instance is enabled.


Loads the scaler state. If this instance is disabled, load_state_dict() is a no-op.
state_dict (dict) – scaler state. Should be an object returned from a call to state_dict().


Multiplies (‘scales’) a tensor or list of tensors by the scale factor.
Returns scaled outputs. If this instance of GradScaler is not enabled, outputs are returned unmodified.
outputs (Tensor or iterable of Tensors) – Outputs to scale.


new_scale (float) – Value to use as the new scale backoff factor.


new_scale (float) – Value to use as the new scale growth factor.


new_interval (int) – Value to use as the new growth interval.


Returns the state of the scaler as a dict. It contains five entries:
If this instance is not enabled, returns an empty dict.
Note
If you wish to checkpoint the scaler’s state after a particular iteration, state_dict() should be called after update().


step() carries out the following two operations:
*args and **kwargs are forwarded to optimizer.step().
Returns the return value of optimizer.step(*args, **kwargs).
Warning
Closure use is not currently supported.


Divides (“unscales”) the optimizer’s gradient tensors by the scale factor.
unscale_() is optional, serving cases where you need to modify or inspect gradients between the backward pass(es) and step(). If unscale_() is not called explicitly, gradients will be unscaled automatically during step().
Simple example, using unscale_() to enable clipping of unscaled gradients:
optimizer (torch.optim.Optimizer) – Optimizer that owns the gradients to be unscaled.
Note
unscale_() does not incur a CPU-GPU sync.
Warning
unscale_() should only be called once per optimizer per step() call, and only after all gradients for that optimizer’s assigned parameters have been accumulated. Calling unscale_() twice for a given optimizer between each step() triggers a RuntimeError.
Warning
unscale_() may unscale sparse gradients out of place, replacing the .grad attribute.


Updates the scale factor.
If any optimizer steps were skipped the scale is multiplied by backoff_factor to reduce it. If growth_interval unskipped iterations occurred consecutively, the scale is multiplied by growth_factor to increase it.
Passing new_scale sets the scale directly.
new_scale (float or torch.cuda.FloatTensor, optional, default=None) – New scale factor.
Warning
update() should only be called at the end of the iteration, after scaler.step(optimizer) has been invoked for all optimizers used this iteration.


Checks if peer access between two devices is possible.


Broadcasts a tensor to specified GPU devices.
Note
Exactly one of devices and out must be specified.
a tuple containing copies of tensor, placed on devices.
a tuple containing out tensors, each containing a copy of tensor.


Broadcasts a sequence tensors to the specified GPUs. Small tensors are first coalesced into a buffer to reduce the number of synchronizations.
A tuple containing copies of tensor, placed on devices.


Gathers tensors from multiple GPU devices.
Note
destination must not be specified when out is specified.
a tensor located on destination device, that is a result of concatenating tensors along dim.
the out tensor, now containing results of concatenating tensors along dim.


Sums tensors from multiple GPUs.
All inputs should have matching shapes, dtype, and layout. The output tensor will be of the same shape, dtype, and layout.
A tensor containing an elementwise sum of all inputs, placed on the destination device.