torch.Tensor.is_leafTensor.is_leaf
   All Tensors that have "requires_grad" which is "False" will be leaf
   Tensors by convention.
   For Tensors that have "requires_grad" which is "True", they will be
   leaf Tensors if they were created by the user. This means that they
   are not the result of an operation and so "grad_fn" is None.
   Only leaf Tensors will have their "grad" populated during a call to
   "backward()". To get "grad" populated for non-leaf Tensors, you can
   use "retain_grad()".
   Example:
      >>> a = torch.rand(10, requires_grad=True)
      >>> a.is_leaf
      True
      >>> b = torch.rand(10, requires_grad=True).cuda()
      >>> b.is_leaf
      False
      # b was created by the operation that cast a cpu Tensor into a cuda Tensor
      >>> c = torch.rand(10, requires_grad=True) + 2
      >>> c.is_leaf
      False
      # c was created by the addition operation
      >>> d = torch.rand(10).cuda()
      >>> d.is_leaf
      True


d.is_leaf
      True
      # d does not require gradients and so has no operation creating it (that is tracked by the autograd engine)
      >>> e = torch.rand(10).cuda().requires_grad_()
      >>> e.is_leaf
      True
      # e requires gradients and has no operations creating it
      >>> f = torch.rand(10, requires_grad=True, device="cuda")
      >>> f.is_leaf
      True
      # f requires grad, has no operation creating it


torch.jit.waittorch.jit.wait(future)
   Forces completion of a torch.jit.Future[T] asynchronous task,
   returning the result of the task. See "fork()" for docs and
   examples. :param future: an asynchronous task reference, created
   through torch.jit.fork :type future: torch.jit.Future[T]
   Returns:
      the return value of the the completed task
   Return type:
      T


torch.Tensor.scatter_addTensor.scatter_add(dim, index, src) -> Tensor
   Out-of-place version of "torch.Tensor.scatter_add_()"


torch.Tensor.reshapeTensor.reshape(shape) -> Tensor
   Returns a tensor with the same data and number of elements as
   "self" but with the specified shape. This method returns a view if
   "shape" is compatible with the current shape. See
   "torch.Tensor.view()" on when it is possible to return a view.
   See "torch.reshape()"
   Parameters:
      shape (tuple of ints or int*...) -- the desired shape


ObserverBaseclass torch.quantization.observer.ObserverBase(dtype)
   Base observer Module. Any observer implementation should derive
   from this class.
   Concrete observers should follow the same API. In forward, they
   will update the statistics of the observed Tensor. And they should
   provide a calculate_qparams function that computes the
   quantization parameters given the collected statistics.
   Parameters:
      dtype -- dtype argument to the quantize node needed to
      implement the reference model spec.
   classmethod with_args(**kwargs)
      Wrapper that allows creation of class factories.
      This can be useful when there is a need to create classes with
      the same constructor arguments, but different instances. Can be
      used in conjunction with _callable_args
      Example:
         >>> Foo.with_args = classmethod(_with_args)
         >>> foo_builder = Foo.with_args(a=3, b=4).with_args(answer=42)
         >>> foo_instance1 = foo_builder()


foo_instance1 = foo_builder()
         >>> foo_instance2 = foo_builder()
         >>> id(foo_instance1) == id(foo_instance2)
         False
   classmethod with_callable_args(**kwargs)
      Wrapper that allows creation of class factories args that need
      to be called at construction time.
      This can be useful when there is a need to create classes with
      the same constructor arguments, but different instances and
      those arguments should only be calculated at construction time.
      Can be used in conjunction with _with_args
      Example:
         >>> Foo.with_callable_args = classmethod(_with_callable_args)
         >>> Foo.with_args = classmethod(_with_args)
         >>> foo_builder = Foo.with_callable_args(cur_time=get_time_func).with_args(name="dan")
         >>> foo_instance1 = foo_builder()
         >>> # wait 50
         >>> foo_instance2 = foo_builder()
         >>> id(foo_instance1.creation_time) == id(foo_instance2.creation_time)
         False


torch.Tensor.igamma_Tensor.igamma_(other) -> Tensor
   In-place version of "igamma()"


torch.Tensor.log10Tensor.log10() -> Tensor
   See "torch.log10()"


torch.cuda.can_device_access_peertorch.cuda.can_device_access_peer(device, peer_device)
   Checks if peer access between two devices is possible.
   Return type:
      bool


torch.linalg.dettorch.linalg.det(A, , out=None) -> Tensor
   Computes the determinant of a square matrix.
   Supports input of float, double, cfloat and cdouble dtypes. Also
   supports batches of matrices, and if "A" is a batch of matrices
   then the output has the same batch dimensions.
   See also:
     "torch.linalg.slogdet()" computes the sign and natural logarithm
     of the absolute value of the determinant of square matrices.
   Parameters:
      A (Tensor) -- tensor of shape (, n, n) where *** is
      zero or more batch dimensions.
   Keyword Arguments:
      out (Tensor, optional) -- output tensor. Ignored if
      None. Default: None.
   Examples:
      >>> A = torch.randn(3, 3)
      >>> torch.linalg.det(A)
      tensor(0.0934)
      >>> A = torch.randn(3, 2, 2)
      >>> torch.linalg.det(A)
      tensor([1.1990, 0.4099, 0.7386])


TripletMarginWithDistanceLossclass torch.nn.TripletMarginWithDistanceLoss(*, distance_function=None, margin=1.0, swap=False, reduction='mean')
   Creates a criterion that measures the triplet loss given input
   tensors a, p, and n (representing anchor, positive, and negative
   examples, respectively), and a nonnegative, real-valued function
   ("distance function") used to compute the relationship between the
   anchor and positive example ("positive distance") and the anchor
   and negative example ("negative distance").
   The unreduced loss (i.e., with "reduction" set to "'none'") can be
   described as:
      \ell(a, p, n) = L = {l_1,\dots,l_N}^\top, \quad l_i = \max
      {d(a_i, p_i) - d(a_i, n_i) + {\rm margin}, 0}
   where N is the batch size; d is a nonnegative, real-valued function
   quantifying the closeness of two tensors, referred to as the
   "distance_function"; and margin is a nonnegative margin


representing the minimum difference between the positive and
   negative distances that is required for the loss to be 0.  The
   input tensors have N elements each and can be of any shape that the
   distance function can handle.
   If "reduction" is not "'none'" (default "'mean'"), then:
      \ell(x, y) = \begin{cases}     \operatorname{mean}(L), &
      \text{if reduction} = \text{mean';}\\
      \operatorname{sum}(L),  &  \text{if reduction} = \text{sum'.}
      \end{cases}
   See also "TripletMarginLoss", which computes the triplet loss for
   input tensors using the l_p distance as the distance function.
   Parameters:
      * distance_function (Callable, optional) -- A
        nonnegative, real-valued function that quantifies the
        closeness of two tensors. If not specified,
        nn.PairwiseDistance will be used.  Default: "None"
      * margin (float, optional) -- A nonnegative margin
        representing the minimum difference between the positive and


negative distances required for the loss to be 0. Larger
        margins penalize cases where the negative examples are not
        distant enough from the anchors, relative to the positives.
        Default: 1.
      * swap (bool, optional) -- Whether to use the distance
        swap described in the paper Learning shallow convolutional
        feature descriptors with triplet losses by V. Balntas, E.
        Riba et al. If True, and if the positive example is closer to
        the negative example than the anchor is, swaps the positive
        example and the anchor in the loss computation. Default:
        "False".
      * reduction (str, optional) -- Specifies the
        (optional) reduction to apply to the output: "'none'" |
        "'mean'" | "'sum'". "'none'": no reduction will be applied,
        "'mean'": the sum of the output will be divided by the number
        of elements in the output, "'sum'": the output will be summed.
        Default: "'mean'"
   Shape:


Default: "'mean'"
   Shape:
      * Input: (N, *) where * represents any number of additional
        dimensions as supported by the distance function.
      * Output: A Tensor of shape (N) if "reduction" is "'none'", or a
        scalar otherwise.
   Examples:
      >>> # Initialize embeddings
      >>> embedding = nn.Embedding(1000, 128)
      >>> anchor_ids = torch.randint(0, 1000, (1,))
      >>> positive_ids = torch.randint(0, 1000, (1,))
      >>> negative_ids = torch.randint(0, 1000, (1,))
      >>> anchor = embedding(anchor_ids)
      >>> positive = embedding(positive_ids)
      >>> negative = embedding(negative_ids)
      >>>
      >>> # Built-in Distance Function
      >>> triplet_loss = \
      >>>     nn.TripletMarginWithDistanceLoss(distance_function=nn.PairwiseDistance())
      >>> output = triplet_loss(anchor, positive, negative)
      >>> output.backward()
      >>>
      >>> # Custom Distance Function
      >>> def l_infinity(x1, x2):


def l_infinity(x1, x2):
      >>>     return torch.max(torch.abs(x1 - x2), dim=1).values
      >>>
      >>> triplet_loss = (
      >>>     nn.TripletMarginWithDistanceLoss(distance_function=l_infinity, margin=1.5))
      >>> output = triplet_loss(anchor, positive, negative)
      >>> output.backward()
      >>>
      >>> # Custom Distance Function (Lambda)
      >>> triplet_loss = (
      >>>     nn.TripletMarginWithDistanceLoss(
      >>>         distance_function=lambda x, y: 1.0 - F.cosine_similarity(x, y)))
      >>> output = triplet_loss(anchor, positive, negative)
      >>> output.backward()
   Reference:
      V. Balntas, et al.: Learning shallow convolutional feature
      descriptors with triplet losses:
      http://www.bmva.org/bmvc/2016/papers/paper119/index.html


torch.rot90torch.rot90(input, k=1, dims=[0, 1]) -> Tensor
   Rotate an n-D tensor by 90 degrees in the plane specified by dims
   axis. Rotation direction is from the first towards the second axis
   if k > 0, and from the second towards the first for k < 0.
   Parameters:
      * input (Tensor) -- the input tensor.
      * k (int) -- number of times to rotate. Default value is 1
      * dims (a list or tuple) -- axis to rotate. Default
        value is [0, 1]
   Example:
      >>> x = torch.arange(4).view(2, 2)
      >>> x
      tensor([[0, 1],
              [2, 3]])
      >>> torch.rot90(x, 1, [0, 1])
      tensor([[1, 3],
              [0, 2]])
      >>> x = torch.arange(8).view(2, 2, 2)
      >>> x
      tensor([[[0, 1],
               [2, 3]],
              [[4, 5],
               [6, 7]]])
      >>> torch.rot90(x, 1, [1, 2])
      tensor([[[1, 3],
               [0, 2]],
              [[5, 7],
               [4, 6]]])


torch.nn.utils.prune.global_unstructuredtorch.nn.utils.prune.global_unstructured(parameters, pruning_method, importance_scores=None, kwargs)
   Globally prunes tensors corresponding to all parameters in
   "parameters" by applying the specified "pruning_method". Modifies
   modules in place by:
   1. adding a named buffer called "name+'_mask'" corresponding to the
      binary mask applied to the parameter "name" by the pruning
      method.
   2. replacing the parameter "name" by its pruned version, while the
      original (unpruned) parameter is stored in a new parameter named
      "name+'_orig'".
   Parameters:
      * parameters (*Iterable of (module, name)
        *tuples) -- parameters of the model to prune in a global
        fashion, i.e. by aggregating all weights prior to deciding
        which ones to prune. module must be of type "nn.Module", and
        name must be a string.


name must be a string.
      * pruning_method (function) -- a valid pruning function
        from this module, or a custom one implemented by the user that
        satisfies the implementation guidelines and has
        "PRUNING_TYPE='unstructured'".
      * importance_scores (dict) -- a dictionary mapping
        (module, name) tuples to the corresponding parameter's
        importance scores tensor. The tensor should be the same shape
        as the parameter, and is used for computing mask for pruning.
        If unspecified or None, the parameter will be used in place of
        its importance scores.
      * kwargs -- other keyword arguments such as: amount (int or
        float): quantity of parameters to prune across the specified
        parameters. If "float", should be between 0.0 and 1.0 and
        represent the fraction of parameters to prune. If "int", it
        represents the absolute number of parameters to prune.
   Raises:


Raises:
      TypeError -- if "PRUNING_TYPE != 'unstructured'"
   Note:
     Since global structured pruning doesn't make much sense unless
     the norm is normalized by the size of the parameter, we now limit
     the scope of global pruning to unstructured methods.
   -[ Examples ]-

from torch.nn.utils import prune
from collections import OrderedDict
net = nn.Sequential(OrderedDict([
   ...     ('first', nn.Linear(10, 4)),
   ...     ('second', nn.Linear(4, 1)),
   ... ]))
parameters_to_prune = (
   ...     (net.first, 'weight'),
   ...     (net.second, 'weight'),
   ... )
prune.global_unstructured(
   ...     parameters_to_prune,
   ...     pruning_method=prune.L1Unstructured,
   ...     amount=10,
   ... )
print(sum(torch.nn.utils.parameters_to_vector(net.buffers()) == 0))
   tensor(10)


TransformerDecoderLayerclass torch.nn.TransformerDecoderLayer(d_model, nhead, dim_feedforward=2048, dropout=0.1, activation=, layer_norm_eps=1e-05, batch_first=False, norm_first=False, device=None, dtype=None)
   TransformerDecoderLayer is made up of self-attn, multi-head-attn
   and feedforward network. This standard decoder layer is based on
   the paper "Attention Is All You Need". Ashish Vaswani, Noam
   Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
   Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you
   need. In Advances in Neural Information Processing Systems, pages
   6000-6010. Users may modify or implement in a different way during
   application.
   Parameters:
      * d_model (int) -- the number of expected features in the
        input (required).
      * nhead (int) -- the number of heads in the
        multiheadattention models (required).
      * dim_feedforward (int) -- the dimension of the


feedforward network model (default=2048).
      * dropout (float) -- the dropout value (default=0.1).
      * activation (Union[str,
        Callable[[Tensor], Tensor]]) -- the
        activation function of the intermediate layer, can be a string
        ("relu" or "gelu") or a unary callable. Default: relu
      * layer_norm_eps (float) -- the eps value in layer
        normalization components (default=1e-5).
      * batch_first (bool) -- If "True", then the input and
        output tensors are provided as (batch, seq, feature). Default:
        "False" (seq, batch, feature).
      * norm_first (bool) -- if "True", layer norm is done prior
        to self attention, multihead attention and feedforward
        operations, respectively. Otherwise it's done after. Default:
        "False" (after).
   Examples::
      >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
      >>> memory = torch.rand(10, 32, 512)


memory = torch.rand(10, 32, 512)
      >>> tgt = torch.rand(20, 32, 512)
      >>> out = decoder_layer(tgt, memory)
   Alternatively, when "batch_first" is "True":
      >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8, batch_first=True)
      >>> memory = torch.rand(32, 10, 512)
      >>> tgt = torch.rand(32, 20, 512)
      >>> out = decoder_layer(tgt, memory)
   forward(tgt, memory, tgt_mask=None, memory_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None, tgt_is_causal=False, memory_is_causal=False)
      Pass the inputs (and mask) through the decoder layer.
      Parameters:
         * tgt (Tensor) -- the sequence to the decoder layer
           (required).
         * memory (Tensor) -- the sequence from the last layer
           of the encoder (required).
         * tgt_mask (Optional[Tensor]) -- the mask for the
           tgt sequence (optional).
         * memory_mask (Optional[Tensor]) -- the mask for


the memory sequence (optional).
         * tgt_key_padding_mask (Optional[Tensor]) -- the
           mask for the tgt keys per batch (optional).
         * memory_key_padding_mask (Optional[Tensor]) --
           the mask for the memory keys per batch (optional).
         * tgt_is_causal (bool) -- If specified, applies a
           causal mask as tgt mask. Mutually exclusive with providing
           tgt_mask. Default: "False".
         * memory_is_causal (bool) -- If specified, applies a
           causal mask as tgt mask. Mutually exclusive with providing
           memory_mask. Default: "False".
      Return type:
         Tensor
      Shape:
         see the docs in Transformer class.


torch.randint_liketorch.randint_like(input, low=0, high, *, dtype=None, layout=torch.strided, device=None, requires_grad=False, memory_format=torch.preserve_format) -> Tensor
   Returns a tensor with the same shape as Tensor "input" filled with
   random integers generated uniformly between "low" (inclusive) and
   "high" (exclusive).
   Parameters:
      * input (Tensor) -- the size of "input" will determine
        size of the output tensor.
      * low (int, optional) -- Lowest integer to be drawn
        from the distribution. Default: 0.
      * high (int) -- One above the highest integer to be drawn
        from the distribution.
   Keyword Arguments:
      * dtype ("torch.dtype", optional) -- the desired data type
        of returned Tensor. Default: if "None", defaults to the dtype
        of "input".
      * layout ("torch.layout", optional) -- the desired layout of
        returned tensor. Default: if "None", defaults to the layout of


"input".
      * device ("torch.device", optional) -- the desired device of
        returned tensor. Default: if "None", defaults to the device of
        "input".
      * requires_grad (bool, optional) -- If autograd should
        record operations on the returned tensor. Default: "False".
      * memory_format ("torch.memory_format", optional) -- the
        desired memory format of returned Tensor. Default:
        "torch.preserve_format".


torch.Tensor.masked_selectTensor.masked_select(mask) -> Tensor
   See "torch.masked_select()"


torch.Tensor.bernoulliTensor.bernoulli(*, generator=None) -> Tensor
   Returns a result tensor where each \texttt{result[i]} is
   independently sampled from \text{Bernoulli}(\texttt{self[i]}).
   "self" must have floating point "dtype", and the result will have
   the same "dtype".
   See "torch.bernoulli()"


torch.fft.fftfreqtorch.fft.fftfreq(n, d=1.0, , out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) -> Tensor
   Computes the discrete Fourier Transform sample frequencies for a
   signal of size "n".
   Note:
     By convention, "fft()" returns positive frequency terms first,
     followed by the negative frequencies in reverse order, so that
     "f[-i]" for all 0 < i \leq n/2` in Python gives the negative
     frequency terms. For an FFT of length "n" and with inputs spaced
     in length unit "d", the frequencies are:
        f = [0, 1, ..., (n - 1) // 2, -(n // 2), ..., -1] / (d * n)
   Note:
     For even lengths, the Nyquist frequency at "f[n/2]" can be
     thought of as either negative or positive. "fftfreq()" follows
     NumPy's convention of taking it to be negative.
   Parameters:
      * n (int) -- the FFT length
      * d (float, optional*) -- The sampling length scale.


The spacing between individual samples of the FFT input. The
        default assumes unit spacing, dividing that result by the
        actual spacing gives the result in physical frequency units.
   Keyword Arguments:
      * out (Tensor, optional) -- the output tensor.
      * dtype ("torch.dtype", optional) -- the desired data type
        of returned tensor. Default: if "None", uses a global default
        (see "torch.set_default_tensor_type()").
      * layout ("torch.layout", optional) -- the desired layout of
        returned Tensor. Default: "torch.strided".
      * device ("torch.device", optional) -- the desired device of
        returned tensor. Default: if "None", uses the current device
        for the default tensor type (see
        "torch.set_default_tensor_type()"). "device" will be the CPU
        for CPU tensor types and the current CUDA device for CUDA
        tensor types.
      * requires_grad (bool, optional) -- If autograd should


record operations on the returned tensor. Default: "False".
   -[ Example ]-

torch.fft.fftfreq(5)
   tensor([ 0.0000,  0.2000,  0.4000, -0.4000, -0.2000])
   For even input, we can see the Nyquist frequency at "f[2]" is given
   as negative:
torch.fft.fftfreq(4)
   tensor([ 0.0000,  0.2500, -0.5000, -0.2500])


torch.Tensor.broadcast_toTensor.broadcast_to(shape) -> Tensor
   See "torch.broadcast_to()".


torch.cuda.nvtx.range_pushtorch.cuda.nvtx.range_push(msg)
   Pushes a range onto a stack of nested range span.  Returns zero-
   based depth of the range that is started.
   Parameters:
      msg (str) -- ASCII message to associate with range


GELUclass torch.nn.GELU(approximate='none')
   Applies the Gaussian Error Linear Units function:
      \text{GELU}(x) = x * \Phi(x)
   where \Phi(x) is the Cumulative Distribution Function for Gaussian
   Distribution.
   When the approximate argument is 'tanh', Gelu is estimated with:
      \text{GELU}(x) = 0.5 * x * (1 + \text{Tanh}(\sqrt(2 / \pi) * (x
      + 0.044715 * x^3)))
   Parameters:
      approximate (str, optional) -- the gelu approximation
      algorithm to use: "'none'" | "'tanh'". Default: "'none'"
   Shape:
      * Input: (), where * means any number of dimensions.
      * Output: (), same shape as the input.
   [image]
   Examples:
      >>> m = nn.GELU()
      >>> input = torch.randn(2)
      >>> output = m(input)


torch.func.functionalizetorch.func.functionalize(func, , remove='mutations')
   functionalize is a transform that can be used to remove
   (intermediate) mutations and aliasing from a function, while
   preserving the function's semantics.
   "functionalize(func)" returns a new function with the same
   semantics as "func", but with all intermediate mutations removed.
   Every inplace operation performed on an intermediate tensor:
   "intermediate.foo_()" gets replaced by its out-of-place equivalent:
   "intermediate_updated = intermediate.foo()".
   functionalize is useful for shipping a pytorch program off to
   backends or compilers that aren't able to easily represent
   mutations or aliasing operators.
   Parameters:
      * func (Callable) -- A Python function that takes one or
        more arguments.
      * remove (str*) -- An optional string argument, that takes
        on either the value 'mutations' or 'mutations_and_views'. If


'mutations' is passed in then all mutating operators will be
        replaced with their non-mutating equivalents. If
        'mutations_and_views' is passed in, then additionally, all
        aliasing operators will be replaced with their non-aliasing
        equivalents. Default: 'mutations'.
   Returns:
      Returns a new "functionalized" function. It takes the same
      inputs as "func", and has the same behavior, but any mutations
      (and optionally aliasing) performed on intermeidate tensors in
      the function will be removed.
   Return type:
      Callable
   functionalize will also remove mutations (and views) that were
   performed on function inputs. However to preserve semantics,
   functionalize will "fix up" the mutations after the transform has
   finished running, by detecting if any tensor inputs "should have"
   been mutated, and copying the new data back to the inputs if
   necessary.
   Example:
      >>> import torch


necessary.
   Example:
      >>> import torch
      >>> from torch.fx.experimental.proxy_tensor import make_fx
      >>> from torch.func import functionalize
      >>>
      >>> # A function that uses mutations and views, but only on intermediate tensors.
      >>> def f(a):
      ...     b = a + 1
      ...     c = b.view(-1)
      ...     c.add_(1)
      ...     return b
      ...
      >>> inpt = torch.randn(2)
      >>>
      >>> out1 = f(inpt)
      >>> out2 = functionalize(f)(inpt)
      >>>
      >>> # semantics are the same (outputs are equivalent)
      >>> print(torch.allclose(out1, out2))
      True
      >>>
      >>> f_traced = make_fx(f)(inpt)
      >>> f_no_mutations_traced = make_fx(functionalize(f))(inpt)
      >>> f_no_mutations_and_views_traced = make_fx(functionalize(f, remove='mutations_and_views'))(inpt)
      >>>
      >>> print(f_traced.code)
      def forward(self, a_1):
          add = torch.ops.aten.add(a_1, 1);  a_1 = None
          view = torch.ops.aten.view(add, [-1])


view = torch.ops.aten.view(add, [-1])
          add_ = torch.ops.aten.add_(view, 1);  view = None
          return add
      >>> print(f_no_mutations_traced.code)
      def forward(self, a_1):
          add = torch.ops.aten.add(a_1, 1);  a_1 = None
          view = torch.ops.aten.view(add, [-1]);  add = None
          add_1 = torch.ops.aten.add(view, 1);  view = None
          view_1 = torch.ops.aten.view(add_1, [2]);  add_1 = None
          return view_1
      >>> print(f_no_mutations_and_views_traced.code)
      def forward(self, a_1):
          add = torch.ops.aten.add(a_1, 1);  a_1 = None
          view_copy = torch.ops.aten.view_copy(add, [-1]);  add = None
          add_1 = torch.ops.aten.add(view_copy, 1);  view_copy = None
          view_copy_1 = torch.ops.aten.view_copy(add_1, [2]);  add_1 = None
          return view_copy_1
      >>> # A function that mutates its input tensor
      >>> def f(a):
      ...     b = a.view(-1)
      ...     b.add_(1)
      ...     return a
      ...


...     return a
      ...
      >>> f_no_mutations_and_views_traced = make_fx(functionalize(f, remove='mutations_and_views'))(inpt)
      >>> #
      >>> # All mutations and views have been removed,
      >>> # but there is an extra copy_ in the graph to correctly apply the mutation to the input
      >>> # after the function has completed.
      >>> print(f_no_mutations_and_views_traced.code)
      def forward(self, a_1):
          view_copy = torch.ops.aten.view_copy(a_1, [-1])
          add = torch.ops.aten.add(view_copy, 1);  view_copy = None
          view_copy_1 = torch.ops.aten.view_copy(add, [2]);  add = None
          copy_ = torch.ops.aten.copy_(a_1, view_copy_1);  a_1 = None
          return view_copy_1
   There are a few "failure modes" for functionalize that are worth
   calling out:
      1. Like other torch.func transforms, functionalize() doesn't
         work with functions that directly use .backward(). The same


is true for torch.autograd.grad. If you want to use autograd,
         you can compute gradients directly with
         functionalize(grad(f)).
      2. Like other torch.func transforms, functionalize() doesn't
         work with global state. If you call functionalize(f) on a
         function that takes views / mutations of non-local state,
         functionalization will simply no-op and pass the
         view/mutation calls directly to the backend. One way to work
         around this is is to ensure that any non-local state creation
         is wrapped into a larger function, which you then call
         functionalize on.
      3. resize_() has some limitations: functionalize will only
         work on programs that use resize_()` as long as the tensor
         being resized is not a view.
      4. as_strided() has some limitations: functionalize will not
         work on as_strided() calls that result in tensors with
         overlapping memory.


overlapping memory.
   Finally, a helpful mental model for understanding functionalization
   is that most user pytorch programs are writting with the public
   torch API. When executed, torch operators are generally decomposed
   into our internal C++ "ATen" API. The logic for functionalization
   happens entirely at the level of ATen. Functionalization knows how
   to take every aliasing operator in ATen, and map it to its non-
   aliasing equivalent (e.g. "tensor.view({-1})" ->
   "at::view_copy(tensor, {-1})"), and how to take every mutating
   operator in ATen, and map it to its non-mutating equivalent (e.g.
   "tensor.add_(1)" -> "at::add(tensor, -1)"), while tracking aliases
   and mutations out-of-line to know when to fix things up.
   Information about which ATen operators are aliasing or mutating all
   comes from https://github.com/pytorch/pytorch/blob/master/aten/src
   /ATen/native/native_functions.yaml.


torch.bernoullitorch.bernoulli(input, , generator=None, out=None) -> Tensor
   Draws binary random numbers (0 or 1) from a Bernoulli distribution.
   The "input" tensor should be a tensor containing probabilities to
   be used for drawing the binary random number. Hence, all values in
   "input" have to be in the range: 0 \leq \text{input}i \leq 1.
   The \text{i}^{th} element of the output tensor will draw a value 1
   according to the \text{i}^{th} probability value given in "input".
      \text{out} \sim \mathrm{Bernoulli}(p = \text{input}_{i})
   The returned "out" tensor only has values 0 or 1 and is of the same
   shape as "input".
   "out" can have integral "dtype", but "input" must have floating
   point "dtype".
   Parameters:
      input (Tensor) -- the input tensor of probability values
      for the Bernoulli distribution
   Keyword Arguments:
      * generator* ("torch.Generator", optional) -- a pseudorandom
        number generator for sampling


number generator for sampling
      * out (Tensor, optional) -- the output tensor.
   Example:
      >>> a = torch.empty(3, 3).uniform_(0, 1)  # generate a uniform random matrix with range [0, 1]
      >>> a
      tensor([[ 0.1737,  0.0950,  0.3609],
              [ 0.7148,  0.0289,  0.2676],
              [ 0.9456,  0.8937,  0.7202]])
      >>> torch.bernoulli(a)
      tensor([[ 1.,  0.,  0.],
              [ 0.,  0.,  0.],
              [ 1.,  1.,  1.]])
      >>> a = torch.ones(3, 3) # probability of drawing "1" is 1
      >>> torch.bernoulli(a)
      tensor([[ 1.,  1.,  1.],
              [ 1.,  1.,  1.],
              [ 1.,  1.,  1.]])
      >>> a = torch.zeros(3, 3) # probability of drawing "1" is 0
      >>> torch.bernoulli(a)
      tensor([[ 0.,  0.,  0.],
              [ 0.,  0.,  0.],
              [ 0.,  0.,  0.]])


torch.minimumtorch.minimum(input, other, , out=None) -> Tensor
   Computes the element-wise minimum of "input" and "other".
   Note:
     If one of the elements being compared is a NaN, then that element
     is returned. "minimum()" is not supported for tensors with
     complex dtypes.
   Parameters:
      * input (Tensor) -- the input tensor.
      * other (Tensor) -- the second input tensor
   Keyword Arguments:
      out (Tensor, optional*) -- the output tensor.
   Example:
      >>> a = torch.tensor((1, 2, -1))
      >>> b = torch.tensor((3, 0, 4))
      >>> torch.minimum(a, b)
      tensor([1, 0, -1])


torch.logical_andtorch.logical_and(input, other, , out=None) -> Tensor
   Computes the element-wise logical AND of the given input tensors.
   Zeros are treated as "False" and nonzeros are treated as "True".
   Parameters:
      * input (Tensor) -- the input tensor.
      * other (Tensor) -- the tensor to compute AND with
   Keyword Arguments:
      out (Tensor, optional*) -- the output tensor.
   Example:
      >>> torch.logical_and(torch.tensor([True, False, True]), torch.tensor([True, False, False]))
      tensor([ True, False, False])
      >>> a = torch.tensor([0, 1, 10, 0], dtype=torch.int8)
      >>> b = torch.tensor([4, 0, 1, 0], dtype=torch.int8)
      >>> torch.logical_and(a, b)
      tensor([False, False,  True, False])
      >>> torch.logical_and(a.double(), b.double())
      tensor([False, False,  True, False])
      >>> torch.logical_and(a.double(), b)
      tensor([False, False,  True, False])


tensor([False, False,  True, False])
      >>> torch.logical_and(a, b, out=torch.empty(4, dtype=torch.bool))
      tensor([False, False,  True, False])


CELUclass torch.nn.CELU(alpha=1.0, inplace=False)
   Applies the element-wise function:
      \text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) -
      1))
   More details can be found in the paper Continuously Differentiable
   Exponential Linear Units .
   Parameters:
      * alpha (float) -- the \alpha value for the CELU
        formulation. Default: 1.0
      * inplace (bool) -- can optionally do the operation in-
        place. Default: "False"
   Shape:
      * Input: (), where * means any number of dimensions.
      * Output: (), same shape as the input.
   [image]
   Examples:
      >>> m = nn.CELU()
      >>> input = torch.randn(2)
      >>> output = m(input)


TransformerDecoderclass torch.nn.TransformerDecoder(decoder_layer, num_layers, norm=None)
   TransformerDecoder is a stack of N decoder layers
   Parameters:
      * decoder_layer -- an instance of the
        TransformerDecoderLayer() class (required).
      * num_layers -- the number of sub-decoder-layers in the
        decoder (required).
      * norm -- the layer normalization component (optional).
   Examples::
      >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
      >>> transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=6)
      >>> memory = torch.rand(10, 32, 512)
      >>> tgt = torch.rand(20, 32, 512)
      >>> out = transformer_decoder(tgt, memory)
   forward(tgt, memory, tgt_mask=None, memory_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None)
      Pass the inputs (and mask) through the decoder layer in turn.
      Parameters:
         * tgt (Tensor) -- the sequence to the decoder


(required).
         * memory (Tensor) -- the sequence from the last layer
           of the encoder (required).
         * tgt_mask (Optional[Tensor]) -- the mask for the
           tgt sequence (optional).
         * memory_mask (Optional[Tensor]) -- the mask for
           the memory sequence (optional).
         * tgt_key_padding_mask (Optional[Tensor]) -- the
           mask for the tgt keys per batch (optional).
         * memory_key_padding_mask (Optional[Tensor]) --
           the mask for the memory keys per batch (optional).
      Return type:
         Tensor
      Shape:
         see the docs in Transformer class.


avg_pool3dclass torch.ao.nn.quantized.functional.avg_pool3d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None)
   Applies 3D average-pooling operation in kD \ times kH \times kW
   regions by step size sD \times sH \times sW steps. The number of
   output features is equal to the number of input planes.
   Note:
     The input quantization parameters propagate to the output.
   Parameters:
      * input -- quantized input tensor (\text{minibatch} ,
        \text{in_channels} , iH , iW)
      * kernel_size -- size of the pooling region. Can be a single
        number or a tuple (kD, kH, kW)
      * stride -- stride of the pooling operation. Can be a single
        number or a tuple (sD, sH, sW). Default: "kernel_size"
      * padding -- implicit zero paddings on both sides of the
        input. Can be a single number or a tuple (padD, padH, padW).
        Default: 0


Default: 0
      * ceil_mode -- when True, will use ceil instead of floor
        in the formula to compute the output shape. Default: "False"
      * count_include_pad -- when True, will include the zero-
        padding in the averaging calculation. Default: "True"
      * divisor_override -- if specified, it will be used as
        divisor, otherwise size of the pooling region will be used.
        Default: None


torch.Tensor.nan_to_numTensor.nan_to_num(nan=0.0, posinf=None, neginf=None) -> Tensor
   See "torch.nan_to_num()".


torch.nn.functional.dropouttorch.nn.functional.dropout(input, p=0.5, training=True, inplace=False)
   During training, randomly zeroes some of the elements of the input
   tensor with probability "p" using samples from a Bernoulli
   distribution.
   See "Dropout" for details.
   Parameters:
      * p (float) -- probability of an element to be zeroed.
        Default: 0.5
      * training (bool) -- apply dropout if is "True". Default:
        "True"
      * inplace (bool) -- If set to "True", will do this
        operation in-place. Default: "False"
   Return type:
      Tensor


torch.Tensor.dotTensor.dot(other) -> Tensor
   See "torch.dot()"


torch.Tensor.fminTensor.fmin(other) -> Tensor
   See "torch.fmin()"


torch.Tensor.expandTensor.expand(sizes) -> Tensor
   Returns a new view of the "self" tensor with singleton dimensions
   expanded to a larger size.
   Passing -1 as the size for a dimension means not changing the size
   of that dimension.
   Tensor can be also expanded to a larger number of dimensions, and
   the new ones will be appended at the front. For the new dimensions,
   the size cannot be set to -1.
   Expanding a tensor does not allocate new memory, but only creates a
   new view on the existing tensor where a dimension of size one is
   expanded to a larger size by setting the "stride" to 0. Any
   dimension of size 1 can be expanded to an arbitrary value without
   allocating new memory.
   Parameters:
      sizes (torch.Size or int...) -- the desired
      expanded size
   Warning:
     More than one element of an expanded tensor may refer to a single
     memory location. As a result, in-place operations (especially


ones that are vectorized) may result in incorrect behavior. If
     you need to write to the tensors, please clone them first.
   Example:
      >>> x = torch.tensor([[1], [2], [3]])
      >>> x.size()
      torch.Size([3, 1])
      >>> x.expand(3, 4)
      tensor([[ 1,  1,  1,  1],
              [ 2,  2,  2,  2],
              [ 3,  3,  3,  3]])
      >>> x.expand(-1, 4)   # -1 means not changing the size of that dimension
      tensor([[ 1,  1,  1,  1],
              [ 2,  2,  2,  2],
              [ 3,  3,  3,  3]])


RReLUclass torch.nn.RReLU(lower=0.125, upper=0.3333333333333333, inplace=False)
   Applies the randomized leaky rectified liner unit function,
   element-wise, as described in the paper:
   Empirical Evaluation of Rectified Activations in Convolutional
   Network.
   The function is defined as:
      \text{RReLU}(x) = \begin{cases}     x & \text{if } x \geq 0 \
      ax & \text{ otherwise } \end{cases}
   where a is randomly sampled from uniform distribution
   \mathcal{U}(\text{lower}, \text{upper}).
      See: https://arxiv.org/pdf/1505.00853.pdf
   Parameters:
      * lower (float) -- lower bound of the uniform
        distribution. Default: \frac{1}{8}
      * upper (float) -- upper bound of the uniform
        distribution. Default: \frac{1}{3}
      * inplace (bool) -- can optionally do the operation in-
        place. Default: "False"
   Shape:
      * Input: (), where * means any number of dimensions.
      * Output: (), same shape as the input.
   [image]
   Examples:


[image]
   Examples:
      >>> m = nn.RReLU(0.1, 0.3)
      >>> input = torch.randn(2)
      >>> output = m(input)


torch.Tensor.transpose_Tensor.transpose_(dim0, dim1) -> Tensor
   In-place version of "transpose()"


torch.nn.functional.max_unpool1dtorch.nn.functional.max_unpool1d(input, indices, kernel_size, stride=None, padding=0, output_size=None)
   Computes a partial inverse of "MaxPool1d".
   See "MaxUnpool1d" for details.
   Return type:
      Tensor


torch.nn.functional.lineartorch.nn.functional.linear(input, weight, bias=None) -> Tensor
   Applies a linear transformation to the incoming data: y = xA^T + b.
   This operation supports 2-D "weight" with sparse layout
   Warning:
     Sparse support is a beta feature and some layout(s)/dtype/device
     combinations may not be supported, or may not have autograd
     support. If you notice missing functionality please open a
     feature request.
   This operator supports TensorFloat32.
   Shape:
      * Input: (, in_features) where *** means any number of
        additional dimensions, including none
      * Weight: (out_features, in_features) or (in_features)
      * Bias: (out_features) or ()
      * Output: (, out_features) or (*), based on the shape of the
        weight


torch.nansumtorch.nansum(input, , dtype=None) -> Tensor
   Returns the sum of all elements, treating Not a Numbers (NaNs) as
   zero.
   Parameters:
      input (Tensor) -- the input tensor.
   Keyword Arguments:
      dtype ("torch.dtype", optional) -- the desired data type of
      returned tensor. If specified, the input tensor is casted to
      "dtype" before the operation is performed. This is useful for
      preventing data type overflows. Default: None.
   Example:
      >>> a = torch.tensor([1., 2., float('nan'), 4.])
      >>> torch.nansum(a)
      tensor(7.)
   torch.nansum(input, dim, keepdim=False, , dtype=None) -> Tensor
   Returns the sum of each row of the "input" tensor in the given
   dimension "dim", treating Not a Numbers (NaNs) as zero. If "dim" is
   a list of dimensions, reduce over all of them.
   If "keepdim" is "True", the output tensor is of the same size as
   "input" except in the dimension(s) "dim" where it is of size 1.


Otherwise, "dim" is squeezed (see "torch.squeeze()"), resulting in
   the output tensor having 1 (or "len(dim)") fewer dimension(s).
   Parameters:
      * input (Tensor) -- the input tensor.
      * dim (int or tuple of ints, optional) -- the
        dimension or dimensions to reduce. If "None", all dimensions
        are reduced.
      * keepdim (bool) -- whether the output tensor has "dim"
        retained or not.
   Keyword Arguments:
      dtype ("torch.dtype", optional) -- the desired data type of
      returned tensor. If specified, the input tensor is casted to
      "dtype" before the operation is performed. This is useful for
      preventing data type overflows. Default: None.
   Example:
      >>> torch.nansum(torch.tensor([1., float("nan")]))
      1.0
      >>> a = torch.tensor([[1, 2], [3., float("nan")]])
      >>> torch.nansum(a)
      tensor(6.)
      >>> torch.nansum(a, dim=0)
      tensor([4., 2.])
      >>> torch.nansum(a, dim=1)
      tensor([3., 3.])


torch.Tensor.maximumTensor.maximum(other) -> Tensor
   See "torch.maximum()"


torch.ttorch.t(input) -> Tensor
   Expects "input" to be <= 2-D tensor and transposes dimensions 0 and
   1.
   0-D and 1-D tensors are returned as is. When input is a 2-D tensor
   this is equivalent to "transpose(input, 0, 1)".
   Parameters:
      input (Tensor) -- the input tensor.
   Example:
      >>> x = torch.randn(())
      >>> x
      tensor(0.1995)
      >>> torch.t(x)
      tensor(0.1995)
      >>> x = torch.randn(3)
      >>> x
      tensor([ 2.4320, -0.4608,  0.7702])
      >>> torch.t(x)
      tensor([ 2.4320, -0.4608,  0.7702])
      >>> x = torch.randn(2, 3)
      >>> x
      tensor([[ 0.4875,  0.9158, -0.5872],
              [ 0.3938, -0.6929,  0.6932]])
      >>> torch.t(x)
      tensor([[ 0.4875,  0.3938],
              [ 0.9158, -0.6929],
              [-0.5872,  0.6932]])
   See also "torch.transpose()".


torch.Tensor.lt_Tensor.lt_(other) -> Tensor
   In-place version of "lt()".


torch.nn.functional.binary_cross_entropytorch.nn.functional.binary_cross_entropy(input, target, weight=None, size_average=None, reduce=None, reduction='mean')
   Function that measures the Binary Cross Entropy between the target
   and input probabilities.
   See "BCELoss" for details.
   Parameters:
      * input (Tensor) -- Tensor of arbitrary shape as
        probabilities.
      * target (Tensor) -- Tensor of the same shape as input
        with values between 0 and 1.
      * weight (Tensor, optional) -- a manual rescaling
        weight if provided it's repeated to match input tensor shape
      * size_average (bool, optional) -- Deprecated (see
        "reduction"). By default, the losses are averaged over each
        loss element in the batch. Note that for some losses, there
        multiple elements per sample. If the field "size_average" is
        set to "False", the losses are instead summed for each


minibatch. Ignored when reduce is "False". Default: "True"
      * reduce (bool, optional) -- Deprecated (see
        "reduction"). By default, the losses are averaged or summed
        over observations for each minibatch depending on
        "size_average". When "reduce" is "False", returns a loss per
        batch element instead and ignores "size_average". Default:
        "True"
      * reduction (str, optional) -- Specifies the reduction
        to apply to the output: "'none'" | "'mean'" | "'sum'".
        "'none'": no reduction will be applied, "'mean'": the sum of
        the output will be divided by the number of elements in the
        output, "'sum'": the output will be summed. Note:
        "size_average" and "reduce" are in the process of being
        deprecated, and in the meantime, specifying either of those
        two args will override "reduction". Default: "'mean'"
   Return type:
      Tensor
   Examples:


Return type:
      Tensor
   Examples:
      >>> input = torch.randn(3, 2, requires_grad=True)
      >>> target = torch.rand(3, 2, requires_grad=False)
      >>> loss = F.binary_cross_entropy(torch.sigmoid(input), target)
      >>> loss.backward()


torch.fmintorch.fmin(input, other, , out=None) -> Tensor
   Computes the element-wise minimum of "input" and "other".
   This is like "torch.minimum()" except it handles NaNs differently:
   if exactly one of the two elements being compared is a NaN then the
   non-NaN element is taken as the minimum. Only if both elements are
   NaN is NaN propagated.
   This function is a wrapper around C++'s "std::fmin" and is similar
   to NumPy's "fmin" function.
   Supports broadcasting to a common shape, type promotion, and
   integer and floating-point inputs.
   Parameters:
      * input (Tensor) -- the input tensor.
      * other (Tensor) -- the second input tensor
   Keyword Arguments:
      out (Tensor, optional*) -- the output tensor.
   Example:
      >>> a = torch.tensor([2.2, float('nan'), 2.1, float('nan')])
      >>> b = torch.tensor([-9.3, 0.1, float('nan'), float('nan')])
      >>> torch.fmin(a, b)
      tensor([-9.3000, 0.1000, 2.1000,    nan])


torch.mintorch.min(input) -> Tensor
   Returns the minimum value of all elements in the "input" tensor.
   Warning:
     This function produces deterministic (sub)gradients unlike
     "min(dim=0)"
   Parameters:
      input (Tensor) -- the input tensor.
   Example:
      >>> a = torch.randn(1, 3)
      >>> a
      tensor([[ 0.6750,  1.0857,  1.7197]])
      >>> torch.min(a)
      tensor(0.6750)
   torch.min(input, dim, keepdim=False, *, out=None)
   Returns a namedtuple "(values, indices)" where "values" is the
   minimum value of each row of the "input" tensor in the given
   dimension "dim". And "indices" is the index location of each
   minimum value found (argmin).
   If "keepdim" is "True", the output tensors are of the same size as
   "input" except in the dimension "dim" where they are of size 1.
   Otherwise, "dim" is squeezed (see "torch.squeeze()"), resulting in
   the output tensors having 1 fewer dimension than "input".
   Note:


Note:
     If there are multiple minimal values in a reduced row then the
     indices of the first minimal value are returned.
   Parameters:
      * input (Tensor) -- the input tensor.
      * dim (int) -- the dimension to reduce.
      * keepdim (bool) -- whether the output tensor has "dim"
        retained or not.
   Keyword Arguments:
      out (tuple, optional) -- the tuple of two output
      tensors (min, min_indices)
   Example:
      >>> a = torch.randn(4, 4)
      >>> a
      tensor([[-0.6248,  1.1334, -1.1899, -0.2803],
              [-1.4644, -0.2635, -0.3651,  0.6134],
              [ 0.2457,  0.0384,  1.0128,  0.7015],
              [-0.1153,  2.9849,  2.1458,  0.5788]])
      >>> torch.min(a, 1)
      torch.return_types.min(values=tensor([-1.1899, -1.4644,  0.0384, -0.1153]), indices=tensor([2, 0, 1, 0]))
   torch.min(input, other, *, out=None) -> Tensor
   See "torch.minimum()".


torch.Tensor.remainderTensor.remainder(divisor) -> Tensor
   See "torch.remainder()"


torch._asserttorch._assert(condition, message)
   A wrapper around Python's assert which is symbolically traceable.


torch.foreach_log10_torch._foreach_log10(self: List[Tensor]) -> None
   Apply "torch.log10()" to each Tensor of the input list.


torch.Tensor.argsortTensor.argsort(dim=- 1, descending=False) -> LongTensor
   See "torch.argsort()"


CosineAnnealingWarmRestartsclass torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0, T_mult=1, eta_min=0, last_epoch=- 1, verbose=False)
   Set the learning rate of each parameter group using a cosine
   annealing schedule, where \eta_{max} is set to the initial lr,
   T_{cur} is the number of epochs since the last restart and T_{i} is
   the number of epochs between two warm restarts in SGDR:
      \eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} -
      \eta_{min})\left(1 +
      \cos\left(\frac{T_{cur}}{T_{i}}\pi\right)\right)
   When T_{cur}=T_{i}, set \eta_t = \eta_{min}. When T_{cur}=0 after
   restart, set \eta_t=\eta_{max}.
   It has been proposed in SGDR: Stochastic Gradient Descent with Warm
   Restarts.
   Parameters:
      * optimizer (Optimizer) -- Wrapped optimizer.
      * T_0 (int) -- Number of iterations for the first restart.
      * T_mult (int, optional) -- A factor increases T_{i}
        after a restart. Default: 1.


after a restart. Default: 1.
      * eta_min (float, optional) -- Minimum learning rate.
        Default: 0.
      * last_epoch (int, optional) -- The index of last
        epoch. Default: -1.
      * verbose (bool) -- If "True", prints a message to stdout
        for each update. Default: "False".
   get_last_lr()
      Return last computed learning rate by current scheduler.
   load_state_dict(state_dict)
      Loads the schedulers state.
      Parameters:
         state_dict (dict) -- scheduler state. Should be an
         object returned from a call to "state_dict()".
   print_lr(is_verbose, group, lr, epoch=None)
      Display the current learning rate.
   state_dict()
      Returns the state of the scheduler as a "dict".
      It contains an entry for every variable in self.dict which
      is not the optimizer.
   step(epoch=None)
      Step could be called after every batch update
      -[ Example ]-


-[ Example ]-
      >>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult)
      >>> iters = len(dataloader)
      >>> for epoch in range(20):
      >>>     for i, sample in enumerate(dataloader):
      >>>         inputs, labels = sample['inputs'], sample['labels']
      >>>         optimizer.zero_grad()
      >>>         outputs = net(inputs)
      >>>         loss = criterion(outputs, labels)
      >>>         loss.backward()
      >>>         optimizer.step()
      >>>         scheduler.step(epoch + i / iters)
      This function can be called in an interleaved way.
      -[ Example ]-
      >>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult)
      >>> for epoch in range(20):
      >>>     scheduler.step()
      >>> scheduler.step(26)
      >>> scheduler.step() # scheduler.step(27), instead of scheduler(20)


torch.inversetorch.inverse(input, *, out=None) -> Tensor
   Alias for "torch.linalg.inv()"


upsample_bilinearclass torch.ao.nn.quantized.functional.upsample_bilinear(input, size=None, scale_factor=None)
   Upsamples the input, using bilinear upsampling.
   Warning:
     This function is deprecated in favor of
     "torch.nn.quantized.functional.interpolate()". This is equivalent
     with "nn.quantized.functional.interpolate(..., mode='bilinear',
     align_corners=True)".
   Note:
     The input quantization parameters propagate to the output.
   Note:
     Only 2D inputs are supported
   Parameters:
      * input (Tensor) -- quantized input
      * size (int or Tuple[int, int]) -- output
        spatial size.
      * scale_factor (int or Tuple[int, int]) --
        multiplier for spatial size


torch.Tensor.diagonal_scatterTensor.diagonal_scatter(src, offset=0, dim1=0, dim2=1) -> Tensor
   See "torch.diagonal_scatter()"


torch.unflattentorch.unflatten(input, dim, sizes) -> Tensor
   Expands a dimension of the input tensor over multiple dimensions.
   See also:
     "torch.flatten()" the inverse of this function. It coalesces
     several dimensions into one.
   Parameters:
      * input (Tensor) -- the input tensor.
      * dim (int) -- Dimension to be unflattened, specified as
        an index into "input.shape".
      * sizes (Tuple[int]) -- New shape of the unflattened
        dimension. One of its elements can be -1 in which case the
        corresponding output dimension is inferred. Otherwise, the
        product of "sizes" must equal "input.shape[dim]".
   Returns:
      A View of input with the specified dimension unflattened.
   Examples::
      >>> torch.unflatten(torch.randn(3, 4, 1), 1, (2, 2)).shape
      torch.Size([3, 2, 2, 1])
      >>> torch.unflatten(torch.randn(3, 4, 1), 1, (-1, 2)).shape
      torch.Size([3, 2, 2, 1])


torch.Size([3, 2, 2, 1])
      >>> torch.unflatten(torch.randn(5, 12, 3), -1, (2, 2, 3, 1, 1)).shape
      torch.Size([5, 2, 2, 3, 1, 1, 3])


torch.nn.functional.interpolatetorch.nn.functional.interpolate(input, size=None, scale_factor=None, mode='nearest', align_corners=None, recompute_scale_factor=None, antialias=False)
   Down/up samples the input to either the given "size" or the given
   "scale_factor"
   The algorithm used for interpolation is determined by "mode".
   Currently temporal, spatial and volumetric sampling are supported,
   i.e. expected inputs are 3-D, 4-D or 5-D in shape.
   The input dimensions are interpreted in the form: mini-batch x
   channels x [optional depth] x [optional height] x width.
   The modes available for resizing are: nearest, linear (3D-
   only), bilinear, bicubic (4D-only), trilinear (5D-only),
   area, nearest-exact
   Parameters:
      * input (Tensor) -- the input tensor
      * size (int or Tuple[int] or Tuple[int,
        int] or Tuple[int, int, int]) -- output
        spatial size.


spatial size.
      * scale_factor (float or Tuple[float]) --
        multiplier for spatial size. If scale_factor is a tuple, its
        length has to match the number of spatial dimensions;
        input.dim() - 2.
      * mode (str) -- algorithm used for upsampling: "'nearest'"
        | "'linear'" | "'bilinear'" | "'bicubic'" | "'trilinear'" |
        "'area'" | "'nearest-exact'". Default: "'nearest'"
      * align_corners (bool, optional) -- Geometrically, we
        consider the pixels of the input and output as squares rather
        than points. If set to "True", the input and output tensors
        are aligned by the center points of their corner pixels,
        preserving the values at the corner pixels. If set to "False",
        the input and output tensors are aligned by the corner points
        of their corner pixels, and the interpolation uses edge value
        padding for out-of-boundary values, making this operation


independent of input size when "scale_factor" is kept the
        same. This only has an effect when "mode" is "'linear'",
        "'bilinear'", "'bicubic'" or "'trilinear'". Default: "False"
      * recompute_scale_factor (bool, optional) -- recompute
        the scale_factor for use in the interpolation calculation. If
        recompute_scale_factor is "True", then scale_factor must
        be passed in and scale_factor is used to compute the output
        size. The computed output size will be used to infer new
        scales for the interpolation. Note that when scale_factor is
        floating-point, it may differ from the recomputed
        scale_factor due to rounding and precision issues. If
        recompute_scale_factor is "False", then size or
        scale_factor will be used directly for interpolation.
        Default: "None".
      * antialias (bool, optional) -- flag to apply anti-


aliasing. Default: "False". Using anti-alias option together
        with "align_corners=False", interpolation result would match
        Pillow result for downsampling operation. Supported modes:
        "'bilinear'", "'bicubic'".
   Return type:
      Tensor
   Note:
     With "mode='bicubic'", it's possible to cause overshoot, in other
     words it can produce negative values or values greater than 255
     for images. Explicitly call "result.clamp(min=0, max=255)" if you
     want to reduce the overshoot when displaying the image.
   Note:
     Mode "mode='nearest-exact'" matches Scikit-Image and PIL nearest
     neighbours interpolation algorithms and fixes known issues with
     "mode='nearest'". This mode is introduced to keep backward
     compatibility. Mode "mode='nearest'" matches buggy OpenCV's
     "INTER_NEAREST" interpolation algorithm.
   Note:
     This operation may produce nondeterministic gradients when given
     tensors on a CUDA device. See Reproducibility for more


information.


torch.not_equaltorch.not_equal(input, other, *, out=None) -> Tensor
   Alias for "torch.ne()".


LPPool2dclass torch.nn.LPPool2d(norm_type, kernel_size, stride=None, ceil_mode=False)
   Applies a 2D power-average pooling over an input signal composed of
   several input planes.
   On each window, the function computed is:
      f(X) = \sqrt[p]{\sum_{x \in X} x^{p}}
   * At p = \infty, one gets Max Pooling
   * At p = 1, one gets Sum Pooling (which is proportional to average
     pooling)
   The parameters "kernel_size", "stride" can either be:
      * a single "int" -- in which case the same value is used for the
        height and width dimension
      * a "tuple" of two ints -- in which case, the first int is
        used for the height dimension, and the second int for the
        width dimension
   Note:
     If the sum to the power of p is zero, the gradient of this
     function is not defined. This implementation will set the
     gradient to zero in this case.
   Parameters:
      * kernel_size (Union[int, Tuple[int*,


int]]*) -- the size of the window
      * stride (*Union[int, Tuple[int, int]]*)
        -- the stride of the window. Default value is "kernel_size"
      * ceil_mode (bool) -- when True, will use ceil instead
        of floor to compute the output shape
   Shape:
      * Input: (N, C, H_{in}, W_{in})
      * Output: (N, C, H_{out}, W_{out}), where
           H_{out} = \left\lfloor\frac{H_{in} -
           \text{kernel_size}[0]}{\text{stride}[0]} + 1\right\rfloor
           W_{out} = \left\lfloor\frac{W_{in} -
           \text{kernel_size}[1]}{\text{stride}[1]} + 1\right\rfloor
   Examples:
      >>> # power-2 pool of square window of size=3, stride=2
      >>> m = nn.LPPool2d(2, 3, stride=2)
      >>> # pool of non-square window of power 1.2
      >>> m = nn.LPPool2d(1.2, (3, 2), stride=(2, 1))
      >>> input = torch.randn(20, 16, 50, 32)
      >>> output = m(input)


default_histogram_observertorch.quantization.observer.default_histogram_observer
   alias of functools.partial(, quant_min=0,
   quant_max=127){}


torch.Tensor.cumsum_Tensor.cumsum_(dim, dtype=None) -> Tensor
   In-place version of "cumsum()"


PixelShuffleclass torch.nn.PixelShuffle(upscale_factor)
   Rearranges elements in a tensor of shape (, C \times r^2, H, W) to
   a tensor of shape (, C, H \times r, W \times r), where r is an
   upscale factor.
   This is useful for implementing efficient sub-pixel convolution
   with a stride of 1/r.
   See the paper: Real-Time Single Image and Video Super-Resolution
   Using an Efficient Sub-Pixel Convolutional Neural Network by Shi
   et. al (2016) for more details.
   Parameters:
      upscale_factor (int) -- factor to increase spatial
      resolution by
   Shape:
      * Input: (, C_{in}, H_{in}, W_{in}), where * is zero or more
        batch dimensions
      * Output: (, C_{out}, H_{out}, W_{out}), where
      C_{out} = C_{in} \div \text{upscale_factor}^2
      H_{out} = H_{in} \times \text{upscale_factor}
      W_{out} = W_{in} \times \text{upscale_factor}
   Examples:
      >>> pixel_shuffle = nn.PixelShuffle(3)
      >>> input = torch.randn(1, 9, 4, 4)


input = torch.randn(1, 9, 4, 4)
      >>> output = pixel_shuffle(input)
      >>> print(output.size())
      torch.Size([1, 1, 12, 12])


default_histogram_fake_quanttorch.quantization.fake_quantize.default_histogram_fake_quant
   alias of functools.partial(,
   observer=, quant_min=0,
   quant_max=255, dtype=torch.quint8, qscheme=torch.per_tensor_affine,
   reduce_range=True){}


torch.Tensor.minTensor.min(dim=None, keepdim=False)
   See "torch.min()"


Conv3dclass torch.ao.nn.qat.Conv3d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', qconfig=None, device=None, dtype=None)
   A Conv3d module attached with FakeQuantize modules for weight, used
   for quantization aware training.
   We adopt the same interface as torch.nn.Conv3d, please see https
   ://pytorch.org/docs/stable/nn.html?highlight=conv3d#torch.nn.Conv3d
   for documentation.
   Similar to torch.nn.Conv3d, with FakeQuantize modules initialized
   to default.
   Variables:
      weight_fake_quant -- fake quant module for weight


PoissonNLLLossclass torch.nn.PoissonNLLLoss(log_input=True, full=False, size_average=None, eps=1e-08, reduce=None, reduction='mean')
   Negative log likelihood loss with Poisson distribution of target.
   The loss can be described as:
      \text{target} \sim \mathrm{Poisson}(\text{input})
      \text{loss}(\text{input}, \text{target}) = \text{input} -
      \text{target} * \log(\text{input})                             +
      \log(\text{target!})
   The last term can be omitted or approximated with Stirling formula.
   The approximation is used for target values more than 1. For
   targets less or equal to 1 zeros are added to the loss.
   Parameters:
      * log_input (bool, optional) -- if "True" the loss is
        computed as \exp(\text{input}) - \text{target}\text{input},
        if "False" the loss is \text{input} -
        \text{target}\log(\text{input}+\text{eps}).
      * full (bool, optional) --
        whether to compute full loss, i. e. to add the Stirling


approximation term
           \text{target}\log(\text{target}) - \text{target} + 0.5 *
           \log(2\pi\text{target}).
      * size_average (bool, optional) -- Deprecated (see
        "reduction"). By default, the losses are averaged over each
        loss element in the batch. Note that for some losses, there
        are multiple elements per sample. If the field "size_average"
        is set to "False", the losses are instead summed for each
        minibatch. Ignored when "reduce" is "False". Default: "True"
      * eps (float, optional) -- Small value to avoid
        evaluation of \log(0) when "log_input = False". Default: 1e-8
      * reduce (bool, optional*) -- Deprecated (see
        "reduction"). By default, the losses are averaged or summed
        over observations for each minibatch depending on
        "size_average". When "reduce" is "False", returns a loss per
        batch element instead and ignores "size_average". Default:
        "True"


"True"
      * reduction (str, optional) -- Specifies the reduction
        to apply to the output: "'none'" | "'mean'" | "'sum'".
        "'none'": no reduction will be applied, "'mean'": the sum of
        the output will be divided by the number of elements in the
        output, "'sum'": the output will be summed. Note:
        "size_average" and "reduce" are in the process of being
        deprecated, and in the meantime, specifying either of those
        two args will override "reduction". Default: "'mean'"
   Examples:
      >>> loss = nn.PoissonNLLLoss()
      >>> log_input = torch.randn(5, 2, requires_grad=True)
      >>> target = torch.randn(5, 2)
      >>> output = loss(log_input, target)
      >>> output.backward()
   Shape:
      * Input: (), where * means any number of dimensions.
      * Target: (), same shape as the input.
      * Output: scalar by default. If "reduction" is "'none'", then
        (*), the same shape as the input.


torch._foreach_acostorch._foreach_acos(self: List[Tensor]) -> List[Tensor]
   Apply "torch.acos()" to each Tensor of the input list.


torch.bincounttorch.bincount(input, weights=None, minlength=0) -> Tensor
   Count the frequency of each value in an array of non-negative ints.
   The number of bins (size 1) is one larger than the largest value in
   "input" unless "input" is empty, in which case the result is a
   tensor of size 0. If "minlength" is specified, the number of bins
   is at least "minlength" and if "input" is empty, then the result is
   tensor of size "minlength" filled with zeros. If "n" is the value
   at position "i", "out[n] += weights[i]" if "weights" is specified
   else "out[n] += 1".
   Note:
     This operation may produce nondeterministic gradients when given
     tensors on a CUDA device. See Reproducibility for more
     information.
   Parameters:
      * input (Tensor) -- 1-d int tensor
      * weights (Tensor) -- optional, weight for each value in
        the input tensor. Should be of same size as input tensor.
      * minlength (int) -- optional, minimum number of bins.


Should be non-negative.
   Returns:
      a tensor of shape "Size([max(input) + 1])" if "input" is non-
      empty, else "Size(0)"
   Return type:
      output (Tensor)
   Example:
      >>> input = torch.randint(0, 8, (5,), dtype=torch.int64)
      >>> weights = torch.linspace(0, 1, steps=5)
      >>> input, weights
      (tensor([4, 3, 6, 3, 4]),
       tensor([ 0.0000,  0.2500,  0.5000,  0.7500,  1.0000])
      >>> torch.bincount(input)
      tensor([0, 0, 0, 2, 2, 0, 1])
      >>> input.bincount(weights)
      tensor([0.0000, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.5000])


torch.triltorch.tril(input, diagonal=0, , out=None) -> Tensor
   Returns the lower triangular part of the matrix (2-D tensor) or
   batch of matrices "input", the other elements of the result tensor
   "out" are set to 0.
   The lower triangular part of the matrix is defined as the elements
   on and below the diagonal.
   The argument "diagonal" controls which diagonal to consider. If
   "diagonal" = 0, all elements on and below the main diagonal are
   retained. A positive value includes just as many diagonals above
   the main diagonal, and similarly a negative value excludes just as
   many diagonals below the main diagonal. The main diagonal are the
   set of indices \lbrace (i, i) \rbrace for i \in [0, \min{d_{1},
   d_{2}} - 1] where d_{1}, d_{2} are the dimensions of the matrix.
   Parameters:
      * input (Tensor) -- the input tensor.
      * diagonal (int, optional*) -- the diagonal to consider
   Keyword Arguments:


Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
   Example:
      >>> a = torch.randn(3, 3)
      >>> a
      tensor([[-1.0813, -0.8619,  0.7105],
              [ 0.0935,  0.1380,  2.2112],
              [-0.3409, -0.9828,  0.0289]])
      >>> torch.tril(a)
      tensor([[-1.0813,  0.0000,  0.0000],
              [ 0.0935,  0.1380,  0.0000],
              [-0.3409, -0.9828,  0.0289]])
      >>> b = torch.randn(4, 6)
      >>> b
      tensor([[ 1.2219,  0.5653, -0.2521, -0.2345,  1.2544,  0.3461],
              [ 0.4785, -0.4477,  0.6049,  0.6368,  0.8775,  0.7145],
              [ 1.1502,  3.2716, -1.1243, -0.5413,  0.3615,  0.6864],
              [-0.0614, -0.7344, -1.3164, -0.7648, -1.4024,  0.0978]])
      >>> torch.tril(b, diagonal=1)
      tensor([[ 1.2219,  0.5653,  0.0000,  0.0000,  0.0000,  0.0000],
              [ 0.4785, -0.4477,  0.6049,  0.0000,  0.0000,  0.0000],
              [ 1.1502,  3.2716, -1.1243, -0.5413,  0.0000,  0.0000],


[-0.0614, -0.7344, -1.3164, -0.7648, -1.4024,  0.0000]])
      >>> torch.tril(b, diagonal=-1)
      tensor([[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
              [ 0.4785,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
              [ 1.1502,  3.2716,  0.0000,  0.0000,  0.0000,  0.0000],
              [-0.0614, -0.7344, -1.3164,  0.0000,  0.0000,  0.0000]])


torch._foreach_expm1torch._foreach_expm1(self: List[Tensor]) -> List[Tensor]
   Apply "torch.expm1()" to each Tensor of the input list.


torch.cuda.max_memory_allocatedtorch.cuda.max_memory_allocated(device=None)
   Returns the maximum GPU memory occupied by tensors in bytes for a
   given device.
   By default, this returns the peak allocated memory since the
   beginning of this program. "reset_peak_memory_stats()" can be used
   to reset the starting point in tracking this metric. For example,
   these two functions can measure the peak allocated memory usage of
   each iteration in a training loop.
   Parameters:
      device (torch.device or int, optional) -- selected
      device. Returns statistic for the current device, given by
      "current_device()", if "device" is "None" (default).
   Return type:
      int
   Note:
     See Memory management for more details about GPU memory
     management.


torch.cuda.caching_allocator_deletetorch.cuda.caching_allocator_delete(mem_ptr)
   Deletes memory allocated using the CUDA memory allocator.
   Memory allocated with "caching_allocator_alloc()". is freed here.
   The associated device and stream are tracked inside the allocator.
   Parameters:
      mem_ptr (int) -- memory address to be freed by the
      allocator.
   Note:
     See Memory management for more details about GPU memory
     management.


torch.Tensor.multinomialTensor.multinomial(num_samples, replacement=False, *, generator=None) -> Tensor
   See "torch.multinomial()"


torch.foreach_zero_torch._foreach_zero(self: List[Tensor]) -> None
   Apply "torch.zero()" to each Tensor of the input list.


torch.Tensor.round_Tensor.round_(decimals=0) -> Tensor
   In-place version of "round()"


torch.Tensor.msortTensor.msort() -> Tensor
   See "torch.msort()"


torch.Tensor.resolve_conjTensor.resolve_conj() -> Tensor
   See "torch.resolve_conj()"


LazyConvTranspose1dclass torch.nn.LazyConvTranspose1d(out_channels, kernel_size, stride=1, padding=0, output_padding=0, groups=1, bias=True, dilation=1, padding_mode='zeros', device=None, dtype=None)
   A "torch.nn.ConvTranspose1d" module with lazy initialization of the
   "in_channels" argument of the "ConvTranspose1d" that is inferred
   from the "input.size(1)". The attributes that will be lazily
   initialized are weight and bias.
   Check the "torch.nn.modules.lazy.LazyModuleMixin" for further
   documentation on lazy modules and their limitations.
   Parameters:
      * out_channels (int) -- Number of channels produced by the
        convolution
      * kernel_size (int or tuple) -- Size of the convolving
        kernel
      * stride (int or tuple, optional) -- Stride of the
        convolution. Default: 1
      * padding (int or tuple, optional) -- "dilation *
        (kernel_size - 1) - padding" zero-padding will be added to


both sides of the input. Default: 0
      * output_padding (int or tuple, optional) --
        Additional size added to one side of the output shape.
        Default: 0
      * groups (int, optional) -- Number of blocked
        connections from input channels to output channels. Default: 1
      * bias (bool, optional) -- If "True", adds a learnable
        bias to the output. Default: "True"
      * dilation (int or tuple, optional) -- Spacing
        between kernel elements. Default: 1
   See also:
     "torch.nn.ConvTranspose1d" and
     "torch.nn.modules.lazy.LazyModuleMixin"
   cls_to_become
      alias of "ConvTranspose1d"


torch.Tensor.erfinv_Tensor.erfinv_() -> Tensor
   In-place version of "erfinv()"


torch.Tensor.rot90Tensor.rot90(k, dims) -> Tensor
   See "torch.rot90()"


torch.Tensor.tril_Tensor.tril_(diagonal=0) -> Tensor
   In-place version of "tril()"


torch.Tensor.floatTensor.float(memory_format=torch.preserve_format) -> Tensor
   "self.float()" is equivalent to "self.to(torch.float32)". See
   "to()".
   Parameters:
      memory_format ("torch.memory_format", optional) -- the
      desired memory format of returned Tensor. Default:
      "torch.preserve_format".


Embeddingclass torch.nn.Embedding(num_embeddings, embedding_dim, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False, _weight=None, _freeze=False, device=None, dtype=None)
   A simple lookup table that stores embeddings of a fixed dictionary
   and size.
   This module is often used to store word embeddings and retrieve
   them using indices. The input to the module is a list of indices,
   and the output is the corresponding word embeddings.
   Parameters:
      * num_embeddings (int) -- size of the dictionary of
        embeddings
      * embedding_dim (int) -- the size of each embedding vector
      * padding_idx (int, optional) -- If specified, the
        entries at "padding_idx" do not contribute to the gradient;
        therefore, the embedding vector at "padding_idx" is not
        updated during training, i.e. it remains as a fixed "pad". For
        a newly constructed Embedding, the embedding vector at


"padding_idx" will default to all zeros, but can be updated to
        another value to be used as the padding vector.
      * max_norm (float, optional) -- If given, each
        embedding vector with norm larger than "max_norm" is
        renormalized to have norm "max_norm".
      * norm_type (float, optional) -- The p of the p-norm
        to compute for the "max_norm" option. Default "2".
      * scale_grad_by_freq (bool, optional) -- If given,
        this will scale gradients by the inverse of frequency of the
        words in the mini-batch. Default "False".
      * sparse (bool, optional) -- If "True", gradient
        w.r.t. "weight" matrix will be a sparse tensor. See Notes for
        more details regarding sparse gradients.
   Variables:
      weight (Tensor) -- the learnable weights of the module of
      shape (num_embeddings, embedding_dim) initialized from
      \mathcal{N}(0, 1)
   Shape:


\mathcal{N}(0, 1)
   Shape:
      * Input: (), IntTensor or LongTensor of arbitrary shape
        containing the indices to extract
      * Output: (, H), where *** is the input shape and
        H=\text{embedding_dim}
   Note:
     Keep in mind that only a limited number of optimizers support
     sparse gradients: currently it's "optim.SGD" (CUDA and CPU),
     "optim.SparseAdam" (CUDA and CPU) and "optim.Adagrad" (CPU)
   Note:
     When "max_norm" is not "None", "Embedding"'s forward method will
     modify the "weight" tensor in-place. Since tensors needed for
     gradient computations cannot be modified in-place, performing a
     differentiable operation on "Embedding.weight" before calling
     "Embedding"'s forward method requires cloning "Embedding.weight"
     when "max_norm" is not "None". For example:
        n, d, m = 3, 5, 7
        embedding = nn.Embedding(n, d, max_norm=True)
        W = torch.randn((m, d), requires_grad=True)
        idx = torch.tensor([1, 2])


idx = torch.tensor([1, 2])
        a = embedding.weight.clone() @ W.t()  # weight must be cloned for this to be differentiable
        b = embedding(idx) @ W.t()  # modifies weight in-place
        out = (a.unsqueeze(0) + b.unsqueeze(1))
        loss = out.sigmoid().prod()
        loss.backward()
   Examples:
      >>> # an Embedding module containing 10 tensors of size 3
      >>> embedding = nn.Embedding(10, 3)
      >>> # a batch of 2 samples of 4 indices each
      >>> input = torch.LongTensor([[1, 2, 4, 5], [4, 3, 2, 9]])
      >>> embedding(input)
      tensor([[[-0.0251, -1.6902,  0.7172],
               [-0.6431,  0.0748,  0.6969],
               [ 1.4970,  1.3448, -0.9685],
               [-0.3677, -2.7265, -0.1685]],
              [[ 1.4970,  1.3448, -0.9685],
               [ 0.4362, -0.4004,  0.9400],
               [-0.6431,  0.0748,  0.6969],
               [ 0.9124, -2.3616,  1.1151]]])
      >>> # example with padding_idx
      >>> embedding = nn.Embedding(10, 3, padding_idx=0)


input = torch.LongTensor([[0, 2, 0, 5]])
      >>> embedding(input)
      tensor([[[ 0.0000,  0.0000,  0.0000],
               [ 0.1535, -2.0309,  0.9315],
               [ 0.0000,  0.0000,  0.0000],
               [-0.1655,  0.9897,  0.0635]]])
      >>> # example of changing pad vector
      >>> padding_idx = 0
      >>> embedding = nn.Embedding(3, 3, padding_idx=padding_idx)
      >>> embedding.weight
      Parameter containing:
      tensor([[ 0.0000,  0.0000,  0.0000],
              [-0.7895, -0.7089, -0.0364],
              [ 0.6778,  0.5803,  0.2678]], requires_grad=True)
      >>> with torch.no_grad():
      ...     embedding.weight[padding_idx] = torch.ones(3)
      >>> embedding.weight
      Parameter containing:
      tensor([[ 1.0000,  1.0000,  1.0000],
              [-0.7895, -0.7089, -0.0364],
              [ 0.6778,  0.5803,  0.2678]], requires_grad=True)


classmethod from_pretrained(embeddings, freeze=True, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False)
      Creates Embedding instance from given 2-dimensional FloatTensor.
      Parameters:
         * embeddings (Tensor) -- FloatTensor containing weights
           for the Embedding. First dimension is being passed to
           Embedding as "num_embeddings", second as "embedding_dim".
         * freeze (bool, optional) -- If "True", the tensor
           does not get updated in the learning process. Equivalent to
           "embedding.weight.requires_grad = False". Default: "True"
         * padding_idx (int, optional) -- If specified, the
           entries at "padding_idx" do not contribute to the gradient;
           therefore, the embedding vector at "padding_idx" is not
           updated during training, i.e. it remains as a fixed "pad".
         * max_norm (float, optional) -- See module


initialization documentation.
         * norm_type (float, optional) -- See module
           initialization documentation. Default "2".
         * scale_grad_by_freq (bool, optional) -- See module
           initialization documentation. Default "False".
         * sparse (bool, optional) -- See module
           initialization documentation.
      Examples:
         >>> # FloatTensor containing pretrained weights
         >>> weight = torch.FloatTensor([[1, 2.3, 3], [4, 5.1, 6.3]])
         >>> embedding = nn.Embedding.from_pretrained(weight)
         >>> # Get embeddings for index 1
         >>> input = torch.LongTensor([1])
         >>> embedding(input)
         tensor([[ 4.0000,  5.1000,  6.3000]])


torch.Tensor.amaxTensor.amax(dim=None, keepdim=False) -> Tensor
   See "torch.amax()"


torch.sparse_csc_tensortorch.sparse_csc_tensor(ccol_indices, row_indices, values, size=None, , dtype=None, device=None, requires_grad=False, check_invariants=None) -> Tensor
   Constructs a sparse tensor in CSC (Compressed Sparse Column) with
   specified values at the given "ccol_indices" and "row_indices".
   Sparse matrix multiplication operations in CSC format are typically
   faster than that for sparse tensors in COO format. Make you have a
   look at the note on the data type of the indices.
   Note:
     If the "device" argument is not specified the device of the given
     "values" and indices tensor(s) must match. If, however, the
     argument is specified the input Tensors will be converted to the
     given device and in turn determine the device of the constructed
     sparse tensor.
   Parameters:
      * ccol_indices (array_like) -- (B+1)-dimensional array of
        size "(batchsize, ncols + 1)".  The last element of each


batch is the number of non-zeros. This tensor encodes the
        index in values and row_indices depending on where the given
        column starts. Each successive number in the tensor subtracted
        by the number before it denotes the number of elements in a
        given column.
      * row_indices (array_like) -- Row co-ordinates of each
        element in values. (B+1)-dimensional tensor with the same
        length as values.
      * values (array_list) -- Initial values for the tensor.
        Can be a list, tuple, NumPy "ndarray", scalar, and other types
        that represents a (1+K)-dimensional tensor where "K" is the
        number of dense dimensions.
      * size (list, tuple, "torch.Size", optional) -- Size of the
        sparse tensor: "(batchsize, nrows, ncols, densesize)". If
        not provided, the size will be inferred as the minimum size
        big enough to hold all non-zero elements.
   Keyword Arguments:


Keyword Arguments:
      * dtype ("torch.dtype", optional) -- the desired data type
        of returned tensor.  Default: if None, infers data type from
        "values".
      * device ("torch.device", optional) -- the desired device of
        returned tensor.  Default: if None, uses the current device
        for the default tensor type (see
        "torch.set_default_tensor_type()"). "device" will be the CPU
        for CPU tensor types and the current CUDA device for CUDA
        tensor types.
      * requires_grad (bool, optional) -- If autograd should
        record operations on the returned tensor. Default: "False".
      * check_invariants (bool, optional) -- If sparse
        tensor invariants are checked. Default: as returned by
        "torch.sparse.check_sparse_tensor_invariants.is_enabled()",
        initially False.
   Example::
      >>> ccol_indices = [0, 2, 4]
      >>> row_indices = [0, 1, 0, 1]
      >>> values = [1, 2, 3, 4]


values = [1, 2, 3, 4]
      >>> torch.sparse_csc_tensor(torch.tensor(ccol_indices, dtype=torch.int64),
      ...                         torch.tensor(row_indices, dtype=torch.int64),
      ...                         torch.tensor(values), dtype=torch.double)
      tensor(ccol_indices=tensor([0, 2, 4]),
             row_indices=tensor([0, 1, 0, 1]),
             values=tensor([1., 2., 3., 4.]), size=(2, 2), nnz=4,
             dtype=torch.float64, layout=torch.sparse_csc)


torch.Tensor.traceTensor.trace() -> Tensor
   See "torch.trace()"


torch.cuda.memory_summarytorch.cuda.memory_summary(device=None, abbreviated=False)
   Returns a human-readable printout of the current memory allocator
   statistics for a given device.
   This can be useful to display periodically during training, or when
   handling out-of-memory exceptions.
   Parameters:
      * device (torch.device or int, optional) --
        selected device. Returns printout for the current device,
        given by "current_device()", if "device" is "None" (default).
      * abbreviated (bool, optional) -- whether to return an
        abbreviated summary (default: False).
   Return type:
      str
   Note:
     See Memory management for more details about GPU memory
     management.


torch.difftorch.diff(input, n=1, dim=- 1, prepend=None, append=None) -> Tensor
   Computes the n-th forward difference along the given dimension.
   The first-order differences are given by out[i] = input[i + 1] -
   input[i]. Higher-order differences are calculated by using
   "torch.diff()" recursively.
   Parameters:
      * input (Tensor) -- the tensor to compute the differences
        on
      * n (int, optional) -- the number of times to
        recursively compute the difference
      * dim (int, optional) -- the dimension to compute the
        difference along. Default is the last dimension.
      * prepend (Tensor, optional) -- values to prepend or
        append to "input" along "dim" before computing the difference.
        Their dimensions must be equivalent to that of input, and
        their shapes must match input's shape except on "dim".
      * append (Tensor, optional) -- values to prepend or


append to "input" along "dim" before computing the difference.
        Their dimensions must be equivalent to that of input, and
        their shapes must match input's shape except on "dim".
   Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
   Example:
      >>> a = torch.tensor([1, 3, 2])
      >>> torch.diff(a)
      tensor([ 2, -1])
      >>> b = torch.tensor([4, 5])
      >>> torch.diff(a, append=b)
      tensor([ 2, -1,  2,  1])
      >>> c = torch.tensor([[1, 2, 3], [3, 4, 5]])
      >>> torch.diff(c, dim=0)
      tensor([[2, 2, 2]])
      >>> torch.diff(c, dim=1)
      tensor([[1, 1],
              [1, 1]])


torch.Tensor.eq_Tensor.eq_(other) -> Tensor
   In-place version of "eq()"


torch.addbmmtorch.addbmm(input, batch1, batch2, , beta=1, alpha=1, out=None) -> Tensor
   Performs a batch matrix-matrix product of matrices stored in
   "batch1" and "batch2", with a reduced add step (all matrix
   multiplications get accumulated along the first dimension). "input"
   is added to the final result.
   "batch1" and "batch2" must be 3-D tensors each containing the same
   number of matrices.
   If "batch1" is a (b \times n \times m) tensor, "batch2" is a (b
   \times m \times p) tensor, "input" must be broadcastable with a (n
   \times p) tensor and "out" will be a (n \times p) tensor.
      out = \beta\ \text{input} + \alpha\ (\sum_{i=0}^{b-1}
      \text{batch1}_i \mathbin{@} \text{batch2}_i)
   If "beta" is 0, then "input" will be ignored, and nan and inf
   in it will not be propagated.
   For inputs of type FloatTensor or DoubleTensor*, arguments
   "beta" and "alpha" must be real numbers, otherwise they should be
   integers.
   This operator supports TensorFloat32.


This operator supports TensorFloat32.
   On certain ROCm devices, when using float16 inputs this module will
   use different precision for backward.
   Parameters:
      * batch1 (Tensor) -- the first batch of matrices to be
        multiplied
      * batch2 (Tensor) -- the second batch of matrices to be
        multiplied
   Keyword Arguments:
      * beta (Number, optional) -- multiplier for "input"
        (\beta)
      * input (Tensor) -- matrix to be added
      * alpha (Number, optional) -- multiplier for batch1 @
        batch2 (\alpha)
      * out (Tensor, optional) -- the output tensor.
   Example:
      >>> M = torch.randn(3, 5)
      >>> batch1 = torch.randn(10, 3, 4)
      >>> batch2 = torch.randn(10, 4, 5)
      >>> torch.addbmm(M, batch1, batch2)
      tensor([[  6.6311,   0.0503,   6.9768, -12.0362,  -2.1653],
              [ -4.8185,  -1.4255,  -6.6760,   8.9453,   2.5743],
              [ -3.8202,   4.3691,   1.0943,  -1.1109,   5.4730]])


ScriptModuleclass torch.jit.ScriptModule
   A wrapper around C++ "torch::jit::Module". "ScriptModule"s contain
   methods, attributes, parameters, and constants. These can be
   accessed the same way as on a normal "nn.Module".
   add_module(name, module)
      Adds a child module to the current module.
      The module can be accessed as an attribute using the given name.
      Parameters:
         * name (str) -- name of the child module. The child
           module can be accessed from this module using the given
           name
         * module (Module) -- child module to be added to the
           module.
   apply(fn)
      Applies "fn" recursively to every submodule (as returned by
      ".children()") as well as self. Typical use includes
      initializing the parameters of a model (see also torch.nn.init).
      Parameters:
         fn ("Module" -> None) -- function to be applied to each
         submodule
      Returns:
         self
      Return type:
         Module


self
      Return type:
         Module
      Example:
         >>> @torch.no_grad()
         >>> def init_weights(m):
         >>>     print(m)
         >>>     if type(m) == nn.Linear:
         >>>         m.weight.fill_(1.0)
         >>>         print(m.weight)
         >>> net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))
         >>> net.apply(init_weights)
         Linear(in_features=2, out_features=2, bias=True)
         Parameter containing:
         tensor([[1., 1.],
                 [1., 1.]], requires_grad=True)
         Linear(in_features=2, out_features=2, bias=True)
         Parameter containing:
         tensor([[1., 1.],
                 [1., 1.]], requires_grad=True)
         Sequential(
           (0): Linear(in_features=2, out_features=2, bias=True)
           (1): Linear(in_features=2, out_features=2, bias=True)
         )
   bfloat16()
      Casts all floating point parameters and buffers to "bfloat16"
      datatype.
      Note:


datatype.
      Note:
        This method modifies the module in-place.
      Returns:
         self
      Return type:
         Module
   buffers(recurse=True)
      Returns an iterator over module buffers.
      Parameters:
         recurse (bool) -- if True, then yields buffers of this
         module and all submodules. Otherwise, yields only buffers
         that are direct members of this module.
      Yields:
         torch.Tensor -- module buffer
      Return type:
         Iterator[Tensor]
      Example:
         >>> for buf in model.buffers():
         >>>     print(type(buf), buf.size())
          (20L,)
          (20L, 1L, 5L, 5L)
   children()
      Returns an iterator over immediate children modules.
      Yields:
         Module -- a child module
      Return type:
         Iterator[Module]
   property code
      Returns a pretty-printed representation (as valid Python syntax)


of the internal graph for the "forward" method. See Inspecting
      Code for details.
   property code_with_constants
      Returns a tuple of:
      [0] a pretty-printed representation (as valid Python syntax) of
      the internal graph for the "forward" method. See code. [1] a
      ConstMap following the CONSTANT.cN format of the output in [0].
      The indices in the [0] output are keys to the underlying
      constant's values.
      See Inspecting Code for details.
   cpu()
      Moves all model parameters and buffers to the CPU.
      Note:
        This method modifies the module in-place.
      Returns:
         self
      Return type:
         Module
   cuda(device=None)
      Moves all model parameters and buffers to the GPU.
      This also makes associated parameters and buffers different
      objects. So it should be called before constructing optimizer if
      the module will live on GPU while being optimized.
      Note:
        This method modifies the module in-place.


This method modifies the module in-place.
      Parameters:
         device (int, optional) -- if specified, all
         parameters will be copied to that device
      Returns:
         self
      Return type:
         Module
   double()
      Casts all floating point parameters and buffers to "double"
      datatype.
      Note:
        This method modifies the module in-place.
      Returns:
         self
      Return type:
         Module
   eval()
      Sets the module in evaluation mode.
      This has any effect only on certain modules. See documentations
      of particular modules for details of their behaviors in
      training/evaluation mode, if they are affected, e.g. "Dropout",
      "BatchNorm", etc.
      This is equivalent with "self.train(False)".
      See Locally disabling gradient computation for a comparison
      between .eval() and several similar mechanisms that may be
      confused with it.
      Returns:
         self
      Return type:
         Module


self
      Return type:
         Module
   extra_repr()
      Set the extra representation of the module
      To print customized extra information, you should re-implement
      this method in your own modules. Both single-line and multi-line
      strings are acceptable.
      Return type:
         str
   float()
      Casts all floating point parameters and buffers to "float"
      datatype.
      Note:
        This method modifies the module in-place.
      Returns:
         self
      Return type:
         Module
   get_buffer(target)
      Returns the buffer given by "target" if it exists, otherwise
      throws an error.
      See the docstring for "get_submodule" for a more detailed
      explanation of this method's functionality as well as how to
      correctly specify "target".
      Parameters:
         target (str) -- The fully-qualified string name of the
         buffer to look for. (See "get_submodule" for how to specify a
         fully-qualified string.)
      Returns:


fully-qualified string.)
      Returns:
         The buffer referenced by "target"
      Return type:
         torch.Tensor
      Raises:
         AttributeError -- If the target string references an
         invalid     path or resolves to something that is not a
         buffer
   get_extra_state()
      Returns any extra state to include in the module's state_dict.
      Implement this and a corresponding "set_extra_state()" for your
      module if you need to store extra state. This function is called
      when building the module's state_dict().
      Note that extra state should be picklable to ensure working
      serialization of the state_dict. We only provide provide
      backwards compatibility guarantees for serializing Tensors;
      other objects may break backwards compatibility if their
      serialized pickled form changes.
      Returns:
         Any extra state to store in the module's state_dict
      Return type:
         object
   get_parameter(target)


object
   get_parameter(target)
      Returns the parameter given by "target" if it exists, otherwise
      throws an error.
      See the docstring for "get_submodule" for a more detailed
      explanation of this method's functionality as well as how to
      correctly specify "target".
      Parameters:
         target (str) -- The fully-qualified string name of the
         Parameter to look for. (See "get_submodule" for how to
         specify a fully-qualified string.)
      Returns:
         The Parameter referenced by "target"
      Return type:
         torch.nn.Parameter
      Raises:
         AttributeError -- If the target string references an
         invalid     path or resolves to something that is not an
         "nn.Parameter"
   get_submodule(target)
      Returns the submodule given by "target" if it exists, otherwise
      throws an error.
      For example, let's say you have an "nn.Module" "A" that looks
      like this:
         A(
             (net_b): Module(


A(
             (net_b): Module(
                 (net_c): Module(
                     (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2))
                 )
                 (linear): Linear(in_features=100, out_features=200, bias=True)
             )
         )
      (The diagram shows an "nn.Module" "A". "A" has a nested
      submodule "net_b", which itself has two submodules "net_c" and
      "linear". "net_c" then has a submodule "conv".)
      To check whether or not we have the "linear" submodule, we would
      call "get_submodule("net_b.linear")". To check whether we have
      the "conv" submodule, we would call
      "get_submodule("net_b.net_c.conv")".
      The runtime of "get_submodule" is bounded by the degree of
      module nesting in "target". A query against "named_modules"
      achieves the same result, but it is O(N) in the number of
      transitive modules. So, for a simple check to see if some
      submodule exists, "get_submodule" should always be used.


Parameters:
         target (str) -- The fully-qualified string name of the
         submodule to look for. (See above example for how to specify
         a fully-qualified string.)
      Returns:
         The submodule referenced by "target"
      Return type:
         torch.nn.Module
      Raises:
         AttributeError -- If the target string references an
         invalid     path or resolves to something that is not an
         "nn.Module"
   property graph
      Returns a string representation of the internal graph for the
      "forward" method. See Interpreting Graphs for details.
   half()
      Casts all floating point parameters and buffers to "half"
      datatype.
      Note:
        This method modifies the module in-place.
      Returns:
         self
      Return type:
         Module
   property inlined_graph
      Returns a string representation of the internal graph for the
      "forward" method. This graph will be preprocessed to inline all


function and method calls. See Interpreting Graphs for details.
   ipu(device=None)
      Moves all model parameters and buffers to the IPU.
      This also makes associated parameters and buffers different
      objects. So it should be called before constructing optimizer if
      the module will live on IPU while being optimized.
      Note:
        This method modifies the module in-place.
      Parameters:
         device (int, optional) -- if specified, all
         parameters will be copied to that device
      Returns:
         self
      Return type:
         Module
   load_state_dict(state_dict, strict=True)
      Copies parameters and buffers from "state_dict" into this module
      and its descendants. If "strict" is "True", then the keys of
      "state_dict" must exactly match the keys returned by this
      module's "state_dict()" function.
      Parameters:
         * state_dict (dict) -- a dict containing parameters and
           persistent buffers.


persistent buffers.
         * strict (bool, optional) -- whether to strictly
           enforce that the keys in "state_dict" match the keys
           returned by this module's "state_dict()" function. Default:
           "True"
      Returns:
         * missing_keys is a list of str containing the missing
           keys
         * unexpected_keys is a list of str containing the
           unexpected keys
      Return type:
         "NamedTuple" with "missing_keys" and "unexpected_keys" fields
      Note:
        If a parameter or buffer is registered as "None" and its
        corresponding key exists in "state_dict", "load_state_dict()"
        will raise a "RuntimeError".
   modules()
      Returns an iterator over all modules in the network.
      Yields:
         Module -- a module in the network
      Return type:
         Iterator[Module]
      Note:
        Duplicate modules are returned only once. In the following
        example, "l" will be returned only once.