akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Different slicing of a tensor at different rows

hzzt: <p>Given a multidimensional tensor, I want to find the max position for each row in the last dimension, and then extract 50 elements before and after each max position.</p>
<pre><code class="lang-auto">x = torch.rand(2, 4, 256)
peak, i = torch.max(x[...,50:-50], dim=-1)
</code></pre>
<p>But when I try to slice according to indices <code>i</code>, I get an error</p>
<pre><code class="lang-auto">x_subset = x[...,i-50 : i+50]

TypeError: only integer tensors of a single element can be converted to an index
</code></pre>
<p>Is there a good way to use different slices on different rows in a tensor? Thanks.</p>

KFrank: <p>Hi hzzt!</p>
<aside class="quote no-group" data-username="hzzt" data-post="1" data-topic="194808" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/0ea827/48.png" class="avatar"> hzzt:</div>
<blockquote>
<p>But when I try to slice according to indices <code>i</code>, I get an error</p>
<pre><code class="lang-auto">x_subset = x[...,i-50 : i+50]

TypeError: only integer tensors of a single element can be converted to an index
</code></pre>
<p>Is there a good way to use different slices on different rows in a tensor?</p>
</blockquote>
</aside>
<p><code>gather()</code> will do this if you prepare an index tensor and then offset it with<br>
<code>i</code> (the <code>argmax()</code> of <code>x</code>):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.1.2
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt;
&gt;&gt;&gt; x = torch.rand (2, 4, 256)
&gt;&gt;&gt; peak, i = torch.max (x[..., 50:-50], dim = -1)
&gt;&gt;&gt;
&gt;&gt;&gt; ind = torch.arange (100).repeat (*x.size()[0:2], 1)
&gt;&gt;&gt; x_subset = x.gather (2, ind + i.unsqueeze (-1))
&gt;&gt;&gt;
&gt;&gt;&gt; peak
tensor([[0.9945, 0.9997, 0.9907, 0.9982],
        [0.9968, 0.9943, 0.9872, 0.9989]])
&gt;&gt;&gt; x_subset[:, :, 50]
tensor([[0.9945, 0.9997, 0.9907, 0.9982],
        [0.9968, 0.9943, 0.9872, 0.9989]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Using Tensorboard with Optuna

Sart: <p>I trained a simple NN using pytorch and did hyperparameter optimization using optuna. I’m not sure how to integrate it with Tensorboard as there are 10 different experiments which ran with the help of optuna. I’m a beginner so if it’s a naive question, I’m sorry in advance.</p>

Amd: <p>I am also having the same problem did you find a solution for this?</p>
DTensor params vs. model.named_parameters()

xayalith: <p>When creating a sharded model with DTensor, should I expect the parameters shown with <code>model.named_parameters</code> to be sharded or unsharded? e.g. In the below toy example where we shard some Linear layer over 2 devices, should the shapes produced by printing <code>param.shape</code> from <code>model.named_parameters()</code> be sharded or unsharded? Should it change if I remove the all-gather? Trying to get a sense of what to expect when looking at sharded modules. Ideally I’d like to shard e.g. a Linear layer column-wise and be able to see its sharded representation, but I’m not sure if that’s expected behaviour. Thanks!</p>
<pre><code class="lang-auto">from torch.distributed._tensor import DeviceMesh
from torch.distributed.tensor.parallel import parallelize_module, ColwiseParallel
from torch.distributed import _functional_collectives as funcol

mesh = DeviceMesh("cuda", torch.arange(2))

class FFN(torch.nn.Module):
  def __init__(self):
    super(FFN, self).__init__()
    self.w = torch.nn.Linear(in_features=128, out_features=128)
  def forward(self, input):
    return self.w(input)


class Network(torch.nn.Module):
  def __init__(self, mesh):
    super(Network, self).__init__()
    self.linear = FFN()

    parallelize_plan = {"w": ColwiseParallel()}

    self.ffn = parallelize_module(self.linear, mesh, parallelize_plan)
    self.ffn_compiled = torch.compile(self.ffn)

  def forward(self, input):
    return self.ffn_compiled(input)

network = Network(mesh)
network.linear.register_forward_hook(lambda _module, _input, output: funcol.all_gather_tensor(output, -1, list(range(2))))
</code></pre>
PrivateUse1 Backend Registration with libtorch

jlui275: <p>I followed <a href="https://pytorch.org/tutorials/advanced/privateuseone.html" rel="noopener nofollow ugc">this tutorial</a> using <a href="https://github.com/pytorch/pytorch/blob/main/test/inductor/extension_backends/extension_device.cpp" rel="noopener nofollow ugc">this test</a> as an example to integrate my custom backend via PrivateUse1. Everything is working as expected in my PyTorch implementation.</p>
<p>However, I wanted to implement a similar dev environment in C++ in the hopes that I’d be able to set breakpoints and step into my custom kernel implementations for debugging. To my understanding in PyTorch, my custom backend is registered when I call <code>torch.utils.cpp_extension.load</code> method where I pass in my registration and kernel implementation code. What is the process that needs to be followed to register my custom backend to libtorch?</p>
Architecture of distributed Pytorch

ph0123: <p>Hi,<br>
I have a question about the architecture of distributed PyTorch!</p>
<p>When I run some examples, I saw that we can send and receive directly from worker A to worker B.<br>
Why do we need MASTER_PORT and MASTER_ADDRESS?<br>
For port, we can understand that they need this number to recognize other workers which belong to the same program or not. However, I do not understand why we need master_add?</p>
<p>if it is a Master-Slave model, I think that is no problem, and Master worker will manage all works.</p>
<p>Thanks,</p>

mrshenli: <p>Hey <a class="mention" href="/u/ph0123">@ph0123</a></p>
<p>The reason is that when we implement <code>torch.distributed.rpc</code>, we would like to abstract out the comm layer and reuse whatever is available in <code>torch.disributed</code>. At that time, <code>ProcessGroup</code> is the only option we have, which requires a rendezvous during initialization. The master port and address is needed for that rendezvous. Subsequent communications do not go through the master address.</p>
<p>As of v1.6.0, we added a new P2P comm backend implementation, <a href="https://pytorch.org/docs/master/rpc.html#tensorpipe-backend" rel="nofollow noopener">https://pytorch.org/docs/master/rpc.html#tensorpipe-backend</a>. And we do plan to remove the requirement for <code>MASTER_PORT</code>/<code>MASTER_ADDRESS</code>.</p>

selineni: <p>hi Shen Li,</p>
<p>After the rendezvous,  is there a way to restrict the connection between some peers in the p2p alignment.   like for example if there are 3 peers, can i restrict communication between p1-p1 and p1-p3 exclusively.   as I see it the process automatically starts ephemeral ports over tcp to talk to all peers even though i programmatically restricted data communication between p2 and p3?  your response if greatly appreciated</p>

selineni: <p>just like my other post, is there way for tensforpipe backend to limit which node can communicate with which?  a quick response is greatly appreciated</p>
[Help] Reaching definition of Aten matmul function

hungpham3112: <p>I’m figuring out where matmul function in Pytorch is and how it works.</p>
<p>Looked at this <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/LinearAlgebra.cpp" rel="noopener nofollow ugc">doc</a>, I found <code>matmul</code> → <code> __matmul_impl</code> → <code>at::mm_out</code>, but I didn’t found any documentation for <code>at::mm_out</code>. The Pytorch repo is just too big to analyze. Is there any expert can explain how to find definition or read source code of Pytorch more efficient? Thank you.</p>

smth: <p>have a look at this answer: <a href="https://dev-discuss.pytorch.org/t/how-to-find-the-c-cuda-implementation-of-specific-operators-in-pytorch-source-code/1551/3?u=smth" class="inline-onebox">How to find the c++/cuda implementation of specific operators in pytorch source code - #3 by bdhirsh - PyTorch Dev Discussions</a></p>
What is Aten, is Aten a standalone library or created by Pytorch team?

hungpham3112: <p>Hi, I’m new to Pytorch and try to figure out structure of Pytorch repo.</p>
<p>I notice that</p>
<blockquote>
<p>ATen is fundamentally a tensor library, on top of which almost all other Python and C++ interfaces in PyTorch are built. It provides a core <code>Tensor</code> class, on which many hundreds of operations are defined.</p>
</blockquote>
<p>Is ATen created 100% by Pytorch core team or Pytorch use this standalone library in C++ and build from that? Thank you.</p>

ptrblck: <p><code>ATen</code> was developed by the PyTorch team and I believe <a class="mention" href="/u/zdevito">@zdevito</a> came up with the name (“A Tensor Library”), but please correct me if I’m wrong. <a class="mention" href="/u/ezyang">@ezyang</a> can certainly add more details to the history of <code>ATen</code>.</p>

smth: <p>as <a class="mention" href="/u/ptrblck">@ptrblck</a> said, its created by pytorch team for pytorch core use</p>
How do image resize/crop transformations handle small images?

mcfn: <p>i have a dataset of large images (i.e. the parameters i pass to transforms.resize or resized crop are smaller than the image dimensions and it works fine. i added small images to the dataset (i.e. smaller than the parameters). how are they handled by the transformations? also, is it/what is the best way to handle them?</p>

tom: <p>I would start with the <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Resize" rel="nofollow noopener">TorchVision documentation of Resize</a>. It explains</p>
<ul>
<li>If size is a sequence like (h, w), output size will be matched to this.</li>
<li>If size is an int, smaller edge of the image will be matched to this number. i.e, if height  width, then image will be rescaled to (size * height / width, size)</li>
</ul>
<p>So then crop will do its thing on the (upscaled) image.</p>
<p>Best regards</p>
<p>Thomas</p>

AishaMohamed2022: <p>I have same problem .but, i can not understand this solution. can you explain more ?</p>
PyTorch Research Study

vbsc: <p>I am looking for participants for a research study on PyTorch usage. The study is a one-hour interview and you are compensated for your time. We are looking for active PyTorch users (no student projects) that use AWS and PyTorch. Let me know if you are interested.</p>

nicole_cis: <p>Hello,</p>
<p>Hope you are doing well.</p>
<p>I would like to assist you.</p>
<p>Please reach me via Skype/Email.<br>
Skype - nicole_15269</p>
<p>Email - <a href="mailto:nicole@cisinlabs.com">nicole@cisinlabs.com</a></p>
<p>Thanks!<br>
Nicole A</p>

vbsc: <p>Hello Nicole,</p>
<p>Thank you for reaching out! (I just sent you an email as well.)</p>
<p>To give you a little more insight, Evans Data Corporation is are a market research firm. We are putting together a research study to better understand the usage, adoption patterns, likes/dislikes, etc. of those that use PyTorch.</p>
<p>The first step is to set up a 5-minute screening via Google Meet with me. After that, you will move on to the 1-on-1 interview. Again, that interview is one hour. Compensation is $250.</p>
<p>I emailed you this info along with a link to my calendar to book the screening interview.</p>
<p>Hope to talk soon! Have a nice weekend.</p>
<p>Best,</p>
<p>Violet</p>

jowenedc: <p>We’re still looking for participants for this study. This hour-long interview looks at such topics and dislikes, how you use PyTorch and challenges you have with using PyTorch and deploying PyTorch workloads.</p>

Daniel_Darbekov: <p>Hello, I hope I’m not too late to help you</p>
<p>I will be glad to support you in some way</p>
<p>My email: <a href="mailto:darvekov.daniel@bk.ru">darvekov.daniel@bk.ru</a></p>
<p>Good Luck!<br>
Daniel D.</p>
What's the best way to serialize and reuse torch.compile'd functions?

ziyu-guo: <p>Context:</p>
<p>In our pipeline there’re numerous light-weight but compute-intensive functions in different nested loops that can be speed-up using torch.compile.</p>
<p>These functions are separated by relatively complicated business logic that tends to change often, so compiling the entire nested loop is out of the question.</p>
<p>These functions are invoked repeatedly for thousands of times in nested loops, each time taking a handful of different input tensor shapes.</p>
<p>We know that compile with <code>dynamic=True</code> avoids recompilation, but brings additional runtime overhead. Without <code>dynamic=True</code>, everytime input shapes changes will ensue recompilation that brings even more runtime overhead.</p>
<p>Given that we only have ~ a dozen of possible input shapes, and we’re ok with a long initial compilation overhead. Is there a way to pre-compile the specialized versions of the same function, serialize them somewhere, and do runtime dispatch based on input shape?</p>

ziyu-guo: <p>It looks <em>output_code</em> flag can be used to generate python-importable code snippet, but feels quick hacky. Is there any formal serialization functionality in the roadmap?</p>

samsja: <p>curious about a solution as well</p>

smth: <p>the formal solution for serialization of compiled code is <code>torch.export</code></p>
Cupti library used by PyTorch

Rithvik_Kolla: <p>Hey folks,</p>
<p>My query is simple. Is there any way to deduce the exact Cupti library that PyTorch uses from a piece of Python code? Most times, torch uses the base version of the Cupti library that comes along with the CUDA toolkit installation. However, in more recent versions, torch has begun to ship a dedicated Cupti library as part of the torch installation.</p>
<p>I can use tools like LD_DEBUG and deduce this in an interactive environment. I wanted to know if there’s a way to automatically get this information from torch (even a hacky way would do)?</p>
<p>PS: I later intend to use this same library to add more tracing functionality and get some additional metrics.</p>

smth: <p>can you derive it from information in <code>torch.__config__.show()</code>? It has info on what CUDA toolkit version the binary was compiled with.</p>

ptrblck: <p><code>pip list</code> should also show the installed dependencies including the full version tags:</p>
<pre data-code-wrap="python"><code class="lang-python">pip list | grep nvidia
nvidia-cublas-cu12            12.1.3.1
nvidia-cuda-cupti-cu12        12.1.105
nvidia-cuda-nvrtc-cu12        12.1.105
nvidia-cuda-runtime-cu12      12.1.105
nvidia-cudnn-cu12             8.9.2.26
nvidia-cufft-cu12             11.0.2.54
nvidia-curand-cu12            10.3.2.106
nvidia-cusolver-cu12          11.4.5.107
nvidia-cusparse-cu12          12.1.0.106
nvidia-nccl-cu12              2.19.3
nvidia-nvjitlink-cu12         12.1.105
nvidia-nvtx-cu12              12.1.105
</code></pre>
<p>E.g. this is the output of the current nightly binary.</p>
Skip a submodule that cannot be compile

samsja: <p>Hey all,</p>
<p>I am using xformers to perform fast scale dot product and the function is not compatible with torch.compile.</p>
<p>Though it is the only place in my code that is not compatible. I would like to be able to call torch.compile() on my model and be able to just “skip” the part that is calling xformers function from compilation.</p>
<p>Right now to achieve the same I have to apply torch.compile manually on every layer that is not the xformers one. Is there a way to achieve the same in an automatic matter ?</p>
<p>Thanks in advance</p>

mclaugha: <p>Depending on your exact use case it sounds like you might want <code>torch._dynamo.disable</code> or <code>torch._dynamo.disallow_in_graph</code>. See here: <a href="https://pytorch.org/docs/stable/torch.compiler_faq.html#how-do-i-graph-break-on-a-function" class="inline-onebox" rel="noopener nofollow ugc">Frequently Asked Questions — PyTorch 2.1 documentation</a></p>

samsja: <p>It is exactly what I was looking for. Thanks !</p>
<p>By any chance do you know why it is a private function here ? Does it just mean it is experimental for now and will be stabilize later ?</p>

mclaugha: <p>Great! I’m not sure how stable these methods are but the fact that they’re documented implies to me that they’re reasonably safe to use.</p>
Bahdanau Attention on 1d input

Noorulain_Islam: <p>Hi, I want to apply Bahdanau atetntion on my two inputs, my input data has not sequence length. How can I apply in a correct way?</p>
<pre data-code-wrap="python"><code class="lang-python">input1 = torch.randn(128, 64)
input2 = torch.randn(128, 64)
in my input, 128 represent the total number of training samples and 64 is the feature dimension. I want to apply Bahdanau attention on these two inputs.
I am applying this way, but it is not giving me output of shape (1,64). 
class BahdanauAttention(nn.Module):
    def __init__(self, hidden_size):
        super(BahdanauAttention, self).__init__()
        self.Wa = nn.Linear(hidden_size, hidden_size)
        self.Ua = nn.Linear(hidden_size, hidden_size)
        self.Va = nn.Linear(hidden_size, 1)

    def forward(self, query, keys):
        scores = self.Va(torch.tanh(self.Wa(query) + self.Ua(keys)))
        print(scores.shape)
        scores = scores.squeeze(1).unsqueeze(0)
        print(scores.shape)
        weights = F.softmax(scores, dim=1)
        context = torch.mm(weights, keys)

        return context, weights
</code></pre>
Why there's a line `scheduler.step(0)` in the code of class `SequentialLR`?

jack_S: <p>I was using a <code>SequentialLR</code> to adjust my learning rate in the training process. However, a warning appeared:</p>
<pre><code class="lang-auto">UserWarning: The epoch parameter in `scheduler.step()` was not necessary
and is being deprecated where possible. Please use `scheduler.step()` to 
step the scheduler. During the deprecation, if epoch is different from None, 
the closed form is used instead of the new chainable form, where available. 
Please open an issue if you are unable to replicate your use case: 
https://github.com/pytorch/pytorch/issues/new/choose.
warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
</code></pre>
<p>I checked my code and there was nothing like <code>scheduler.step(epoch)</code>. After debugging, I found the problem was in <code>torch\optim\lr_scheduler.py</code>. The line <code>scheduler.step(0)</code> in <code>SequentialLR.step</code> caused the warning.</p>
<pre><code class="lang-auto">def step(self):
    self.last_epoch += 1
    idx = bisect_right(self._milestones, self.last_epoch)
    scheduler = self._schedulers[idx]
    if idx &gt; 0 and self._milestones[idx - 1] == self.last_epoch:
        scheduler.step(0)
    else:
        scheduler.step()

    self._last_lr = scheduler.get_last_lr()
</code></pre>
<p>I wonder why there’s a line <code>scheduler.step(0)</code> in <code>SequentialLR</code>. After all, the <code>epoch</code> parameter should be deprecated.</p>

albanD: <p>Hi!</p>
<p>Most likely an oversight on our end during the deprecation process. Could you please open an issue about this on github?</p>

jack_S: <p>Sure, my pleasure. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Specify retain_graph=True not working for RuntimeError: Trying to backward through the graph a second time

shrbrh: <p>Hello, I am trying to train a CycleGAN model in PyTorch. Following is the code snippet for training the two generators alternately:</p>
<pre><code># Training G_ST and G_TS
    if epoch % 5 == 0:
        optimizer_G_ST.zero_grad()
        L_GAN_ST_forGST = - torch.mean(torch.log(D_T_G_ST_E_x_s_forGST))
        L_cyc_forGST = cycleLoss(G_ST_G_TS_E_x_t, E_x_t)
        L_GAN_ST = L_GAN_ST_forGST + L_cyc_forGST
        L_GAN_ST.backward(retain_graph=True)
        optimizer_G_ST.step()

    if epoch % 5 == 1:
        optimizer_G_TS.zero_grad()
        L_GAN_TS_forGTS = - torch.mean(torch.log(D_S_G_TS_E_x_t_forGTS))
        L_cyc_forGTS = cycleLoss(G_TS_G_ST_E_x_s, E_x_s)            
        L_GAN_TS = L_GAN_TS_forGTS + L_cyc_forGTS
        L_GAN_TS.backward()
        optimizer_G_TS.step()
</code></pre>
<p>For the line of code <code>L_GAN_ST.backward(retain_graph=True)</code>, I am getting the following error:<br>
<code>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</code></p>
<p>What is the reason for this error and how do I resolve this? A few answers have suggested using <code>retain_graph=True</code> which I tried but it is not working. Please help.</p>

ptrblck: <aside class="quote no-group" data-username="shrbrh" data-post="1" data-topic="194685">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b5e925/48.png" class="avatar"> shrbrh:</div>
<blockquote>
<p>A few answers have suggested using <code>retain_graph=True</code> which I tried but it is not working.</p>
</blockquote>
</aside>
<p>Which posts suggested this as a solution as it’s generally wrong?</p>
<p>You might be running into <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this issue</a>.</p>

shrbrh: <p>I was talking about <a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795">this post</a>.<br>
Also, the discussion in <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/5">this issue</a> didn’t help. I am still facing the same thing.</p>

ptrblck: <p>Did you read my linked post and did you try to check if your training also wants to use stale gradients?</p>
How to read file only once in DDP

hcrot: <p>When I run the training code for DDP, each process reads the dataset, which comes down to N times file read. When the file size is too large, I run out of memory (not GPU memory but CPU’s).</p>
<p>Are there any better ways to load the dataset file in the memory-efficient way?<br>
(To be clear, I’m not talking about the dataloader class when I say ‘load’ - I’m saying about the pre-stage for dataloading - reading the dataset file from disk.)</p>

ptrblck: <p>If you are lazily loading samples you would use a <code>DistributedRandomSampler</code> and would thus load only the samples assigned to the corresponding rank.</p>
UninitializedTensorMixin - requires_grad allowed?

tjoseph: <p>Hi <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I am wondering why <code>requires_grad_</code> is not in <code>_allowed_methods</code>? Is there a reason or has it just been forgotten?</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/91bbcf8c718ff245ec85e33874dc4c1a1a28c4f3/torch/nn/parameter.py#L86">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/91bbcf8c718ff245ec85e33874dc4c1a1a28c4f3/torch/nn/parameter.py#L86" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/91bbcf8c718ff245ec85e33874dc4c1a1a28c4f3/torch/nn/parameter.py#L86" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/91bbcf8c718ff245ec85e33874dc4c1a1a28c4f3/torch/nn/parameter.py#L86</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="76" style="counter-reset: li-counter 75 ;">
          <li></li>
          <li>        return (</li>
          <li>            torch._utils._rebuild_parameter_with_state,</li>
          <li>            (self.data, self.requires_grad, hooks, state)</li>
          <li>        )</li>
          <li></li>
          <li>    __torch_function__ = _disabled_torch_function_impl</li>
          <li></li>
          <li></li>
          <li>class UninitializedTensorMixin:</li>
          <li class="selected">    _allowed_methods = [</li>
          <li>        torch.Tensor.__hash__,</li>
          <li>        torch.Tensor.size,</li>
          <li>        torch.Tensor.copy_,</li>
          <li>        torch.Tensor.is_floating_point,</li>
          <li>        torch.Tensor.half,</li>
          <li>        torch.Tensor.float,</li>
          <li>        torch.Tensor.double,</li>
          <li>        torch.Tensor.char,</li>
          <li>        torch.Tensor.short,</li>
          <li>        torch.Tensor.int,</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Is it possible to get the model weights from optimizer

Xiaowen_Sun: <p>I accidentally only saved the optimizer and forgot to save the model.state_dict(). Is it possible to rebuild the model.state_dict() with the weights saved in the optimizer?  Any idea is helpful.</p>

albanD: <p>Hey!</p>
<p>I’m afraid it is not possible no. The optimizer explicitly does not save model weights to avoid saving them twice.</p>
Inheriting functional transforms for custom TVTensor

polo5: <p>The latest pytorch v2 transforms allow creating custom TVTensors (see <a href="https://pytorch.org/vision/stable/auto_examples/transforms/plot_custom_tv_tensors.html" rel="noopener nofollow ugc">here</a>). which allows registering functional transforms specific to a TVTensor type.</p>
<p>But what about applying the default F.transform on my custom TVTensor? I would need to convert TVTensor → torch.Tensor first, but I think the way I’m doing it creates a copy and <code>torch.tensor()</code> is deprecated.</p>
<pre><code class="lang-auto">from torchvision import tv_tensors
from torchvision.transforms.v2 import functional as F

class DepthmapVideoTVTensor(tv_tensors.TVTensor):
    """
    Modified from the Video TVTensor here:
    https://github.com/pytorch/vision/blob/main/torchvision/tv_tensors/_video.py
    """
    def __new__(cls, data: Any, *, dtype: Optional[torch.dtype] = None, 
                device: Optional[Union[torch.device, str, int]] = None,
                requires_grad: Optional[bool] = None):
        
        tensor = cls._to_tensor(data, dtype=dtype, device=device, requires_grad=requires_grad)
        if data.ndim != 4: #(seq_len, C, H, W)
            raise ValueError
        return tensor.as_subclass(cls)

    def __repr__(self, *, tensor_contents: Any = None) -&gt; str: 
        return self._make_repr()

@F.register_kernel(functional=F.pad, tv_tensor_cls=DepthmapVideoTVTensor)
def pad_depthmap(depthmap, *args, **kwargs):
    print("Padding Depthmap!")
    depthmap_new = torch.tensor(depthmap)
    depthmap_new = F.pad(depthmap_new, *args, **kwargs)
    return tv_tensors.wrap(depthmap_new, like=depthmap)

</code></pre>
<p>Any better ideas? Thanks!</p>
How to reduce the size of the PyTorch package after using PyInstaller with -D option, resulting in a 3GB size for the torch file?

make-a: <p>How to reduce the size of the PyTorch package after using PyInstaller with -D option, resulting in a 3GB size for the torch file?</p>

albanD: <p>Hey,</p>
<p>I’m not very familiar with PyInstaller, is it pulling in all dependencies? Also you should be careful if you build with CUDA to limit which architecture it uses as that can quickly increase the binary size.</p>
TorchRL duplicates model weights in LossModule's functional paramers

wbinventor: <p>The TorchRL <code>LossModule.convert_to_functional(...)</code> method creates a deep copy of the parameters. If I understand correctly, this leads to the parameters being duplicated in memory and a larger memory footprint than necessary. Is my understanding correct? If so, why is this necessary? Is there any way for the <code>LossModule</code> simply contain a single reference to the weights for, e.g., its actor and critic <code>TorchModule</code>s?</p>
<p>This deepcopy’ing of parameters can be seen in the following line of code: <a href="https://github.com/pytorch/rl/blob/80c63ad40c7356559a2d4991215eb331ecff0f95/torchrl/objectives/common.py#L290" rel="noopener nofollow ugc">rl/torchrl/objectives/common.py</a></p>
<p>This is the specific snippet of code containing the deep copy that this question pertains to:</p>
<pre><code class="lang-auto"># set the functional module: we need to convert the params to non-differentiable params
# otherwise they will appear twice in parameters
with params.apply(
    self._make_meta_params, device=torch.device("meta")
).to_module(module):
    # avoid buffers and params being exposed
    self.__dict__[module_name] = deepcopy(module)
</code></pre>

vmoens: <p>Thanks for posting this!<br>
Please see the <a href="https://github.com/pytorch/rl/issues/1769#issuecomment-1876672355" rel="noopener nofollow ugc">answer on github</a>.</p>
Worse C++ performance in simple conv2d implementation

nclslvr: <p>I am trying to check the performance gain of using a c++ extension instead of just python, when implementing custom conv2d operations.<br>
When benchmarking it for a simple conv2d implementation, I although get worse performance with the c++ extension (83µs vs 56µs), despite virtually identical code. What could cause this ?</p>
<p>Here’s my python code:</p>
<pre><code class="lang-auto">import torch
from torch import nn, Tensor
import torch.nn.functional as F
import time
import conv2d_cpp

def time_fn(fn, n=10000):
    t0 = time.time()
    for _ in range(n):
        fn()
    t = time.time()-t0
    torch.cuda.synchronize()
    print(f"{t/n*10**6:.2f}µs per loop")

def conv_2d(x: Tensor, kernel: Tensor, bias: Tensor) -&gt; Tensor:
    """get each individual conv window to compute the whole conv in one matmul"""
    assert(x.shape[0] == kernel.shape[0] and kernel.shape[1] == bias.shape[0])
    output_shape = (x.shape[1] - kernel.shape[2]) + 1
    im2col_matrix = F.unfold(x, kernel.shape[2])
    im2col_conv = torch.matmul(kernel.moveaxis(0, 1).flatten(start_dim=1), im2col_matrix) + bias
    im2col_conv = im2col_conv.reshape(kernel.shape[1], output_shape, output_shape)
    return im2col_conv

def main():
    device = 'cuda'
    input = torch.randint(-255, 255, size=(1, 256, 256)).float().to(device)
    kernel = torch.tensor([[[[8., 9., 7.], [4., 4., 2.], [-1., 6., -3.]]]]).to(device)
    bias = torch.tensor([[0.06]]).to(device)
    di, do = 16, 32
    input, kernel, bias = input.repeat(di, 1, 1), kernel.repeat(di, do, 1, 1), bias.repeat(do, 1)
    time_fn(lambda: conv2d_cpp.forward(input[None], kernel, bias))
    time_fn(lambda: conv_2d(input, kernel, bias))
</code></pre>
<p>and my cpp code:</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;

namespace F = torch::nn::functional;

torch::Tensor conv2d_forward(
  torch::Tensor im, // should be of shape (bs, di, side, side)
  torch::Tensor kernel, // (di, do, ks, ks)
  torch::Tensor bias
) {
  auto ks = kernel.size(3);
  auto stride = ks/2;
  auto side = im.size(2);
  auto output_shape = side - ks + 1;
  auto im2col_matrix = F::unfold(im, F::UnfoldFuncOptions({ks, ks}).padding(0).stride(stride));
  auto im2col_conv = torch::matmul(kernel.moveaxis(0, 1).flatten(1), im2col_matrix) + bias;
  return im2col_conv.reshape({kernel.size(1), output_shape, output_shape});
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
    m.doc() = "Implementation of a forward pass of conv2d in C++";
    m.def("forward", &amp;conv2d_forward, "Conv2d forward");
}
</code></pre>

eqy: <p>You might be missing a <code>torch.cuda.synchronize</code> at the beginning of your timing function which could lead you to include previous CUDA kernels in your timing such as the host to device copies.</p>
<p>Additionally, there may be a first-time cost of loading the kernels/heuristics for running the matmul, so you may want to include a few warmup runs so that the approach that is tested first is not penalized.</p>
Trying to Use & Understand iNaturalist Dataset

Cory_Root: <p>Hello. I’m trying to understand the iNaturalist dataset and I can’t figure out what the target variable means.</p>
<p>For example, if <code>target_type</code> is set to ‘genus’, then I would expect the target value to equal that entry’s <em>genus</em> id. But it doesn’t appear so:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a056f453bd9072622f22d80cc3a3aa2986abfa20.png" data-download-href="https://discuss.pytorch.org/uploads/default/a056f453bd9072622f22d80cc3a3aa2986abfa20" title="Screenshot 2024-01-02 at 3.21.31 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a056f453bd9072622f22d80cc3a3aa2986abfa20_2_690x251.png" alt="Screenshot 2024-01-02 at 3.21.31 PM" data-base62-sha1="mSqH2AcPZwdy94VvMbXVilYP4PK" width="690" height="251" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a056f453bd9072622f22d80cc3a3aa2986abfa20_2_690x251.png, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a056f453bd9072622f22d80cc3a3aa2986abfa20.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a056f453bd9072622f22d80cc3a3aa2986abfa20.png 2x" data-dominant-color="F8F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-01-02 at 3.21.31 PM</span><span class="informations">772×281 21.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What is this target value of 464?</p>
<p>Thank you,<br>
Cory</p>
How can I do a masked scatter operation without a memory allocation?

galv: <p>Hi all,</p>
<p>Basically, I would like to do this in CUDA in eager mode, without performing a memory allocation:</p>
<pre><code class="lang-auto">dest[mask] = src[mask]
</code></pre>
<p>Unfortunately, this does a memory allocation. You can confirm this as follows:</p>
<pre><code class="lang-auto">import torch

dest = torch.tensor([-1, -1, -1, -1, -1], device="cuda")
src  = torch.tensor([0, 1, 2, 3, 4], device="cuda")
mask = torch.tensor([True, False, False, False, True], device="cuda")

torch.cuda.set_sync_debug_mode(2)

dest[mask] = src[mask]
</code></pre>
<p>This outputs <code>RuntimeError: called a synchronizing CUDA operation</code>.</p>
<p>I understand why this does an allocation, at least in eager mode. Since the execution is imperative, there is no way to know that the output of executing src[mask] can be written directly into the relevant coefficients of dest[mask], since they are both indexed by the same mask. Therefore, an allocation must happen, and of course this allocation is dependent upon the number of “True” values in mask.</p>
<p>I originally thought that I should use <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.masked_scatter_.html#torch.Tensor.masked_scatter_" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.masked_scatter_ — PyTorch 2.1 documentation</a>, but this does not perform a “scatter” as most people think of it, without going into details. This stack overflow question explains in more detail: <a href="https://stackoverflow.com/questions/68675160/torch-masked-scatter-result-did-not-meet-expectations" class="inline-onebox" rel="noopener nofollow ugc">pytorch - torch.masked_scatter result did not meet expectations - Stack Overflow</a></p>
<p>The motivation is that I want to use cuda graphs to speed up a particular model inference that has several short-running cuda kernels (so much so that 90% of the time is spent on the CPU). However, without a way to do a “masked select” that avoids a memory allocation, which is synchronous, I seem to be out of luck.</p>
<p>torch.index_select and friends don’t work because I need to create the index tensor in the first place from the mask tensor, which involves doing a memory allocation (since I need to count the number of instances of True in the mask tensor).</p>

albanD: <p>Hey!</p>
<p>The problem is indeed that the operation you’re doing depends on the value of the content.<br>
You can avoid that by doing Tensor-wide ops (that will of course not be as efficient for very sparse mask):</p>
<pre><code class="lang-auto">res = dest - dest * mask + src * mask
</code></pre>
<p>You can make that into inplace to reduce extra allocations as needed. But that won’t trip debug mode/cudagraph for sure.</p>

galv: <p>Actually, someone offline told me that <code>torch.where(mask, src, dest, out=dest)</code> is exactly what I need, so that is what I went with.</p>
Confused about default values in `torch.nn.init_kaiming_normal_`

ado_sar: <p>I am reading the <a href="https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_" rel="noopener nofollow ugc">docs</a> for initializing the weights according to <code>torch.nn.init.kaiming_normal_</code> and I have trouble to understand the following:</p>
<p>Why the default value for <code>nonlinearity</code> is <code>leaky_relu</code> while the default value for <code>a</code> is <code>0.01</code>?</p>

albanD: <p>Hi,</p>
<p>It might be a bit confusing  but the idea is that the default indeed a regular relu layer but we also want users to be able to change <code>a</code> without having to change the nonlinearity flag. Hence the default there being leaky_relu.</p>

ado_sar: <p><a class="mention" href="/u/alband">@albanD</a> Thanks for the reply! I had a typo in the OP. The default value for <code>nonlinearity</code> is <code>leaky_relu</code> but the default value for <code>a</code> is <code>0</code>.</p>