akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Complex indexing to avoid for-loop on GPU

cmetzner: <p>Hello,</p>
<p>I am trying to perform complex indexing to avoid using for-loops on GPU. My problem is as follows, I have a large matrix X = [B, N, H], a smaller matrix H = [B, N, M, H] (which is initialized with as zeros), and a list containing indices L, where each list has the coordinates l = [b, n, m], where B: batch size, N: number of tokens in sequence, M: number of sub_tokens - M is a subset of N, H: hidden dimension.</p>
<p>Specifically, I want to populate H with samples from X; however, not every N in H samples M times. Also there is no sampling across the batch B. Let me illustrate with an example.</p>
<p>X = tensor(</p>
<pre><code>              [[[ 0,  1,  2,  3],
                [ 4,  5,  6,  7],
                [ 8,  9, 10, 11]],

               [[12, 13, 14, 15],
                [16, 17, 18, 19],
                [20, 21, 22, 23]]]
</code></pre>
<p>)</p>
<p>Index = [[0, 0, 0], [0, 0, 1], [0, 1, 0], [0, 2, 0], [0, 2, 1], [1, 0, 0], [1, 1, 1], [1, 2, 1], [1, 2, 2]]</p>
<p>H = tensor(</p>
<pre><code>              [[[[0, 1, 2, 3],
                 [4, 5, 6, 7]],

                [[0, 1, 2, 3],
                 [0, 0, 0, 0]],

                [[0, 1, 2, 3],
                 [4, 5, 6, 7]]],

              [[[12, 13, 14, 15],
                [0, 0, 0, 0]],

               [[16, 17, 18, 19],
                [0, 0, 0, 0]],

              [[16, 17, 18, 19],
               [20, 21, 22, 23]]]])
</code></pre>
<p>If anyone has an idea how to solve this that would be amazing!</p>
<p>Thank you,<br>
Christoph</p>

cmetzner: <p>Hi,</p>
<p>I found a different solution that tackles the problem from a different angle and does not require complex indexing as mentioned in the post.</p>
<p>Thanks.</p>
Bazel build of Python library

nzmsv: <p>I’m trying to use Bazel to build PyTorch and I have it mostly working.  Running into an annoying issue though: I am not sure what target to use in a <code>py_binary</code> so that I can depend on Torch from Python.</p>
<p>There’s the <code>:torch</code> target which produces <code>libtorch.so</code>.  There’s the <code>:torch_python</code> that produces the library with the right Python bindings.  There’s the <code>:_C</code> target that wraps the latter in a <code>pybind_extension</code>, and that is <em>almost</em> what I need, but not quite.  That’s the prerequisite for doing <code>import torch.C_</code> from Python… but as far as I can tell there is nothing that packages up the <code>torch</code> module itself (and would let me do <code>import torch</code>).</p>
<p>I feel like that would be crazy - it’s just something that can’t be missing.  And yet I cannot find it.  I found an older post that seems to be adding what I want (<a href="https://discuss.pytorch.org/t/pytorch-and-bazel/42592" class="inline-onebox">PyTorch and Bazel</a>) but I’m just wondering if there is a proper solution already upstream.</p>
Pytorch bazel build can not generate the libtorch.so and etc

ArtIntAI: <p>As the title says, when we use the bazel build, there are no libtorch.so and etc *.so generated.<br>
This makes it hard use the custom operator .so in the situation of source code build by bazel.<br>
Will you guys to do the same .so generate in bazel comparing with the cmake?</p>

glaringlee: <p><a class="mention" href="/u/malfet">@malfet</a> Can you pls help?</p>

nzmsv: <p><code>bazel build //:torch</code> produces the .so files for me (under <code>bazel-out</code>).  Any errors you are running into?</p>
Use of CUDA Graphs for forward and for backward Dynamo/Inductor-compiled modules

vadimkantorov: <p>As far as I understood, Inductor-generated code does not yet make use of CUDA graphs for futher speed-ups of multi-kernel code paths.</p>
<p>Is there somewhere a practical example of how to still use CUDA graphs for Inductor/Triton-generated code paths? (I read somewhere that e.g. the same input/output tensors must be used, so it should probable deal with this and with fixed collection of input shapes?)</p>

marksaroufim: <p>It should work yeah, might just need to call <code>torch._dynamo.reset()</code> in between different calls</p>

vadimkantorov: <p>Btw does torch.compile support different compilation options for different subgraphs, e.g. will it work as exptected if I first compile with some option a sub-module (e.g. with reduce-overhead), and then compile the whole module (without such options)?</p>
<p>Also, a question on whether it imposes constraints on using the same input/output tensors + multiple possible batch shapes?</p>
How to limit the number of CPUs used by PyTorch?

f3ba: <p>I am running my training on a server which has 56 CPUs cores. When I train a network PyTorch begins using almost all of them.</p>
<p>I want to limit PyTorch usage to only 8 cores (say). How can I do this?</p>

albanD: <p>Hi,</p>
<p>You can use <a href="https://pytorch.org/docs/stable/torch.html#torch.set_num_threads"><code>torch.set_num_threads()</code></a> to do this.</p>
Attribute error occurs in a function about normalizing flow

jack_S: <p>I wanted to use <code>torch.compile</code> to acclerate my training then I got an attribute error. Here’s the code and the traceback:</p>
<pre><code class="lang-auto">model = torch.compile(model)

def fun(model, data):
    fparam = model.flow.sample(data.shape[0])

opt_fun = torch.compile(fun)
opt_fun(model)
</code></pre>
<p>where <code>model.flow</code> is a <code>distribution</code></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
    out_code = transform_code_object(code, transform)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object
    transformations(instructions, code_options)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
    tracer.run()
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run
    super().run()
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run
    and self.step()
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step
    getattr(self, inst.opname)(inst)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1030, in LOAD_ATTR
    result = BuiltinVariable(getattr).call_function(
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 566, in call_function
    result = handler(tx, *args, **kwargs)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 930, in call_getattr
    return obj.var_getattr(tx, name).add_options(options)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 124, in var_getattr
    subobj = inspect.getattr_static(base, name)
  File "/root/miniconda3/envs/myconda/lib/python3.10/inspect.py", line 1769, in getattr_static
    raise AttributeError(attr)
AttributeError: flow

from user code:
   File "/mnt/VADA/test.py", line 152, in fun
     fparam = model.flow.sample(data.shape[0])


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True
</code></pre>

jack_S: <p>Wasn’t I misunderstand what is a compiled model? I used to think <code>torch.compile</code> gives a faster model that is almost the same as before. But now it seems that <code>model</code> has attribute <code>flow</code> doen’t mean the compiled model also has it.</p>
Training losses remain unchanged

Debt: <p>I have built an image enhancement network based on the paper  , which requires convolution of the generated filter kernel and input image. However, after writing the convolution process, the model gradient will not change during training</p>
<p>my code is :</p>
<pre><code>def extract_patche_onechannel(img, k_size, stride=1):

    kernel_size = torch.zeros(k_size, k_size).cuda()

    kernel_list = []
    for i in range(k_size):
        for j in range(k_size):
            k = kernel_size.clone()
            k[i][j] = 1
            kernel_list.append(k)

    kernel = torch.stack(kernel_list)
    kernel = kernel.unsqueeze(1)
    weight = nn.Parameter(data=kernel, requires_grad=True)
    out = F.conv2d(img[:, :1, ...], weight, padding=int((k_size - 1) / 2), stride=stride)
    return out

def forward(self, input_img):
    dynamic_filters = self.general_model(input_img)

    dynamic_filters = self.softmax(dynamic_filters)
    input_transformed = self.extract_patche_onechannel(input_img, k_size=1) 
    output_dynconv = torch.sum(dynamic_filters * input_transformed.cuda(), dim=1, keepdim=True)
    return output_dynconv
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e57f875144c6f57da29b642cb0692a07ab2f2f34.png" data-download-href="https://discuss.pytorch.org/uploads/default/e57f875144c6f57da29b642cb0692a07ab2f2f34" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e57f875144c6f57da29b642cb0692a07ab2f2f34_2_690x307.png" alt="image" data-base62-sha1="wKezj16tSokdkV5Q1TAkubQaNP6" width="690" height="307" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e57f875144c6f57da29b642cb0692a07ab2f2f34_2_690x307.png, https://discuss.pytorch.org/uploads/default/original/3X/e/5/e57f875144c6f57da29b642cb0692a07ab2f2f34.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/5/e57f875144c6f57da29b642cb0692a07ab2f2f34.png 2x" data-dominant-color="E6E1E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">929×414 95.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on your code snippet you are initializing the trainable <code>weight</code> parameter from scratch in each new forward pass which would break the training process. Instead, initialize the <code>weight</code> once and reuse it in the <code>forward</code> method.</p>

Debt: <p>Thank you for your answer. I set the weight parameter attribute to False for retraining, and the training loss will not decrease. I suspect that my code may have caused gradient loss. Another possibility is that the code I used to use convolutional kernels to convolution images is an error code</p>

ptrblck: <p>Sorry, I’m unsure how you are setting the parameter to <code>False</code>, but did you check my previous recommendation in initializing the parameter once?</p>
Error called AttributeError:__enter__

peony: <p>I am practicing with a code, and below is the snippet:</p>
<p><code>def readfile(data_pt): return open(data_pt, "r")</code></p>
<pre><code class="lang-auto">    for file in files:       
        with readfile(file) as openfile:
            molecules.append(process_file_fn(openfile))
</code></pre>
<p>However, I get this error <code>with readfile(file) as openfile: AttributeError: __enter__</code></p>
<p>I tried to define the mode which is read in both the function and the loop but I am met with the same error. Any insights on how to fix this error? Thank you in advance</p>

eqy: <p>Have you overridden <code>readfile</code> or <code>open</code> somewhere else? I cannot reproduce this issue:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; with open('a', 'r') as f:
...   pass
...
&gt;&gt;&gt; d = open('a', 'r')
&gt;&gt;&gt; with d as f:
...   pass
...
&gt;&gt;&gt; def readfile(s):
...   return open(s, 'r')
...
&gt;&gt;&gt; with readfile('a') as f:
...   pass
...
&gt;&gt;&gt; with readfile('a') as f:
...   print("hello")
...
hello
</code></pre>

peony: <p>oh sorry I posted the wrong first part of the code that was causing the error. This one is the correct part that is probably causing the error:</p>
<pre data-code-wrap="if"><code class="lang-plaintext">        tardata = tarfile.open(data, "r")
        files = tardata.getmembers()

        def readfile(data_pt): return tardata.extractfile(data_pt)
</code></pre>
<p>basically, I am trying to process a .tar file</p>
TorchScript hardcoded tensor shapes

Sylvain55: <p>Hello !</p>
<p>Let’s say I have a warping module that inherits from nn.Module.<br>
In the constructor of this module I create a meshgrid of a particular shape as a registered buffer.<br>
Then in the forward method of this module, which takes a batch of images and a batch of homographies as inputs, I compute a new grid of coordinates using input homographies and the base meshgrid defined in the constructor. Then I use grid_sample to warp input images.</p>
<p>When I use torch.jit.trace() on this module, it seems only usable with the size given to torch.jit.trace(), “input_sample”. So the .pt export seems not dynamic, <strong>the size of the meshgrid in the constructor seems hardcoded in the TorchScript.</strong></p>
<p>How can I do to have a “dynamic” TorchScript that would allow me to warp any shapes ?</p>
<p>Thanks for your help !</p>

marksaroufim: <p>you want to check up <code>torch.jit.script()</code> instead - also keep in mind torchscript is in maintenance mode so compiler efforts are mostly under <code>torch.compile()</code>now which has a <code>dynamic</code> parameter you can use</p>

Sylvain55: <p>Ok thank you I will check this out !</p>
Why not move torchvision download utils to torchdata?

biphasic: <p>Hello,<br>
I’m the maintainer of a datasets/transforms library for event-based data called Tonic: <a href="https://github.com/neuromorphs/tonic" class="inline-onebox" rel="noopener nofollow ugc">GitHub - neuromorphs/tonic: Publicly available event datasets and transforms.</a><br>
I modeled the API after torchvision to a great extent, which was so helpful! One thing though that always bugged me is that I relied on a copy of the <a href="https://github.com/pytorch/vision/blob/main/torchvision/datasets/utils.py" rel="noopener nofollow ugc">download utils</a> to be able to download and check datasets. These days I’m working on migrating Tonic to depend on torchdata and datapipes, which I’m really looking forward to. My hope was that I could rely on torchdata functionality to download and check data. Unfortunately I saw that torchvision relies on its own download utils <a href="https://github.com/pytorch/vision/blob/main/torchvision/prototype/datasets/utils/_resource.py" rel="noopener nofollow ugc">scripts</a>. I was wondering, why not move those scripts to torchdata directly? At least Tonic, but likely also audio/text could benefit from that code.<br>
Thank you so much!</p>

ptrblck: <p>I think your idea is great and am sure the code owners would be interested in discussing it further. Would you mind creating a GitHub issue in the <code>vision</code> or <code>data</code> repository and would you be interested in working on an implementation?</p>

biphasic: <p>Thank you for the suggestion <a class="mention" href="/u/ptrblck">@ptrblck</a> , I opened an issue <a href="https://github.com/pytorch/vision/issues/7549" rel="noopener nofollow ugc">here</a>.</p>
Are the class weights of the CE loss optimizble?

setsailforfail: <p>The cross entropy loss of PyTorch has a optional parameter “weights”, that multiplies the loss of the predicted class by a user defined value.<br>
For a class that is more present in the data, we can put a lower weight on that class, and the model will more likely predict other, weighted, classes.</p>
<p>That is my current understanding. I am curious however, if this parameter is optimizble?<br>
After all, it appear in the loss function and one could potentially calculate the loss w.r.t to the class weight and change them during training.</p>
<p>Does this make sense? Or will it simply result in predicting the majority class, as in the initiall setup without the weights?<br>
Lastly, can this be implemented with a torch optimizer like Adam?</p>

ptrblck: <p>You could apply a manual weighting as seen in <a href="https://discuss.pytorch.org/t/does-weightedrandomsampler-increase-interations/51514/16">this post</a> and optimize the <code>weights</code> during the training.<br>
However, I would assume the model could try to move the weight values to zeros or large negative values to decrease the weighted loss, so you might need to e.g. normalize them or add a penalty.</p>
Why tensor indexing on GPU calls GPU->CPU memcpy and cudaStreamSync?

rhwang: <p>I found some results on PyTorch that are very interesting and cannot be easily understood.</p>
<p>Here’s the code example.</p>
<pre><code class="lang-auto">import torch
import torch.cuda.nvtx as nvtx_cuda
torch.cuda.cudart().cudaProfilerStart()

idx = torch.tensor([2,3,4], device="cuda")

tensor_0 = torch.rand(6, device="cuda")
tensor_1 = torch.rand(6, device="cuda")

nvtx_cuda.range_push("S0")
for i in idx:
    tensor_0[i] = tensor_0[i] + tensor_1[i]
nvtx_cuda.range_pop()

nvtx_cuda.range_push("S1")
for i in range(4):
    tensor_0[i] = tensor_0[i] + tensor_1[i]
nvtx_cuda.range_pop()

idx = [2,3,4]
nvtx_cuda.range_push("S2")
for i in idx:
    tensor_0[i] = tensor_0[i] + tensor_1[i]
nvtx_cuda.range_pop()


torch.cuda.cudart().cudaProfilerStop()
</code></pre>
<p>And this is the Nsys output of this code.</p>
<p><a href="https://i.stack.imgur.com/UR3OX.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/UR3OX.png" alt="enter image description here" width="690" height="118"></a></p>
<p>As we can see in this figure, for case “S0”, even though all the tensors are already on the GPU, including <code>idx,</code> but it calls GPU-&gt;CPU memcpy and cudaSteamSync.</p>
<p>However, in the case of “S1” and “S2”, it doesn’t call GPU-&gt;CPU memcpy calls. Rather, it calls DtoD calls. How does that happen?</p>
<p>So here’re my question.</p>
<ul>
<li>
<p>Why does only tensor idx call GPU-&gt;CPU memcpy and cudaStreamSync?</p>
</li>
<li>
<p>Why “S1” and “S2” doesn’t call memcpy? are they (=idx) placed on GPU by any chance?</p>
</li>
</ul>
Torch.compile() slows down

ymw9754: <p>Hi, I’m trying to apply torch.compile() into my model.<br>
I’ve seen guaranteed speedups with simple CNN structures, but there was a slowdown when ‘for loop’ is contained in the model class.<br>
In other words, if there is a loop in the forward part of the model class (like this structure: <a href="https://github.com/yinboc/liif" class="inline-onebox" rel="noopener nofollow ugc">GitHub - yinboc/liif: Learning Continuous Image Representation with Local Implicit Image Function, in CVPR 2021 (Oral)</a>), the time per epoch is increased when torch.compile() is applied.<br>
Is there any solution that I can avoid this trouble?<br>
Unfortunately, it is not possible to remove the ‘for loop’.</p>
Instantiate with pre-trained layers vs loading weights

abhisek: <p>Consider the following <code>pytorch</code> model:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self. layer1: nn.Module, layer2: nn.Module):
        self.l1 = layer1
        self.l2 = layer2
    
    def forward(self, x):
        return self.l2(self.l1(x))
</code></pre>
<p>With pre-trained <code>lyr1</code> and <code>lyr2</code> if I instantiate</p>
<pre><code class="lang-python">model = MyModel(layer1=lyr1, layer2=lyr2)
</code></pre>
<p>Will the <code>model</code> in this case work as a pre-trained model preserving weights of <code>lyr1</code> and  <code>lyr2</code>? If not how do I construct a pre-trained <code>MyModel</code> instance from <code>lyr1</code> and <code>lyr2</code> objects?</p>

arunppsg: <p>I think it will work. If we do not want to update the weights of the pretrained layers during finetuning, we can set <code>require_grad</code> as False for finetuning on the selected layers. Example:</p>
<pre><code class="lang-auto">model = Model()
for param in model.l1.parameters():
    param.requires_grad = False
for param in model.l2.parameters():
    param.requires_grad = False
</code></pre>

abhisek: <p>Is there any possibility of undefined behavior? Would it be wise to call <code>model.l1.load_state_dict(lyr1.state_dict())</code>  and <code>model.l2.load_state_dict(lyr2.state_dict())</code> just to be safe?</p>

ptrblck: <p>Assuming <code>lyrx</code> are already properly initialized with the pretrained weights your code should work fine.<br>
Loading the <code>state_dict</code> afterwards again would not be needed if it was already done.</p>
Contribution: Mini-Tutorial

Avilay_Parekh: <p>I have created a mini-tutorial series for learning how to use data utilities in PyTorch. I’d love to get your feedback on this.</p>
<p><a href="https://www.youtube.com/playlist?list=PLWv7dKcFUITuj5o29j3Ak2CHG4nlEH-0f" rel="noopener nofollow ugc">PyTorch Data Utils - YouTube</a></p>
<p>Here is the course content, with a video for each module. I have tried to keep the videos to around 5mins.</p>
<ol>
<li>Motivation</li>
<li>Datasets
<ol>
<li>Introduction</li>
<li>Map Style</li>
<li>Iterable Style</li>
<li>Conclusion</li>
</ol>
</li>
<li>DataLoader
<ol>
<li>Introduction</li>
<li>Usage</li>
<li>Batch</li>
<li>Sample</li>
<li>Collate</li>
<li>Iterable</li>
</ol>
</li>
<li>Multiple Workers
<ol>
<li>In-Proc Workers</li>
<li>MultiWorkers</li>
</ol>
</li>
</ol>
Does `model.to('cuda')` move the models' weights to cuda, too?

NightMachinery: <p>If so, what is the use of <code>map_location</code> in this code snippet form the docs?</p>
<pre><code class="lang-auto">device = torch.device("cuda")
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH, map_location="cuda:0"))  # Choose whatever GPU device number you want
model.to(device)
# Make sure to call input = input.to(device) on any input tensors that you feed to the model
</code></pre>

eqy: <p>I believe this question was previously answered here:</p><aside class="quote" data-post="2" data-topic="99983">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/is-map-location-in-torch-load-and-model-load-state-dict-independent-from-device-in-to/99983/2">Is `map_location` in `torch.load()` and `model.load_state_dict()` independent from `device` in `.to()`?</a> 
  </div>
  <blockquote>
    Hi, 
The map_location changes the device of the Tensors in the state dict that is returned. 
But when you load_state_dict(), then these values are loaded (and only values) into the model. But that does not change the model’s device! you will need to move the model itself with .to() if you want to have it on a different device.
  </blockquote>
</aside>
Trying to understand model description

nmnir: <p>I am trying to learn pytorch and can’t understand how the yaml file translates to model description. I am running yolov5 with yolov5n.yaml. here are the first two layers of the model description:</p>
<pre><code>             from  n    params  module                                  arguments                     
</code></pre>
<p>0                -1  1      1760  models.common.Conv                      [3, 16, 6, 2, 2]<br>
1                -1  1      4672  models.common.Conv                      [16, 32, 3, 2]</p>
<p>and here are the first two layers in yolov5n.yaml:</p>
<pre><code class="lang-auto">
# YOLOv5 v6.0 backbone
backbone:
  # [from, number, module, args]
  [[-1, 1, Conv, [64, 6, 2, 2]],  # 0-P1/2
   [-1, 1, Conv, [128, 3, 2]],  # 1-P2/4
</code></pre>
<p>How [64, 6, 2, 2] translates to  [64, 6, 2, 2]?<br>
How [16, 32, 3, 2]  translates to  [128, 3, 2]?</p>
Loading model to cuda is extremely slow on A100

LuoXin-s: <p>Hi, I’m testing my code on a new A100 cluster, I use a 3090 cluster before.</p>
<p>It is extremely slow on A100, after staring at log files, I found it takes 5 minutes loading a model to cuda on A100. However, the time cost on 3090 is around several seconds.</p>
<p>Do I have a wrong environments?</p>
<p>Below is my environments:<br>
PyTorch version: 2.0.0+cu117<br>
CUDA used to build PyTorch: 11.7</p>
<p>OS: Ubuntu 22.04.2 LTS (x86_64)<br>
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0<br>
Libc version: glibc-2.35</p>
<p>Python version: 3.9.12 (main, Apr 5 2022, 06:56:58) [GCC 7.5.0] (64-bit runtime)<br>
Python platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35<br>
Is CUDA available: True<br>
CUDA runtime version: 12.0.140<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration:<br>
GPU 0: NVIDIA A100-SXM4-80GB</p>
<p>Nvidia driver version: 525.60.13<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>

eqy: <p>Are you able to compile and run the CUDA samples in both the 3090 and A100 environments? As a quick sanity check I would test the host-device bandwidth on both setups and see if something looks wrong there:</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/NVIDIA/cuda-samples/tree/master/Samples/1_Utilities/bandwidthTest">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/NVIDIA/cuda-samples/tree/master/Samples/1_Utilities/bandwidthTest" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/NVIDIA/cuda-samples/tree/master/Samples/1_Utilities/bandwidthTest" target="_blank" rel="noopener nofollow ugc">cuda-samples/Samples/1_Utilities/bandwidthTest at master · NVIDIA/cuda-samples</a></h3>

  <p><a href="https://github.com/NVIDIA/cuda-samples/tree/master/Samples/1_Utilities/bandwidthTest" target="_blank" rel="noopener nofollow ugc">master/Samples/1_Utilities/bandwidthTest</a></p>

  <p><span class="label1">Samples for CUDA Developers which demonstrates features in CUDA Toolkit</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
TorchRL documentation website

krateg: <p><a href="https://pytorch.org/rl/" class="inline-onebox" rel="noopener nofollow ugc">Welcome to the TorchRL Documentation! — torchrl main documentation</a> - The documentation for torchrl is broken, please fix.</p>

ptrblck: <p>Thank you for letting us know! The <code>torchrl</code> team is already on it. CC <a class="mention" href="/u/vmoens">@vmoens</a> fir viz</p>

vmoens: <p>‪Fixed! And there’s a new tutorial</p>
<p><a href="https://pytorch.org/rl/tutorials/dqn_with_rnn.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/rl/tutorials/dqn_with_rnn.html</a></p>
nn.MultiheadAttention to get heatmap

Youguang_Xing: <p>Hello, I have a quetion about MultiheadAttention(short for MA). Not about the <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html?highlight=multiheadattention#torch.nn.MultiheadAttention" rel="noopener nofollow ugc">doc explaination</a>, but is about using this module. I want to plot a heatmap(CAM) for my neural network based on transformer. In this process, I need to get the MA mid layer output, especially the dot product results for query-key pairs(similarity matrix). How can I get it? If can’t get it, I have to calculate the output dot product to estimate the result for the self attention layers. But this estimation may cause some errors. So do you have any idea to get the mid-layer results？<br>
I want to use <code>register_forward_hook</code>, but this module architecture output really makes me confused cause it doesn’t show me the component layer that I need.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(self_attn)
MultiheadAttention(
  (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
)
</code></pre>
<p>So can you help me? Thank you very much!</p>

_Bustion: <p>I think you can use the returned weights from MultiheadAttention forward call to calculate the projections, and then calculate the attention matrix by yourself or use the ScaledDotProduct from torchtext library.</p>
<p>Hopefully it helps.</p>

Youguang_Xing: <p>I am so sorry to omit to check the nn.MultiheadAttention.forward() API. Thank you very much. I got the solution!!!</p>
Q-Table Values becoming Nan

NHarris: <p>I am training models, and I am noticing that it is not choosing the optimal policy. I have made several modifications, however, now I’m noticing that the Q-table values are becoming Nan; any insight as to why?</p>

J_Johnson: <p>Can you share some more information?</p>
<ol>
<li>Loss function;</li>
<li>Optimizer and lr;</li>
<li>Reward function;</li>
<li>Batch size.</li>
</ol>

NHarris: <p>I am using the default TD3 algorithm &amp; parameters provided by the author:</p>
<p>with torch.no_grad():</p>
<pre><code>		# Select action according to policy and add clipped noise
		noise = (
			torch.randn_like(action) * self.policy_noise
		).clamp(-self.noise_clip, self.noise_clip)
		
		next_action = (
			self.actor_target(next_state) + noise
		).clamp(-self.max_action, self.max_action)

		# Compute the target Q value
		target_Q1, target_Q2 = self.critic_target(next_state, next_action)
		target_Q = torch.min(target_Q1, target_Q2)
		target_Q = reward + not_done * self.discount * target_Q

	# Get current Q estimates
	current_Q1, current_Q2 = self.critic(state, action)

	# Compute critic loss
	critic_loss = F.mse_loss(current_Q1, target_Q) + F.mse_loss(current_Q2, target_Q)

	# Optimize the critic
	self.critic_optimizer.zero_grad()
	critic_loss.backward()
	self.critic_optimizer.step()

	# Delayed policy updates
	if self.total_it % self.policy_freq == 0:

		# Compute actor losse
		actor_loss = -self.critic.Q1(state, self.actor(state)).mean()
		
		# Optimize the actor 
		self.actor_optimizer.zero_grad()
		actor_loss.backward()
		self.actor_optimizer.step()

		# Update the frozen target models
		for param, target_param in zip(self.critic.parameters(), self.critic_target.parameters()):
			target_param.data.copy_(self.tau * param.data + (1 - self.tau) * target_param.data)

		for param, target_param in zip(self.actor.parameters(), self.actor_target.parameters()):
			target_param.data.copy_(self.tau * param.data + (1 - self.tau) * target_param.data)
</code></pre>
<p>The lr is 3e-4</p>
<p>The reward function is:<br>
if self.avg_hit &gt;= 0.80 and self.avg_use &gt;= 0.80:<br>
reward = 9<br>
elif self.avg_hit &lt; 0.70 or self.avg_use &lt; 0.70:<br>
reward = -9<br>
else:<br>
reward = 0</p>
<p>The beach size is 256</p>

NHarris: <p>I would like to add that it trains after every episode instead of every step and appears to be getting stuck in the actions that keep it in the zero rang rather move toward 9</p>
Computing Gradients of Weighted State Averages in PyTorch Models

Amin1091: <p>Hi,</p>
<p>I have two trained models with the same architecture and different performances. I want to construct a new model by taking the weighted average of their states (s1, s2, s1+s2=1) and calculate the gradients of the new model’s loss with respect to s1 and s2. Here’s my current approach:</p>
<ol>
<li>
<p>Create a simple model:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

class MultiplyByWeight(nn.Module):
    def __init__(self, input_size, output_size):
        super(MultiplyByWeight, self).__init__()
        self.linear = nn.Linear(input_size, output_size)
        
    def forward(self, x):
        return self.linear(x)
</code></pre>
</li>
<li>
<p>Define the states:</p>
<pre><code class="lang-python">state_1 = {
    'linear.weight': 2.4,
    'linear.bias': 0.1
}
state_2 = {
    'linear.weight': 0,
    'linear.bias': -0.5,
}
</code></pre>
</li>
<li>
<p>Set the weights of the states:</p>
<pre><code class="lang-python">state_weights = torch.tensor([0.5, 0.5], requires_grad=True)
</code></pre>
</li>
<li>
<p>Set the weighted average of the states as the new model’s state. I encountered a problem here: when I update the state_dict or modify the named_parameters, the tensors are treated as leaf nodes and detached, losing the computation graph needed to calculate gradients with respect to state_weights.</p>
<ul>
<li>
<p>With load_state_dict:</p>
<pre><code class="lang-python">d = model.state_dict()
for k in d:
    d[k] = d[k] + state_1[k] * state_weights[0] + state_2[k] * state_weights[1]
model.load_state_dict(d)
</code></pre>
</li>
<li>
<p>With named_parameters:</p>
<pre><code class="lang-python">for k, param in model.named_parameters():
    param.data.add_(state_1[k] * state_weights[0])
    param.data.add_(state_2[k] * state_weights[1])
</code></pre>
</li>
</ul>
</li>
<li>
<p>Calculate loss and gradients:</p>
<pre><code class="lang-python">def loss_pt(y_hat, y):
    return (y_hat - y) ** 2

r = model(torch.tensor([1.]))
l = loss_pt(r, 2)
l.backward()
state_weights.is_leaf, state_weights.grad
# (True, None)
</code></pre>
</li>
</ol>
<p>I would greatly appreciate any help or suggestions on how to tackle this issue. I tried searching for a quick workaround in the PyTorch source code, but it doesn’t seem to be a straightforward approach.</p>

ptrblck: <p>You might be able to use the <code>functional_call</code> approach as explained <a href="https://discuss.pytorch.org/t/losing-gradients-when-using-auxiliary-model-to-compute-loss/178509/2">here</a>.</p>
What KLDivLoss expects as input?

0piero: <p>I’ve never used Pytorch’s <a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html#kldivloss" rel="noopener nofollow ugc">KLDivLoss</a>. What input and target should be? It says</p>
<blockquote>
<p>this function expects the first argument <code>input</code>, to be the output of the model (e.g. the neural network) and the second, <code>target</code>, to be the observations in the dataset</p>
</blockquote>
<p>But what exactly should be those ones? Probability densities under different distributions?</p>

ptrblck: <p>The model output should represent log-probabilities (e.g. the output of e.g. <code>F.log_softmax(logits)</code>) while the targets are either probabilities by default or also log-probabilities if <code>log_target=True</code> is used.</p>

0piero: <p>I don’t know if I’m using it correctly.</p>
<pre><code class="lang-auto">        sigma = torch.exp(std)
        
        z = mu + sigma * eps
       
        kl_input = F.log_softmax(z, dim=0)
        kl_target = F.log_softmax(eps, dim=0)

        self.kl = self.kl_lossfn(
            kl_input,
            kl_target
        )
</code></pre>
<p>I’m using the KLDivLoss function with the batchmean reduction in an example of a Variational Autoencoder. In this example, mu and std are output parameters of my model, and eps is a standard normal sample from the distribution that I’m trying to approximate. In my experiments with MNIST, I’ve noticed that some of the sampled values are too far from the standard mean. Am I doing something wrong with the parameters of the loss function?</p>

ptrblck: <p>The usage looks alright assuming you have used <code>log_target=True</code> while creating the criterion.</p>
Pytorch installed but GPU not recognized

John_Deatherage: <p>I have successfully installed pytorch but my GPU is not recognized.  I followed the instructions from <a href="https://developer.nvidia.com/cuda-downloads" class="inline-onebox" rel="noopener nofollow ugc">CUDA Toolkit 12.1 Update 1 Downloads | NVIDIA Developer</a>.  I downloaded and ran cuda_12.1.1_windows_network  It appeared to have worked but something went wrong.</p>
<h1>
<a name="device-config-uses-gpu-if-available-1" class="anchor" href="#device-config-uses-gpu-if-available-1"></a>device config  uses GPU if available</h1>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
print("device = ", device)   ### This prints “cpu”</p>
<p>print(torch.version.cuda)  ### this line prints  none</p>
<p>print (torch.<strong>version</strong>)  #### this line prints 2.0.0+cpu</p>
<p>any ideas as to what I’ve done wrong and how to fix it?</p>
<p>THANK YOU!!!</p>

eqy: <p>How did you install PyTorch? Note that you do not need to install a CUDA toolkit to install and use prebuilt PyTorch binaries, as those usually will ship with their own CUDA toolkit.</p>
<p>I would try uninstall PyTorch and installing a binary e.g., via <code>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</code></p>

John_Deatherage: <p>thank you!  I took your advice and now pytorch recognizes my GPU</p>
<p>THANK YOU!</p>