akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Conversion the 16 bit image to 8 bit image

username: Sushmanth_reddy_Mere
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  i have been using read_image function to read an 16 bit image but it was throwing the following error that it could only read 8 bit image…<br>
how to convert the 16 bit image to 8 bit image</p>
<p>error <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b214288cb1084f7415fc5dbc8a6d1d8833a7b7e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/b214288cb1084f7415fc5dbc8a6d1d8833a7b7e0" title="Screenshot 2023-06-07 at 11.26.36 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b214288cb1084f7415fc5dbc8a6d1d8833a7b7e0_2_690x251.png" alt="Screenshot 2023-06-07 at 11.26.36 PM" data-base62-sha1="ppmb5IgYmkbSy7TFJEM7GwbhsCk" width="690" height="251" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b214288cb1084f7415fc5dbc8a6d1d8833a7b7e0_2_690x251.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b214288cb1084f7415fc5dbc8a6d1d8833a7b7e0_2_1035x376.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b214288cb1084f7415fc5dbc8a6d1d8833a7b7e0_2_1380x502.png 2x" data-dominant-color="F0D8D5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-06-07 at 11.26.36 PM</span><span class="informations">2518×918 364 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Based on the error message it seems <code>read_image</code> doesn’t support 16 bit images and you might need to use e.g. <code>PIL</code> or <code>OpenCV</code>.<br>
I would generally not recommend tagging specific users as it could demotivate others to post a valid response.</p>
Torch.nn.functional.one_hot does not work as expected with ROCM

username: Thu_Nguyen
cooked: <p>I installed the pytorch 2.1 with rocm 5.4.2.<br>
When I tested the one_hot function, it returned this:<br>
print(torch.nn.functional.one_hot(torch.tensor(0,device=‘cuda:0’), 2)) = [0,0]</p>
<p>Do you know why it behaved like this? Thank you in advance</p>

username: eqy
cooked: <p>This looks like a bug and given that you already have minimal reproducible example I would recommend opening an issue @ <a href="https://github.com/ROCmSoftwarePlatform/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ROCmSoftwarePlatform/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a></p>
Distributed libtorch

username: ph0123
cooked: <p>Dear all,</p>
<p>I see some interesting examples with python (<a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/dist_tuto.html</a>).<br>
I want to make a example with libtorch c++.<br>
Please share with me the tutorial or similar functions with c++ to do that.</p>
<p>Thank you so much,<br>
Best regards,</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/ph0123">@ph0123</a> <code>DistributedDataParallel</code> API is not yet available in C++. We are working on that.</p>

username: ph0123
cooked: <aside class="quote no-group" data-username="mrshenli" data-post="2" data-topic="75029">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p><code>DistributedDataParallel</code></p>
</blockquote>
</aside>
<p>Thank you so much! Could you estimate when the API will release?</p>

username: mrshenli
cooked: <p>We can aim for in 3 months, but cannot promise that. If we see more people requesting this, we can try to allocate more resources to it and bump up the priority.</p>

username: kydyahah
cooked: <p>I really need this feature!</p>

username: llm_Warrior
cooked: <p>Do we have a distributed libtorch implementation now? Please let me know.</p>
Which stream will be synchronized when ‘.cuda()’ is called? The current stream or all the streams?

username: Nanlalalalalal
cooked: <p>Hi,<br>
I hope to transfor data from cpu to gpu in the current stream, and synchronize the current stream.<br>
So I wonder which stream will be synchronized when ‘.cuda()’ is called? The current stream? or it just act as torch.cuda.synchronize()?</p>

username: ptrblck
cooked: <p>The copy kernel should use the current stream as seen <a href="https://github.com/pytorch/pytorch/blob/28f43c767c0de48885f8de285ca4af4424d28f29/aten/src/ATen/native/cuda/Copy.cu#L235">here</a> in <code>copy_kernel_cuda</code>.</p>
Indexing tensor with tensor : without for loop

username: Hyeonuk_Woo
cooked: <p>I have a tensor A with shape of (32,200,50,2), and tensor B with shape of (32,100,100,128).<br>
Tensor A is integer tensor. The lower bound of tensor A is 0 and upper bound of tensorA is 99. Therefore tensor A can be used as index tensor for tensor B.<br>
From tensor A and tensor B, I want to generate tensor C with shape of (32,200,50,128).<br>
The code for get tensor C is below.</p>
<pre><code class="lang-auto">C=torch.zeros(*A.shape[:-1], B.shape[-1])
for b in range( A.shape[0]):
       for i in range(A.shape[1]):
              for j in range(A.shape[2]):
                     C[b, i, j, :]= B[b, A[b,i,j,0], A[b,i,j,1], :] 
</code></pre>
<p>How can I do the same job without using ugly for loop?</p>

username: ptrblck
cooked: <p>Direct indexing should work:</p>
<pre><code class="lang-python">A = torch.randint(0, 100, (32, 200, 50, 2))
B = torch.randn(32, 100, 100, 128)

C=torch.zeros(*A.shape[:-1], B.shape[-1])
for b in range( A.shape[0]):
       for i in range(A.shape[1]):
              for j in range(A.shape[2]):
                     C[b, i, j, :]= B[b, A[b,i,j,0], A[b,i,j,1], :] 
    

out = B[torch.arange(B.size(0))[:, None, None], A[:, :, :, 0], A[:, :, :, 1]]
print((out == C).all())
# tensor(True)
</code></pre>
Bias required when using `Conv2d` and `LayerNorm` / `GroupNorm`?

username: kfshr
cooked: <p>I stumbled upon the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html" rel="noopener nofollow ugc">Performance Tuning Guide</a> and read that the <code>bias</code> can be set to true when using <code>Conv2d</code> followed by a <code>BatchNorm2d</code>.</p>
<p>I wondered if that is true for <code>LayerNorm</code> and <code>GroupNorm</code> as well? And what about <code>Linear</code> followed by <code>LayerNorm</code>? Can I set <code>bias=False</code> when using <code>LayerNorm</code> right after <code>Linear</code>? How can I check if the bias term is actually required?</p>
Switching between random and sequential sampling

username: scottrs
cooked: <p>i would like my code to be able to switch easily between using random and sequential sampling with dataloader.</p>
<p>was hoping to do something like the following.  based on a user entered parameter (useseq) i can switch from using random to sequential.  of course this doesn’t work.  there must be a relatively easy way to do this.</p>
<p>can anyone help?</p>
<pre><code>    auto dlisrnd = torch::data::make_data_loader&lt;torch::data::samplers::RandomSampler&gt;(std::move(dsis), cunbs);
    auto dlisseq = torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt;(std::move(dsis), cunbs);
    auto dlisuse = dlisrnd;
    if (useseq) {
        dlisuse = dlisseq;
    }
</code></pre>

username: eqy
cooked: <p>I assume the problem is that the returned types of the dataloaders don’t match?<br>
Is there an issue with just passing both dataloaders in a struct, tuple, or other iterable structure and using one or the other? If you must use just a single object, you could e.g., use a Union, create a wrapper class/template, or even use raw pointers and cast them back to the selected dataloader type?</p>

username: scottrs
cooked: <p>thanks.  will try those.  appreciate the help.</p>
Mixture of Expert in Sgementation

username: Youness_EL_BRAG
cooked: <p>I wondered how could i implement a Mixture of Experts in Segmentation</p>
<p>the model Arch</p>
<pre><code class="lang-python">class LeNet(nn.Module):

    # network structure
    def __init__(self, output_dim):
        super(LeNet, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5, padding=2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, output_dim)

    def forward(self, x):
        '''
        One forward pass through the network.
        
        Args:
            x: input
        '''
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        '''
        Get the number of features in a batch of tensors `x`.
        '''
        size = x.size()[1:]
        return np.prod(size)
</code></pre>
<p>MoE build model</p>
<pre><code class="lang-python">class MoE(nn.Module):
    def __init__(self, num_experts, output_dim):
        super(MoE, self).__init__()
        self.num_experts = num_experts
        self.output_dim = output_dim
        self.softmax = nn.Softmax()

        self.experts = nn.ModuleList([LeNet(output_dim) for i in range(num_experts)])

        self.gating = LeNet(num_experts)

    def forward(self, x):

        weights = self.softmax(self.gating(x))    ### Outputs a tensor of [batch_size, num_experts]

        out = torch.zeros([weights.shape[0], self.output_dim])
        for i in range(self.num_experts):
            out += weights[:, i].unsqueeze(1) * self.experts[i](x)    ### To get the output of experts weighted by the appropriate gating weight

        return out    ### out will have the shape [batch_size, output_dim
</code></pre>
<p>My question : is that i have Unet model 3D Per-Trained</p>
Matrix multiplication in convolution

username: chris666
cooked: <p>Hi,</p>
<p>I want to employ matrix convolution in the convolutional layer. For example, if X has the shape of (20, 200), and I want each feature map to have the shape of (20, 100). How can I design the kernel so that the backpropagation can update the weights?</p>

username: J_Johnson
cooked: <p>Hello <a class="mention" href="/u/chris666">@chris666</a>!</p>
<p>There are many ways you can take an input of (20, 200) —&gt; (20, 100) using 1-dimensional convolution and related layers. Here are just a few:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
import torch.nn as nn

x = torch.rand(20,200)

model1 = nn.Conv1d(20, 20, kernel_size=(2,), stride=(2,))
model2 = nn.Conv1d(20, 20, kernel_size=(4,), stride=(4,), padding=100)
model3 = nn.Conv1d(20, 20, kernel_size=(101,))

print(model1(x).size())
print(model2(x).size())
print(model3(x).size())
print(F.max_pool1d(x, kernel_size=(2,), stride=(2,)).size())
print(F.avg_pool1d(x, kernel_size=(2,), stride=(2,)).size())
</code></pre>
<blockquote>
<p>torch.Size([20, 100])<br>
torch.Size([20, 100])<br>
torch.Size([20, 100])<br>
torch.Size([20, 100])<br>
torch.Size([20, 100])</p>
</blockquote>

username: chris666
cooked: <p>Hi Johnson,</p>
<p>Thanks for your reply.</p>
<p>It seems I didn’t explain it clearly. I want a matrix as the kernel, like (20, <em>). So, for example, the input has 20</em>500 (rows*columns). Suppose now we have output_channel=1. I don’t want there to be only one filter to go through all rows in the input. There should be one filter for each row, and a total of 20 filters.</p>
<p>Do you know how to design this? thanks.</p>

username: J_Johnson
cooked: <p>That depends on the kernel_size, stride, padding, etc. Take, for instance, the first example, adjusted for out_channels=1:</p>
<pre><code class="lang-auto">model1 = nn.Conv1d(20, 1, kernel_size=(2,), stride=(2,), bias=False)
</code></pre>
<p>We can print the size of the weights with:</p>
<pre><code class="lang-auto">for param in model1.parameters():
    print(param.data.size())
</code></pre>
<blockquote>
<p>torch.Size([1, 20, 2])</p>
</blockquote>
<p>That’s the learnable weights portion of the layer.</p>
Parallellism compatible with autograd?

username: sovgud
cooked: <p>I am using pytorch to do stochastic gradient descent in order to maximaze a target function (evidence lower bound for a variational method). I am using CPU tensors</p>
<p>I have a model with parameters. In each iteration I sample a number of trees and corresponding branch lengths using the model’s parameters. For each tree <em>t</em> I apply funtion <em>f</em> that computes a probability over the tree to get <em>f(t)</em>, and use them to compute the loss. The computations in <em>f</em> are totally independent for different trees.</p>
<p>My question: Is there any way to run the function <em>f</em> for differenct trees in parallell?</p>
<p>I tried following:</p>
<pre><code class="lang-auto">from torch.multiprocessing import Process, Pool
def logprior(trees, log_branch):
    n_particles = len(trees)
    pool = Pool(n_particles)
    logp_trees = pool.map(f, zip(trees, log_branch))
</code></pre>
<p>But got</p>
<blockquote>
<p>RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).</p>
</blockquote>
<p>I understand the reason for this error. But I want gradient computation through <em>f</em>.</p>
<p>Thanks in advance.</p>
Using the nn.Transformer during the eval

username: Krishu_Thapa
cooked: <p>I have daily data for n locations to predict some weather values for each location daily based on those daily data. My custom model is using the transformer model for it. In the training time I have the ground truth value of that weather variable which I can pass on to the tgt attribute required by the transformer model.Hiowever, during the evaluation/inference phase I will not hace the ground truth value to be sent to the tgt attribute.</p>
<p>Shouldn’t transformer generate the value based on my input features only during the inference without sending the tgt attribute or should I send the placeholder like torch.zeros() for that case? I am confused , please help me with that issue</p>
optimize to obtain the maximum of a well trained neural network output 

username: AmosJoseph
cooked: <p>Obtain the maximum of the output of a well trained neural network in the domain.</p>
<p>I trained a neural network and tested it with good accuracy.<br>
Based on the built neural network, how to obtain the maximum of the output of it in the domain?</p>
<p>Is there any such example?<br>
Could you please provide a minimal code example?</p>
How to contribute to creating explanation videos for PyTorch in GitHub

username: Killpit
cooked: <p>For PyTorch docathon, how can we create videos for GitHub and do a pull request simultaneously</p>

username: ptrblck
cooked: <p>I’m unsure what exactly you want to do, but to contribute to PyTorch you could check <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">this guide</a>.</p>

username: Killpit
cooked: <p>For the docathon, I decided to contribute on Torch Autograd video and it was originally published on GitHub. I’m going to shoot up a video on it and probably will have to make a pull request to GitHub along with code.</p>
Load model trained on GPU (via c++ frontend) into CPU (also via C++ frontend)

username: Willem
cooked: <p>I run my code on several nodes on a cluster. On GPU enabled nodes, I train a nn, and save the parameters via torch::save. Now I want to load them on a CPU enabled node, where I have no GPU support. I get errors complaining that I don’t have the GPU support (which is correct). I think I need to tell torch::load that it needs to convert what it loads to CPU model. But I can’t find anywhere how.</p>
<p>For python, the approach is clear in a few posts. But how to do this for c++ frontend.</p>
<p>I also tried converting the model to cpu (with model-&gt;to(device)), where device was a CPU device, before saving it on GPU. But that didn’t help either. Any ideas on how this can be done?</p>
<p>Error I get while loading:<br>
terminate called after throwing an instance of ‘c10::Error’<br>
what():  Cannot initialize CUDA without ATen_cuda library. PyTorch splits its backend into</p>

username: Willem
cooked: <p>So I ran some more tests. This problem only occurs when a model was trained at one point on the GPU. If I use the GPU version of pytorch c++, but never train on GPU (just CPU), and save the model, then it is possible to load the model using the pytorch CPU-only version. But once a model is trained on GPU, saving it results in a file that cannot be loaded with the CPU only version without triggering the warning. I would be very much helped with a workaround. Thanks!</p>

username: Willem
cooked: <p>Since my models are not very complex (i use a dense linear architecture), it would be very simple to overcome this problem by writing my own “save” and “load” functions. (Just saving the entries of the linear layers and biases…) I am about to do so, but it would be a waste if I am overlooking something.</p>
<p>Can anybody confirm that the above is at present an issue?</p>

username: Willem
cooked: <p>Ok, somehow I missed a rather obvious solution: using the GPU pytorch library on a CPU only device. This is possible, and allows me to load whatever I want.</p>

username: phaogo
cooked: <p>I used it as below and it worked.</p>
<pre><code> c10::Device device(c10::DeviceType::CPU);
// Deserialize the ScriptModule from a file using torch::jit::load().
module = torch::jit::load(argv[1], device);
</code></pre>
Problem arises during installation of pytorch on the windows

username: Rahul_Srivastava
cooked: <p>pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu117" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu117</a></p>
<p>this code shows an installation error while running it on the anaconda prompt.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb301703855eaa5a964a155d733c7c4c745dfe4a.png" data-download-href="https://discuss.pytorch.org/uploads/default/bb301703855eaa5a964a155d733c7c4c745dfe4a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb301703855eaa5a964a155d733c7c4c745dfe4a.png" alt="image" data-base62-sha1="qHWjMDrVAPM48pZWCps1MwyVRUe" width="690" height="414" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1108×665 20.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>It seems you are trying to run the <code>pip</code> command in a Python script instead of your terminal, so try to execute it directly in your <code>Anaconda Prompt</code> instead of in a Python REPL.</p>

username: Rahul_Srivastava
cooked: <p>Thank, my query has got resolved</p>
Centos6.9+python3.7+pytorch1.12.1 build succeed,but import error

username: zhb0920
cooked: <p>import torch<br>
Traceback (most recent call last):<br>
File “”, line 1, in<br>
File “/meda_home/hongbin/miniconda3/lib/python3.7/site-packages/torch/<strong>init</strong> .py”, line 202, in<br>
from torch._C import * # noqa: F403<br>
ImportError: /meda_home/hongbin/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so: undefined symbol: aligned_alloc</p>

username: ptrblck
cooked: <p><a href="https://en.cppreference.com/w/cpp/memory/c/aligned_alloc"><code>aligned_alloc</code></a> seems to be introduced in C++17, so you might need to check if your <code>libc.so</code> defines these symbols or which <code>libc.so</code> is used.</p>
Backpropagation gives null due to trigonometric division

username: Salman_Abbasi
cooked: <p>Hi</p>
<p>I am using a trigonometric values in my code, and after first iteration, the loss is nan. I figured out it is due to trigonometric term being divided.</p>
<pre><code>            cosAngle=torch.cos(Chi)
            sinAngle=torch.sin(Chi)
            cosAngle=torch.unsqueeze(cosAngle,1)
            sinAngle=torch.unsqueeze(sinAngle,1)
            angle=(cosAngle)/(sinAngle)
</code></pre>
<p>If i remove sinAngle, it works. I tried torch.asin and also tried to add and multiply episol term, but no luck.</p>
<p>I greatly appreciate the response</p>
<p>regards<br>
Sal</p>

username: eqy
cooked: <p>I think one issue is that simply adding an epsilon wouldn’t be able to guarantee there is no division by zero because there could be cancellation. You may want to take a look at the clamp function: <a href="https://pytorch.org/docs/stable/generated/torch.clamp.html" class="inline-onebox" rel="noopener nofollow ugc">torch.clamp — PyTorch 2.0 documentation</a> if you want to ensure that <code>sinAngle</code> remains in a certain range.</p>

username: Salman_Abbasi
cooked: <p><a class="mention" href="/u/eqy">@eqy</a>, thanks. it is good to know this function. i tried angle range -90 to 90 . but still it gives nan. the problem is not the trigonometric range but the division. without division it works just fine. Why the loss is ok but its backpropagation for division gives nan?</p>

username: eqy
cooked: <p>I think you may have misunderstood my point. You can use clamp to ensure that the divisor does not becomes too small—though it may be simpler to do something like <code>cosAngle/(signAngle + torch.sign(signAngle)*eps)</code>.</p>
<p>If the divisor becomes too small it can still cause numerical issues as the loss can cause the gradients of each layer to explode / vanish depending on the depth of the model.<br>
E.g., a contrived example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.ones(1, requires_grad=True)
&gt;&gt;&gt; loss = ((1/torch.sin((a/1e30))).sum())
&gt;&gt;&gt; loss
tensor(1.0000e+30, grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; a
tensor([1.], requires_grad=True)
&gt;&gt;&gt; a.grad
tensor([-inf])
&gt;&gt;&gt;
</code></pre>

username: Salman_Abbasi
cooked: <aside class="quote no-group" data-username="eqy" data-post="4" data-topic="181487">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>torch.sign(signAngle)*eps)</p>
</blockquote>
</aside>
<p>Thanks. My loss before backpropagation is tensor(2.7658, grad_fn=).</p>
<p>Should it explode gradient?<br>
thanks</p>

username: eqy
cooked: <p>It depends on the depth of the network, I would recommend inspecting the <code>.grad</code> fields of layers in your network and checking if there is any trend such as shallower layers having higher values.</p>
Running TorchScript models in C++

username: mirzadeh
cooked: <p>I have a torch script model that I load and run in C++. I recently found that the torch script file itself (which appears to be a zip file) contains several Python source files which prompted the following question</p>
<p>When running a torch script model using libtorch, is Python runtime involved in any way or do all operations run natively in C++? For instance, are the Python source files in the serialized model actually executed in a Python (or Python-like) interpreter?</p>
<p>I would appreciate any clarification here.</p>
PyTorch compile requires fixed input naming?

username: zhihao_zhu
cooked: <p>Hi,</p>
<p>My model inference has two steps, and each step digests input tensors with different naming: def forward(input, step), let’s say the first step takes in an input tensor named f’input_{step}'(i.e.input_1), and second step takes in an input tensor named input_2. When I run first inference(with torch.compile) using the input_1 as naming, and when I run second step, the model seems to automatically assume the input tensor naming should be input_1, and overwrite the input_2. So my question is: should we always keep a consistent naming for a compiled model?</p>
<p>thanks</p>

username: marksaroufim
cooked: <p>Could you share a code snippet for whatt you mean exactly? Are the inputs the same shape? If not then <code>dynamic=True</code> should help but otherwise a JIT compiler will typically specialize on properties of the input so would just like to understand a bit better what you’re trying to do</p>

username: zhihao_zhu
cooked: <p>Thanks for the reply. Yes the input shapes are the same, here I attached the code to replicate the issue:</p>
<pre><code class="lang-auto">import torch
torch._dynamo.config.verbose=True

class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = torch.nn.Linear(100, 10)

    def forward(self, x, step):
        return {f'output_{step}': self.lin(x[f'input_{step}'])}

mod = MyModule()
opt_mod = torch.compile(mod)

my_input = {
    'input_0': torch.ones([100]),
    'input_1': torch.ones([100])
}

for step in range(2):
    output = opt_mod(my_input, step)
    print(output.keys())
</code></pre>
<p>My expected output would be:</p>
<pre><code class="lang-auto">dict_keys(['output_0'])
dict_keys(['output_1'])
</code></pre>
<p>But the actual output from the code above is:</p>
<pre><code class="lang-auto">dict_keys(['output_0'])
dict_keys(['output_0'])
</code></pre>
<p>where the second key name is changed from output_1 to output_0.</p>

username: marksaroufim
cooked: <p>Interesting, I x-posted this on github since I’m not sure if this a bug or intended behavior <a href="https://github.com/pytorch/pytorch/issues/103099" class="inline-onebox" rel="noopener nofollow ugc">torch.compile specializes on input name · Issue #103099 · pytorch/pytorch · GitHub</a></p>
How to load only the first 500 samples from dataloader

username: peony
cooked: <p>I have implemented this code below:</p>
<p><code>for idx, (data, target, meta) in list(enumerate(tqdm(dataloader_test)))[:500]:</code></p>
<p>but it still loaded the entire dataset instead of the first 500. How do we limit the number of samples to load?</p>
<p>Thank you in advance.</p>

username: ptrblck
cooked: <p>This is expected since you are slicing the <code>list</code> after creating it.<br>
You could create the iterator via <code>loader_iter = iter(dataloader_test)</code> and iterate it 500 times:</p>
<pre><code class="lang-python">for _ in range(500):
    data, target, meta = next(iter)
</code></pre>
<p>To be safe you could also wrap it into a <code>try</code> block and catch the <code>StopIteration</code> exception.</p>

username: peony
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> , I tried creating an iterator and editing my for loop like this:</p>
<pre><code class="lang-auto">    loader_iter = iter(dataloader_test)
    
    
    for _ in range(500):
        idx, (data, target, meta) = next(iter)

        for d, t, m in zip(data, target, meta):
            d, t = map(lambda x: x.to(device), (d, t))
</code></pre>
<p>but I get this error:</p>
<pre><code class="lang-auto">
    idx, (data, target, meta) = next(iter)

TypeError: 'builtin_function_or_method' object is not an iterator
</code></pre>
<p>my dataloader have batch_size by default so I can’t remove idx</p>

username: ptrblck
cooked: <p>Sorry, my second code snippet used bad variable names. You should use <code>next(loader_iter)</code>, i.e. call it on the actual iterator object.</p>
Got stucks while loading a big tensor in the subprocess

username: SharpKoi
cooked: <h2>
<a name="environment-1" class="anchor" href="#environment-1"></a>Environment</h2>
<ul>
<li>OS: Linux Mint 21 (based on Ubuntu 22.04)</li>
<li>Python: 3.9.15</li>
<li>PyTorch: 2.0.0</li>
</ul>
<h2>
<a name="description-2" class="anchor" href="#description-2"></a>Description</h2>
<p>I’m currently working on an asynchronous reinforcement learning algorithm, and I want to create multiple processes for agents interacting with the environment and collecting the trajectories into a replay buffer.<br>
In each subprocess, The agent will periodically load the learned and shared actor parameters to collect data.<br>
However, the issue is that the subprocess will hang if the agent loads a shared actor with a large tensor.<br>
How could I solve this issue?</p>
<p>Below is a simple example:</p>
<pre><code class="lang-python">class FakeAgent:
    def __init__(self):
        self._actor = nn.Sequential(
            nn.Linear(512, 512, bias=False)
        )

def collect_traj(shared_actor):
    pid = os.getpid()
    agent_behav = FakeAgent()

    # [START] Repeat for some fixed number of iterations
    print(f"pid {pid}: Start loading shared model ...")
    agent_behav._actor.load_state_dict(shared_actor.state_dict())
    print(f"pid {pid}: Model loaded.")
    
    # collect trajectories here using agent_behav
    # ...
    
    # [END]   Repeat for some fixed number of iterations


if __name__ == '__main__':
    agent = FakeAgent()
    agent._actor.share_memory()
    
    ppool = []
    for _ in range(2):
        p = mp.Process(target=collect_traj, args=(agent._actor,))
        ppool.append(p)

    for p in ppool:
        p.start()

    for p in ppool:
        p.join()

    print("All process done.")
</code></pre>
Customized Convolution with Dynamic Kernels

username: Kai_Lan
cooked: <p>Hi all,</p>
<p>I am trying to implement a customized convolution. I have two inputs, <code>image</code> and <code>x</code>, with shape <code>1 x N x N</code> and <code>batch_size x 1 x N x N</code>, respectively. The kernels depend on <code>image</code>, and will be applied to <code>x</code>. That means, we learn a map (9 x 9 linear map) from a <code>3 x 3</code> patch of the image to a <code>3 x 3</code> kernel. Then apply that kernel to the corresponding patch of <code>x</code>.</p>
<p>The following is my current implementation. I have two versions: the <code>forward</code> version  takes up too much memory and is relatively slow compared to a built-in Conv2D, the <code>forward1</code> version saves memory, but due to naive for-loops. it is VERY slow. How can I make it fast with limited CUDA memory?</p>
<pre><code class="lang-auto">class SmallSMBlock(nn.Module):
    def __init__(self):
        super().__init__()
        self.KL = nn.Conv2d(1, 9, kernel_size=3, padding='same', bias=True)
    def forward1(self, image, x): # 1 x N x N, bs x 1 x N x N
        N = x.shape[-1]
        y = torch.zeros_like(x)
        image = F.pad(image, (1,)*4)
        x = F.pad(x, (1,)*4)
        for i in range(1, N+1):
            for j in range(1, N+1):
                K = ((self.KL.weight.squeeze() * image[0, i-1:i+2, j-1:j+2])).sum(dim=(1, 2)) + self.KL.bias
                K = K.view((3, 3))
                xx = x[:, 0, i-1:i+2, j-1:j+2]
                y[:, 0, i-1, j-1] = (xx * K).sum(dim=(-2, -1))
        return y
    def forward(self, image, x): # 1 x N x N, bs x 1 x N x N
        K = self.KL(image) # 1 x N x N -&gt; 9 x N x N
        K = K.permute((1, 2, 0)) # 9 x N x N -&gt; N x N x 9
        K = K.unflatten(2, (3, 3)) # N x N x 9 -&gt; N x N x 3 x 3
        x = F.pad(x, (1, 1, 1, 1)) # bs x 1 x N x N -&gt; bs x 1 x (N+2) x (N+2)
        x = x.unfold(2, 3, 1).unfold(3, 3, 1) # bs x 1 x (N+2) x (N+2) -&gt; bs x 1 x N x N x 3 x 3
        y = (x * K).sum(dim=(-2, -1))
        return y
</code></pre>
Discrepancy between Conv3D and my custom implementation

username: Kai_Lan
cooked: <p>Hi all,</p>
<p>I am trying to implement Conv3D with <code>unfold</code> function. However, when I compare the results with the build-in Conv3D, they agree on CPU but NOT on CUDA. Is there a reason behind this?<br>
You can reproduce with the following code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

channels = 5
h, w, d = 4, 4, 4

def test(device):
    image = torch.randn(channels, h, w, d).to(device) # input image

    kh, kw, kd = 3, 3, 3 # kernel size
    dh, dw, dd = 1, 1, 1 # stride

    # Create conv
    conv = nn.Conv3d(channels, 10, (kh, kw, kd), padding='same', bias=False).to(device)
    filt = conv.weight

    # Manual approach
    patches = F.pad(image, (1,)*6)
    patches = patches.unfold(1, kh, dh).unfold(2, kw, dw).unfold(3, kd, dd)

    patches = patches.contiguous().view(channels, -1, kh, kw, kd)

    nb_windows = patches.size(1)

    # Now we have to shift the windows into the batch dimension.
    # Maybe there is another way without .permute, but this should work
    patches = patches.permute(1, 0, 2, 3, 4)

    # Calculate the conv operation manually
    res = patches.flatten(1) @ filt.flatten(1).transpose(0, 1)
    res = res.transpose(0, 1) # out_channels, output_pixels


    res = res.unflatten(1, (h, w, d))

    # Module approach
    out = conv(image)
    print('max abs error ', (out - res).abs().max())

print('Test on CPU')
test(torch.device("cpu")) # 4.7684e-07
print('Test on CUDA')
test(torch.device("cuda")) # 0.0005
</code></pre>

username: ptrblck
cooked: <p>If you are using an Ampere GPU or newer, disable <code>TF32</code> and the error should be smaller via <code>torch.backends.cudnn.allow_tf32 = False</code>.</p>

username: Kai_Lan
cooked: <p>Does it mean my implementation will lose accuracy if we have `torch.backends.cudnn.allow_tf32 = True’?</p>

username: ptrblck
cooked: <p>Yes, precision will be lost as only 10 bits of mantissa will be stored for the trade-off of using TensorCores as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices">here</a>. The link also links to more detailed blog posts about TF32.</p>
Robustness metrics

username: teio
cooked: <p>Hello,</p>
<p>Is there any metric provided to assess robustness / stability of a ML module? As far as I understand the ones provided are more related with the functionality.<br>
I understand reliability as the ability to perform the intended function in the presence of abnormal or unknown inputs, related to ‘reliability’.</p>
<p>Any comment is welcome<br>
Thanks in advance</p>

username: ptrblck
cooked: <p>You could check <a href="https://torchdrift.org/"><code>TorchDrift</code></a> created by <a class="mention" href="/u/tom">@tom</a> which might be useful for your use case.</p>

username: teio
cooked: <p>thanks you very much for your answer</p>
Conv*d returns drastically different results on ROCm vs CPU

username: vdi
cooked: <p>I encounter a weird inconsistency in the result of <code>nn.Conv2d</code> operation on CPU and AMD GPU.</p>
<p>Here is the snippet showing the drastic difference in an environment with ROCm:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.0.0+rocm5.4.2'
&gt;&gt;&gt; layer = torch.nn.Conv2d(1, 768, kernel_size=16, stride=10)
&gt;&gt;&gt; x = torch.rand(10, 1, 128, 66)
&gt;&gt;&gt; layer.to('cuda:0')(x.to('cuda:0')).sum()
tensor(10210.0078, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; layer.to('cpu')(x.to('cpu')).sum()
tensor(2468.2070, grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>Locally, I have an NVidia GPU and here is the snippet with the results with CUDA:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.0.0'
&gt;&gt;&gt; layer = torch.nn.Conv2d(1, 768, kernel_size=16, stride=10)
&gt;&gt;&gt; x = torch.rand(10, 1, 128, 66)
&gt;&gt;&gt; layer.to('cuda:0')(x.to('cuda:0')).sum()
tensor(244.4510, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; layer.to('cpu')(x.to('cpu')).sum()
tensor(244.4510, grad_fn=&lt;SumBackward0&gt;)
# `torch.allclose` returns True
</code></pre>
<p>Interestingly, <code>nn.Linear</code> does not exhibit this behaviour, ie the numbers match.</p>

username: vdi
cooked: <p>It seems to be a bug and is now tracked here: <a href="https://github.com/pytorch/pytorch/issues/102968" class="inline-onebox" rel="noopener nofollow ugc">Conv2d returns drastically different results on ROCm (MI250X) vs CPU · Issue #102968 · pytorch/pytorch · GitHub</a></p>
<p>It was suggested to turn off implicit GEMM by setting <code>MIOPEN_DEBUG_CONV_IMPLICIT_GEMM=0</code></p>
Issue with Registering Forward Hooks in a Transformer

username: Alex_F
cooked: <p>Hello!</p>
<p>I am currently trying to track layer activations as part of a project I’m working on.</p>
<p>Previously I had no issues with using the register_forward_hooks method for convolutional layers in AlexNet. I am now trying to reuse this code for a transformer layer in ViT_b32 but the activations dict is left empty. Below is an excerpt of the code:</p>
<p>def register_hooks(model_layers, features, feature_layer):<br>
def get_features(name):<br>
def hook(model, input, output):<br>
features[name] = output.detach()<br>
return hook<br>
model_layers[feature_layer].register_forward_hook(get_features(‘txr_layer’))</p>
<p>model = models.vit_b_32()<br>
feature_layer = ‘encoder_layers_encoder_layer_11_self_attention_out_proj’<br>
layers = get_model_layers(model)  <span class="hashtag">#This</span> just stores layers in a dict with key = layer name and val = layer</p>
<p>activations = {}<br>
register_hooks(layers, activations, feature_layer)</p>
<p>out = model(img_tensor)</p>
<p>print(f’activations dict size: {len(activations)}')</p>
<p>This output a dict size of 0 for ViT and 1 for Alexnet, indicating that the hook is not registering as intended.</p>
<p>Any advice offered would be greatly appreciated, and if you need additional information please don’t hesitate to ask!<br>
Cheers!</p>

username: ptrblck
cooked: <p>Your code is not executable as e.g. <code>model_layers</code> is undefined and it’s also not properly formatted.<br>
However, it seems you want to access the <code>NonDynamicallyQuantizableLinear</code> layer, which is only used for error handling in the <code>MultiHeadAttention</code> layer.<br>
This code works for me:</p>
<pre><code class="lang-python">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output[0].detach()
    return hook


model = models.vit_b_32()
model.encoder.layers.encoder_layer_11.self_attention.register_forward_hook(get_activation("mha"))

x = torch.randn(1, 3, 224, 224)
out = model(x)
print(out.shape)
# torch.Size([1, 1000])
print(activation["mha"].shape)
# torch.Size([1, 50, 768])
</code></pre>