akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loss.backward() if multiple losses from multiple networks are present

Daksh_Maheshwari: <p>I have a model which can be divided into two submodels.<br>
Model1 takes input, then k number of Model2 takes input from Model1.<br>
So, I have k number losses.<br>
Currently, I’m doing loss = loss_1 + … + loss_k and then doing loss.backward().<br>
My query is, how loss.backward will make sure that ith Model2 gradient calculation happens by loss_i, not summed up loss.</p>

srishti-git1110: <p>Hi Daksh, on calling <code>loss.backward()</code>, the grad attribute of all the leaf tensors in the computation graph of <code>loss</code> is populated.<br>
The answer to your question can be best answered if you could provide some code, please.</p>

eqy: <p>If the ith model contributes to the only ith loss value, the derivative of the other components of the loss with respect to that model are zero so it would happen automatically.</p>
<p>Another way of viewing this is that by the chain rule, <code>loss.backward()</code> when loss is the sum of the other losses is equivalent to calling <code>loss_i.backward()</code> for each loss individually.</p>
<p>Note that model1 would be affected by all losses as it contributes to each term by virtue of producing one output that is used by the downstream models.</p>
<pre><code class="lang-auto">import torch

torch.manual_seed(0)
m1 = torch.nn.Conv2d(1,1,1,1)
m2 = torch.nn.Conv2d(1,2,1,1)
inp = torch.randn(1,1,1,1)
loss1 = m1(inp).prod()
loss2 = m2(inp).prod()
loss = loss1 + loss2
loss.backward()
print(m1.weight.grad, m2.weight.grad)

torch.manual_seed(0)
m1 = torch.nn.Conv2d(1,1,1,1)
m2 = torch.nn.Conv2d(1,2,1,1)
inp = torch.randn(1,1,1,1)
loss1 = m1(inp).prod()
loss2 = m2(inp).prod()
loss1.backward()
loss2.backward()
print(m1.weight.grad, m2.weight.grad)
</code></pre>
<pre><code class="lang-auto">$ python3 multiple.py
tensor([[[[1.2645]]]]) tensor([[[[-0.8376]]],


        [[[-1.8029]]]])
tensor([[[[1.2645]]]]) tensor([[[[-0.8376]]],


        [[[-1.8029]]]])
</code></pre>
Convert n4hw:uint8 to n1hw:int for image processing?

jxtps: <p>I’m working with images, and sometimes it’s useful to view them as one int per pixel, and sometimes as four bytes per pixel.</p>
<p>Is there a way to convert between these two views in pytorch?</p>
<p>Specifically I’m looking to go from four bytes per pixel to one int per pixel. Ideally while re-using the underlying memory.</p>
<p>Thanks!</p>

ptrblck: <p>Would <code>view</code> work?</p>
<pre><code class="lang-python">&gt;&gt;&gt; a = torch.tensor([1, 2, 3, 4]).to(torch.uint8)
&gt;&gt;&gt; a
tensor([1, 2, 3, 4], dtype=torch.uint8)
&gt;&gt;&gt; a.view(torch.int32)
tensor([67305985], dtype=torch.int32)
&gt;&gt;&gt; a = torch.tensor([4, 3, 2, 1]).to(torch.uint8)
&gt;&gt;&gt; a.view(torch.int32)
tensor([16909060], dtype=torch.int32)
</code></pre>

jxtps: <p>That works excellently, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p>For future readers: you have to rearrange your data so that it’s channels last, and you have to have four channels for this to work. E.g:</p>
<pre><code class="lang-python">a = torch.ones((1, 4, 5, 5)).to(torch.uint8).permute(0, 2, 3, 1).contiguous()  # shape: [1, 5, 5, 4], NHWC
b = a.view(torch.int32)  # shape: [1, 5, 5, 1] NHWC
</code></pre>
<p>If you mess up your channel dimension or the tensor isn’t contiguous then you get all sorts of quite sensible error messages, e.g.:</p>
<ul>
<li>
<p><code>RuntimeError: self.size(-1) must be divisible by 4 to view Byte as Int (different element sizes), but got 5</code></p>
</li>
<li>
<p><code>RuntimeError: self.stride(-1) must be 1 to view Byte as Int (different element sizes), but got 25</code></p>
</li>
</ul>
Multiple output MLP

ays: <p>I have an MLP model and my goal is to predict 2 variables(so ideally I should have 2 neurons in my final layer). However, the first variable (lets call it var1) has sub values(up to 12) and the second variable(var2) has just a single value. Does it make sense to have 13 neurons in my final layer and during back prop, I’ll calculate 2 losses(one w.r.t to var1 and the second wrt var2), then sum them up?</p>
<p><strong>For better intuition- my scenario is a bit complex so I’ll use the house prediction analogy</strong><br>
Imagine we’re trying to predict the price of houses and the number of rooms in each house(just for the sake of intuition)- we should have just 2 neurons in the final layer. However, lets say we want to be more specific(and we have enough data) to predict the prices of houses in 12 different states along side the number of rooms. So we’ll have 13 neurons in our final layer(12 for the prices and 1 for # of rooms).</p>
<ol>
<li>Does this architecture make sense?</li>
<li>Does it make sense then to compute our loss wrt to the 2 variables independently and sum them up?</li>
</ol>
<p><strong>Something like this</strong></p>
<pre><code class="lang-auto">
output_dims = 12
input_dims = 491776
mse_loss = nn.MSELoss()

model = MLP(input_dims, output_dims) #we get a tensor with 13 values
l1 = mse_loss(model[0][:-1], true_prices) #prices
l2  = mse_loss(model[0][-1], true_rooms) #number of rooms
loss = l1 + l2
optimizer.zero_grad() 
loss.backward() 
optimizer.step()
</code></pre>

eqy: <p>This looks OK at a glance—I would also check that the loss scales are comparable across <code>l1</code> and <code>l2</code> since one is prices and the other is rooms or rather that they are not in wild disagreement over how you would weight the two tasks.</p>
Is a lower number of CPU cores better?

tanabe.d.zc32s: <p>Hi all.</p>
<p>A strange thing happened.</p>
<p>I am using Pytorch on a shared PC and the CPU usage was very high and monopolizing resources during machine learning.<br>
So I limited the number of CPU cores used (72 → 8) and the CPU utilization dropped as expected compared to before the limitation. And strangely enough, the GPU utilization became higher, and the time for learning was also greatly reduced.<br>
This is a departure from my expectation that the more cores on the CPU, the better.<br>
What could be the cause of this phenomenon?</p>
<p>Best regards.</p>

eqy: <p>It could on the mechanism you used. If it was e.g., <code>OMP_NUM_THREADS</code>, you could be reducing the contention for physical cores between different threads. It would also depend on the workload e.g., if you are relying on the CPU for actual numerical computation (rather than just Python/dispatching), it isn’t recommended to use more threads than the number of <em>physical</em> cores on the machine.</p>
Torch tensor print

Dr_Vladyslav_Danilov: <p>Dear torchers,</p>
<p>I have a short comment/thought on printing pytorch tensors. The thing is that usually these tensors are big, and print function in python shows a shortened version of the tensor, which makes sense.<br>
For example:<br>
embs_txt =  tensor([[ 0.0010,  0.0058, -0.0007,  …,  0.0090, -0.0016,  0.0083],<br>
[-0.0112,  0.0075, -0.0007,  …,  0.0104,  0.0091,  0.0010],<br>
[-0.0016,  0.0067, -0.0073,  …,  0.0081,  0.0051,  0.0004],<br>
[ 0.0019,  0.0009, -0.0082,  …,  0.0181,  0.0193,  0.0007]],<br>
device=‘cuda:0’)</p>
<p>However, I would like to suggest to remove rounding of the weights itself, because sometimes you check your code and want to have a short comparison if order, or weights itself match. So you compare such prints with weights recoded in a file, for example. In that case, one may find that the unrolled weights have no such weights that starts with 0.0010 or ends with 0.0007. The thing is that these numbers are rounded. I think from time to time it may confuse a bit, especially newcomers.<br>
Please, share what you think.</p>
<p>Cheers,<br>
V.D.</p>

ptrblck: <p>You could use <code>torch.set_printoptions(precision=...)</code> to increase the printed precision of the values in case that’s needed.</p>
How to enable torch.profiler.profile using flag?

jayz: <p>There is the context manager <code>profile</code> (<a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 1.11.0+cu102 documentation</a>) which allows us to monitor cpu and gpu times. I’d like to integrate it into my script such that I can turn it on / off flexibly:</p>
<pre><code class="lang-python">from torch.profiler import profile
use_prof = True
with profile(enabled=use_prof,...): 
    # ...
</code></pre>
<p>Only there is no <code>enabled</code> option, as opposed to the <code>torch.autograd.profile</code> where such option exists (<a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.profiler.profile" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.11.0 documentation</a>).</p>
<p>Is it possible / is there a workaround to flag the <code>torch.profiler.profile</code> somehow, just like <code>torch.autograd.profile</code>?</p>
<p>Thanks!</p>

psoulos: <p>You can use a <a href="https://docs.python.org/3.7/library/contextlib.html#contextlib.nullcontext" rel="noopener nofollow ugc">nullcontext</a> and switch on that.</p>
<pre><code class="lang-auto">from torch.profiler import profile
import contextlib
use_prof = True
cm = profile(enabled=use_prof,...) if use_prof else contextlib.nullcontext()
with cm as prof:
   ...
</code></pre>
TMUX and GPU memeory occupancy issues

papoo13: <p>I used <code>tmux new-session</code> to create a tmux process and then run my PyTorch code on GPU using <code>CUDA_VISIBLE_DEVICES=0 CUBLAS_WORKSPACE_CONFIG=:16:8 python main.py</code>. After using <code>nvidia-smi</code> to check the GPUs’ status, I noticed that some of the GPUs memories are still allocated to the tmux processes, despite the fact that they were finished running. If I use <code>ctrl+c</code> to terminate a tmux process, the memory gets freed up. If I kill the tmux processes, the memory releases too. I don’t get what the issue is. Is there a way to guarantee that when a tmux process is finished, the GPU memory is fully released again?</p>

ptrblck: <p>Based on your description it seems that <code>tmux</code> might keep the Python process alive for some reason.<br>
Do you still see the python process via <code>ps aux</code> in the <code>tmux</code> session?</p>

papoo13: <p>Thank you for the reply. Could you explain a bit further how I should check this? Should I go inside the tumx session and look for the pids?</p>

ptrblck: <p>Yes, that’s how I would check for running processes and see if something still uses GPU memory.</p>
PyTorch: Scattering a 3D tensor

alvarogutyerrez: <p>I have the following 3D tensor (<code>tensor_3D</code>), which I want to scatter (product) over the first two dimensions using the index (<code>index</code>).</p>
<pre><code class="lang-auto">import torch
# create a 3D tensor of size 3x3x3 with numbers from 1 to 27 
tensor_3D = torch.arange(1,28).reshape(3,3,3) 
#tensor([[[ 1,  2,  3],
#         [ 4,  5,  6],
#         [ 7,  8,  9]],
#
#        [[10, 11, 12],
#         [13, 14, 15],
#         [16, 17, 18]],
#
#        [[19, 20, 21],
#         [22, 23, 24],
#         [25, 26, 27]]])

# create a 1D tensor of size 3 to be used as index when scattering
index = torch.tensor([0,0,1]).reshape(3,1) 
#tensor([[0],
#        [0],
#        [1]])
</code></pre>
<p>This is my desired tensor is:</p>
<pre><code class="lang-auto"># permute the result to get the desired output
d1 = torch.tensor ([4,  10,  18, 7,   8,   9 ]).reshape(2,3)
d2 = torch.tensor ([130, 154, 180, 16,  17,  18 ]).reshape(2,3)
d3 = torch.tensor ([418, 460, 504, 25,  26,  27 ]).reshape(2,3)
desired = torch.stack([d1,d2,d3], dim=2)
#tensor([[[  4, 130, 418],
#         [ 10, 154, 460],
#         [ 18, 180, 504]],
#
#        [[  7,  16,  25],
#         [  8,  17,  26],
#         [  9,  18,  27]]])
</code></pre>
<hr>
<p>As of now, I am doing the following but I am not getting the correct dimensions</p>
<pre><code class="lang-auto">import torch 
# create a 3D tensor of size 3x3x3 with numbers from 1 to 27 uisng torch.arange
tensor_3D = torch.arange(1,28).reshape(3,3,3) 
# create a 1D tensor of size 3 with numbers 
index = torch.tensor([0,0,1]).reshape(3,1) 
#Expand index to 3x3x3
index_expand = index.expand(3,3,3)
print(index_expand)
# use index to scatter_reduce product the values of tensor_3D to a new tensor of size 3x2x3
new_tensor = torch.ones(3,2,3, dtype=torch.int64)
print(new_tensor)
res = new_tensor.scatter_reduce(
    1,
    index_expand, 
    tensor_3D, 
    reduce="prod")
</code></pre>

ptrblck: <p><code>res.permute(1, 2, 0)</code> should work and would create your desired output.</p>

alvarogutyerrez: <p>Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a>, for your answer. This actually produces the desired output. However, I would like to understand how <code>scatter_reduce()</code> works for 3D tensors. Could you help me to find a solution that does not require this ex-post modification (<code>.permute()</code>) to find the desired output?</p>
<p>To be concrete, I would like to know what kind of <code>index</code> I need to provide to modify the <code>new_tensor </code> object if it were 2x3x3 from the beginning (differently from my attempted solution where it starts being 3x2x3 and then it was <code>permute()</code> ex-post to 2x3x3)</p>
<p>Additionally, I couldn’t find examples for <code>scatter_reduce()</code> in the documentation where tensors larger than two dimensions were used. Am I simply not looking at the right places or there are they simply not provided?</p>
<p>Thank you a lot for your time.</p>
Source for the torch.diff() operator

ifeherva: <p>I would like to build a custom operator in c++ that is largely based on the torch.diff operator. I was trying to find the source code of that operator but github search failed me. Can anyone point me to the right source file please? Thank you!</p>
<p>/ I am also okay if someone can point me some documentation that explains the project structure and where one usually go for different parts /</p>

ifeherva: <p>Found it here: <a href="https://github.com/pytorch/pytorch/blob/v1.13.1/aten/src/ATen/native/ReduceOps.cpp#L856" class="inline-onebox" rel="noopener nofollow ugc">pytorch/ReduceOps.cpp at v1.13.1 · pytorch/pytorch · GitHub</a></p>
What does the custom Observed module do?

111357: <p>I am a little confused about the custom observed module when I read the <a href="https://pytorch.org/docs/stable/quantization.html#quantization-custom-module-api" rel="noopener nofollow ugc">Quantization Custom Module API</a> docs.</p>
<p>The code snippet in the docs are as follows:</p>
<pre><code class="lang-python"># custom observed module, provided by user
class ObservedCustomModule(torch.nn.Module):
    def __init__(self, linear):
        super().__init__()
        self.linear = linear

    def forward(self, x):
        return self.linear(x)

    @classmethod
    def from_float(cls, float_module):
        assert hasattr(float_module, 'qconfig')
        observed = cls(float_module.linear)
        observed.qconfig = float_module.qconfig
        return observed
</code></pre>
<p>The <code>ObservedCustomModule</code> class seems to do the same as <code>CustomModule.linear</code> (<code>torch.nn.Linear</code>) except add the <code>qconfig</code> attribute.</p>
<p>Can someone provide a built-in observed module for reference?</p>
<p>If I want to customize an observed module for quantizing <code>conv2d</code> with 2 bits, what should I do?</p>

Vasiliy_Kuznetsov: <p>hi <a class="mention" href="/u/111357">@111357</a> , this is a hook for user to insert customizable quantization modules.</p>
<p>If you wanted to set up something like conv2d with 2 bit weights, you’d need</p>
<ol>
<li>kernels that can do inference with 2 bit weights (or kernels that emulate this).  There is no such thing in PyTorch today, but you could provide a custom function.  You could write this in Python to at least get correct numerics.</li>
<li>either an observer or a fake_quantize module which is set up for 2 bits.  You can do this by setting qmin/qmax settings in the default observers provided by quantization.</li>
<li>once you have those things, you can plug them in to the framework using <a href="https://pytorch.org/docs/stable/quantization.html#quantization-custom-module-api" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.13 documentation</a>
</li>
</ol>
Is there a way to make models.[anynet](pretrained=True) [anynet] configureable as the parameter from the function?

Song_Song: <p>i.e</p>
<pre><code class="lang-auto">From torchvision import models
model = models.vgg19(pretrained=True)
</code></pre>
<p>vgg19 is the net I set here to be used. Currently, I have a list of net(s) I want to use as parameters in my function below. Each time it will take one value from the list and return the above example.</p>
<p>Example list:[VGG19, resnet50 ,vit_b_16]</p>
<pre><code class="lang-auto">def fe_net(self, extractor):
    model = str(models + '.' + extractor + 'pretrained=True')
    modules = list(model.children())[:-1]  # delete the last fc layer.
    feature_extractor = nn.Sequential(*modules)
    for param in feature_extractor.parameters():
        param.requires_grad = False
    return feature_extractor
</code></pre>
<p>The current error message shows:</p>
<blockquote>
<pre><code class="lang-auto">model = str(models + '.' + extractor + 'pretrained=True') TypeError: unsupported operand type(s) for +: 'module' and 'str'
</code></pre>
</blockquote>

ptrblck: <p><a href="https://pytorch.org/blog/easily-list-and-initialize-models-with-new-apis-in-torchvision/">This blog post</a> might be helpful as it explains how util. methods can be used to get all models.</p>

Song_Song: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks so much! That definitely helped! One question is that post does not have a parameter as</p>
<p><code>pretrained=True</code> . Does</p>
<p><code>weights="DEFAULT"</code> works exactly same?</p>

ptrblck: <p>Not exactly, as the <code>DEFAULT</code> weights might change between different <code>torchvision</code> releases as described in <a href="https://pytorch.org/vision/stable/models.html#initializing-pre-trained-models">Initializing pre-trained models</a>.<br>
Currently <code>DEFAULT</code> seems to refer to <code>IMAGENET1K_V2</code> for e.g. <code>resnet18</code> while the deprecated <code>pretrained=True</code> argument should refer to <code>IMAGENET1K_V1</code> as given in the warning:</p>
<pre><code class="lang-python">&gt;&gt;&gt; torchvision.models.resnet18(pretrained=True)
UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet18_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet18_Weights.DEFAULT` to get the most up-to-date weights.
</code></pre>
AMD radeon not working with pytorch

Mohammed_Zidane: <p>Hi,<br>
I installed PyTorch for AMD ROCm™ using this<br>
pip3 install torch torchvision torchaudio --extra-index-url <a href="https://download.pytorch.org/whl/rocm5.2" rel="noopener nofollow ugc">https://download.pytorch.org/whl/rocm5.2</a></p>
<p>and ran a simple code<br>
import os<br>
import torch<br>
os.environ[“CUDA_VISIBLE_DEVICES”]=“0,1” # ‘0’ is the NVIDIA GPU card, ‘1’ is AMD radeon GPU<br>
t1 = torch.zeros(2,3, device=‘cuda:1’)</p>
<p>But got this error<br>
RuntimeError: Unexpected error from hipGetDeviceCount(). Did you run some cuda functions before calling NumHipDevices() that might have already set an error? Error 101: hipErrorInvalidDevice</p>
<p>Could you help?<br>
Thanks</p>

OuttaBrainz: <p>Here’s a guide I wrote: <a href="https://medium.com/@damngoodtech/amd-rocm-pytorch-and-ai-on-ubuntu-the-rules-of-the-jungle-24a7ab280b17" class="inline-onebox" rel="noopener nofollow ugc">AMD, ROCM, PyTorch, and AI on Ubuntu: The Rules of the Jungle | by Jordan H (Principal, Damn Good Tech) #openforwork | Feb, 2023 | Medium</a></p>
<p>If you experience anything hip-related, then you usually need to set the <code>HSA_OVERRIDE_GFX_VERSION</code> flag.</p>
Change embedding's weights

Rami_Nasser: <p>is there a way to change nn.Embedding weights and keep the gradient ?<br>
for example, passing the nn.Embedding layer through an mlp layer and keep the gradient so the mlp is updated.<br>
If I update the nn.Embedding.weight.data this doesn’t keep the gradient to the mlp.</p>

dreidizzle: <p>Can you elaborate on what you want to do exactly? So if you have an Embedding layer and you do a forward pass and then a backward pass you’ll get all the gradients for all parameters. You’d then step to update all the parameters with the gradients that you computed. The loss ideally would go down.</p>
<p>You want to get gradients, change the embedding data, and then update? I mean then you can’t guarantee that you make progress on the loss. Why can you just do this manually?</p>
<p>old_grad = e.grad.data<br>
e.data = new data<br>
e.data -= old_grad</p>
<p>This is pseudo code. If you explain more, maybe I can comment more but unsure of the idea here.</p>

Rami_Nasser: <pre><code class="lang-auto">encoder = nn.Embedding()
W = nn.Parameters()
# once in epoch, I want to make the following update to the encoder:
encoder.weight.data = W.mm(encoder)
</code></pre>
<p>The problem is that the gradient is lost and the W is not updated.</p>

dreidizzle: <p>Wait so the encoder is an embedding layer. So, I think what you want to do is to grab the weight inside of the encoder and multiply it by W and then compute the loss? This way W and the weight in the encoder are in the computational graph, and you’ll get gradients. This would work, but still not sure if it’s what you want to do.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
e = nn.Embedding(10, 5)
w = nn.Parameter(torch.rand(5, 5))
p  = (e.weight @ w).sum()
l = (1 - p) ** 2
l.backward()
w = w - 0.01*w.grad
e.weight.data = e.weight.data - 0.01 * e.weight.grad
</code></pre>
<p>Now, you have gradients for w and e.weight and you can update w and e.weight. The goal here is so the product-sum of e.weight and w is 1 in MSE. You need to adapt this to what you want. The learning rate about is 0.01.</p>
Lane Detection Model Not Producing Any Results

j15big: <p>Hello! I’m very new to this so please excuse any obvious mistakes.</p>
<p>I want to create a computer vision model in pytorch that will detect and highlight lane markers on the road using a live video feed. This is mainly for learning purposes as I know models like this already exist.</p>
<p>I wanted to start training with a very low count of training images because I want to make sure this all works before I spend time creating masks for more images.</p>
<p>I was able to train a simple segmentation model that I found online. During training, the loss was going down. Once the training finished, I tried to pass one of the training images through the model, but the results I got were unexpected.</p>
<p>After “denormalizing” the output mask and applying it to the input image using cv2, it was just completely blank.</p>
<p>Can you take a look at my code and see if anything jumps out at you as being incorrect?</p>
<p>Some things I can think of that might be causing this:</p>
<ul>
<li>Model is too simple for this task</li>
<li>Not enough training data (I know this is true, but I was expecting at least some results)</li>
<li>My normalization/denormalization is wrong</li>
</ul>
<p>Thanks in advance!</p>
<p><a href="https://github.com/joeysantana3/lane-detection/blob/master/Lane-Detection-Complete.ipynb" rel="noopener nofollow ugc">https://github.com/joeysantana3/lane-detection/blob/master/Lane-Detection-Complete.ipynb</a></p>

ksmdanl: <p>Hi Joey, did you check the prediction output whether it’s normalized?<br>
If I understand it correctly, your prediction should be a sigmoid output according to <code>predict()</code> function. In your case the mask ranges between 0. and 1. Thus I would transform the output to 0…255 value range and check if it works.</p>
xpected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 

keertika-11: <pre><code class="lang-auto">def accuracy(predictions, labels):
    classes = torch.argmax(predictions, dim=1)
    print("prediction",classes)
    print("ground truth",labels)
    return torch.mean((classes == labels).float())

def test_loop(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataloader:
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            print(pred.argmax(1))
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()

    test_loss /= num_batches
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):&gt;0.1f}%, Avg loss: {test_loss:&gt;8f} \n")

if __name__ == '__main__':
    "Image Visualization"
    # iter_l = iter(train_loader)
    # input, classes = next(iter_l)
    # out = torchvision.utils.make_grid(input)
    # print(out.shape)
    # print([x.item() for x in classes])
    # imshow(out, title=[x.item() for x in classes])
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    
    "Creating CoAtNet Model"
    model = timm.create_model("coatnet_2_rw_224.sw_in12k_ft_in1k", pretrained=True, num_classes=1981, scriptable=True,
                              exportable=True)
    model = torch.nn.DataParallel(model)
    model.to(device)
    model.cuda()
    print(device)
    # print(model.get_classifier())
    data_config = timm.data.resolve_model_data_config(model)
    print(data_config)
    "Creating optimizer"
    optimizer = timm.optim.create_optimizer_v2(model, opt='adam', lr=0.0001)
    "creating Loss function"
    loss_fn = torch.nn.CrossEntropyLoss()
    for epoch in range(1):
        running_loss = 0.00
        running_accuracy = 0.00
        size = len(train_loader.dataset)
        for batch,(inputs, targets) in enumerate(train_loader):
            inputs, targets = inputs.to(device), targets.to(device)
            outputs = model(inputs)
            loss = loss_fn(outputs, targets)
            # print(loss)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            
            if batch % 100 == 0: 
                loss, current = loss.item(), batch * len(inputs)
                # running_loss += loss.item()
                running_accuracy += accuracy(outputs, targets)
                print("Accuracy",running_accuracy)
                print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
        # running_loss /= len(trainloader)
        running_accuracy /= len(train_loader)
        test_loop(val_loader, model,loss_fn)
        
    model = model.eval()
    # saving the entire model
    torch.save(model,"test_coatnet.pt")
</code></pre>

ptrblck: <p>You are not pushing the data to the GPU in your <code>test_loop</code>:</p>
<pre><code class="lang-python">for X, y in dataloader:
    pred = model(X)
</code></pre>
<p>which is most likely causing the issue.</p>

keertika-11: <p>Thank you It worked!</p>
Compute capability 3.5

akshaypaygude: <p>Is there a way to use PyTorch with compute capability 3.5, other than PyTorch source code compilation?</p>

ptrblck: <p>Unfortunately, you would need to build from source or use old PyTorch binaries, which still shipped with compute capability 3.5. Some older (community provided) binary builds are also provided <a href="https://github.com/pytorch/pytorch/issues/30532">here</a>.</p>

akshaypaygude: <p>Thanks for the help.<br>
I did install one of the pre-built PyTorch. However, on <em>conda list</em> it shows PyTorch as <strong>unknown</strong> and I’m not able to <em>import torch</em> as well. It says no module found.</p>
<p>Is there a way around this?</p>

akshaypaygude: <p>Sorry to bother. I know there are other similar discussions but I have not found clear explanation or solution to the import issue.</p>

ptrblck: <p>Sorry, I don’t know what might be causing these issues using the community built binaries.</p>
Efficient KNN tensor

L-Reichardt: <p>I am looking for a memory efficient way to construct a KNN tensor prior to a Linear Layer.</p>
<p>An example with K = 5:<br>
I have a pointcloud tensor (N, 3) and KNN-Indices Tensor (N, 5). If I concatenated the pointcloud neighbors the tensor becomes (N, 3, K=5). However, each of the K-neighbors represents data in the original (N, 3) pointcloud, just in different order.</p>
<p>Is there a memory efficient way to construct this (N, 3, K) tensor compatible with linear layers? E.g. using something like K-views of the original pointcloud.</p>
<p>I am unsure what would be the best way to approach this problem and am looking for some ideas/guidance.</p>

L-Reichardt: <p>Found and solved partiall with .expand().</p>
<p>However, .Expands just views the original pointcloud multiple times in the same order. I tried some tests, changing the order of each K without using additional memory does not seem possible, as each view after expand changes the other Ks as well.</p>
What is the specific difference between wait steps and warmup steps in torch.profiler.schedule?

songyuc: <p>Hi, guys!<br>
I am learning about using the <code>torch.profiler.schedule</code> API. But from its introduction, I cannot figure out the specific difference between wait steps and warmup steps inside it.</p>
<p>What I can get it that the profiler would not do recording in both wait steps and warmup steps, but is there any other specific differences of operations between them?<br>
In other words, why are the wait steps are esstianl for profiling?</p>
<p>Your answer and guide will be appreciated!</p>

ptrblck: <p>During the wait steps the profiler should be disabled and not try to profile any workloads at all.<br>
Afterwards, during the warmup the profiler is active and profiles the workload but nothing will be recorded. After this warmup, the profiler keeps profiling the workload and this time the results are recorded.<br>
That’s at least my understanding.</p>

songyuc: <p>Thanks sincerely for your response!</p>
'WeightNormInterfaceBackward0' returned nan values in its 0th output

a15695293281: <p>I’m training a TCN model for time series prediction, the structure is as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/0988159faa666ddfbde505328e39ec41a18247a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/0988159faa666ddfbde505328e39ec41a18247a4" title="图片 1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/0988159faa666ddfbde505328e39ec41a18247a4_2_544x500.png" alt="图片 1" data-base62-sha1="1mjR7LZMgokMwGQuGimZ4pnWaXO" width="544" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/0988159faa666ddfbde505328e39ec41a18247a4_2_544x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/9/0988159faa666ddfbde505328e39ec41a18247a4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/9/0988159faa666ddfbde505328e39ec41a18247a4.png 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">图片 1</span><span class="informations">789×725 67.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But as the training progressed, I suddenly got the following error report:</p>
<p><strong>‘WeightNormInterfaceBackward0’ returned nan values in its 0th output.</strong></p>
<p>I carefully checked the parameters of the model and found that some of them were particularly strange, the values of the parameters were particularly small （1e-16, 1e-17），and the corresponding gradients were almost 0. After backward, they became nan</p>
<p>I have tried to lowe the learning rate, but this time it just took a little longer to train and I still get the same reported error! And I dont’t know why this happen…</p>
<p>Can anyone who also met similar problems help me?<br>
What’s the problem and How can I correct it?</p>

hygr: <p>I meet the similar problem, have you solve this probelm?</p>

hygr: <p>**‘WeightNormInterfaceBackward0’ maybe mean the WeightNorm layer make some error(it cause nan)，so I delete the WeightNorm layer of Conv1d ,it works!</p>
Error in Transformer tutorial building vocab

drj3122: <p>I’m following the <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and TorchText — PyTorch Tutorials 1.12.1+cu102 documentation</a> and am running into an error in the vocab portion:</p>
<pre><code class="lang-auto">from torchtext.datasets import WikiText2
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator
train_iter = WikiText2(split='train')
tokenizer = get_tokenizer('basic_english')
vocab = build_vocab_from_iterator(map(tokenizer, train_iter), specials=['&lt;unk&gt;'])
</code></pre>
<pre><code class="lang-auto">**Exception** : OnDiskCache Exception: C:\Users\XXXXX/.cache\torch\text\datasets\WikiText2\wikitext-2-v1.zip expected to be written by different process, but file is not ready in 300 seconds. This exception is thrown by __iter__ of MapperIterDataPipe(datapipe=UnBatcherIterDataPipe, fn=functools.partial(&lt;function _wait_promise_fn at 0x0000016F75AFF2E0&gt;, 300), input_col=None, output_col=None)
</code></pre>
<p>I thought maybe it’s a network issue since I’m on a work network but I was able to download it from <a href="https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip" rel="noopener nofollow ugc">https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip</a>.</p>

Aryan_AMG: <p>Hi there were you able to solve this problem if so how did you do it</p>
The PyTorch Model keeps appearing KeyError error

astorwang0705: <pre><code class="lang-auto">
import torch
import torch.nn as nn
import torch.optim as optim

#導入訓練模型
text = ""
#將text中的字數轉換成數字
chars = list(set(text))
print(chars)
#建立labels，用以儲存不同的句子
labels = [""]
char = list(set(labels))
print(char)
int2label = dict(enumerate(char))
label2int = {label: index for index, label in int2label.items()}
encoded_labels = [label2int[label] for label in labels]
int2char = dict(enumerate(chars))
char2int = {char: index for index, char in int2char.items()}
encoded_text = [char2int[char] for char in text]

#定義訓練環境
class SentenceRecognitionModel(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, num_layers,num_classes):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size,embedding_dim)
        self.lstm = nn.LSTM(embedding_dim,hidden_dim, num_layers)
        self.fc = nn.Linear(hidden_dim, num_classes)
    def forward(self, x):
        x = self.embedding(x)
        x, _=self.lstm(x)
        x = self.fc(x)
        return x
    
#創建模型
model = SentenceRecognitionModel(len(chars),embedding_dim = 256, hidden_dim = 64, num_layers = 2, num_classes=len(labels))
#創建優化器
optimizer = optim.Adam(model.parameters())
#紀錄模型訓練損耗
loss_fn = nn.CrossEntropyLoss()

#訓練模型
num_epochs = 25000 #訓練二萬五千次模型
for epoch in range(num_epochs):
    #將每一次訓練前的梯度設為零，這樣在下次迭代時就不會衝突到了
    optimizer.zero_grad()
    input_seq = torch.tensor(encoded_text).unsqueeze(0)
    target_label = torch.tensor(encoded_labels).unsqueeze(0)
    output = model(input_seq)
    loss = loss_fn(output.squeeze(), target_label.squeeze())
    loss.backward()
    optimizer.step()
    if (epoch+1) % 10 == 0:
        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')
#將模型設為推理模式，以讓模型的輸出保持一致
model.eval()
while True:
  # 接收用戶輸入的句子
  input_sentence = input("請輸入句子：")
  if input_sentence not in labels:
    print("輸入的句子不存在於訓練數據中")
  else:
    start_index = label2int[input_sentence]
    generated_text = int2char[start_index]
    # 選擇要生成的文本長度
    num_generated = 100

    # 對輸入的句子進行預處理，將其編碼為數字
    encoded_sentence = [label2int[start_index] for generated_text in input_sentence]
    input_seq = torch.tensor(encoded_sentence).unsqueeze(0)

    # 將輸入的句子傳遞到模型中
     
    with torch.no_grad():
         output = model(input_seq)

    #使用模型去辨識句子
    output_idx = torch.argmax(output.squeeze()).item()
    recognized_label = int2label[output_idx]

    # 輸出結果
    print(f"Recognized Label: {recognized_label}")
    FILE = 'model_state_dict.pt'
    model.save(model.state_dict(), 'SentenceRecognizedModel.pt') 
</code></pre>
<p>encoded_sentence = [label2int[start_index] for generated_text in input_sentence] keep appearing KeyError error</p>
?Compiling misses parent vars? limitation of inspect.getattr_static?

aifartist: <p>If ‘parent’ vars isn’t the right term then replace it with the correct term in what follows.</p>
<blockquote>
<p>M = &lt;ldm.models.diffusion.ddpm.LatentDiffusion model&gt;<br>
print(f"xxx in dir {‘xxx’ in dir(M)}")  # True<br>
print(M.xxx)     # no problem</p>
<p>M = torch.compile(M) # M now torch._dynamo.eval_frame.OptimizedModule<br>
print(f"xxx in dir {‘xxx’ in dir(M)}")  # <strong>False</strong><br>
print(M.xxx)     # no problem even though no longer in M</p>
</blockquote>
<p>Does torch compile, which produces an OptimizedModule object, make this a subclass of the original<br>
LatentDiffusion object?</p>
<p>In any case the following then fails for the compiled model if using a compiled function:</p>
<blockquote>
<p>print(f"M.xxx = {M.xxx}“)  # Still ok<br>
<span class="mention">@torch.compile</span><br>
def _wacko(sdm):<br>
print(f"sdm.xxx = {sdm.xxx}”)<br>
<strong>wacko(M)</strong></p>
</blockquote>
<p>This crashes with “AttributeError: xxx” and the following stack:<br>
File “~/a1test/webui.py”, line 111, in initialize<br>
modules.sd_models.load_model()<br>
File “~/a1test/modules/sd_models.py”, line 443, in load_model<br>
wacko(sd_model)<br>
File “~/a1test/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py”, line 209, in _fn<br>
return fn(*args, **kwargs)<br>
File “~/a1test/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py”, line 330, in catch_errors<br>
return callback(frame, cache_size, hooks)<br>
File “~/a1test/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py”, line 404, in _convert_frame<br>
result = inner_convert(frame, cache_size, hooks)<br>
File “~/a1test/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py”, line 104, in _fn<br>
return fn(*args, **kwargs)<br>
File “~/a1test/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py”, line 262, in _convert_frame_assert<br>
return _compile(<br>
File “~/a1test/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py”, line 163, in time_wrapper<br>
r = func(*args, **kwargs)<br>
File “~/a1test/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py”, line 394, in _compile<br>
raise InternalTorchDynamoError() from e<br>
torch._dynamo.exc.InternalTorchDynamoError</p>

ptrblck: <p>Could you create an issue on GitHub, since an internal error is raised, so that we could track and fix it, please?</p>

aifartist: <p>Ok.  I just wanted to make sure there was a better than 50/50 change I wasn’t making a mistake.</p>

ptrblck: <p>Even if your code would be broken in some way, I wouldn’t expect to see internal errors so at least the error handling should be checked. Also, thanks for forwarding this issue in the first place.</p>

aifartist: <p><a href="https://github.com/pytorch/pytorch/issues/94478" class="inline-onebox" rel="noopener nofollow ugc">@torch.compile has problems accessing compiled model vars · Issue #94478 · pytorch/pytorch · GitHub</a> is now created</p>
How to concatenate CNN features and reduce size

lanka: <p>hi,<br>
suppose I have vgg features of images like 4096 and some other features 2048, so I need to concatenate both together and reduce the feature size to 512. how can I do that?<br>
<strong><code>f1=4096,f2=2048</code></strong></p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.fc1 = nn.Linear(4096,2048)
        self.fc2 = nn.Linear(4096,512)
        
    def forward(self, f1,f2):
        x1 = self.fc1(f1)
        x=torch.cat(x1,f2,1)
        x = F.relu(self.fc1(x))
       
        return x
</code></pre>
<p>and after starting the training process, do we need set feature x before feed the proposed model;<br>
<code>x=Variable(x)</code> ?   //(<code>from torch.autograd import Variable</code>)</p>

J_Johnson: <p>It looks to me like you’re having trouble with your <code>torch.cat</code> function. To fix it, you can change that line to add brackets:</p>
<pre><code class="lang-auto">x = torch.cat([x1, f2], dim=1)
</code></pre>

lanka: <p>Thank you. what about autograd?</p>

J_Johnson: <p>What about it? Anything you pass through torch objects are caught in the graph unless you explicitly <code>.detach()</code> them.</p>
Where are input ids and attention masks stored during extra large scale finetuning?

Prudhvi_Raj: <p>Hi, I am working on a project that involves fine-tuning a sentence transformers model on 1B pairs. Usually in practice, how are these pairs processed? It seems futile to load the sentences from a generate and generate the tokenized encodings for each of them in every epoch? Is there a practical way to do this? Please help. Thanks</p>
Can the discriminator of a GAN produce high probability values for both real and fake images?

shrbrh: <p>Hi,<br>
I am training a GAN model for image style transfer. The discriminator tries to classify between real and stylized images. Ideally, the discriminator should give high probability values for the real images and low values for the stylized images as the training starts. These are some of the values I got:</p>
<pre><code class="lang-auto">For real images: [[[0.8925122618675232]], [[0.8756555318832397]], [[0.8772845268249512]], [[0.8345637321472168]]]
For corresponding stylized images: [[[0.4710726737976074]], [[0.47152334451675415]], [[0.471288800239563]], [[0.47164005041122437]]
</code></pre>
<p>But as the training proceeds, after around 60 iterations, the discriminator starts producing weird results and is stuck at probability 1.0 for the stylized images:</p>
<pre><code class="lang-auto">For real images after 60 iterations: [[[0.9994499683380127]], [[0.9997959136962891]], [[0.9631960391998291]], [[0.9862018823623657]]]
For corresponding stylized images after 60 iterations: [[[1.0]], [[1.0]], [[1.0]], [[1.0]]
</code></pre>
<p>I trained upto 50000 iterations and it is still stuck at 1.0. The adversarial loss function I used is:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c26d44a575cee8cf35a6f4618e04f9f5dcaefcf9.png" alt="GANLossFunction" data-base62-sha1="rJYJA6jN2XGCE7WYn1E8QIOS5AZ" width="364" height="72"><br>
where x is the real images and G(x,y) is the stylized image.<br>
Please provide suggestions on when and why the discriminator behaves in this manner? Is there a way to rectify this issue?</p>
Does weight sharing occur if I simply assign the weights of a layer to another?

seankala: <p>I had a question regarding weight sharing. The current paper that I’m reimplementing has an option to use the embedding layer as a classification layer. Currently what I’ve done so far is use this if-else block inside my forward method (the code is a simplified version to get my point across):</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self,
                 share_weights: bool,
                 hidden_dim: int,
                 num_items: int) -&gt; None:
        super().__init__()

        self.share_weights = share_weights

        self.embedding = nn.Embedding(num_items, hidden_dim)
        self.classifier = nn.Linear(hidden_dim, num_items)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x_emb = self.embedding(x)
        if self.share_weights:
            outputs = x_emb @ self.embedding.weight.transpose(1, 0)
        else:
            outputs = self.classifier(x_emb)

        return outputs
</code></pre>
<p>What I was wondering if I could simply assign the weights of the embedding layer to the classifier layer:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self,
                 share_weights: bool,
                 hidden_dim: int,
                 num_items: int) -&gt; None:
        super().__init__()

        self.embedding = nn.Embedding(num_items, hidden_dim)
        self.classifier = nn.Linear(hidden_dim, num_items)

        if share_weights:
            self.classifier.weight = self.embedding.weight.transpose(1, 0)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x_emb = self.embedding(x)
        outputs = self.classifier(x_emb)

        return outputs
</code></pre>
<p>and receive the same results since I’m not performing a deep copy or anything?</p>

eqy: <p>Yes, this should implement weight sharing and you can verify this by e.g., checking the <code>.grad</code> field of the weights when using the same inputs with separate vs. shared weights.</p>
How to convert a meta tensor to normal tensor?

HeyangQin: <p>If we simply try to cast a meta tensor to another device, it doesn’t work</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a=torch.empty([10,10],device="meta")
&gt;&gt;&gt; a.to("cpu")
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
NotImplementedError: Cannot copy out of meta tensor; no data!
</code></pre>
<p>What is the recommended way to convert a meta tensor to normal tensor?</p>

bdhirsh: <p>The issue with converting a meta tensor to a cpu tensor is that… the meta tensor doesn’t have any data! What data do you want your tensor to contain once you “move” into cpu?</p>
<p>One option would be to just construct a fresh tensor on the cpu device, using the metadata from your meta tensor.</p>
<p>Cpu_tensor = torch.empty_like(meta_t, device=“cpu”)</p>
ATen IR and mutation/in-place

BowenBao: <p>This is regarding <a href="https://pytorch.org/docs/master/ir.html" class="inline-onebox" rel="noopener nofollow ugc">IRs — PyTorch master documentation</a>.</p>
<p>I found it somewhat contradicting that the doc states</p>
<blockquote>
<p>Core aten IR is fully functional, and there is no inplace or _out variants in this opset.</p>
</blockquote>
<p>However, in the list of ops, there are still a few annotated with aliasing, naming a few</p>
<blockquote>
<p>aten.alias | alias(Tensor(a) self) → Tensor(a)<br>
aten.permute | permute(Tensor(a) self, int<span class="chcklst-box fa fa-square-o fa-fw"></span> dims) → Tensor(a)<br>
aten.expand | expand(Tensor(a) self, SymInt<span class="chcklst-box fa fa-square-o fa-fw"></span> size, *, bool implicit=False) → Tensor(a)<br>
…</p>
</blockquote>
<p>So which statement is more accurate?</p>

bdhirsh: <p>I think aliasing ops are still considered part of “core aten IR” (maybe this is just a nomenclature issue, where you read “functional” IR as “non-aliasing”?</p>
<p>Core aten IR doesn’t include mutations, as you pointed out.</p>
Multiprocessing args issue

Meraki: <p>Hi, I tried to run multiprocessing on cpu for my network, but confused about the issue below:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
import numpy as np
import time

params2 = ...
params5 = ...

images = ...

def run():
    times = []
    for i in [1,2, 3, 4]:
        start = time.time()
        mp.spawn(train, args=(images, params2, params5, 20, 10), nprocs=i)
        #train(images, params2, params5, batch_size, epochs=1000)
        end = time.time()
        times.append(end - start)
    np.savetxt('./times.txt', np.array(times))

if __name__ == '__main__':
    run()
</code></pre>
<p>The error is <code>TypeError: train() takes from 4 to 5 positional arguments but 6 were given</code>. Actually, I provided 5 args, how it could be 6?</p>