akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Computing multiple losses error

username: nn_beginner
cooked: <p>Hi,<br>
I am designing a NN controller which is trained using a state space model (SSM) representation NN already built. The controller makes predictions for the next N outputs, where N = 2. The SSM using one of the inputs at a time to compute the next state, and so on for N inputs. After each output I change the input by one time step to accommodate for this.</p>
<p>This is where the issue arises, making updates to the tensor means the autograd can no longer compute the gradient.</p>
<p>Ideally, I would like a tensor of the N outputs caused by the N inputs, to compare to a desired tensor from where the loss can be computed, is there a way to do this?</p>
<p>Below, I have my code:</p>
<p>For context, X is the N previous input and state tensor</p>
<code>
<pre><code>for dd in range(0, len(x_train) - batch, batch):
    X = x_train[dd:dd + batch]
    optimizer.zero_grad()
    output = dpc(X)
    optimizer.zero_grad()

    history = torch.zeros((batch, 21))
    #fill this array with the current states
    for r in range(batch):
        for ii in range(20):
            history[r][ii] = X[r][ii]
        history[r][20] = output[r][0]
    y = ss_model(history[0:4][0:21])
    y = (y * (max_a[1][0] - min_a[1][0])) + min_a[1][0]
    desired = torch.zeros((batch, 1))
    loss = (F.mse_loss(y, desired))
    print(y)

    for gg in range(batch):
        for ll in range(19):
            history[gg][ll] = history[gg][ll+2]
        history[gg][19] = history[gg][20]
        history[gg][20] = y[gg][0]

    y2 = ss_model(history[0:4][0:21])
    y2 = (y2 * (max_a[1][0] - min_a[1][0])) + min_a[1][0]
    print(y2)
    loss2 = (F.mse_loss(y2, desired))
    loss2.backward()
    optimizer.step()
</code></pre>
</code>
<p>Many thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>It would help if you could provide a code sample that we could run here. With dummy Modules/Optimizers if you don’t want to share the ones you use.</p>
<p>Note that you can simplify your code as follows already:</p>
<pre><code class="lang-auto">for dd in range(0, len(x_train) - batch, batch):
    X = x_train[dd:dd + batch]
    optimizer.zero_grad()
    output = dpc(X)
    optimizer.zero_grad()

    history = torch.zeros((batch, 21))
    #fill this array with the current states
    history.narrow(1, 0, 20).copy_(X)
    history.narrow(1, -1, 1).copy_(output)

    y = ss_model(history[0:4])
    y = (y * (max_a[1][0] - min_a[1][0])) + min_a[1][0]
    desired = torch.zeros((batch, 1))
    loss = (F.mse_loss(y, desired))
    print(y)

    history.narrow(1, 0, 19).copy_(history.narrow(1, 2, 19))
    history.select(1, 19).copy_(history.select(1, 20))
    history.narrow(1, 20, 1).copy_(y)

    y2 = ss_model(history[0:4])
    y2 = (y2 * (max_a[1][0] - min_a[1][0])) + min_a[1][0]
    print(y2)
    loss2 = (F.mse_loss(y2, desired))
    loss2.backward()
    optimizer.step()
</code></pre>
<p>That should at least be faster. But if you want to keep around the history, you can just <code>.clone()</code> that Tensor whenever you need that value saved.</p>
PyG batch generation on the fly

username: AleTL
cooked: <p>Hello!</p>
<p>I have tabular data and what I want is to take these data, transform them into several graphs, and input these graphs to some GNN. Pytorch Geometric allow to generate my own dataset both in memory and storing it. But what I want is to generate the graphs on the fly. I don’t want to process the whole tabular data, generate the graphs and then feed the GNN; what I want is to generate the graphs on the fly and directly feed them to the GNN. Everytime I need a new batch of graphs, I take the tabular data, generate the graphs, feed them and then remove them and start again with new data.</p>
<p>Does anyone know if this is possible or how should I do it? I’ve been trying to look for information or code snips but I’ve found nothing.</p>
<p>Thank you!</p>

username: ejguan
cooked: <p>If you choose to use <code>Dataset</code> in PyTorch, you can implement all data retrieving logic inside <code>__getitem__</code> function to do data generation on the fly.</p>
<p>Or, you can choose to use  torchdata <a href="https://github.com/pytorch/data" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/data: A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.</a>, which utilize iterator-style datapipe to help you construct data pipeline on the fly.</p>
How can I call the backward function of operations in torch.nn.functional

username: 12wang3
cooked: <p>We know that we can implement our own custom autograd functions by subclassing torch.autograd.Function and implementing the forward and backward passes which operate on Tensors. As shown in <a href="https://pytorch.org/docs/stable/autograd.html#function" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html#function</a></p>
<pre><code class="lang-auto">class Exp(Function):

    @staticmethod
    def forward(ctx, i):
        result = i.exp()
        ctx.save_for_backward(result)
        return result

    @staticmethod
    def backward(ctx, grad_output):
        result, = ctx.saved_tensors
        return grad_output * result
</code></pre>
<p>The question is how can I call the backward function of operations in torch.nn.functional? Is this possible? And what is the simplest way?<br>
For example, I want to call the backward function of torch.nn.functional.softmax and get the Jacobian matrix. I have tried torch.autograd.functional.jacobian, but it is experimental now and gets a wrong result when I apply it to softmax and a 3D tensor. Is there an existing backward function that I can directly call?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The functional methods in torch.nn are not all elementary functions for the autograd. So there isn’t a single backward function to call for them.<br>
What you can do though is get the gradient with the regular autograd:</p>
<pre><code class="lang-auto"># inp that requires_grad=True and grad_output that match what you want to compute.
# If you want a full jacobian, you will need a for loop to reconstruct it line by line.
out = functional.softmax(inp)
grad = torch.autograd.grad(out, inp, grad_output)
</code></pre>
<p>What is the issue you have with functional.jacobian? What do you mean by wrong result?</p>

username: 12wang3
cooked: <p>Hi albanD,<br>
Thank you for your answer and torch.autograd.grad indeed works for me!<br>
I checked the docs of functional.jacobian again, and I found that I misunderstood the use of it. After I change the code, the result of functional.jacobian is all good now.</p>

username: Liang
cooked: <p>Hi, how to calculate the backward gradient of the softmax operation by pytorch operations? (not autograd)? Or is there a quick operator for it?</p>

username: albanD
cooked: <p>There is no operation that you can call directly from python no.</p>
Dynamically add dimensions for broadcasting of a vector to a tensor

username: FeryET
cooked: <p>Imagine I have this vector:</p>
<pre><code class="lang-auto">import torch
x = torch.rand(10)
</code></pre>
<p>and I want to multiply it by this tensor:</p>
<pre><code class="lang-auto">y = torch.rand(10, 20, 30, 3)
</code></pre>
<p>I can’t do <code>x * y</code> directly. I can’t do <code>x.expand_as(y) * y</code> either, expand can apparently only stack dims on top of the current dim, and there will be a mismatch.</p>
<p>I can do <code>x.expand_as(y.T).T* y</code> but it isn’t that elegant. Any other solution? Can I somehow make x from 1 dimension to 4 dimension (and more if y was more) without some very unreadable tricks?</p>
<p>Thanks!</p>

username: KFrank
cooked: <p>Hi Fery!</p>
<aside class="quote no-group" data-username="FeryET" data-post="1" data-topic="147067" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/feryet/48/39151_2.png" class="avatar"> FeryET:</div>
<blockquote>
<p>I can do <code>x.expand_as(y.T).T* y</code> but it isn’t that elegant. Any other solution?</p>
</blockquote>
</aside>
<p>I don’t know if this would be any more elegant, but you can let<br>
broadcasting do the equivalent of the <code>expand()</code> for you if you use<br>
<code>.T</code> (transpose) to make the dimensions over which to broadcast be<br>
the leading dimensions:</p>
<pre data-code-wrap="python"><code class="lang-python">(x * y.T).T
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: FeryET
cooked: <p>Another problem with using <code>.T</code> is that it’s deprecated for tensors, so there must be another way to do this, with at least the same readability?</p>

username: KFrank
cooked: <p>Hi Fery!</p>
<aside class="quote no-group" data-username="FeryET" data-post="4" data-topic="147067" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/feryet/48/39151_2.png" class="avatar"> FeryET:</div>
<blockquote>
<p>Another problem with using <code>.T</code> is that it’s deprecated for tensors, so there must be another way to do this, with at least the same readability?</p>
</blockquote>
</aside>
<p>At the cost of typing a few more characters you could use:</p>
<pre data-code-wrap="python"><code class="lang-python">(x * y.transpose (0, -1)).transpose (0, -1)
</code></pre>
<p>Note that <code>.T</code> and <code>.transpose (0, -1)</code> don’t do exactly the same thing,<br>
but they do both move the common dimension (in your case, <code>10</code>) to<br>
be the trailing dimension, which is all you need to get broadcasting to<br>
work for this use case.</p>
<p>Best.</p>
<p>K. Frank</p>
Assigning values to tensor sliced by indices and mask

username: h4ns
cooked: <p>Hello everyone,</p>
<p>I am searching for way to make this assignment to a slice of a tensor work, without copying data (and also understand why my version doesn’t work):</p>
<pre><code class="lang-auto">import torch

indices = torch.tensor([2,0])
lengths_new = torch.tensor([1,2])
lengths_old = torch.tensor([2,2,3])
tensor_new = torch.tensor([[3,1,2,4], [2,1,3,4]])
tensor_old = torch.tensor([[1,2,3,4], [1,2,3,4], [2,1,3,4]])

mask = lengths_new &lt; lengths_old[indices]
print("old value", tensor_old[indices][mask])
print("new value", tensor_new[mask])
tensor_old[indices][mask] = tensor_new[mask]
print("result value", tensor_old[indices][mask])
print("old value", lengths_old[indices][mask])
print("new value", lengths_new[mask])
lengths_old[indices][mask] = lengths_new[mask]
print("result value", lengths_old[indices][mask])
</code></pre>
<p>The output is this:</p>
<pre><code class="lang-auto">old value tensor([[2, 1, 3, 4]])
new value tensor([[3, 1, 2, 4]])
result value tensor([[2, 1, 3, 4]])
old value tensor([3])
new value tensor([1])
result value tensor([3])
</code></pre>
<p>My desired output would be that the result value is the same as the new value. Any suggestions how to achieve this are very welcome.</p>
<p>Best regards<br>
H4ns</p>

username: ptrblck
cooked: <p>If I understand your code correctly you are currently working on a copy of the data since you are indexing the tensor sequentially.<br>
This should work and yields the same results for <code>new</code> and <code>result</code>:</p>
<pre><code class="lang-python">indices = torch.tensor([2,0])
lengths_new = torch.tensor([1,2])
lengths_old = torch.tensor([2,2,3])
tensor_new = torch.tensor([[3,1,2,4], [2,1,3,4]])
tensor_old = torch.tensor([[1,2,3,4], [1,2,3,4], [2,1,3,4]])

mask = lengths_new &lt; lengths_old[indices]
print("old value", tensor_old[indices][mask])
print("new value", tensor_new[mask])
tensor_old[indices[mask]] = tensor_new[mask]
print("result value", tensor_old[indices][mask])
print("old value", lengths_old[indices][mask])
print("new value", lengths_new[mask])
lengths_old[indices[mask]] = lengths_new[mask]
print("result value", lengths_old[indices][mask])
</code></pre>

username: h4ns
cooked: <p>Ah very nice thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
RTX 3050 woth cuda 11.3 install solution and test ok

username: Jesse_Stone
cooked: <p>ASUS Dual GeForce RTX 3050 OC<br>
NVIDIA-Linux-x86_64-510.47.03.run<br>
cuda_11.3.1_465.19.01_linux.run<br>
libcudnn8_8.2.1.32-1+cuda11.3_amd64.deb<br>
libcudnn8-dev_8.2.1.32-1+cuda11.3_amd64.deb<br>
libcudnn8-samples_8.2.1.32-1+cuda11.3_amd64.deb<br>
pip3 install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f <a href="https://download.pytorch.org/whl/cu113/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu113/torch_stable.html</a><br>
install and test AI in cuda OK</p>

username: ptrblck
cooked: <p>I’m not sure what the actual question is, but if you want to confirm that the mentioned pip wheels are working for your 3050 then yes, they should work as they are shipping with the right compute capability (8.6).</p>

username: Jesse_Stone
cooked: <p>just a sharing method~~~thank you</p>
Pytorch model to tensorflow conversation

username: Sonali
cooked: <p>Hi,<br>
I want to convert one pytorch model into tensorflow model. But i don’t have the structure of model. Can it possible to convert without knowing the architecture of model?</p>
<p>I have only pytorch model and i loaded it using jit but i am not able to do convert in tensorflow using onnx library.</p>
<p>Can anyone suggest the solution?</p>
Torchscript model need more GPu memory than python(checkpoint)

username: wangyunxiaa
cooked: <p>I have exported my model to torchscript formate by torch.jit.trace<br>
BUT, when i do inference use the script model, I need 3 times GPU memory than checkpoint,<br>
However when i add  <code>with torch.no_grad()</code> at the begining of inference I cat just do once, the second time do inference I always get the error:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbd6c951a81f4df40d1e0e69da45f927155d86f2.png" alt="image" data-base62-sha1="zVS1pKm4aip4Ymmytm6atDRva8y" width="620" height="94"></p>
<p>I have add model.eval and  with torch.no_grad() when i traced my model. But, how to deal with the memory usage problem?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="wangyunxiaa" data-post="1" data-topic="147024">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wangyunxiaa/48/47773_2.png" class="avatar"> wangyunxiaa:</div>
<blockquote>
<p>when i do inference use the script model, I need 3 times GPU memory than checkpoint,</p>
</blockquote>
</aside>
<p>The <code>state_dict</code> only stores the registered buffers and parameters of the model, which might be smaller than the forward activations needed to calculate the gradients created during the forward pass.</p>
<aside class="quote no-group" data-username="wangyunxiaa" data-post="1" data-topic="147024">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wangyunxiaa/48/47773_2.png" class="avatar"> wangyunxiaa:</div>
<blockquote>
<p>However when i add <code>with torch.no_grad()</code> at the begining of inference I cat just do once</p>
</blockquote>
</aside>
<p><code>no_grad()</code> disables storing the forward activations as no gradients are calculated during inference which would explain why this approach seems to work.</p>
<aside class="quote no-group" data-username="wangyunxiaa" data-post="1" data-topic="147024">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wangyunxiaa/48/47773_2.png" class="avatar"> wangyunxiaa:</div>
<blockquote>
<p>the second time do inference I always get the error:</p>
</blockquote>
</aside>
<p>I don’t see the error message so don’t know what’s happening.<br>
Could you post the error message by wrapping it into three backticks ```, please?</p>
Understanding tied weights

username: Jay_Timbadia
cooked: <p>Hey,</p>
<p>I was coding some models using pytorch and came across this</p>
<pre><code class="lang-auto">model.set_tied()
</code></pre>
<p>I did searched for the documentation on pytorch, but didn’t found anything.<br>
Can you please let me know what does this line of code does.<br>
If there is any brief resource, can you please point me out?</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/graykode/gpt-2-Pytorch/blob/master/GPT2/utils.py">
  <header class="source">

      <a href="https://github.com/graykode/gpt-2-Pytorch/blob/master/GPT2/utils.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/graykode/gpt-2-Pytorch/blob/master/GPT2/utils.py" target="_blank" rel="noopener nofollow ugc">graykode/gpt-2-Pytorch/blob/master/GPT2/utils.py</a></h4>


      <pre><code class="lang-py">'''
    code by TaeHwan Jung(@graykode)
    Original Paper and repository here : https://github.com/openai/gpt-2
    GPT2 Pytorch Model : https://github.com/huggingface/pytorch-pretrained-BERT
'''
import logging

logger = logging.getLogger(__name__)

def load_weight(model, state_dict):
    old_keys = []
    new_keys = []
    for key in state_dict.keys():
        new_key = None
        if key.endswith(".g"):
            new_key = key[:-2] + ".weight"
        elif key.endswith(".b"):
            new_key = key[:-2] + ".bias"
        elif key.endswith(".w"):
            new_key = key[:-2] + ".weight"
</code></pre>



  This file has been truncated. <a href="https://github.com/graykode/gpt-2-Pytorch/blob/master/GPT2/utils.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thank you!</p>

username: ptrblck
cooked: <p><code>set_tied</code> is not a built-in <code>nn.Module</code> method and defined in the repository <a href="https://github.com/graykode/gpt-2-Pytorch/blob/401078fde5e85475590dc04c9ed11b79332b8717/GPT2/model.py#L198-L201">here</a>.</p>
Extending PyTorch with Persistent Memory support

username: Ashikee_Ghosh
cooked: <p>I am looking for extending PyTorch with the feature of using persistent memory for allocating tensors. As of now, we can use CPU-Memory or Cuda-Memory for the allocations of tensors. What should be the starting path for using persistent memory?  When some new allocation of tensor will be made that allocation will be in Persistent memory, not in CPU-DRAM. What are files need to be modified? I went through the tutorials of extending PyTorch. However, they talked about things like custom c++ extensions, custom c++ classes, new backend dispatcher, etc.  I am not sure how should I proceed with it.</p>

username: ptrblck
cooked: <p>Could you explain what exactly you mean by “persistent memory”?</p>
Question about Resnet's Bottleneck incoming object

username: shenao1994
cooked: <p>I’m optimizing Resnet50, and besides input x, I want to input another objective at each layer.I’m optimizing Resnet50, and besides input x, I want to input another objective at each layer. That is, I want to pass parameters in Bottleneck’s forward function.Every layer of Resnet in many example codes is built by <code>make_layer</code>, which makes it impossible for me to pass other parameters.</p>
<pre><code class="lang-auto">class Bottleneck(nn.Module):
    expansion = 4

    def __init__(self, inplanes, planes, stride=1, downsample=None):
        super(Bottleneck, self).__init__()
        ...........

    def forward(self, x, mask):
        residual = x
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn3(out)
        if self.downsample is not None:
            residual = self.downsample(x)
        out += residual
        out = self.relu(out)

        return out
</code></pre>
<p>All in all, I just want to know how to pass in the <em><strong><code>mask</code></strong></em> in Bottleneck.</p>

username: ptrblck
cooked: <p>You could create a custom <code>nn.Module</code> which accepts the <code>mask</code> object and apply it in the <code>forward</code>.<br>
If you keep the module names equal and only modify the <code>forward</code> method loading the pre-trained <code>state_dict</code> should still work.</p>
Train same network for two data-set

username: Gaurav_Prasad
cooked: <p>Is it possible to train same network architecture using two different dataset. For the first dataset the forward function is different in comparison to the second dataset training.</p>

username: ptrblck
cooked: <p>Yes, you could theoretically add a condition into the <code>forward</code> method and pass e.g. a flag to it to switch between different execution paths.<br>
However, while this would be possible during training you would have to think about the validation/testing use case and if you would be able to pass this flag to the <code>forward</code> method as well.<br>
If not, you might be able to use some data conditions to switch between these execution paths.</p>
Kfold method validation

username: jasg
cooked: <p>good day how to apply kfold to a convolutional model in pytorch to validate it for example convolutions=6, kernel=5, epochs=100, image size=240, thank you for your kind answer.</p>

username: ptrblck
cooked: <p><a href="https://github.com/skorch-dev/skorch">Skorch</a> provides a “scikit-learn compatible neural network library that wraps PyTorch” and can thus be used with the random or grid search k-fold CV in sklearn.</p>
Importing torch in Python3.7. OSError

username: Sreekanth
cooked: <p>Hi,</p>
<p>We have built the torch and torchvision from scratch using CUDA10.2, and have GT730 GPU. When we are installing the torch and torchvision wheels in new system we are getting “OSERROR: libmkl_intel_lp64.so: cannot open shared object file: no such file or directory” error.</p>
<p>We used Ubuntu 18.04 for building torch and torchvision and our new system is also having ubuntu 18.04.</p>
<p>We tried to install mkl and also updated few packages/libraries but the error is still there. Can anyone please let us know how to resolve this error.</p>
<p>Thanks,<br>
Sreekanth</p>

username: ptrblck
cooked: <p>Usually <code>pip install mkl</code> (and adding the lib path to your <code>LD_LIBRARY_PATH</code> if needed) should work.<br>
If not, check which library is missing <code>libmkl</code> via <code>ldd</code> and check if <code>libmkl</code>.</p>

username: Sreekanth
cooked: <p>Hi,</p>
<p>We installed mkl but still we are getting the same error. When we installed the Anaconda and installed the CUDNN the issue got resolved.</p>
<p>Not sure why we need to install the Anaconda and CUDNN.</p>
<p>Thanks,<br>
Sreekanth</p>
About memory overloading during test time

username: Sadegh_ab
cooked: <p>I want to spot a better deep neural network architecture for a custom Siamese object tracker which is using a correlation filter. So I put the training and testing function in a loop (new architectures are designed, trained, and tested by each iteration).<br>
After some iteration, the RAM and SWAP memory begins to fill during testing, and then the process is killed by Linux. How can I solve this problem?</p>

username: ptrblck
cooked: <p>Based on the description it seems you are storing data on the host in each iteration.<br>
This could be the case e.g. if you are training on the CPU and are storing tensors with their computation graph (e.g. via <code>loss_total += loss</code>). In such a case, <code>detach()</code> the losses before accumulating them or call <code>item()</code> on them.</p>

username: Sadegh_ab
cooked: <p>thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. both procedures training and testing run on the GPU and I am not storing any unusable data during testing and training also the garbage collector is called manually during the loop.<br>
I wonder why my system fills the memory(RAM and SWAP) capacity.</p>

username: ptrblck
cooked: <p>That’s hard to tell without seeing the code. Could you try to create a minimal, executable code snippet which would reproduce the increase in host memory usage?</p>
How to calculate PSNR after normalization

username: dachun
cooked: <p>We know that PSNR as below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08984ea8b841c6b0ab0df238297d76a2d614b490.png" alt="image" data-base62-sha1="1e28sXPxgDJh876IwIoSwcREhvW" width="340" height="82"></p>
<p>where<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2e0a2651b0745c5c111c44a02aa415b7446786e.png" alt="image" data-base62-sha1="yEAP8htVgb8ip0dgIiMCW2PG2dw" width="467" height="103"> and<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec5a371a4ef5b2187e68a809ce48c442522b596b.png" alt="image" data-base62-sha1="xIRY28nbmM7jDtORCsnxwgP1I8z" width="197" height="46"></p>
<p>。<br>
So if I normalize the image before calculating PSNR, I won’t know<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec5a371a4ef5b2187e68a809ce48c442522b596b.png" alt="image" data-base62-sha1="xIRY28nbmM7jDtORCsnxwgP1I8z" width="197" height="46"><br>
. In this case, how to calculate PSNR?</p>
Creating own dataset with Pytorch Geometric Temporal for Graph Neural Network

username: Ditsuhi
cooked: <p>Hello everyone,</p>
<p>I recently started using graph neural network with PyTorch. I am trying to create my dataset  based on the following link: <a href="https://bit.ly/36DsYSb" class="inline-onebox" rel="noopener nofollow ugc">torch_geometric_temporal.signal.static_graph_temporal_signal — PyTorch Geometric Temporal documentation</a>,  however I am getting error. I also tried to create  METRLADatasetLoader dataset separately without importing from torch_geometric_temporal.dataset (by simply copying and pasting two classes StaticGraphTemporalSignal and METRLADatasetLoader (<a href="https://bit.ly/37CO5EL" class="inline-onebox" rel="noopener nofollow ugc">torch_geometric_temporal.dataset.metr_la — PyTorch Geometric Temporal documentation</a>) and got the following error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c58667245aad5e8f6632559be43a1ce7ded7da6.png" data-download-href="https://discuss.pytorch.org/uploads/default/8c58667245aad5e8f6632559be43a1ce7ded7da6" title="Screenshot 2022-03-22 at 02.30.08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c58667245aad5e8f6632559be43a1ce7ded7da6_2_690x307.png" alt="Screenshot 2022-03-22 at 02.30.08" data-base62-sha1="k1yf1HlqEByQ9h5UbAzs2GESYDQ" width="690" height="307" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c58667245aad5e8f6632559be43a1ce7ded7da6_2_690x307.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c58667245aad5e8f6632559be43a1ce7ded7da6_2_1035x460.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c58667245aad5e8f6632559be43a1ce7ded7da6_2_1380x614.png 2x" data-dominant-color="F1F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-03-22 at 02.30.08</span><span class="informations">1632×728 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could you please tell me why I am getting the above error if this code works fine?</p>
<p>from torch_geometric_temporal.dataset import METRLADatasetLoader<br>
loader = METRLADatasetLoader()<br>
dataset = loader.get_dataset(num_timesteps_in=12, num_timesteps_out=12)<br>
print("Dataset type: ", dataset)<br>
print("Number of samples / sequences: ", len(set(dataset)))</p>
<p>Thanks.</p>
Project Development Cycle

username: andreasoie
cooked: <p>Greetings PyTorchians,</p>
<p>I was curious about how people structure their development cycle for Deep Learning project from start to finish - so thought I’d ask you guys!</p>
<p>Personally, I tend to lean towards something in the lines of:</p>
<ul>
<li>Dataset exploration and visualization</li>
<li>Model development</li>
<li>Model evaluation</li>
</ul>
<p>Furthermore, as a project grows, I’m curious to ask how you guys set up your folder structure. Maintaining a logical folder structure to keep things ordered and orderly, increasingly becomes more important as your writing new code, creating new training configs etc …</p>
Pre_transform adds feature dimension pytorch geometric

username: FloorEijkelboom
cooked: <p>Hi! When using a pre_transform on my dataset, the resulting graphs have different feature dimensions than when the transform is applied to some graph (i.e. outside the pre_transform of a dataset). The same happens when considering batches, the pre_transformed batches have an extra feature dimension, where the transform applied by the dataloader real-time does not have this extra dimension.</p>
<p>Example:</p>
<pre><code class="lang-auto">def some_transform(graph: Data) -&gt; Data:
    feature_dimension = graph.x.shape[1]
    feature = torch.ones((1, feature_dimension))

    graph.extra_feature = feature

    return graph

dataset = TUDataset('./datasets/TUDataset/PROTEINS', name='PROTEINS', pre_transform=some_transform)
dataset = dataset.shuffle()
loader = DataLoader(dataset, batch_size=batch_size, shuffle=True)

for batch in train_loader:
      print(batch.x.shape)
      print(batch.extra_feature.shape)
</code></pre>
<p>yields</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.Size([4313, 3])
&gt;&gt;&gt; torch.Size([128, 4])
</code></pre>
<p>Why could this be the case? Thanks in advance!</p>

username: nivek
cooked: <p>It is hard to tell without seeing what each sample and <code>TUDataset</code> look like.</p>
<p>It may be because <code>DataLoader</code> <a href="https://pytorch.org/docs/stable/data.html#working-with-collate-fn" rel="noopener nofollow ugc">performs collation</a> on each batch of data. Specifically, you may want to check if the <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/_utils/collate.py#L84" rel="noopener nofollow ugc">default collation</a> is applied.</p>
Pytorch Geometric - accessing global node embeddings

username: Victor_Suciu
cooked: <p>I’m trying to make a GNN where, after a few convolutions, it computes the dot product between one of the node embeddings and all the rest. This is to do link prediction on nodes that might not exist in the current graph. However, to my understanding Pytorch graphs only contain the node representations for their own nodes and not other nodes that exist somewhere else in the dataset.</p>
<p>How can you access node embeddings for nodes that don’t exist in the current graph?</p>
<pre><code class="lang-auto"># example
def forward(x, edge_index):
    x = conv1(x, edge_index)
    x = conv2(x, edge_index)

    # dot product vector x[0] with all other vectors x[1:]
    return x[0] @ x[1:]
    
# x[1:] only has the nodes contained in this particular input graph, but I want all the nodes in the entire dataset
</code></pre>
<p>Edit: Some clarification.</p>
<p>The initial node values are ints, and are stored in whatever <code>data.x</code> you pass into the <code>forward()</code> function. I’m assuming there is some sort of global <code>torch.nn.Embedding</code> layer that converts the node integers into the initial embeddings that <code>self.conv1</code> acts upon. I am looking for these initial embeddings for all possible nodes.</p>

username: Victor_Suciu
cooked: <p>Never mind, I found the answer. GCN layers have a simple Linear layer to project the input node representations into the right dimension. There dooes not seem to be an actual Embedding layer unless you add a custom one yourself. Here’s a snippet from GCNConv.</p>
<pre><code class="lang-auto">def __init__(self, in_channels: int, out_channels: int,
                 improved: bool = False, cached: bool = False,
                 add_self_loops: bool = True, normalize: bool = True,
                 bias: bool = True, **kwargs):

        kwargs.setdefault('aggr', 'add')
        super().__init__(**kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels

        # ... irrelevant stuff

        self.lin = Linear(in_channels, out_channels, bias=False,
                          weight_initializer='glorot')
        # ... irrelevant stuff


def forward(self, x: Tensor, edge_index: Adj,
                edge_weight: OptTensor = None) -&gt; Tensor:
        """"""
        if self.normalize:
                # ... irrelevant stuff

        x = self.lin(x) # &lt;--- project input

        # ... irrelevant stuff
</code></pre>
MSE (L2) loss on two masked images

username: numpee
cooked: <p>I’m working on an Inpainting project.<br>
I’m trying to calculate the L2 reconstruction loss of only the regions that have been masked.<br>
I can mask the output with</p>
<pre><code class="lang-auto">criterion = nn.MSELoss()
masked_pred = torch.mul(prediction, mask) 
loss = criterion(masked_input, masked_pred) 
</code></pre>
<p>But the problem is that MSELoss averages the loss over all pixels, and since most of my pixels are masked the loss becomes very small.<br>
Is there a way to calculate the MSELoss only on non-masked pixels?</p>
<p>Just as a side note, the mask that I use is a randomly generated mask (not fixed)<br>
Thanks in advance</p>

username: numpee
cooked: <p>In case the question wasn’t clear:<br>
I’m looking for a feature similar to ignore_index in nn.CrossEntropyLoss() so that I can ignore the loss on certain pixel values.</p>

username: tumble-weed
cooked: <p>you can multiply the <code>loss</code> by <code>masked_input.numel()/mask.sum()</code> to correct for the number of ON pixels.</p>

username: numpee
cooked: <p>Thanks for the reply.<br>
I was actually gonna reply that I solved the issue.<br>
Basically the same solution, but I set the MSELoss reduction parameter to ‘sum’<br>
and then divided by the total number of nonzero pixels in the mask.<br>
your method works too</p>

username: Daniel_Ortega
cooked: <p>Hi, this method work with a batch on gpu? Or have to do something else? I have diferent result with batch size 1 and greater batch size</p>

username: yuzhi_guo
cooked: <p>did you try to make the reduce=False in <code>torch.nn.</code> <code>MSELoss</code> ( <em>size_average=None</em> ,  <em>reduce=None</em> ,  <em>reduction=‘mean’</em> )? It will be return a vector. You can make a mask by yourself. For example, your output is <code>[1,2,3,4]</code>, you don’t want to calculate the loss for <code>1</code> and <code>4</code>, you can just set <code>mask = [0,1,1,0]</code> and your <code>loss_matrix = output * mask</code> and then <code>loss = avg(loss_matrix)</code></p>
<p>here is the Doc:<br>
<strong>reduce</strong>  (<a href="https://docs.python.org/3/library/functions.html#bool" rel="nofollow noopener"> <em>bool</em> </a> <em>,</em>  <em>optional</em> ) – Deprecated (see  <code>reduction</code> ). By default, the losses are averaged or summed over observations for each minibatch depending on  <code>size_average</code> . When  <code>reduce</code>  is  <code>False</code> , returns a loss per batch element instead and ignores  <code>size_average</code> . Default:  <code>True</code></p>

username: jmontoyaz
cooked: <p>could you pls post your solution?</p>

username: Mohammad_Hassan_Soha
cooked: <p>Hey numpee</p>
<p>I suggest that instead of multiplying mask tensor with the prediction, either use fancy indexing or <code>torch.gather</code> to get rid of unwanted regions. Yout suggested solution is totally waste of computation!</p>
Training DeepLabV3+ on Pascal Voc 2012 dataset with pytorch

username: avatar
cooked: <p>I’m trying to train the DeepLabV3+ architecture with ResNet101 as the backbone on Pascal Voc 2012 semantic segmentation dataset. I’m using the pretrained weights on imagenet and i freeze the weights of the backbone in training. I also perform some transformations on the training data such as random flip and random rotate. The problem is that the model mIoU metric is very low in both training and validation set,approximately 40% and does not improve more. It seems like underfitting. Besides this, when i’am not using data augmention the model overfits in a large scale. Does anyone has any suggestions about how I can reproduce good results on Pascal Voc dataset with the deeplabv3+ architecture? It will help so much. Here to note that I’m referring to the voc dataset with 1464 train and 1449 validation images, not the augmented dataset.</p>
<p>Thank you a lot.</p>
Avoid negative output from yolo model

username: hyman9090
cooked: <p>I am currently building a YOLO version 1, following the research paper. According to their model, the fully connected layer of YOLO version 1 is as follow:</p>
<pre><code class="lang-auto">return nn.Sequential(
nn.Flatten(),
nn.Linear(1024*S*S, 4096),
nn.Dropout(),
nn.LeakyReLU(0.1),
nn.Linear(4096, S*S*(C+B*5) )
)
</code></pre>
<p>In the end, the output would be a size (7x7x30) tensor, where each 30-parameter long array presents the information of an image. Thus, all the parameters are either 0 or +ve values.</p>
<p>However, the nn.linear() would yield negative values, which result in an error. Because the loss function contains square root calculation, a negative value would result in nan.</p>
<p>I would like to ask by what means can avoid receiving errors; at the same time, the model can still stick on the research paper?</p>

username: Akbar_Shah
cooked: <p>I have seen people using a sigmoid activation function after the final layer to avoid negative predictions.</p>

username: hyman9090
cooked: <p><a class="mention" href="/u/akbar_shah">@Akbar_Shah</a> Thanks for your reply. Do you mean to append sigmoid activation function after nn.Linear()?</p>
<p>As the research paper didn’t mention the sigmoid activation function, so I didn’t apply it to the network. Because I don’t know if doing so is legit?</p>

username: Akbar_Shah
cooked: <p>Yes, that’s what I meant. Also, if you are regressing normalized center-form coordinates, maybe you could clamp them to be within [0, 1] to avoid negatives? Since the width/height of the box can’t be negative.</p>

username: hyman9090
cooked: <p>Though I still have some problems with my model… as it somehow outputs ‘nan’ after a few iterations. I have no idea why does it happen.<br>
But the issue of negativity has been solved. Thank you very much.</p>
MSELoss when mask is used

username: sajastu
cooked: <p>I’m trying to calculate MSELoss when mask is used. Suppose that I have tensor with batch_size of 2: <code>[2, 33, 1]</code> as my target, and another input target with the same shape. Since sequence length might differ for each example, I have also a binary mask indicating the existence of each element in the input sequence. So here is what I’m doing:</p>
<pre><code class="lang-auto">mse_loss = nn.MSELoss(reduction='none')

loss = mse_loss(input, target)
loss = (loss * mask.float()).sum() # gives \sigma_euclidean over unmasked elements

mse_loss_val = loss / loss.numel()

# now doing backpropagation
mse_loss_val.backward()
</code></pre>
<p>Is <code>loss / loss.numel()</code> a good practice? I’m skeptical, as I have to use <code>reduction='none'</code> and when calculating final loss value, I think I should calculate the loss only considering those loss elements that are nonzero (i.e., unmasked), however, I’m taking the average over all tensor elements with <code>torch.numel()</code>. I’m actually trying to take <code>1/n</code> factor of MSELoss into account. Any thoughts?</p>

username: Mitchell_DeHaven
cooked: <p>Maybe a bit late, but I don’t think you want numel if you want the mean with respect to non-padded elements. Why not try mask.sum() instead of loss.numel()?</p>

username: Mohammad_Hassan_Soha
cooked: <p>Hey sajad</p>
<p>I genuinely believe you’re using mask in a wrong way. In the snippet you sent, you were multiplying the mask with the loss, that will impose a float multiplication. The most efficient way of using mask is obtained by fancy indexing. Also take a look at <code>torch.gather</code> function.</p>
Is there anyway to get the first element of a tensor as a scalar

username: Mika_S
cooked: <p>When I declare:<br>
x = Variable(torch.ones(2), requires_grad=False)</p>
<p>And then do : x[0]<br>
I still get a tensor of size 1. Indexing further x[0][0] further leads to the same tensor. Is there a way to get the scalar element that is x[0]</p>

username: trypag
cooked: <p>Hi <a class="mention" href="/u/mika_s">@Mika_S</a>,</p>
<p>You can use <code>x.data</code> to access the underlying storage tensor. As a result, what you are looking for is <code>x.data[0]</code>.</p>

username: Mika_S
cooked: <p>Awesome. THats what i needed,</p>

username: albanD
cooked: <p>Keep in mind that doing this will not work with the autograd and no gradient will be backpropagated through this number.</p>

username: Shubham_Samanta
cooked: <p>You can use the torch.Tensor.item method to get the desired result<br>
eg.<br>
x.item()<br>
Note: This will only work if the x is of type torch.Tensor</p>
Overfitting to first batch

username: Mohammad_Farazi
cooked: <p>Hi,</p>
<p>I am training my model on a 3D dataset consisting of 100 data. The batch size I use is 1 (I cannot use a larger batch size). Although I use shuffle=True in dataloader, when I test my model, my model overfits to the first batch (i.e. data sample). So the test accuracy of the first batch in the test set is considerably higher than the rest. It seems my training data is overfitting to the first data sample. Any help or thoughts is appreciated.</p>

username: nivek
cooked: <p>It is difficult to debug without see the data, but here are some things you can check:</p>
<ol>
<li>Are you initializing with random weights?</li>
<li>Have you tried lowering your learning rate?</li>
</ol>

username: John_Palmer
cooked: <p>Hard to tell without data an details of what you are doing. However, this doesn’t seem like overfitting to me, it sounds more like your model is not converging. This could be due to poor hyper-parameter choice such as the learning rate, or some other error.</p>

username: Mohammad_Farazi
cooked: <p>Yes, random weights and uniform. I did try lowering the learning rate but it did not change anything. I just tried changing weight decay to different values.</p>

username: Mohammad_Farazi
cooked: <p>I have a shape correspondence task with the FAUST dataset (a total of 100 shapes). batch size is one (almost all models use this batch size and I cannot increase it). I initially use a U-net and two graph CNN layer in tandem (U-net is for learning point-wise features). I think it does converge as loss decrease and I stop at some point. here is the accuracy of each batch size in test set:</p>
<p>accuracy: 0.7264<br>
accuracy: 0.6181<br>
accuracy: 0.5443<br>
accuracy: 0.5376<br>
accuracy: 0.5251<br>
accuracy: 0.5210<br>
accuracy: 0.5014<br>
accuracy: 0.4616<br>
accuracy: 0.4720<br>
accuracy: 0.4741<br>
accuracy: 0.4778<br>
accuracy: 0.4814<br>
accuracy: 0.4782<br>
accuracy: 0.4676<br>
accuracy: 0.4705<br>
accuracy: 0.4721<br>
accuracy: 0.4744<br>
accuracy: 0.4756<br>
accuracy: 0.4768<br>
accuracy: 0.4784</p>
How to associate onnx with pytorch code

username: ynjiun_wang
cooked: <p>I tried to port a pytorch trained model to a RockChip 3566 using <a href="https://github.com/rockchip-linux/rknn-toolkit2/issues/10" rel="noopener nofollow ugc">rknn toolkit</a>.<br>
I am able to torch.onnx._export my pytorch model to model.onnx successfully.<br>
When building the model.rknn for RK3566 inference using the rknn toolkit from model.onnx, I encounter an error:</p>
<blockquote>
<p>Fatal: Meet unsupported operator: Sqrt(name=“Sqrt_941”).</p>
</blockquote>
<p>I am curious where this “sqrt” operator came from in my original pytorch code/model. I did a “grep -r sqrt *.py” in my development folders and cannot find any sqrt function in my code. Is there a way/tool can help me to associate this “Sqrt_941” in model.onnx to my original pytorch code? and how?</p>
<p>Thanks a lot for your help.</p>
How do train a CNN/LSTM combination with long sequences?

username: Alexander_Soare
cooked: <p>Tough question to frame in one line but here is the context:</p>
<p>I have an LSTM on varying length image sequences. A CNN generates the inputs to the LSTM. I want to train the CNN on the fly, rather than use it just as an embeddings generator.</p>
<p>To do this I take my <code>(batch_size, seq_length, *image_dims)</code> and view it as one big batch of images like this <code>(batch_size*seq_length, *image_dims)</code>. Then after it comes out of the CNN I unpack it for feeding to the LSTM.</p>
<p>This works just fine apart from the fact that some of the sequences are too long to fit in memory for the CNN (because I’m also storing gradients for backprop). I’d love to get some feedback on how to solve it. Here’s what I’ve been thinking about:</p>
<ul>
<li>
<p>Break the “big batch” into sub batches. Do back-prop on the LSTM alone, then backprop on the sub-batches. Seems like a lot of work and overcomplicating things. Also, I’m not sure it would work as I still need to store gradients for everything.</p>
</li>
<li>
<p>Randomly select a maximum of 256 samples from the “big batch”, pick those out and run them through. All the rest runs through CNN with no grad. That way the CNN is not learning for every single image in the sequence at once, but at least I can get the inputs through to the LSTM without killing my memory.</p>
</li>
</ul>
<p>Any ideas or guidance would be appreciated!</p>

username: ekmungi
cooked: <p><a class="mention" href="/u/alexander_soare">@Alexander_Soare</a> , Did you figure out what strategy worked for you? Could you share some insights? Also what is your GPU memory? Thanks!</p>

username: Alexander_Soare
cooked: <p><a class="mention" href="/u/ekmungi">@ekmungi</a> I think I was working with an RTX 3090 with 24 GB of memory. I don’t think I ever solved this problem as I moved onto using a transformer instead.</p>
I install pytorch by .whl file, but torch.version.cuda does not match the whl cuda version. Can anyone help me?

username: dachun
cooked: <p>I have installed torch from whl, downloads from <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">pytorch.org</a></p>
<pre><code class="lang-bash">pip install torch-1.10.2+cu111-cp37-cp37m-linux_x86_64.whl
</code></pre>
<p>But when I check the torch cuda version by</p>
<pre><code class="lang-bash">torch.version.cuda
</code></pre>
<p>it returns <code>10.2</code>. But I install cu111 version. What’s wrong?</p>

username: ptrblck
cooked: <p>Most likely you have multiple PyTorch installations in your current environment. Uninstall them all and reinstall the right wheel or create a new virtual environment where you could install the desired wheel.</p>

username: dachun
cooked: <p>Thanks a lot. I really have another pytorch installation.</p>
Transformer Encoder for Seq2Vec Problems

username: blueeagle
cooked: <p>I want to learn a fixed size representation from a variable-length sequence of vectors.<br>
Util now, I used a bidirectional LSTM for this purpose, however as my sequences are rather long (up to 2000 vectors) I now want to try a Transformer Encoder instead.<br>
While using a LSTM, I used the final hidden state of the LSTM cell as representation for the whole sequence. Unfortunately this method is not suitable for the transformer encoder, as it outputs one vector for each element in my input sequence.<br>
In <a href="https://discuss.pytorch.org/t/transformers-to-encode-a-sequence-into-a-fixed-lenght-vector/102290">this post here</a>, it is described that BERT inserts a special [CLS] token into the sequence, uses the representation output for this token as the representation for the whole sequence.<br>
However, in the post they use the BERT tokenizer to get that token, which is not suitable for me as my raw sequence does not represent words (it is just a sequence of vectors).</p>
<p>Can anybody tell me how the special [CLS] should look like?</p>
<p>Thanks for your help!</p>