akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Validation Loss is not decreasing in my RNN model

username: WirryWoo
cooked: <p>I am currently working on training an RNN to map sequential data to real-valued positive values, but unfortunately my validation loss is not decreasing. I have tried increasing the number of layers in my RNN and adding DropOut to my model, but no success. Some possible reasons why I think this happens:</p>
<ul>
<li>
<p>I am incorrectly using <code>WeightedRandomSampler</code>. I have an imbalanced representation of observations where a set of variable sized sequences, <code>X</code>, with associated response <code>y</code> are more represented than others. <code>y</code> is a tensor containing positive integers, and some of these integers are more represented than others in the overall dataset. I think I’m conflating the regression task with classification strategies, which may motivate the model to not learn patterns in the dataset, but I’m not sure. Would the default sampler be the preferred strategy for this task?</p>
</li>
<li>
<p>The gradients are not properly being updated in the training process. Part of my strategy with this mapping from sequential data to a real-value response is to approach my modeling by using the RNN to transform sequential data into a sequence of nonnegative real values, and then summing these values to generate the final model prediction. However, when computing the loss and calling <code>loss.backward()</code>, I’m not sure if the gradients are properly being updated. It’s difficult for me to visualize how to compute the gradients under this approach, and unfortunately, I am unable to find working solutions online that successfully train the model to learn the mapping I am interested in developing. Some sample code:</p>
</li>
</ul>
<pre><code class="lang-auto">import torch
import torch.nn as nn

batch_size = 4
sequence_len = 10
num_features = 8
num_outputs = 1

model = nn.GRU(num_features, num_outputs, batch_first = True)
criterion = torch.nn.MSELoss()

x_test = torch.randn(batch_size, sequence_len, num_features)
y_test = torch.randn(batch_size, num_outputs)

model_output, hidden = model(x_test)
y_pred = torch.sum(model_output, dim=1)
loss = criterion(y_pred, y_test)

loss.backward()
</code></pre>
<p>In this case, is this the correct way to properly update the gradients in the model? If not, what is the recommendation here?</p>
<p>I can provide a Colab notebook in case if anyone wants to view my development.</p>
<p>Thanks!</p>
importError:Key already registered with the same priority

username: NPU-Franklin
cooked: <p>When I import torch, it shows:“<strong>Key already registered with the same priority: GroupSpatialSoftmax</strong>”, I don’t know what’s going on.</p>

username: ptrblck
cooked: <p>How did you install PyTorch? Did you install the binaries or build form source?<br>
Are you using the <code>GroupSpatialSoftmax</code> in a standalone module or just from <code>pytorch/modules/detectron</code>?</p>

username: srodriguez142857
cooked: <p>I had the same problem.</p>
<p>I installed PyTorch following this command:<br>
<code>pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html</code></p>
<p>However, I noticed that my Python environment had a previous installation of PyTorch+CUDA which was later removed (uninstalled).</p>
<p>What I found is that the uninstalled task was not able to remove all the content (library files such as *.DLLs, etc.) from <em>torch</em> folder in the <em>site-packages</em> directory.  I had to delete manually that folder.  Aftewards, I installed again PyTorch (with the correct version for my environment: no CUDA), and it started to work fine.</p>
<p>Conclusion: This problem is due to existent library files in the <em>torch</em> folder due to previous installations.  One needs to remove them manually before reinstall again the latest version of PyTorch.</p>
<p>Cheers!</p>

username: NPU-Franklin
cooked: <p>Thanks for your advice! It really help!</p>

username: NPU-Franklin
cooked: <p>Thank you for asking, but I can’t remember that. I reinstalled my environment after a disastrous breakdown and now It works well!</p>

username: R.A.I.D
cooked: <p>I got the same error using anaconda.<br>
The solution for me was to create a new virtual environment and run:<br>
<code>%conda install pytorch torchvision cudatoolkit=10.2 -c pytorch</code><br>
to set up pytorch again.</p>

username: seven-mile
cooked: <p>I had the same problem but for different reason. Tips from users who build from source:<br>
TVM that built with LibTorch loads libtorch_cpu.so / torch_cpu.dll etc. But you are probably loading them twice.<br>
In python, import order matters. Call <code>import torch</code> before <code>import tvm</code> to ensure single loading behavior.</p>
DirectML and CrossEntropyLoss

username: Victor_Roman
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c55e9c4baee2f1d6b395fb98285a9a45e9bfbb27.png" data-download-href="https://discuss.pytorch.org/uploads/default/c55e9c4baee2f1d6b395fb98285a9a45e9bfbb27" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c55e9c4baee2f1d6b395fb98285a9a45e9bfbb27.png" alt="image" data-base62-sha1="sa0Lm3Ph5pkhf2dMDz2oOMjHBht" width="690" height="460" data-dominant-color="222323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1000×667 33.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Hey there, I’ve been trying to make my own first model and I eventually realized I’d need to do a multioutput model.</p>
<p>I’ve done some tinkering and came to this train function but when trying to use directML I’ve ran into this error code:<br>
RuntimeError: tensor.device().type() == at::DeviceType::PrivateUse1 INTERNAL ASSERT FAILED at “/home/vsts/work/1/s/pytorch-directml-plugin/torch_directml/csrc/dml/DMLTensor.cpp”:30, please report a bug to PyTorch. unbox expects Dml at::Tensor as inputs</p>
<p>I was wondering if there’s other people in my situation and whether I’ve missed something very simple</p>
Error with multiple GPUs: runtime error: nccl error 2: unhandled system error

username: cslxiao
cooked: <p>When I run my code with multiple GPUs, it crashes <strong>occasionally</strong> with the following error:</p>
<pre><code class="lang-auto">File "main.py", line 132, in train
    model.train(train_loader, val_loader)
  File "/mnt/DATA/code/bitbucket/drn_seg/segment/seg_model.py", line 54, in train
    self.train_epoch(epoch, train_loader, val_loader)
  File "/mnt/DATA/code/bitbucket/drn_seg/segment/seg_model.py", line 93, in train_epoch
    output = self.model_seg(image)[0]
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 113, in forward
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 118, in replicate
    return replicate(module, device_ids)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/replicate.py", line 12, in replicate
    param_copies = Broadcast.apply(devices, *params)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/_functions.py", line 17, in forward
    outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
  File "/usr/local/lib/python3.5/dist-packages/torch/cuda/comm.py", line 40, in broadcast_coalesced
    return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
RuntimeError: NCCL Error 2: unhandled system error
</code></pre>

username: fermat97
cooked: <p>Hi could you figure out the problem?</p>

username: RachelGomez161999
cooked: <p>Here are some steps you can take to troubleshoot this error:</p>
<p>Check that all GPUs are properly connected and recognized by the system. Use the nvidia-smi command to verify that all GPUs are visible and have sufficient memory.</p>
<p>Verify that the network connection between the GPUs is working properly. If the GPUs are connected through a network, check that the network settings are configured correctly and that there are no connectivity issues.</p>
<p>Check that there are no resource allocation issues, such as insufficient memory or disk space. Make sure that the software is not trying to use more resources than are available.</p>
<p>Upgrade to the latest version of NCCL and ensure that it is properly installed and configured.</p>
<p>If the error persists, try reducing the number of GPUs used to see if the error disappears. This can help isolate the source of the problem.</p>
<p>Consult the documentation of the software that you are using and the NCCL library to look for known issues and solutions.</p>
<p>Finally, if none of the above steps resolves the issue, consider seeking help from the community or the developers of the software you are using.</p>
<p>Regards,<br>
Rachel Gomez</p>
Issue loading .pt ScriptModule in older torch

username: Theix
cooked: <p>Hello. I have created a PyTorch Model to perform pupil segmentation, and I managed to saved it as a ScriptModule under torch 2.0.0 (current version). But I would like to use it in a bigger project, where we are developping under torch 1.10.0. And it appeared that my .pt file cannot be loaded using torch::jit::load() (in C++) or torch.jit.load() (in python) with torch 1.10.0.<br>
Does someone know what could be the difference in ScriptModule between these two versions that could explain this loading problem ?<br>
Thanks !</p>

username: ptrblck
cooked: <p>This is expected since PyTorch is not forward compatible.<br>
Saving a model in a new version and trying to load it in an older one could break as it’s not a supported use case.<br>
PyTorch is backwards comaptible, i.e. saving a model in an older PyTorch release should allow you to load it in a new release.</p>

username: Theix
cooked: <p>Thanks ! I managed to find a solution for this problem. I’m not sure it is very clean but it provides a great result. I saved my torch 2.0.0 model to onnx format, and then I downgraded all my python environment to versions compatible with torch 1.10.0. In this environment I managed to convert my onnx model back to a torch model, and saved it as a torch 1.10.0 compatible ScriptModule.</p>
How to do data preprocessing for large amount of data?

username: Niraja
cooked: <p>I’m working with a significant amount of image data. This is my first time working on something like this. I had never worked with such a vast amount of data before. So, how do you conduct data preparation on massive amounts of data?</p>

username: ptrblck
cooked: <p>The common approach would be to lazily load and process the data to keep the needed memory usage low. This would also mean that each transformation is applied on the loaded sample only and not during an offline preprocessing step.<br>
The <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/imagenet/main.py">ImageNet example</a> uses such an approach and might be a good reference.</p>
Dropout Question - Val accuracy > Train accuracy

username: Sammmm
cooked: <p>Hi all,</p>
<p>I have been working on a CNN classification model for MRI scans that has a very small dataset and has been experiencing severe overfitting. Accordingly, I have been using dropout and data augmentation.</p>
<p>At the moment my model overfits with dropout = 0.8 but does not with dropout = 0.9. However, with such high dropout, the training accuracy appears mostly random (see the following plots)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9ab74dc3bcb4b4afe9ef776f700155c405cd9221.png" alt="image" data-base62-sha1="m4GfVKWM2qzvNC5KAl9pp6Lx4BP" width="501" height="382"></p>
<p>My question is, is it okay to disregard the bad training accuracy because everything else is fine (aka is it just a visual result of the dropout but the model is still performing well under the hood)? If not how should I rectify the model?</p>

username: Sammmm
cooked: <p>Disregard, there was a data-leak where both datasets were the same because I accidentally named them the same thing. The only difference was that model.eval() was applied on the validation dataset. Thus the plot is just the training set with and without dropout.</p>
How to create heatmap by given coordinates in pytorch

username: Tomato_Gow
cooked: <p>Given a list of 28 key point’s coordinates from a landmark detection model. Could anyone suggest any method of how to generate a gaussian heat map for each key point?<br>
The purpose is to comparing the heat maps later on.</p>

username: amasl
cooked: <p>Check this discussion out: <a href="https://discuss.pytorch.org/t/how-to-create-heatmap-by-given-coordinates-in-mat-file-in-pytorch/65546" class="inline-onebox">How to create heatmap by given coordinates in mat file in pytorch</a></p>
Input shape of target mask in nn.Transformer

username: temps1101
cooked: <p>I am trying to use <code>torch.nn.Transformer()</code> to build a model of a transformer, but I have a question.</p>
<p>I want to batch input to the transformer function, so I tried to run the code with <code>target_mask</code> as <code>(batch size, seq_len, seq_len)</code>, but I got the following runtime error</p>
<pre><code class="lang-auto">RuntimeError: The size of the 3D attn_mask is not correct.
</code></pre>
<p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html" rel="noopener nofollow ugc">pytorch documentation</a> says that the shape of the <code>target_mask</code> should be <code>(T, T)</code> (which means <code>(seq_len, seq_len)</code>), but there is no description of the batch size, so I’m not sure how to input the target_mask, so I want to know the shape of transformer’s target_mask shape with batch.</p>
<p>Sorry for my terribe English, I’m a junior high school student in japan <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: xjtupanda
cooked: <p>I ran into the same problem. I’ve looked into the source code and found that if the attention_mask (target_mask here) is 3D tensor, num_heads should also be taken into consideration. So instead of (batch_size, seq_len, seq_len), it should be (batch_size * num_heads, seq_len, seq_len).<br>
Here’s the documentation in the source code:</p>
<p>attn_mask: If specified, a 2D or 3D mask preventing attention to certain positions. Must be of shape<br>
:math:<code>(L, S)</code> or :math:<code>(N\cdot\text{num\_heads}, L, S)</code>, where :math:<code>N</code> is the batch size,<br>
:math:<code>L</code> is the target sequence length, and :math:<code>S</code> is the source sequence length. A 2D mask will be<br>
broadcasted across the batch while a 3D mask allows for a different mask for each entry in the batch.<br>
Binary, byte, and float masks are supported. For a binary mask, a <code>True</code> value indicates that the<br>
corresponding position is not allowed to attend. For a byte mask, a non-zero value indicates that the<br>
corresponding position is not allowed to attend. For a float mask, the mask values will be added to<br>
the attention weight.</p>
Select elements of a 2-d tensor by rows and columns

username: mahsa
cooked: <p>I am new to pytorch. I have a 2-d tensor and a list of rows and columns, I want to select elements in the pair of rows and columns like follow:</p>
<pre><code class="lang-auto">x = [[1,2,3]
      ,[4,5,6],
       [7,8,9]]
row = [0,1], col=[1,2] 
</code></pre>
<p>I want to have:</p>
<pre><code class="lang-auto">output = [1,6]
</code></pre>

username: realdoug
cooked: <p>Sounds like you want <code>index_select</code> or <code>masked_select</code>?</p>
<p><a href="https://pytorch.org/docs/stable/torch.html#torch.index_select" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.index_select</a></p>
<p><a href="https://pytorch.org/docs/stable/torch.html#torch.masked_select" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.masked_select</a></p>

username: mahsa
cooked: <p>My input is too large and I have memory problem to create a mask from rows and columns.</p>
<p>index_select and gather needs to select data on a specific dimension.</p>

username: o2h4n
cooked: <p>If I understand the question correctly, the out should be:<br>
out = [2,6]<br>
and can be done like:</p>
<pre><code class="lang-auto">x = torch.tensor([[1,2,3], [4,5,6], [7,8,9]])
row = torch.tensor([0,1])
col= torch.tensor([1,2])
res=[]
for idx in range(len(row)):
    res.append(x[row[idx]][col[idx]])
print(res)
out = torch.tensor(res)
print(out)
</code></pre>
<p>[tensor(2), tensor(6)]<br>
tensor([2, 6])</p>

username: Dhawgupta
cooked: <p>doesnt this break the gradient flow computation ?</p>

username: 07hyx06
cooked: <p>I meet the same problem but don’t know how to implement it using PyTorch tensor operation. Did you solve it?</p>

username: David_Gilbertson
cooked: <p>Assuming you actually want the output of <code>[2, 6]</code> (not <code>[1, 6]</code>), you can use indexing like <code>x[rows, cols]</code>:</p>
<pre><code class="lang-python">x = torch.tensor(
    [
        [1, 2, 3],
        [4, 5, 6],
        [7, 8, 9],
    ]
)
rows = torch.tensor([0, 1])
cols = torch.tensor([1, 2])

assert x[rows, cols].tolist() == [2, 6]
</code></pre>
Some in-place operations work with autograd?

username: ArchieGertsman
cooked: <p>The following example does not result in an autograd error:</p>
<pre><code class="lang-auto">lin = nn.Linear(2,2)
y = lin(torch.rand(2))
y[1] = 0 # in-place operation
y.sum().backward()
</code></pre>
<p>I’m just curious, how come this works? When, in general, do in-place operations not interfere with autograd? Thanks!</p>

username: KFrank
cooked: <p>Hi Archie!</p>
<aside class="quote no-group" data-username="ArchieGertsman" data-post="1" data-topic="178338" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/archiegertsman/48/51153_2.png" class="avatar"> ArchieGertsman:</div>
<blockquote>
<p>When, in general, do in-place operations not interfere with autograd?</p>
</blockquote>
</aside>
<p>Here is a post with some examples that illustrate when inplace operations<br>
will and won’t break autograd:</p>
<aside class="quote quote-modified" data-post="3" data-topic="169583">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/when-inplace-operation-are-allowed-and-when-not/169583/3">When inplace operation are allowed and when not</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Jie (and Srishti)! 

The part that can be confusing is that a given tensor used in the forward 
pass may or may not be used in the backward pass.  It depends on the math 
of the gradient computation (and the details of how the relevant backward() 
function is implemented).  It’s not always obvious. 
Here are some explanatory examples: 
The derivative of exp (x) is exp (x).  So it makes sense that pytorch saves 
the output of exp (x) when it is computed during the forward pass so that it 
can …
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>
How to design CNN with MSELoss?

username: YPBS
cooked: <p>Hi,</p>
<p>CNN is mostly taught for image classification, so we use Softmax + Cross Entropy loss to get probabilities. If we need to design a CNN where we want real-value outputs like in Object boundary detection or colorization where we need to compare original color with deduced colors, then how should we design the CNN?</p>
<p>Should output dimensions be same size as original image size for MSE to work? If so, what would be the point of hierarchical representation in CNN. If we make output dimensions of CNN smaller, then we won’t be able to use MSE. Please let me know if you have any suggestions for this.</p>
<p>Currently I am working on colorization problem and MSE loss is in range 200-300 while CE loss starts with 5~6, I am using ConvTranspose2d to scale the image back to original size in the last layer when using MSE.</p>
<p>Thanks in advance,</p>
Reset parameters during k-fold cross validation

username: SimCan
cooked: <p>I would like to know whether is needed to reset the learning rate scheduler, the optimizer parameters, and the CUDA cache on every Kth iteration during <code>k-fold cross validation</code>.</p>
<p>Currently, I am using this code:</p>
<pre><code class="lang-auto">    for fold, (train_idx,test_idx) in enumerate(splits.split(np.arange(len(dataset)))):
        .
        .
        # Reset params
        net.apply(reset_wgts)
        optimizer = OPTIMIZER 
        scheduler = SCHEDULER
        .
        .
        for epoch in range(NUM_EPOCHS):
</code></pre>
How to correctly annotate subclasses of nn.Module?

username: pmeier
cooked: <p>I’m trying to annotate subclasses of <code>nn.Module</code> inline, but for now I unable to get it to work. For my project I create an abstract subclass of <code>nn.Module</code></p>
<pre><code class="lang-python">from typing import Any
from torch import nn


class Foo(nn.Module):
    def forward(self, *input: Any, **kwargs: Any) -&gt; Any:
        pass
</code></pre>
<p>Running <a href="http://mypy-lang.org/" rel="nofollow noopener"><code>mypy</code></a> on this succeeds. In a second step I add a more concrete class</p>
<pre><code class="lang-python">import torch


class Bar(Foo):
    def forward(self, input: torch.Tensor) -&gt; torch.Tensor:
        pass
</code></pre>
<p>Now <code>mypy</code> errors with</p>
<pre><code class="lang-auto">error: Signature of "forward" incompatible with supertype "Foo"
error: Signature of "forward" incompatible with supertype "Module" 
</code></pre>
<p>while pointing to <code>forward()</code> of <code>Bar</code> both times. I’m aware that this is valid error since <code>Bar</code> violates the <a href="https://en.wikipedia.org/wiki/Liskov_substitution_principle" rel="nofollow noopener">Liskov Substitution Principle</a>. Looking at the <a href="https://github.com/pytorch/pytorch/blob/d6377b7cef0c7bc822df80eedd86689bc51e83d2/torch/nn/modules/module.pyi.in#L12-L15" rel="nofollow noopener">stub of <code>nn.Module</code></a> I think the marked paragraph is related, but for now I was unable to comprehend it.</p>
<p>But since I think it is not intended for every subclass of <code>nn.Module</code> to have that exact signature, I’m puzzled how to get around this. Can someone help me out here?</p>

username: adamjstewart
cooked: <p>I’m seeing a similar problem. I have code that looks like:</p>
<pre><code class="lang-auto">import torch.nn as nn

class Foo(nn.Module):
    ...
</code></pre>
<p>but <code>mypy</code> errors out with:</p>
<pre><code class="lang-auto">error: Name "nn.Module" is not defined
error: Class cannot subclass "Module" (has type "Any")
</code></pre>
<p>No idea what this means or how to solve it.</p>

username: rqtz
cooked: <p>Any update here? I’ve got the same error on torch 2.0</p>
How to retrieve number of outputs of a model?

username: Roos
cooked: <p>Hi!</p>
<p>I tried m_ptrModule-&gt;get_method(“forward”).graph()-&gt;outputs().size();<br>
But that returns 1 also for a model with two outputs.</p>
<p>Any help would be appreciated.</p>
<p>Regards,</p>
<p>Roos</p>

username: tom
cooked: <p>Hi Roos,</p>
<p>Very likely, the Python inheritance of TorchScript shows, i.e. that functions pretending to be returning multiple values actually return a tuple.</p>
<p>So</p>
<pre><code class="lang-auto">import torch

@torch.jit.script
def d(x):
    return x,x

print(d.graph)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">graph(%x.1 : Tensor):
  %3 : (Tensor, Tensor) = prim::TupleConstruct(%x.1, %x.1)
  return (%3)
</code></pre>
<p>So a proper check would check whether <code>auto tt = graph()-&gt;output()-&gt;type()-&gt;cast&lt;TupleType&gt;()</code> and then check <code>tt-&gt;elements().size()</code> or somesuch. The <code>elements()</code> also has the types of the returns if it matters.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Roos
cooked: <p>Hi Thomas,</p>
<p>Thanks for your answer. Unfortunately your solution does not compile, e.g.<br>
<code>'output() is not a member a torch::jit::Module</code> and more.</p>
<p>In the end I solved it by passing dummy input through the module like this:</p>
<pre><code class="lang-auto">    auto outputs = m_ptrModule-&gt;forward(inputs);
    if (outputs.isTuple())
    {
        m_nNumOutputs = outputs.toTuple()-&gt;elements().size();
    }
    else
    {
        m_nNumOutputs = 1;
    }
</code></pre>
<p>But personally I think this is a hack. I can ask the number of inputs:</p>
<pre><code class="lang-auto">m_ptrModule-&gt;get_method("forward").num_inputs();
</code></pre>
<p>But not the number of outputs…</p>
<p>Greetings,</p>
<p>Roos</p>

username: tom
cooked: <p>Sorry, <code>output()</code> is a shorthand for asserting <code>outputs.size()==1</code> and then using <code>outputs[0]</code> - but for node, not graph…  The other bits should still work.</p>
<p>Besteht regards</p>
<p>Thomas</p>

username: Roos
cooked: <p>Hi Thomas,</p>
<p>Thank you. Now it works:</p>
<pre><code class="lang-auto">    auto outputs = m_ptrModule-&gt;get_method("forward").graph()-&gt;outputs();
    auto asTuple = outputs[0]-&gt;type()-&gt;cast&lt;c10::TupleType&gt;();
    if (asTuple)
    {
        m_nNumOutputs = asTuple-&gt;elements().size();
    }
    else
    {
        m_nNumOutputs = 1;
    }
</code></pre>
<p>Greetings,</p>
<p>Roos</p>

username: lcxywfe
cooked: <p>Just try</p>
<p><code>auto outputs = m_ptrModule-&gt;get_method("forward").graph()-&gt;outputs().size();</code></p>
LSTM/GRU/RNN (dropout) Pytorch implementation following TensorFlow one

username: nsacco
cooked: <p>Hi guys! It is some months that I’ve moved from TF to Pytorch. While I am enjoying speed and flexibility, I am struggling in replicating results of one of my previous TF works in Pytorch. Specifically, I am talking about a seq2seq model (which I am now extending with attention, but let’s forget about this). I’ve fixed the “basic” discrepancy given by different weights initialization. My major concern is about dropout. As you might now, TF implements two different (variational) dropouts:</p>
<blockquote>
<ul>
<li>dropout - Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. Default: 0.</li>
<li>recurrent_dropout - Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state. Default: 0.</li>
</ul>
</blockquote>
<p>default <code>nn.LSTM</code> Pytorch implementation is, as far as I’ve understood, completely different. In fact, I am not able to reproduce the same results I had in TF just using standard Pytorch dropout (which should not be variational). Using LSTM implementation <a href="https://github.com/keitakurita/Better_LSTM_PyTorch/blob/master/better_lstm/model.py" rel="noopener nofollow ugc">https://github.com/keitakurita/Better_LSTM_PyTorch/blob/master/better_lstm/model.py</a>, as suggested in <a href="https://discuss.pytorch.org/t/variational-dropout/23030/9" class="inline-onebox">Variational dropout? - #9 by Iridium_Blue</a> I get a huge performance improvement, even if I am still struggling in replicating the results that I got in TF.</p>
<p>So, my question: Is anyone aware of an implementation of recurrent layers (so, not just LSTM) that is very close to the TF one?</p>
<p>I think it should be not so much difficult to extend <a href="https://discuss.pytorch.org/t/variational-dropout/23030/9" class="inline-onebox">Variational dropout? - #9 by Iridium_Blue</a> implementation to GRU and RNN, but I am anyhow interested in other implantations, if available.</p>

username: torch_torch
cooked: <p>Same issue. After I migrate from TF to torch, the performance suffers a larger drop. Basically, the dropout plays a role here.</p>

username: vdw
cooked: <p>Do you have some code of your model to show? There are typically 2 issues that often crop up yielding subpar results:</p>
<ul>
<li>
<p>The incorrect output from the LSTM layer is used for further processing</p>
</li>
<li>
<p>The data gets scrambled up due to <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">inconsiderate use of <code>view()</code> or <code>reshape()</code></a></p>
</li>
</ul>
Does DDP supports dual-socket machine?

username: JasonLin
cooked: <p>I know DDP can easily support training on multiple GPUs. And I am wondering if something similar can be done on a platform with two CPUs. In particular, can each CPU have a model replica, run the training process on its own, and then synchronize the gradients? Thanks.</p>

username: wanchaol
cooked: <p>Theoratically DDP probably supports two CPUs as long as you initialize gloo process group, but I think it’s better to train on GPUs for the best support</p>

username: marksaroufim
cooked: <p>should work just fine, take a look at torchrun docs for examples on how to do this <a href="https://pytorch.org/docs/stable/elastic/run.html" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 2.0 documentation</a></p>
Dynamic shape in vmap

username: zihaozou
cooked: <p>Hi,<br>
I am trying to implement an algorithm that needs to deal with dynamic shapes for every sample in a batch. e.g: only process a part of the data that exceeds a threshold in the sample and place back to its original position after processing the data.<br>
I first tried to do this by using vmap. However, it seems like vmap does not support dynamic shapes.<br>
Though I could work around this problem by sequentially processing data sample by sample, is there any better solution, or any future release that will probably solve this issue?</p>
<p>Thanks.</p>

username: marksaroufim
cooked: <aside class="quote no-group" data-username="zihaozou" data-post="1" data-topic="177887">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zihaozou/48/59278_2.png" class="avatar"> zihaozou:</div>
<blockquote>
<p>only process a part of the data that exceeds a threshold in the sample and place back to its original position after processing the data.</p>
</blockquote>
</aside>
<p>Sounds like you probably want to implement a mask, something like this</p>
<pre><code class="lang-auto">mask = (input_tensor &gt; threshold).float()
output_tensor = mask * custom_operation_if_mask_is_true(input_tensor) + (1 - mask) * input_tensor
``</code></pre>
Strides are corrupted after to_dlpack / from_dlpack

username: masahi
cooked: <p>Using PyTorch 2.0, I’m seeing weird behavior from <code>to_dlpack</code> / <code>from_dlpack</code>:</p>
<pre><code class="lang-auto">In [56]: from torch.utils.dlpack import to_dlpack, from_dlpack

In [57]:  a = torch.randint(0, 100, (1, 77), dtype=torch.int32)

In [58]:  b = from_dlpack(to_dlpack(a))

In [59]:  a.stride()
Out[59]: (77, 1)

In [60]: b.stride()
Out[60]: (1, 1)  # ???
</code></pre>
<p>According to <a href="https://discuss.tvm.apache.org/t/a-pytorch-and-tvm-version-incompatibility-problem-while-using-from-dlpack-and-to-dlpack/14771" class="inline-onebox" rel="noopener nofollow ugc">A PyTorch and TVM version incompatibility problem while using from_dlpack and to_dlpack - Troubleshooting - Apache TVM Discuss</a>, the problem can be fixed by downgrading to PT 1.12.</p>
<p>Is this a bug?</p>

username: ptrblck
cooked: <p>Could you create an issue on GitHub so that code owners could take a look at it, please?</p>

username: masahi
cooked: <p>Done <a href="https://github.com/pytorch/pytorch/issues/99803" class="inline-onebox" rel="noopener nofollow ugc">[DLPACK] Strides are corrupted after `from_dlpack` / `to_dlpack` · Issue #99803 · pytorch/pytorch · GitHub</a></p>
Is DistributedDataParallel taking up more CPU memory

username: xiaohao-lin1
cooked: <p>Hi, I have a time series data shaped as follows (n_samples, n_frequency, n_channels, n_timesteps) = (21600, 9, 62, 1000).</p>
<p>When I load the data using a Customised Dataset on a single machine, I consumed about 40GB of CPU memory.</p>
<p>When I use torchrun and DDP, the CPU memory just multiplies by the number of GPUs I have.</p>
<p>Using torchrun --standalone --nnodes=1 --nproc_per_node=n_gpu, It seems like the DDP process calls the scripts n_gpu times simulatenously, and therefore, loads the data n_gpu times.</p>
<p>Is there anyway to resolve this issue?</p>

username: marksaroufim
cooked: <p>Take a look at <code>DistributedSampler</code> it should help restrict data loading to a subset of data <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 2.0 documentation</a></p>
Implementation predication to repo HDenseUet

username: aviadp
cooked: <p>I try train repo that working on LiTs dataset, the repo is this: <a href="https://github.com/thangylvp/HDenseUet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - thangylvp/HDenseUet: H-DenseUnet pytorch implementation</a><br>
This is a convert code from source keras repo (<a href="https://github.com/xmengli/H-DenseUNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - xmengli/H-DenseUNet: TMI 2018. H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation from CT Volumes</a>) to pytorch.<br>
The train working fine, i get 91%+ success, but when i try write the predict i don’t success.<br>
when i try implementation predict the input only time return tensor with other values to what I enter.<br>
Can someone please to:</p>
<ol>
<li>understand who the input tensor looks like?</li>
<li>And who i can a get the summary?</li>
<li>How i can Implementation predication for this repo?</li>
<li>And why the source code in keras Implementation strange predication with window slice?</li>
</ol>
<p>I would be very happy for your help,<br>
Aviad</p>
Trying to mimic a transformation

username: Fabrice_Auzanneau
cooked: <p>Hi<br>
I’d like to create and train a network to reproduce the effects of a (complex) sort of filter applied to sound files.<br>
My idea is to create a dataset with several input files containing (for example) 512 floats in [-1, +1] representing some sounds, and their output counterparts generated by applying the filter to them.</p>
<p>I intend to try at first an MLP network. It would have 512 input neurons and 512 output neurons, with several hidden layers. The objective is that the output of the network fits the filtered sounds.</p>
<p>Is this correct ? How can I begin this project ?<br>
Should I use MSE loss?<br>
Are there any useful links to show me how to do that ?<br>
Thanks for your help</p>
Soft Margin Ranking Triplet Loss

username: eaah
cooked: <p>Hi,</p>
<p>I’ve Implemented the following loss function. But noted on my last training that this is the reason for my loss to be NaN. When I changed the loss function to a hard triplet margin loss the network started training with no issue. Would you please help me point where can this loss be wrong?</p>
<pre><code class="lang-auto">class SoftMarginRankingLoss(torch.nn.Module):
    def __init__(self, weight=None, size_average=True, alpha =1 ):
        super(SoftMarginRankingLoss, self).__init__()
        
        self.apha = alpha
        
    def forward(self, anchor, positive_match, negative_match):
        # Calculate the euclidean distance between anchor and positive
        distance_pos = torch.square(F.pairwise_distance(anchor, positive_match, keepdim = True))
        # Calculate the euclidean distance between anchor and negative
        distance_neg = torch.square(F.pairwise_distance(anchor, negative_match, keepdim = True))
        
        distance = distance_pos - distance_neg
        loss = torch.log(1+torch.exp(self.apha*distance))
        
        # mean is required as the backward() function expects a scalar value for the loss
        return loss.mean()

</code></pre>

username: ptrblck
cooked: <p>I don’t know which values <code>distance</code> is expected to have, but note that you could easily create an overflow creating an <code>Inf</code> loss and thus <code>NaN</code> gradients:</p>
<pre><code class="lang-python">distance = torch.tensor(100., requires_grad=True)

loss = torch.log(1+torch.exp(distance))
print(loss)
# tensor(inf, grad_fn=&lt;LogBackward0&gt;)

loss.backward()
print(distance.grad)
# tensor(nan)
</code></pre>

username: eaah
cooked: <p>Thanks so much for pointing that out. Is there a way to avoid this of happening?</p>

username: hrbigelow
cooked: <p>Hi,</p>
<p>Maybe instead of <code>loss = torch.log(1+torch.exp(self.apha*distance))</code>, you could use <code>torch.logsumexp</code> but stack in a dummy value of 0 alongside the quantity self.apha*distance.  This then exponentiates to 1, thus mathematically giving you <code>log(e^0 + e^{apha*distance}) = log(1 + e^{apha*distance})</code>.</p>
<p>or, it looks like you could hack together using <code>torch.logaddexp(apha*distance, torch.zeros(...))</code></p>