akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Question about use of torch.max function to calculate accuracy

username: seberino
cooked: <p>The CNN sample code here <a href="https://blog.paperspace.com/writing-cnns-from-scratch-in-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Writing CNNs from Scratch in PyTorch</a><br>
has a snippet where they calculate the accuracy of the model</p>
<pre><code class="lang-auto">with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in train_loader:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
    
    print('Accuracy of the network on the {} train images: {} %'.format(50000, 100 * correct / total))
</code></pre>
<p>I don’t understand the ‘<em>_, predicted = torch.max(outputs.data, 1)</em>’ line.<br>
I know outputs.data has the output data for the batch but why<br>
necessary to get torch.max?  Also, what does the 2nd arg of “1” in torch.max do?</p>
<p>Thanks!</p>
<p>Chris</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/seberino">@seberino</a>,</p>
<p>The docs for <code>torch.max</code> can be found here: <a href="https://pytorch.org/docs/stable/generated/torch.max.html" class="inline-onebox" rel="noopener nofollow ugc">torch.max — PyTorch 2.0 documentation</a></p>
<p>In short, <code>torch.max</code> returns,</p>
<p>“Returns a namedtuple <code>(values, indices)</code> where <code>values</code> is the maximum value of each row of the <code>input</code> tensor in the given dimension <code>dim</code>. And <code>indices</code> is the index location of each maximum value found (argmax).”</p>
<p>As you’re probably doing a classification problem (given from your code snippet), you want the class id (or the max index) of your model’s prediction rather than the value of the class id itself.</p>

username: seberino
cooked: <p>Thanks!<br>
I’m guessing PyTorch CNN classification models do not just return the one class with the most<br>
likely chance of being correct, but rather, the probability for <strong>each</strong> class?</p>
<p>And, that is why the user must use torch.max to drill down to the <strong>one</strong> best guess of the model?</p>
<p>Chris</p>

username: Arunprakash-A
cooked: <p><a class="mention" href="/u/seberino">@seberino</a></p>
<aside class="quote no-group" data-username="seberino" data-post="3" data-topic="187500">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/df788c/48.png" class="avatar"> seberino:</div>
<blockquote>
<p>do not just return the one class with the most<br>
likely chance of being correct</p>
</blockquote>
</aside>
<p>It depends on whether we use softmax activation or sigmoid at the network’s output layer. If it is a multi-class classification problem, softmax is used. So yes, it returns the probability score for all the classes. We can also use <code>torch.argmax()</code> .</p>
Does FSDP2 support shared modules

username: cy886
cooked: <p>Hi, I am running into an issue when using FSDP2’s <code>fully_shard</code> API with models that share submodules. Specifically, when I try to shard two models that share a submodule, I encounter an error related to DTensors already existing in the shared module</p>
<p><strong>Example code</strong></p>
<pre data-code-wrap="python"><code class="lang-python">import os

import torch
import torch.distributed as dist
import torch.nn as nn
from torch.distributed._composable.fsdp import fully_shard

class Block(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.nested_lin1 = nn.Linear(16, 16)
        self.nested_lin2 = nn.Linear(16, 16)

    def forward(self, input):
        lin_out = self.nested_lin1(input)
        lin_out = self.nested_lin2(lin_out)
        return lin_out
    
class MyModel(nn.Module):
    def __init__(self, block: Optional[nn.Module] = None) -&gt; None:
        super().__init__()
        self.lin1 = nn.Linear(4, 16)
        self.block = block or Block()
        self.lin3 = nn.Linear(16, 7)

    def forward(self, input):
        output = self.lin1(input)
        output = self.block(output)
        output = self.lin3(output)
        return output

def main():
    rank = int(os.environ["LOCAL_RANK"])
    world_size = int(os.environ["WORLD_SIZE"])
    dist.init_process_group("nccl", rank=rank, world_size=world_size)
    torch.cuda.set_device(rank)
    torch.set_default_device("cuda:" + str(rank))

    root_1 = MyModel()
    root_2 = MyModel(block=root_1.block)  # Shared submodule

    root_1 = fully_shard(root_1)
    root_2 = fully_shard(root_2)  # Error occurs here

    # Example forward pass
    for i in range(num_iters):
        if i % 2:
            output = root_2(input)
        else:
            output = root_1(input)
        ...

if __name__ == "__main__":
    main()
</code></pre>
<p><strong>Error message</strong></p>
<p>When applying <code>fully_shard</code> to <code>root_2</code>, I get the following error:</p>
<pre><code class="lang-auto">AssertionError: FSDP requires named DeviceMesh dims for ND parallelism
</code></pre>
<p><strong>Question</strong></p>
<ul>
<li>Is sharing submodules between models unsupported in FSDP2’s <code>fully_shard</code>?</li>
<li>Is there a recommended way to handle shared modules in this scenario?</li>
<li>Should I avoid sharing modules entirely, or is there a workaround or best practice for this use case?</li>
</ul>
<p>Any clarification or suggestions would be greatly appreciated. Thank you!</p>

username: cy886
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, do you have any thoughts or suggestions regarding this question? Any input would be greatly appreciated. Thanks.</p>
My model is taking too much time in calculating FFT to find top k 

username: M.A_Shanawaz
cooked: <p>so basically my batch size is 32<br>
d_model is 128<br>
d_ff is 256<br>
enc_in = 5<br>
seq_len = 128 and pred_len is 10</p>
<p>I narrow downed the bottle neck and found that my FFT step is taking too much time. i can’t use autocast to make f32 → bf16 (assume that its not currently supported).</p>
<p><strong>but frankly its taking too much time to train. and that too total steps per epoch is 700 - 902 and there are 100 epoch’s.</strong><br>
roughly the FFT is taking 1.5 secs. so</p>
<pre><code class="lang-auto">for i in range(1,4):
     calculate FFT()


</code></pre>
<p>can someone help me?</p>

username: Arunprakash-A
cooked: <p>What does FFT stand for here, Fast Fourier Transform? Aren’t you using <code>torch.topk()</code> instead?</p>
Libtorch uses much more GPU memory than python?

username: Riddick_Gao
cooked: <p>Recently, I am using libtorch to do the prediction. I have saved the original model to .pt model using torch::jit::trace( )  API.</p>
<pre><code class="lang-auto">    traced_script_module = torch.jit.trace(base_model, example)
    traced_script_module.save(ouput_path)
</code></pre>
<p>Then I loaded it in C++ using torch::jit::load() API  and do the model.eval().</p>
<pre><code class="lang-auto">    module = torch::jit::load(model_path);
    module-&gt;eval()
</code></pre>
<p>But I found that libtorch occupied much more GPU memory to do the forward( ) with same image size than original module in python.</p>
<p>So I reloaded the .pt model in python. I found it have the same performance as the original module.</p>
<p>It seems like that C++  uses much more memory than python with same model and same image size.</p>
<p>How can I deal with it?  Does any one has the same problem? thanks.</p>
<p>pytorch version: 1.1.0<br>
python version: 3.7.0<br>
OS: windows 10<br>
IDE: VS2015<br>
cuda10 + cudnn10.0 v7.5.1.10</p>

username: tholzmann
cooked: <p>Hi,</p>
<p>I am observing the same behaviour. Did you find out the reason for this?</p>
<p>Best,<br>
Thomas</p>

username: yf225
cooked: <p><a class="mention" href="/u/riddick_gao">@Riddick_Gao</a> <a class="mention" href="/u/tholzmann">@tholzmann</a> Could you shared the code that can reproduce this problem? And does it only happen on Windows?</p>
<p>cc. <a class="mention" href="/u/peterjc123">@peterjc123</a></p>

username: tholzmann
cooked: <p>On my side it happens on Windows, I’m currently not able to test it on Linux.</p>
<p>A snipped of my code:</p>
<pre><code class="lang-auto">std::ifstream in( fileName, std::ios_base::binary);
m_TorchDevice = std::unique_ptr&lt;torch::Device&gt;( new torch::Device( torch::DeviceType::CPU) );
m_Model = torch::jit::load( in, *m_TorchDevice ); 
m_Model.eval();
torch::Tensor resultTensor = m_Model.forward( inputsVec ).toTensor(); 
</code></pre>
<p>Note that I have observed the same behaviour with GPU and CPU.</p>
<p>Best,<br>
Thomas</p>

username: andy2002ua
cooked: <p>If anyone is struggling with this problem, try putting<br>
<code>torch::NoGradGuard no_grad;</code><br>
right before the <code>forward</code> call.</p>