akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Converting from .pth to .onnx failed on 3d-input. Support 3d-conv?

Vaen_Joe: <p>Hi,<br>
Below picture is the result of a transformation from pth to onnx who are shown in Neutron:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://user-images.githubusercontent.com/46133615/90018739-fbe30b80-dcdf-11ea-95b4-66877623689d.png" title="image" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/46133615/90018739-fbe30b80-dcdf-11ea-95b4-66877623689d.png" alt="image" width="476" height="500"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">image</span><span class="informations">594×623</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div><br>
Left is pth’s architecture and right is onnx’s architecture. You noticed that the first layer size of left input is different from the right. In my exprienment, as for the onnx model, the input data flowing from the bottom layer(32x1x3x3x3) which is opppsite from the correct situation (flowing from (320x640x3x3x3)).<br>
My origin network of .pth includes conv3d and the input data is 5-dim. Can the onnx be converted to onnx  on conv3d?</p>

ptrblck: <p>Are you sure the right image shows the graph from top to bottom?<br>
If so, how would e.g. the module with <code>weight (320x320x3x3x3)</code> be connected to <code>weight (256x512x3x3x3)</code>, which should yield a shape mismatch error?</p>
When I have to stop learning

Korsarz: <p>Hello,<br>
I have basic question about my network. I have four parameters during trenning process: Train Loss, Acc and Validation Loss, Acc. And I have heard about the overtraining, when i set too much train epoch. When I have to stop learning process ? Meyby if I have validation data overtrain is impossible ?<br>
Does anyone have intresning docs about it ?<br>
Thanks for help.</p>

a_d: <p>Plot your val_accuracy and see after how many epochs it starts dropping, that is where you should cut the training off, alternatively, train_loss will start becoming very less when your model starts overfitting</p>

Korsarz: <p>Thanks for your answer, but in my solution, I use pretrain resnet 50, and I have Validation Acc. equal ~1 after 3-4 epoch, (In my datasheet i have 6 class of 1500 picture each). So i must continue trenning process untill validation acc will start drop?</p>

a_d: <p>You may start by setting epochs to 10, seeing the val_acc then start decreasing it…<br>
If your val_acc at higher than lets say epoch 5 than at epoch 10 then at epoch 10 it is overfitting… see after how many epochs it falls…</p>
Model freezes right at loss.backward()

xdwang0726: <p>When I train the model, training freezes at loss.backward() without any error message or warnings. It just freeze… I have print out the loss, and have attached one of the example here<br>
<code>loss tensor(0.6277, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyBackward&gt;)</code><br>
I am wondering does anyone have faced the same problem and is there any suggestions to solve it?</p>
<p>Thank you!!</p>

ptrblck: <p>Could you run your code on the CPU and check, if it’s working or yields any error message?<br>
In case it’s working, could you post an executable code snippet as well as your PyTorch, CUDA, and cudnn version?</p>

xdwang0726: <p>I have tested the codes on CPU, and it appears the same issue. Training freezes at loss.backward() yields no error message and warnings.</p>

xdwang0726: <p>I made some changes in  the forward and the model runs. Basically, I need the main diagonal of a matrix multiplication, so I customized a <code>  col_wise_mul</code> function.</p>
<p>Before:</p>
<pre><code class="lang-auto">def col_wise_mul(m1, m2):
    result = torch.zeros(0).to('cpu')
    for i in range(m1.shape[1]):
        v1 = m1[:, i, :]
        v2 = m2[:, i]
        v = torch.matmul(v1, v2).unsqueeze(1)
        result = torch.cat((result, v), dim=1)
    return result

x = element_wise_mul(x_feature, label_feature)
</code></pre>
<p>Now:</p>
<pre><code class="lang-auto"> x = torch.diagonal(torch.matmul(x_feature, label_feature), offset=0).transpose(0, 1)
</code></pre>
<p>I am wondering why the customized function does not work? Is there any other ways to accomplish the goal as I have two very large matrixes (400, 30000). If I do multiplication first and then get the diagonal, it cost too much memory.</p>

ptrblck: <p>Are you running out of memory using the first approach?<br>
Also, could you add a print statement inside the loop and check, if the code stops at a specific iteration?</p>

xdwang0726: <p>It does not return any out of memory messages when I used the customized function, the model just freezes at the first batch in the first epoch (I have added print statement after each line in order to see which line causes the problem.)</p>

ptrblck: <p>Could you post a minimal code snippet, which would show this behavior in the latest PyTorch version, please?</p>
Lipschitz constant as regularization term

soklyras: <p>Hello everyone!<br>
I want to add as regularization term the Lipschitz constant of a neural network which I compute after each weight update. If I just modify the loss function in the training function as : loss += Lipschitz_constant , I think that this won’t affect the training as I just added a number to the loss.<br>
How can I add the Lipschitz constant (which of course is a function with respect to the weights W) to loss function as a function of weights and not just as a number (so the optimizer does not ignore the Lipschitz constant when optimizing).<br>
Thanks a lot.</p>

albanD: <p>Hi,</p>
<p>You will have to compute the Lipschitz constant in a differentiable manner.<br>
Or provide the formula for the gradient yourself via a custom autograd.Function.</p>

soklyras: <p>What do you mean by differentiable manner ?</p>

albanD: <p>I mean that you’re using only Tensors and pytorch ops for which we can compute gradients. So that the automatic differentiation will be able to compute the gradient of the Lipschitz constant wrt to the parameters.</p>
Torch.cuda.amp - How to set optimization level?

bfeeny: <p>In CUDA/Apex AMP, you set the optimization level:</p>
<p><code>model, optimizer = amp.initialize(model, optimizer, opt_level="O1")</code></p>
<p>In the examples I read on PyTorch’s website, I don’t see anything analogous to this.  How is this accomplished?</p>

ptrblck: <p>Native AMP is similar to the recommended <code>O1</code> level and doesn’t use any other <code>opt_levels</code>.</p>

bfeeny: <p>Thanks, it would seem 01 is the way to go most of the time, so I don’t think I’m losing anything.</p>

ptrblck: <p>Let us know, if you encounter any issues. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

bfeeny: <p>I just have one issue I have seen, which I posted here:</p><aside class="quote quote-modified" data-post="1" data-topic="92746">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bfeeny/48/26059_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torch-cuda-amp-and-accuracy/92746">Torch.cuda.amp and Accuracy</a> 
  </div>
  <blockquote>
    I converted my training loop to use AMP, and I notice my accuracy numbers are now all 0.  What needs to be changed to work with AMP? z was calculated within a with_autocast(): clause so I thought it should be fine, but apparently it is not. 
        print("Training..............................\r", end='') 
        train_iter = iter(train_loader)
        next_batch = train_iter.next()
        next_batch = [_.cuda(non_blocking=True) for _ in next_batch ]
        for idx in range(len(train_loader)…
  </blockquote>
</aside>
Should I add @autoncast() in sub-module with mutiple GPUs

Yu_Shen: <p>Hi,<br>
I am using automatic mixed precision with DataParallel in a single process.<br>
I read the example <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#dataparallel-in-a-single-process" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/amp_examples.html#dataparallel-in-a-single-process</a> and it says the <span class="mention">@autocast</span> should be added in MyModel before forward.<br>
My question is, should I add the <span class="mention">@autocast</span> in the subModel.<br>
For example.<br>
‘’’<br>
MyModel(nn.Module)<br>
…<br>
self. conv1 = subModel(…)</p>
<pre><code>@autocast()
def forward()
</code></pre>
<p>‘’’</p>

mrshenli: <p>cc AMP author <a class="mention" href="/u/mcarilli">@mcarilli</a></p>

mcarilli: <p>Anything that runs under autocast in a particular thread will have autocast enabled.</p>
<p><code>MyModel.forward</code> is what DP runs in a side thread.  If <code>MyModel.forward</code> is decorated with <code>@autocast()</code>, that takes care of enabling autocast for the side thread.</p>
<p>If <code>subModel.forward</code> runs within MyModel’s forward, you don’t need to additionally decorate <code>subModel.forward</code>.</p>
AMP and GPU's with low half-precision and/or no Tensor Cores

bfeeny: <p>My understanding was that having GPU’s with poor half-precision would be a big obstacle for running AMP.  For example, compare the following GPU’s:</p>
<p>NVIDIA 1080Ti<br>
Single Precision GFLOPS.    10609<br>
Half Precision GFLOPS         <strong>166</strong><br>
Tensor Cores                          0</p>
<p>Compare this with an RTX GPU like the 2080 Ti</p>
<p>RTX 2080Ti<br>
Single Precision GFLOPS.    11750<br>
Half Precision GFLOPS         <strong>23500</strong><br>
Tensor Cores                          544</p>
<p>In AMP/Mixed Precision many of the operations are taking place at half-precision, this gives you incredible memory savings and speed.</p>
<p>I tested this with some models tonight on my 1080Ti’s (4 using DataParallel), and I was blown away.  Epochs complete 30% faster, and I could almost double my batch size.  So I don’t understand why it wasn’t dogged slow since the half-precision GFLOPS on a 1080Ti are abysmal.  Is AMP doing some sort of trickery, such as using 16bit and stuff two numbers into a 32bit register or something?</p>

mcarilli: <p>It’s possible your network is mostly bandwidth bound rather than compute bound, in which case the 1/2X memory traffic for fp16 ops would allow a speedup.  It’s also probable that many Pytorch kernels compute internally in fp32 even if the input/output is fp16, so the lack of raw fp16 throughput isn’t a problem.  It’s also possible the same is true for cuda library calls (gemms/convolutions):  they may use fp32 compute internally for fp16 input/output, so the compute throughput is not worse than fp32 and the required bandwidth is reduced.</p>