akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Translate/rotate a 2D tensor without converting it to PIL, numpy or openCV format?

ru6928: <p>Is there a way to translate/rotate a 2D tensor without converting it to PIL, numpy or openCV format?</p>
<p>I am trying to perform rigid registration using CNN. The input to the CNN are two images concatenated. The outputs are 3 regressed values - 2 values for translations and one for angle (rotation). Then, I use these values to warp the moving image and calculate the normalized correlation coefficient using the warped and fixed image. However, the loss does not change at all and the output is the same for every iteration. I suspect it is because of the way I calculate my loss. Any help is appreciated. Thank you.</p>
<p>class nccLoss(th.nn.modules.Module):<br>
def <strong>init</strong>(self, feature_scale=1, is_deconv=True, n_images=2, is_batchnorm=True):<br>
super(nccLoss, self).<strong>init</strong>()</p>
<pre><code>def forward(self, fix, warped):
    fixed_image_valid = th.FloatTensor(fix)
    moving_image_valid = th.FloatTensor(np.array(warped))
    NCC = -1.*th.sum((fixed_image_valid - th.mean(fixed_image_valid))*(moving_image_valid - th.mean(moving_image_valid)))\
                /th.sqrt(th.sum((fixed_image_valid - th.mean(fixed_image_valid))**2)*th.sum((moving_image_valid - th.mean(moving_image_valid))**2) + 1e-10)
    return NCC
</code></pre>
<p>ncc = nccLoss()</p>
<p>avgLoss = []<br>
epl = np.zeros((4196, 3))<br>
for epoch in range(3):<br>
epochLoss = []<br>
for phase in [‘train’]:<br>
model.train()<br>
for subj in dataloaders[phase]:<br>
perm = itertools.permutations(np.arange(subj.shape[3]),2)<br>
perm = np.array(list(perm))<br>
for i in range(perm.shape[0]):<br>
fix = loadSlice2D(subj, perm[i, 0])<br>
fix = fix.to(cuda)<br>
mov = loadSlice2D(subj, perm[i, 1])<br>
mov = mov.to(cuda)<br>
optimizer.zero_grad()<br>
outputs = model(fix, mov)<br>
warp = dat.translate(mov.cpu().squeeze(0).squeeze(0), outputs[0][0], outputs[0][1], 1)<br>
warp = dat.rotate(warp, outputs[0][2], 1, PIL.Image.BILINEAR)<br>
loss = ncc(fix.cpu().squeeze(0).squeeze(0), torch.tensor(np.array(warp)))<br>
loss.backward<br>
optimizer.step()<br>
epochLoss.append(loss.detach().numpy())<br>
epl[:, epoch] = np.array(epochLoss)<br>
avgLoss.append(np.mean(np.array(epochLoss)))<br>
print(np.mean(np.array(epochLoss)))<br>
print(outputs)</p>
<p>print(avgLoss)<br>
time_elapsed = time.time() - since<br>
print(time_elapsed)</p>

Joshua_Clancy: <p>I also have this problem. Any help would be appreciated, if I make progress I’ll post about it here.</p>

Nikronic: <p>Hi, you can use <a href="https://kornia.readthedocs.io/" rel="nofollow noopener">Kornia</a> library for this.<br>
Here is the link to <a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html" rel="nofollow noopener">geometric transforms</a> such rotation and translation.</p>

Joshua_Clancy: <p>Very cool and it is PyTorch differentiable which is what I needed. Thank you!!</p>
Installing PyTorch 1.4 on Tesla K40m

Ajay_Madhavan: <p>Hi,</p>
<p>I know that PyTorch does not support Tesla K40m starting from version 1.3. Is there any workaround for this?. I want to install PyTorch 1.4 in the cluster server which has Tesla K40m using pip command. Currently using PyTorch 1.2 with CUDA9.2. The reason for asking the question is that I need to use the COGDL library which required PyTorch 1.4 minimum.</p>
<p>Any help will be much appreciated.</p>

ptrblck: <p>You could build PyTorch from source as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.</p>

Ajay_Madhavan: <p>Thank you will check details!!</p>
DDP debugging advice

Scott_Hoang: <p>Hi everyone,<br>
I want to solstice some useful advice on how to debug a strange model behavior in my training. Recently, I have taken up a ML project, and in an effort to speed up training time, I converted the original training script’s Data parallel with torch’s distributed data-parallel with Nvidia Apex. My new model will run, however, I noted that no matter how long I trained it, the results are never any better than when it started, unlike the original implementation which was able to get good results.<br>
The only thing peculiar about this model is that it used 2 separate Adam optimizers, one to optimize the layer’s parameters, and the other is used to optimize a user-defined parameters with torch grad on.<br>
I am not quite sure why this is happening. Any advice is appreciated.</p>

ptrblck: <p>I would recommend to scale down the problem first e.g. by removing apex (and potentially other utilities) and test if the DDP model is working as expected.<br>
Note that we also recommend to use the native automatic mixed precision training in the current master branch or from the nightly binaries. You can find the documentation <a href="https://pytorch.org/docs/master/amp.html">here</a>.</p>

Scott_Hoang: <p>thanks! I did not know the Pytorch now offers native mix precision</p>
Can this operation change the value of grad_input

Salient: <p>Tensor grad_input_n = grad_input.select(0, elt);</p>
<pre><code class="lang-auto">        col2hvol&lt;scalar_t, dim&gt;(
            stream,
            columns.data_ptr&lt;scalar_t&gt;(),
            nInputPlane,
            input_size,
            output_size,
            kernel_size,
            stride_size,
            pad_size,
            dilation_size,
            grad_input_n.data_ptr&lt;scalar_t&gt;());
      }
</code></pre>
<p>file “pytorch\aten\src\ATen\native\cuda\NaiveDilatedConvolution.cu”</p>

albanD: <p>Hi,</p>
<p>Yes, from what I understand, it copies the results from the gemm stored in <code>columns</code> back into the <code>grad_input</code> Tensor that will be returned as the gradient wrt the input.</p>
Autograd.grad get None for some variable occassionally

FC_Inter: <p>I’m using the <code>autograd.grad</code> of <code>torch</code> to perform analysis on my network. But I met with the problem that, when I run grad() on different data items, I occassionally get None for some layers on some certain data items, which will further cause errors.</p>
<p>I’m confused why this happens. Why does the same network get None grad on some data items, while does not on other data items? There is not noise items in the dataset, no missing values.</p>
<p>Some part of my network is as follows (to make it short, I did not post the whole network, but it’s enough to tell the problem):</p>
<pre><code>def forward(self, input):
    x = input
    out = self.backbone(x)
    predict_up_conv = self.relu2(self.bn2(self.conv_up_conv(out)))
    predict_down_conv = self.relu3(self.bn3(self.conv_down_conv(out)))
    predict_cls_conv = self.relu4(self.bn4(self.conv_cls_conv(out)))
    predict_up = self.conv_up(predict_up_conv)
    predict_down = self.conv_down(predict_down_conv)
    predict_cls = self.conv_cls(predict_cls_conv)
    if self.phase == 'test':
        predict_cls_softmax = self.softmax(predict_cls)
        return predict_up, predict_down, predict_cls_softmax
    else:
        return predict_up,predict_down,predict_cls
</code></pre>
<p>the code I use to calculate the gradient is:</p>
<pre><code>grads = grad(y, w, retain_graph=True, create_graph=True, allow_unused=True)
</code></pre>
<p><code>y</code> is the target value, and <code>w</code> is the list of all parameters of the network.</p>
<p><code>grads</code> is the resulting list of gradients. In normal cases, it is a list of pytorch tensors with shapes matching the feature map of each layer in the network. However, on some data items when I run this code, the <code>grads</code> has some elements which is <code>None</code>. And the associated layers are always the <code>self.conv_up_conv</code>, <code>self.conv_down_conv</code>, <code>self.conv_up</code>, <code>self.conv_down</code>.</p>
<p>I’m confused what’s wrong with my code.</p>
<p>Thank you all for helping me!!!</p>

albanD: <p>Hey,</p>
<p>First, <code>retain_graph=True</code> is not needed if you use <code>create_graph=True</code>.<br>
Also if you expect that all the parameters were used to compute <code>y</code>, you should not pass <code>allow_unused=True</code>.</p>
<p>Can you share how <code>y</code> is computed based on <code>predict_up</code>, <code>predict_down</code> and <code>predict_cls</code> ?<br>
Because if <code>y</code> only depends on <code>predict_cls</code>, given the structure of your forward, it looks expected that these conv layers wouldn’t get gradients as their result will not be used.</p>
Multiprocessing/ Distributed question regarding loss reporting

mobius: <p>Hello there,</p>
<p>so I’ve been following the imagenet example <a href="https://github.com/pytorch/examples/blob/e9e76722dad4f4569651a8d67ca1d10607db58f9/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/e9e76722dad4f4569651a8d67ca1d10607db58f9/imagenet/main.py</a>) on how to use multiprocessing and I have a question on loss/ statistics reporting.</p>
<p>Currently, to my understanding, the way the example is being structured, every GPU on every Node gets one instance of the entire model in memory by spawning a separate process and executing the <code>main_worker()</code> method.</p>
<p>The imagenet example uses some custom classes like <code>AverageMeter</code> and <code>ProgressMeter</code> to report progress during the training/ validation. However from what I can tell each process will have its own meters/ progress to report.</p>
<p>Thus if I were to execute the example I would get a progress report for each of the processes running.</p>
<p>Instead of getting multiple progress reports and using console logging, I would like to use <code>SummaryWriter</code> and Tensorboard to monitor the progress of my training.</p>
<p>Going through documentation (<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#save-and-load-checkpoints" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#save-and-load-checkpoints</a>) I came across a section on loading checkpoints that states:</p>
<p><code>When using DDP, one optimization is to save the model in only one process and then load it to all processes, reducing write overhead. This is correct because all processes start from the same parameters and gradients are synchronized in backward passes, and hence optimizers should keep setting parameters to the same values. </code></p>
<p>That got me thinking, that perhaps I could create a <code>SummaryWriter</code> only on the process with <code>rank == 0</code> and use that writer to only report statistics to tensorboard. I’ve implemented it as such and it seems to be working.</p>
<p>However I’d like to ask whether my approach is correct or not. Is there a better/ different way to do this?</p>

mrshenli: <p>Hey <a class="mention" href="/u/mobius">@mobius</a>, I believe this is correct. With DDP, every <code>backward()</code> pass is a global synchronization point. So all processes will run the same number of iterations and hence should have the same number of progress steps. Therefore, reporting the progress in one rank should be sufficient. (more details <a href="https://pytorch.org/docs/master/notes/ddp.html" rel="nofollow noopener">link1</a>, <a href="https://arxiv.org/abs/2006.15704" rel="nofollow noopener">link2</a>)</p>

mobius: <p>Thank you for the quick response <a class="mention" href="/u/mrshenli">@mrshenli</a>! Great job on the distributed/ multiprocessing part of PyTorch, its really intuitive. Also your tutorials have been super helpful! I was not aware of the paper you’ve mentioned, I will definitely check it out!</p>
Loss is fluctuating when during stochastic training

SageAgastya: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, I am finetuning a pretrained model over my custom dataset which takes paired sentence as input and classifies them as 1 if first sentence is answer of second sentence else it classifies them as zero.<br>
I am not creating mini-batch and taking only one training pair at a time for training (stochastic way) and I find that the loss is fluctuating in the range of 0.4 to 1.1 and I have tried almost all the loss functions. Can you help with this?</p>
<p>I have tried learning rate varying from 0.01 to 0.0001 and also tried to use learning rate decay.  It seems as it is stuck in some local optima but at the same time the amount of changes I have done in hyperparameter tuning, tells me that problem is something else and not local optimal trap.</p>