akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
About log and softmax

Hannibal_046: <p>Hello, guys!<br>
I am incorporating <a href="https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/reports/custom/15784595.pdf" rel="noopener nofollow ugc">copy mechanism</a> into transformer model and use CrossEntropy Loss to do optimization.<br>
Typically, I would use</p>
<pre data-code-wrap="python"><code class="lang-python">loss_fn = torch.nn.NLLloss()
log_softmax = torch.nn.LogSoftmax()
predictions = log_softmax(logits)
loss = loss_fn(predictions,targets)
</code></pre>
<p>but in copy mechanism: I have to calculate log probability myself by</p>
<pre data-code-wrap="python"><code class="lang-python">probability = p_gen * generation_probability + (1-p_gen)*copy_probability
loss_fn = torch.nn.NLLloss()
loss = loss_fn(torch.log(probability),targets)
</code></pre>
<p>which means I can not use log_softmax which can bring numerical stability .<br>
So my training process get nan problem , what should I do to avoid this?<br>
BTW,i use fp16.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dcc6a685212c6231989241c5180bb004a9fdcd3b.png" data-download-href="https://discuss.pytorch.org/uploads/default/dcc6a685212c6231989241c5180bb004a9fdcd3b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcc6a685212c6231989241c5180bb004a9fdcd3b_2_690x156.png" alt="image" data-base62-sha1="vv4EjeEfCmcTUm0UKjv1YxJPgL1" width="690" height="156" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcc6a685212c6231989241c5180bb004a9fdcd3b_2_690x156.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcc6a685212c6231989241c5180bb004a9fdcd3b_2_1035x234.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcc6a685212c6231989241c5180bb004a9fdcd3b_2_1380x312.png 2x" data-dominant-color="131313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2156×490 67.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Permuting a data set

legoh: <p>Hi, I’m generating a data set of size [batch_size, 2], where each row entry is a 2D coordinate from 0-1 (first column is x-coordinate, and second is y-coordinate). Is there a way to quickly rearrange this data set such that the first entry is always at the furthest right corner (e.g. closest to (1,1))?</p>
Calculation of gate outputs in GRU layer

Vishvas: <p>Hey, I am trying to figure out the calculations that take place in a GRU layer.</p>
<p>I obtained a pre-trained model and it has a GRU layer define as GRU(96, 96, bias=True).</p>
<p>I checked the dimensions of the weights and bias:<br>
weight_ih_l0 = [288, 96]<br>
weight_hh_l0 = [288, 96]<br>
bias_ih_l0 = [288]<br>
bias_hh_l0 = [288]</p>
<p>The input that is fed to the layer is of size [1000, 8, 96]<br>
The batch_first variable is ‘False’, this would mean:<br>
Sequence Length = 1000<br>
Batch size = 8<br>
Input size = 96</p>
<p>I tried to follow the equations in <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html" class="inline-onebox" rel="noopener nofollow ugc">GRU — PyTorch 1.9.0 documentation</a><br>
In r(t) we multiply W with X, but my W is 2 dimensions and X is 3 dimensions which makes them incompatible for matrix multiplication.</p>
<p>I know that there are multiple time steps involved, but how is the input X (which is 3D) split so that it is compatible for multiplication with the weight martix</p>

googlebot: <p>in optimized implementations it is not sliced, instead batch matrix multiplication is done: (1000,8,96) @ (96,hs*3) = (1000, 8, hs*3) = (1000, 8, hs*3), this is Wx summands for three gates and all timesteps at once (hs=hidden size, it can differ from input size 96) . This is a “precalculation”, as Wh summands must be computed step by step.</p>

Vishvas: <p><a class="mention" href="/u/googlebot">@googlebot</a>  Thank you. I got the calculations</p>
How to fully optimize a custom convolution layer?

arman-yekkehkhani: <p>I have implemented a custom conv layer, much like <a href="https://discuss.pytorch.org/t/implementing-a-custom-convolution-using-conv2d-input-and-conv2d-weight/18556/24">here</a>. However it is still memory inefficient compared to native pytorch convlution layer, and slower. How can I fix both speed and memory consumption issues? Any drastic measure is accepted, like coding in the lowest level and building from that, since these issues are really important to me. An ideal situation is where my conv layer is as fast and memory efficient as the native one. I would appreciate any suggestions.</p>

ptrblck: <aside class="quote no-group" data-username="arman-yekkehkhani" data-post="1" data-topic="129785">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arman-yekkehkhani/48/32661_2.png" class="avatar"> arman-yekkehkhani:</div>
<blockquote>
<p>However it is still memory inefficient compared to native pytorch convlution layer, and slower.</p>
</blockquote>
</aside>
<p>You could directly reuse the native convolutions used in PyTorch to speed up your slower approach.<br>
The cuDNN conv implementations are closed source, so you would need to check (open) reference implementations in CUDA (e.g. you could check the convs in <code>cutlass</code> as another reference).</p>

arman-yekkehkhani: <p>Thanks, however, I need to modify the computations in forward and backward passes of the conv layer. I need to know exactly how functional.conv2d is calling the cuDNN API.</p>

ptrblck: <p>The cuDNN backend API calls are defined <a href="https://github.com/pytorch/pytorch/blob/bec75daa77ade04ccef4f3de67fcae216bfd122d/aten/src/ATen/native/cudnn/Conv_v7.cpp#L654-L660">here</a> so you might want to check this file to see how cuDNN is used.</p>

arman-yekkehkhani: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Thanks! I followed the link and it was helpful. However, I have another idea for the optimization of memory consumption. In this <a href="https://discuss.pytorch.org/t/implementing-a-custom-convolution-using-conv2d-input-and-conv2d-weight/18556/24">code</a>, we save <code>input, weight, bias</code> in the <code>ctx</code> for backward pass computations, which is supposedly the source of huge memory consumption. Since we have already allocated these tensors elsewhere(e.g. <code>weight</code> is th the torch.Parameter in a conv layer). I wonder how we can avoid this redundancy. At least we know that the native PyTorch implementation of Conv layer has already solved this problem. Do you know any alternates for this purpose?</p>

ptrblck: <p>I haven’t checked the code in detail, but would expect the <code>weight</code> and <code>bias</code> to be generally tiny compared to the stored activation, since the spatial size is usually larger in the incoming activation.</p>
(2, 384, 512) shape show with PIL

Samuel_Bachorik: <p>Hi I want to ask you, when I have image of this size - torch.Size([2, 384, 512]) - (it is mask for image)<br>
How can I show image with this shape with PIL library ?</p>

arya47: <ol>
<li>Transform to PIL image ( 2  pil images for shape[0] and shape[1]). You can use torchvision.transforms</li>
<li>image.show</li>
</ol>
Remap the normalized data [-1,1] to [0,1]

danishnazir: <p>Hi I have a normalized data which is from [-1,1]  with mean 0 and standard deviation 1. I want to map it to [0,1] in order to give to the network. Can anyone tell me here how does this remapping work and will we get any benefit of normalization even after changing the range to [0,1]?</p>

KFrank: <p>Hi Danish!</p>
<aside class="quote no-group" data-username="danishnazir" data-post="1" data-topic="129988" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/danishnazir/48/33515_2.png" class="avatar"> danishnazir:</div>
<blockquote>
<p>I have a normalized data which is from [-1,1]  with mean 0 and standard deviation 1.</p>
</blockquote>
</aside>
<p>Note that taking what you’ve written literally this can only be true if half<br>
of your data is <code>-1</code> and half is <code>1</code>.  You probably mean something a little<br>
different.</p>
<aside class="quote no-group">
<blockquote>
<p>I want to map it to [0,1] in order to give to the network.</p>
</blockquote>
</aside>
<pre data-code-wrap="python"><code class="lang-python">mapped_data = (data + 1) / 2
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>will we get any benefit of normalization even after changing the range to [0,1]?</p>
</blockquote>
</aside>
<p>If you are feeding your data to an already-trained network that was<br>
trained with data in the range <code>[0, 1]</code>, you will want to normalize your<br>
data to this range.</p>
<p>If you will be training your network from scratch, your training should<br>
accommodate your <code>[-1, 1]</code> data as easily as it does your data that’s<br>
been mapped to <code>[0, 1]</code> (unless you’re doing something perverse).</p>
<p>Best.</p>
<p>K. Frank</p>
Should learning rate changes according GPU number?

Ardeal: <p>Hi,</p>
<p>on 1 GPU, using the parameters batch_size=32 and  lr=0.0001, I get a good accuracy.<br>
I would like to use 8 GPU to retrain the model.<br>
My question is:<br>
should be change lr to <strong>0.0001*8</strong> when I use 8 GPU?</p>

James_W: <p>yes ofc LR should change (follow the linear rule in the paper): <a href="https://arxiv.org/pdf/1706.02677.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1706.02677.pdf</a></p>