akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the optimal options for training epoch?

Aiman_Mutasem-bellh: <p>Dear friends,</p>
<p>I have a question about how to set the optimal options for training epoch like (Number of layers, batch size, number of epochs, etc.) to get better results and avoid data overtraining.</p>
<p>Regards,<br>
Aiman</p>

vdw: <p>There are no optimal, one-size-fits-all settings. When it comes to training a network, there’s a lot of trial-and-error involved. There are whole research papers just about how to set/update the learning rate, for example. Here just some very basic guidelines of mine:</p>
<ul>
<li>Can you overtrain the network on a small dataset? That just to see if the network, loss calculation and backprop is (most likely) working fine.</li>
<li>After each eopch, calculate both training and test loss and trace their trends over time. The training loss should essentiall go down and maybe converge at some point. The test loss should also go down, at least at first. If/once the test loss increases again, your network is overtraining, and you might want to stop the training.</li>
<li>A good batch size depends on the learning rate and/or vice versa. If the batch size is large, the loss gets averaged over more samples and is therefore typically smaller than for small batch sizes.</li>
<li>In the end, you have to try: If the training/test loss is going down but slowly, increase the learning rate. If the tranining/test loss jumps up and down, decrease the learning rate.</li>
<li>I’m not sure what you mean by “number of layers” here since it refers to the network model and not the training itself. In general, however, the more complex a network (e.g., in terms of numbers of layers), the more training is needed, both in terms of the dataset and the computation (i.e., epochs).</li>
</ul>
<p>I hope that helps a bit.</p>

Aiman_Mutasem-bellh: <p>Dear Chris,</p>
<p>Thank you so much, that all I’m looking for. This is my first deep learning project, and there are many things I’ve to learn.</p>
How can I effectively take slices of a tensor?

Andrea_Rosasco: <p>I’ve noticed that if I take slices of a big tensor with the standard numpy syntax (a = b[0:2]) and then save it to a file with pickle, the size is for the two tensors is the same.<br>
What is actually happening?<br>
I found that I can make the change of size effective by converting the newly created tensor to a numpy vector and then back to a tensor again but I’m sure there’s an easier way.</p>

ptrblck: <p><code>a</code> and <code>b</code> will share the same memory, but some attributes in <code>a</code> such as the size and possibly <code>storage_offset</code> will be changed. Here is a small example</p>
<pre><code class="lang-python">b = torch.zeros(3, 2)
a = b[1:3]
print(b)
&gt; tensor([[0., 0.],
        [0., 0.],
        [0., 0.]])
a[:] = 1.
print(b)
&gt; tensor([[0., 0.],
        [1., 1.],
        [1., 1.]])
print(a.storage_offset())
&gt; 2
</code></pre>
<p>Note that I used other indices for slicing to demonstrate the <code>storage_offset</code>.<br>
If you want to create a copy, use <code>a = b[0:2].clone()</code></p>

Andrea_Rosasco: <p>Thank you, now it’s clear!</p>
How to use torch.utils.checkpoint

Small_Home: <p>Just as the title says, how should I modify my forward pass to use this function?</p>
<p>My current forward pass is:</p>
<pre><code class="lang-python">    def forward(self, x):
        out = F.relu(self.pool1(self.conv1(x)))
        out = F.relu(self.pool2(self.conv2(out)))
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)
        return out
</code></pre>
<p>Many thanks!</p>

ptrblck: <p>I like <a class="mention" href="/u/priya_goyal">@Priya_Goyal</a>’s <a href="https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb" rel="nofollow noopener">tutorial</a> on checkpointing.<br>
Note that it wasn’t updated in a while and uses an old PyTorch version, but the general workflow should be the same.</p>
Reshape a tensor

MANSUM: <p>Hi I want to make the following tensor<br>
tensor([[1., 1., 1., 1., 1.],<br>
[2., 2., 2., 2., 2.],<br>
[3., 3., 3., 3., 3.],<br>
[4., 4., 4., 4., 4.],<br>
[5., 5., 5., 5., 5.],<br>
[6., 6., 6., 6., 6.]])</p>
<p>to</p>
<p>tensor([[1., 1., 1., 1., 1., 4., 4., 4., 4., 4.]<br>
[2., 2., 2., 2., 2., 5., 5., 5., 5., 5.],<br>
[3., 3., 3., 3., 3., 6., 6., 6., 6., 6.]])</p>
<p>I tried to do with view, but failed to do so. I want to do it so that I can compute backward() on the tensor.</p>
<p>Can anyone help?</p>

ptrblck: <p><code>x.view(2, 3, 5).permute(1, 0, 2).contiguous().view(3, 10)</code> should work.</p>

MANSUM: <p>Thanks a lot!<img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"><img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"><img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

MANSUM: <p>One more question.<br>
what about if I want to add the first half and the second half?<br>
That is,</p>
<p>[5,5,5,5,5]<br>
[7,7,7,7,7]<br>
[9,9,9,9,9]</p>
<p>Thanks a lot!</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">x.view(2, 3, 5).sum(0)
</code></pre>

MANSUM: <p>thanks a lot!!<img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"><img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

MANSUM: <p>Sorry for coming back with a similar question.</p>
<p>How do I make</p>
<p>tensor([[1., 1., 1., 1., 1., 4., 4., 4., 4., 4.]<br>
[2., 2., 2., 2., 2., 5., 5., 5., 5., 5.],<br>
[3., 3., 3., 3., 3., 6., 6., 6., 6., 6.]])</p>
<p>to</p>
<p>tensor([[1., 1., 1., 1., 1.],<br>
[2., 2., 2., 2., 2.],<br>
[3., 3., 3., 3., 3.],<br>
[4., 4., 4., 4., 4.],<br>
[5., 5., 5., 5., 5.],<br>
[6., 6., 6., 6., 6.]])</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">x.view(3, 2, 5).permute(1, 0, 2).reshape(6, 5)
</code></pre>
How to skip backward if loss is very small in training

autocyz: <p>When training, if loss is very small, I want loss not backward to save training time by this code:</p>
<pre><code class="lang-auto">total_loss = criterion(anchor_f, pos_f, neg_f)        
if total_loss.item() &gt; 1e-6:
    optimizer.zero_grad()
    total_loss.backward()
    optimizer.step()
else:
    pass 
</code></pre>
<p>this will raise cuda memory error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 10.76 GiB total capacity; 9.93 GiB already allocated; 22.19 MiB free; 39.84 MiB cached)
</code></pre>
<p>Is my code not correct?</p>

justusschock: <p>Since you don’t actually call backward, your graph doesn’t get destroyed. I think it would be better, to always backward and (depending on the value of your loss) call an additional zero_grad (or don’t call the optimizer step):</p>
<pre><code class="lang-auto">total_loss = criterion(anchor_f, pos_f, neg_f)        
optimizer.zero_grad()
total_loss.backward()
if total_loss.item() &gt; 1e-6:
    optimizer.step()
else:
    optimizer.zero_grad()
</code></pre>

autocyz: <p>Is there a way to skip backward?</p>

ptrblck: <p>You could put <code>total_loss.backward</code> inside the condition using <a class="mention" href="/u/justusschock">@justusschock</a>’s code.</p>
Has no attribute 'Transformer'

slavavs: <p>I tried to add a layer ‘Transformer’, but got an error:<br>
module ‘torch.nn’ has no attribute ‘Transformer’</p>
<p>how can i fix this?</p>

ywatanabe1989: <p>I’m not too sure that pytorch version &gt;= 1.2.0 (nightly now) includes Transformer modules.</p>

spanev: <p>It should do <a href="https://github.com/pytorch/pytorch/releases/tag/v1.2.0" rel="nofollow noopener">https://github.com/pytorch/pytorch/releases/tag/v1.2.0</a></p>
<p>Which version are you running?</p>