akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Some questions about PyTorch's mechanism of indexing

username: Rancho_Xia
cooked: <p>Recently, I ran into a little problem of tensor indexing. Here’s the code:</p>
<pre><code class="lang-auto">import torch
 a = torch.tensor([[3, 6, 1, 9],[4, 9, 4, 0],[4, 3, 8, 1]])
# index method 1
print(a[:, [0, 2, 3]])
# index method 2
print(a[[0, 1, 2], [0, 2, 3]])
# index method 3
print(a[[0, 1, 2], [[0], [2], [3]]])
# index method 4
print(a[:, [[0], [2], [3]]])
</code></pre>
<p>Output of index method 1:</p>
<pre><code class="lang-auto">tensor([[3, 1, 9],
        [4, 4, 0],
        [4, 8, 1]])
</code></pre>
<p>Output of index method 2:</p>
<pre><code class="lang-auto">tensor([3, 4, 1])
</code></pre>
<p>Output of index method 3:</p>
<pre><code class="lang-auto">tensor([[3, 4, 4],
        [1, 4, 8],
        [9, 0, 1]]
</code></pre>
<p>Output of index method 4:</p>
<pre><code class="lang-auto">tensor([[[3],
         [1],
         [9]],

        [[4],
         [4],
         [0]],

        [[4],
         [8],
         [1]]])
</code></pre>
<p>The mechanism od index methods 1 and 3 are clear, but I’m stuck to understand how the other two methods work. It seems that the 3rd index method takes the 0, 2nd, 3rd elements of every row respectively. The 4th index method is similar to the 3rd one, but adds an extra dimension. Thanks for your help!</p>

username: tom
cooked: <p>2: The result has the “shape” of the index lists (also works with tensors, then it is the shape proper) and each location is the value at the corresponding indices.<br>
In formulas: if we call <code>idx1 = [0, 1, 2]</code> and <code>idx2 = [0, 2, 3]</code> then <code>result[i] = a[idx1[i], idx2[i]]</code>.<br>
4: is essentially an extension of 2, in place of the second dimension, the shape of the second index is used and for each of them, the tensor indexed with that is used.<br>
In formulas: call <code>idx = [[0], [2], [3]]</code>, this has shape 3x1 and the original tensor is 3x4.<br>
Then the result is 3x[3x1] = 3x3x1 and <code>result[:, i, j] = a[:, idx[i, j]]</code> where I gloss over the fact that lists can’t be indexed with multiple indices.<br>
As you would, perhaps expect, 4 is the same as the result in 1 but unsqueezed to add a (singleton) last dimension.<br>
Personally, I think 3 is more elaborate than 2 because it adds another dimension similar to 1 vs. 4 but what do I know about what is difficult or easy. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>If you are used to think about lists and tuples as very similar, note that their indexing behaviour is very different, tuples (where the elements operate on different dimensons) and lists (where the elements operate on the same dimensions and the list “shape” informs the result shape). Tensors can take the role of lists here, but you’d want a tuple of tensors if you have multiple dimensions to index.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Rancho_Xia
cooked: <p>That’s very clear and thorough. Thanks a lot!</p>
Global sum pooling

username: Najeh_Nafti
cooked: <p>How can I perform global sum pooling in pytorch (with and without the view() function).</p>

username: ptrblck
cooked: <p>Assuming the same operation is applied as in a “global avg pooling” layer w.r.t. the shape, you could just use: <code>x = x.sum(dim=1)</code> where <code>dim</code> is specified as the dimension you would like to reduce.</p>

username: Najeh_Nafti
cooked: <p>just this works for me     <code>x=x.sum(0)      </code><br>
is that correct?</p>

username: ptrblck
cooked: <p>If <code>dim0</code> is the batch dimension your code looks wrong since you would not reduce in the batch dimension but in the others (i.e. channels, spatial dimensions, feature dimension etc.).</p>

username: Najeh_Nafti
cooked: <p><code>x= x.sum(-2)</code> works also with my code.</p>
<p>I got the below error if I choose a value  not between [-2,1]</p>
<pre><code class="lang-auto">IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 3)
</code></pre>
<p>And the below error with dimension =1</p>
<pre><code class="lang-auto">RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x8 and 16384x1)
</code></pre>
<p>How can I solve that?</p>

username: ptrblck
cooked: <p>Sure, you are more familiar with your use case so take my post with a grain of salt. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"><br>
I don’t know which operation causes the shape mismatch and also don’t know if you are even dealing with a batch dimension. Just check the shape before and after the reduction and make sure the shapes are explainable and make sense. E.g. if you are dealing with a simple model using linear layers, using <code>sum(0)</code> on an activation would remove the batch dimension and you would have to think about if that’s the right approach as your model would “map” <code>N</code> samples to a single output.</p>
Can multiple training runs, all reading the same data on disk, slow each other down?

username: RylanSchaeffer
cooked: <p>I have multiple training runs, all with DataLoaders accessing the same underlying Dataset on disk, and every training run is moving excruciatingly slowly (one CIFAR10 validation epoch takes ~10 minutes). Each DataLoader has 48 workers, in case that matters. I’m wondering whether the training runs are potentially blocking one another, slowing down training. Is this possibly the cause of the slow progress? If so, what should I do about it?</p>

username: RylanSchaeffer
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> would you happen to know the answer?</p>

username: ptrblck
cooked: <p>The <code>torchvision</code> CIFAR10 dataset is loading the data into the memory (as it’s quite small), so I wouldn’t expect to see a huge speedup using this large number of workers. Generally, 48 workers for each <code>DataLoader</code> sounds quite excessive, so I would recommend to play around with this value and check, if you are creating the slowdown. <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> is also a very good reference when it comes to data loading bottlenecks.</p>

username: RylanSchaeffer
cooked: <p>How could multiple workers create a slowdown?</p>

username: ptrblck
cooked: <p>From the linked post:</p>
<blockquote>
<p>Beyond an optimal number (experiment!), throwing more worker processes at the IOPS barrier WILL NOT HELP, it’ll make it worse. You’ll have more processes trying to read files at the same time, and you’ll be increasing the shared memory consumption by significant amounts for additional queuing, thus increasing the paging load on the system and possibly taking you into thrashing territory that the system may never recover from</p>
</blockquote>
When we nned to use optimizer.zero_grad() and no Model.zero_grad()?

username: laro
cooked: <p>According to:</p><aside class="quote" data-post="1" data-topic="28426">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kunasiramesh/48/13813_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/model-zero-grad-or-optimizer-zero-grad/28426">Model.zero_grad() or optimizer.zero_grad()?</a> 
  </div>
  <blockquote>
    Hi everyone, 
I have confusion when to use model.zero_grad() and optimizer.zero_grad()? I have seen some examples they are using model.zero_grad() in some examples and optimizer.zero_grad() in some other example. Is there any specific case for using any one of these?
  </blockquote>
</aside>

<p><strong>model.zero_grad() and optimizer.zero_grad() are the same IF all your model parameters are in that optimizer.</strong>.</p>
<ol>
<li>In which situation the model parameters will not be in the optimizer ?</li>
<li>Can you give me an example of this situation ?</li>
</ol>

username: tom
cooked: <p>As you note, most of the time it should make no difference.</p>
<ul>
<li>You might have more than one model in the same optimizer, GAN’s typically use two separate optimizers, but you might have a feature extractor and two heads that are separate and you train them together. Here the optimizer contains more than just one model’s parameters.</li>
<li>Or just part of a model (e.g. finetuning a pretrained model). No need to zero grad for parameters that don’t require gradients and so cannot have one.</li>
</ul>
<p>But I’d not worry about it too much. In my experience, it is more common to use <code>optimizer.zero_grad()</code>, so I’d probably take that unless you have a reason not to.</p>
<p>Best regards</p>
<p>Thomas</p>