akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Disk(SSD or HDD) space increase(decrease)ing during training

username: Sherzod_Bek
cooked: <p>Hi,<br>
I was using yolov5. Disk space is increasing (sometimes decreasing) during training. is it normal? I think have enough gpu(8gb) and ram(32) memory and dataset is small(custom dataset).</p>

username: ptrblck
cooked: <p>You could check if the <code>swap</code> is used or if you are storing any data explicitly.<br>
Besides these use cases I would not be aware of any disk usage.</p>

username: Sherzod_Bek
cooked: <p>swap is not used(I think). I think it’s because of data augmentation. During data augmentation(before training) process → disk(SSD) memory reduced, after training(finish) increased.</p>

username: ptrblck
cooked: <p>What kind of library for the data augmentation are you using? It sounds strange that your local SSD would be used as I don’t know why it would be necessary and it could also decrease the performance if you need to write/read from the disk while transforming data.</p>

username: Sherzod_Bek
cooked: <p>I think I found reason,  it is because of → number of dataloader workers.<br>
if I increase num_workers it takes more space from disk, when reduce num_workers it takes less space from disk.</p>

username: ptrblck
cooked: <p>Increasing the <code>num_workers</code> would use shared memory for the communication. Take a look at <a href="https://pytorch.org/docs/stable/data.html#module-torch.utils.data">these docs</a> for more information.<br>
I don’t know which OS you are using, but note this section:</p>
<blockquote>
<p>Since workers rely on Python <a href="https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing"><code>multiprocessing</code></a>, worker launch behavior is different on Windows compared to Unix.</p>
<ul>
<li>On Unix, <code>fork()</code> is the default <a href="https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing"><code>multiprocessing</code></a> start method. Using <code>fork()</code>, child workers typically can access the <code>dataset</code> and Python argument functions directly through the cloned address space.</li>
<li>On Windows or MacOS, <code>spawn()</code> is the default <a href="https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing"><code>multiprocessing</code></a> start method. Using <code>spawn()</code>, another interpreter is launched which runs your main script, followed by the internal worker function that receives the <code>dataset</code>, <code>collate_fn</code> and other arguments through <a href="https://docs.python.org/3/library/pickle.html#module-pickle"><code>pickle</code></a> serialization.</li>
</ul>
</blockquote>
<p>I don’t know exactly how the Windows implementation was done, but in case memory mapping is used your hard drive could be used to store a copy of the data.</p>
Can we replace Relu6 with hardtanh(0,6)

username: bigtree
cooked: <p>Can we replace Relu6 with Hardtah(0,6) since both clamp the value in the range (0~6)</p>

username: Eta_C
cooked: <p>YES, we can see that <code>ReLU6</code> is a subclass of <code>Hardtanh</code>.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/11ca64149120b68cc612ebbdb8e0a24c908709f1/torch/nn/modules/activation.py#L242-L269">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/11ca64149120b68cc612ebbdb8e0a24c908709f1/torch/nn/modules/activation.py#L242-L269" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/11ca64149120b68cc612ebbdb8e0a24c908709f1/torch/nn/modules/activation.py#L242-L269" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/11ca64149120b68cc612ebbdb8e0a24c908709f1/torch/nn/modules/activation.py#L242-L269</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="242" style="counter-reset: li-counter 241 ;">
          <li>class ReLU6(Hardtanh):</li>
          <li>    r"""Applies the element-wise function:</li>
          <li>
          </li>
<li>    .. math::</li>
          <li>        \text{ReLU6}(x) = \min(\max(0,x), 6)</li>
          <li>
          </li>
<li>    Args:</li>
          <li>        inplace: can optionally do the operation in-place. Default: ``False``</li>
          <li>
          </li>
<li>    Shape:</li>
          <li>        - Input: :math:`(*)`, where :math:`*` means any number of dimensions.</li>
          <li>        - Output: :math:`(*)`, same shape as the input.</li>
          <li>
          </li>
<li>    .. image:: ../scripts/activation_images/ReLU6.png</li>
          <li>
          </li>
<li>    Examples::</li>
          <li>
          </li>
<li>        &gt;&gt;&gt; m = nn.ReLU6()</li>
          <li>        &gt;&gt;&gt; input = torch.randn(2)</li>
          <li>        &gt;&gt;&gt; output = m(input)</li>
      </ol>
    </code>
  </pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/11ca64149120b68cc612ebbdb8e0a24c908709f1/torch/nn/modules/activation.py#L242-L269" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Efficient kernel density estimation

username: sbelharbi
cooked: <p>hi,<br>
i have an rgb image.<br>
is there an efficient way to compute a <a href="https://en.wikipedia.org/wiki/Kernel_density_estimation" rel="noopener nofollow ugc">kde</a> over the colors?</p>
<p>assume that we have a maximum of 256 colors, and that we consider discretize them into 128 bins.<br>
this will form a 3d histogram of a color space: 128**3, 3.</p>
<p>i made a naive implementation but it is impractical in term of memory.</p>
<pre><code class="lang-python">ndim = 3  # rgb

 def _get_color_space() -&gt; torch.Tensor:
        x = torch.linspace(start=0., end=256, steps=128, dtype=torch.float32,
                           requires_grad=False)
        tensors = [x for _ in range(ndim)]
        return torch.cartesian_prod(*tensors)

color_space = _get_color_space()
dim, h, w = img.shape  # dim = ndim = 3
x = img.contiguous().view(h * w, dim)

# op1
ki = (x.unsqueeze(1) - color_space)**2  # h*w, nbin**ndim, ndim  //MEMORY ISSUE.
# op2
ki = ki.sum(dim=-1) / const2  # h*w, nbin**ndim   //MEMORY ISSUE.
# op3
ki = const1 * torch.exp(-ki)  # h*w, nbin**ndim   //MEMORY ISSUE.
# op4
prob = ki.mean(dim=0)  # 1, nbin**ndim
</code></pre>
<p>the issue is that we store in memory the difference between ALL pixels and ALL kernels.<br>
it is unnecessary.</p>
<p>another approach is to LOOP over kernels to avoid storing all results in memory. something like this:</p>
<pre><code class="lang-python">prob = torch.zeros((1, nbin**ndim))
for kernel_i in range(color_space.shape[0]):
      prob[kernel_i] = op_1_2_3_4(img, kernel_i)  # scalar
# where op_1_2_3_4(img, kernel_i) is simply
# ~ (exp((img - v)**2).sum(dim=0)).mean() 
# where v is a vector of 3 components.
</code></pre>
<p>the issue with this is the loop over a large number that is 128**3 ~ 2millions.<br>
i wonder if there is an efficient way to do it.<br>
i can minibatch it (in addition to looping over each image in minibatch…). but it is still expensive.<br>
this is similar to convolution.<br>
but we do not need to store the ‘feature maps’ that is the result of <code> exp((img - v)**2).sum(dim=0)</code>.<br>
the result of each convolution, that is a feature map, should be averaged and stored.<br>
we dont need to have any feature map in memory (just instantly, then it is averaged).</p>
<p>is it possible to design an operation that is similar to 1d convolution, but reduces the feature map right away via mean? == <code> exp((img - v)**2).sum(dim=0).mean()</code></p>
<p>any ideas?</p>
<p>in tensorflow, they have a kde dist. <a href="https://arxiv.org/pdf/1711.10604.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1711.10604.pdf</a></p>
<p>thanks</p>
Can someone review my PR?

username: Vansh_Sharma
cooked: <p>I have created this <a href="https://github.com/pytorch/pytorch/pull/68557" rel="noopener nofollow ugc">PR</a> to fix all the typos in pytorch/docs/source/<br>
Can someone please review ?</p>
<p>Edit: It has been approved <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Models on Different GPUs

username: dendisuhubdy
cooked: <p>I wonder if there’s a way to load two (or n) different models on two (or n) different GPUs. For example, if I’d want to load my CNN on GPU device ID 0 and an RNN on GPU device ID 1? Is there a way to do this in Pytorch?</p>

username: ptrblck
cooked: <p>When calling <code>model.cuda()</code> you can pass a device id as argument.<br>
E.g.: <code>cnn.cuda(0)</code> and <code>rnn.cuda(1)</code>.</p>

username: dendisuhubdy
cooked: <p>I think <a class="mention" href="/u/apaszke">@apaszke</a>’s answer on a previous post here helps too, and is also closer to what I’m looking for although I’m not looking to split over a model [<a href="https://discuss.pytorch.org/t/model-parallelism-in-pytorch-for-large-r-than-1-gpu-models/778">Model parallelism in pytorch for large(r than 1 GPU) models?</a>](<a href="http://Model" rel="nofollow noopener">http://Model</a> Parallelism in Pytorch)</p>

username: RylanSchaeffer
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is there a less manual way of placing different models on different devices?</p>

username: ptrblck
cooked: <p>You could take a look at native <a href="https://pytorch.org/docs/stable/pipeline.html">Pipeline Parallel</a> implementation and also check out e.g. Megatron-LM (also available in apex) or DeepSpeed, which apply different parallelism approaches.</p>