akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Send and receive tensor with indefinite length

username: lecoan
cooked: <p>Hello, I am doing some work about gradient sparsification and compression in model training. After compress, tensor size may be different among workers. Now I have to send tensor size to others at first and then send the compressed tensor. However it looks ugly. I want to send and receive tensor with indefinite length in one communication, any ideas?</p>

username: mrshenli
cooked: <p>Unfortunately, with ProcessGroup send/recv, this needs to be done in two comms (size and data) for now. We are building a new <a href="https://github.com/pytorch/pytorch/issues/23110" rel="nofollow noopener">RPC API</a> which could simplifies it a bit at the API level, but internally it still uses ProcessGroup send/recv currently until we have a better comm primitive for that.</p>
[SOLVED] Invalid for input at training

username: 111137
cooked: <p>Hi,</p>
<p>I try to replace softmax with binary-representation, repo is here;<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/IAMAl/MCWOOHE" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/52284359?s=400&amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/IAMAl/MCWOOHE" target="_blank" rel="nofollow noopener">IAMAl/MCWOOHE</a></h3>

<p>Experiment to replace Softmax with Integer-Representation - IAMAl/MCWOOHE</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Now it works for training, but in a way, it makes an error of</p>
<pre><code class="lang-auto">RuntimeError: shape '[64, 1, 324]' is invalid for input of size 10368
</code></pre>
<p>Why the input size was changed? although the input is “batch * image_size”.</p>
<p>Best,<br>
S.Takano</p>

username: 111137
cooked: <p>I repaired with np.shape() to get batch size and use this for “for-loop” boundary.<br>
But really why the batch size is changed?</p>
<p>Best,<br>
S.Takano</p>

username: ptrblck
cooked: <p>The last batch might be smaller, if the number of samples in your <code>Dataset</code> is not divisible without a remainder by your batch size.<br>
If you don’t want this behavior, you could specify <code>drop_last=True</code> in your <code>DataLoader</code>, or alternatively use the (maybe) smaller batch size for the last batch with <code>x = x.view(x.size(0), your_size)</code>.</p>

username: 111137
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> -san,</p>
<p>Thank you, I understand.<br>
It means larger batch size can lost more training data, isn’t it?<br>
So, there is optimal batch size for every model?</p>
<p>Best,<br>
S.Takano</p>

username: ptrblck
cooked: <p>Only the very last batch can be smaller for any batch size, so if you use <code>drop_last=True</code>, you’ll lose <code>(batch_size-1)</code> samples max, which shouldn’t be a problem.<br>
As I said, you could alternatively use the smaller batch and change your reshaping code, but it really depends on your preferences, if your model is not sensitive to the size of the batches (e.g. the batch norm layers might be updated with a noisy stats estimate for small batches).</p>

username: 111137
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> -san,</p>
<p>Thank you very much for your advices!</p>
<p>Best,<br>
S.Takano</p>
Best Laptop for new programmer

username: Fakhar50
cooked: <p>i am new in programming I want to buy a laptop for programming I found a website <a href="https://www.findclickread.com/" rel="nofollow noopener">findclickread</a> and article name is Best Inexpensive Laptop For College which have 4 GB ram and 128 GB hard disk can these specification are enough</p>

username: ptrblck
cooked: <p>I would be quite skeptical when it comes to articles starting with “Best …”.<br>
For deep learning, you would usually use a GPU. Google Colab provides a free GPU runtime for a limited time frame (24h if I’m not mistaken), which should be a good starter to experiment a bit.</p>
Non local denoising

username: Smitha_A
cooked: <p>I want to implement non-local denoising of an image. The algorithm is to break the image into windows, then extract patches within windows and then compare the dissimlarity between patches to compute the weight matrix.</p>
<p>Can anyone please help me to implement the same using pytorch or tensorflow for single image?</p>

username: ptrblck
cooked: <p><code>unfold</code> can be used to get patches on your input tensor.<br>
Here is a small example:</p>
<pre><code class="lang-python">x = torch.randn(1, 3, 12, 12)  # batch, c, h, w
kh, kw = 3, 3  # kernel size
dh, dw = 3, 3  # stride
patches = x.unfold(2, kh, dh).unfold(3, kw, dw)
print(patches.size())
&gt; torch.Size([1, 3, 4, 4, 3, 3])
patches = patches.contiguous().view(
    patches.size(0), patches.size(1), -1, kh, kw)
print(patches.shape)
&gt; torch.Size([1, 3, 16, 3, 3])
</code></pre>
<p>This code will create 16 <code>3-by-3</code> patches, which can be used for further calculation.<br>
Note that you can adjust the kernel size and stride for your use case.</p>

username: Smitha_A
cooked: <p>Thank you very much for this code. I tried this and it works as you mentioned. When I change the stride to dh, dw to 1, 1 (stride) we get 100 3 by 3 patches. But I need 144 patches according to input  size. Could you please suggest the modififcation to be done ?</p>

username: ptrblck
cooked: <p>100 patches will be created as you will get 10 patches in each spatial dimension.<br>
If you need 12, you could add padding via <code>F.pad</code> to the input before the patch generation.</p>
<p>Note that these “outer” patches will have the padding values, which might or might not influence your further calculations.</p>

username: Smitha_A
cooked: <p>ok i will try that. thank you</p>
Multi Term Loss for Policy Gradient Algorithm

username: wykle
cooked: <p>I’ve been attempting to implement a somewhat exotic Actor-Critic policy gradient algorithm (here’s the paper if you’re interested <a href="https://arxiv.org/pdf/1802.05313.pdf" rel="noopener nofollow ugc">RL from Imperfect Deomonstrations</a>). The algorithm is similar to DQN, in that we have a network predicting Q values and use a target network to stabilize the results, but instead of updating with mse loss we calculate V from Q and do an update based on the following update equation:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/7315b61b60a8f1b1e8d7ce8d6a9df71a31f4e6d4.png" alt="image" data-base62-sha1="gq5qnuoAloBdz2PAh4eGOY8cHcw" width="648" height="62"><br>
This part is tripping me up. How would I go about writing the loss function for this? I can calculate values for the V and Q terms, but I’m not sure how to actually handle something in this form.</p>
[SOLVED] Element 0 of tensors does not require grad

username: 111137
cooked: <p>Hi,</p>
<p>I try an experiment to replace softmax with integer-representation, my repo is here;<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/IAMAl/MCWOOHE" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/52284359?s=400&amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/IAMAl/MCWOOHE" target="_blank" rel="nofollow noopener">IAMAl/MCWOOHE</a></h3>

<p>Experiment to replace Softmax with Integer-Representation - IAMAl/MCWOOHE</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I met an error of;</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>What does the error mean? and how to solve the error?</p>
<p>Best,<br>
S.Takano</p>

username: albanD
cooked: <p>Hi,</p>
<p>This means that you tried to backward on a Tensor that does not require gradients.<br>
This is most likely because you did a non-differentiable operation on your Tensor.<br>
In particular, your <code>BinaryRepresentation</code>. The output does not depend on the input in any differentiable way.</p>

username: 111137
cooked: <p><a class="mention" href="/u/alband">@albanD</a> -san,</p>
<p>Thank you for your pointing out my misunderstanding.<br>
It was caused by <strong>if</strong> statement which makes the impossible to differentiate.<br>
Now it works (but I get other error).</p>
<p>Best,<br>
S.Takano</p>
How to seperate effect of gradients from different loss

username: nima_rafiee
cooked: <p>Hi</p>
<p>I have a network like out = h2 (h1(x)). I have two losses  l1 and l2 computed using the out. Is it possible to<br>
have the gradients of l2 only perform on h2 and the gradients of l1 only on h1  as h1 is an input of h2? at first I though it might be possible by defining different optimizers but the problem is that the gradients from both losses will be accumulate for h1( if I understand the autograd function correctly) and apparently the detach() is not a solution.</p>

username: nima_rafiee
cooked: <p>I think I figured it out. I need to do it sequentially with two optimizer but each time I need to zero the gradients.</p>

username: albanD
cooked: <p>Yes exactly.<br>
You can backward one loss, do the step for that network, zero the gradients, backward the second one and do the step on the second network.</p>
Does pytorch has layer like "LocallyConnected2D" in keras?

username: YXAAA
cooked: <p>Does pytorch has layer like “LocallyConnected2D” in keras?<br>
The  <code>LocallyConnected2D</code>  layer works similarly to the  <code>Conv2D</code>  layer, except that weights are unshared, that is, a different set of filters is applied at each different patch of the input.<br>
<a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/LocallyConnected2D" class="onebox" target="_blank" rel="nofollow noopener">https://www.tensorflow.org/api_docs/python/tf/keras/layers/LocallyConnected2D</a></p>
Distributed DataLoader

username: octopusyo
cooked: <p>Hello,</p>
<p>I want to do some heavy data augmentations during training, which results into significant slow down of the training as even having 20-24 workers does not help in preparing a batch of size 32 so GPU would not stale. I am thinking about having an architecture where the data is prepared continuously on several worker nodes and then fed to the machine with GPUs for training. I am curious if there is any existing frameworks which allow doing something like this?</p>

username: Felix_Kreuk
cooked: <p>I have wondering the same thing. Did you find a solution?</p>

username: Oli
cooked: <p>I’m also looking for something like this.</p>
<p><a href="https://pytorch.org/docs/stable/notes/cuda.html#use-nn-dataparallel-instead-of-multiprocessing" rel="noopener nofollow ugc">From the docs</a></p>
<blockquote>
<p>Most use cases involving batched inputs and multiple GPUs should default to using DataParallel to utilize more than one GPU. Even with the GIL, a single Python process can saturate multiple GPUs.</p>
<p>As of version 0.1.9, large numbers of GPUs (8+) might not be fully utilized. However, this is a known issue that is under active development. As always, test your use case.</p>
</blockquote>
<p>That’s essentially what’s happening for me. I have a computer with many GPUs and the CPU becomes the bottleneck</p>
TensorBoard- Training with Validation Scalars

username: aksg87
cooked: <p>Hello all,</p>
<p>I am using TensorBoard and had a question about structuring scalars.</p>
<p>I am using this repository (<a href="https://github.com/usuyama/pytorch-unet" rel="nofollow noopener">https://github.com/usuyama/pytorch-unet</a>) as starter code for another problem and am seeing good results based on my model output and loss. My question is, what is the best way to show a validation loss on-top of the training loss?</p>
<p>I was essentially converting the print function</p>
<pre><code class="lang-python">def print_metrics(metrics, epoch_samples, phase):
    outputs = []
    for k in metrics.keys():
        outputs.append("{}: {:4f}".format(k, metrics[k] / epoch_samples))

    print("{}: {}".format(phase, ", ".join(outputs)))
</code></pre>
<p>to a metrics to tensorboard function, but I am not seeing training and validation separated.</p>
<pre><code class="lang-python">def metrics2tb(metrics, epoch_samples, phase, tb, epoch):
    for k in metrics.keys():
        tb.add_scalar(k, metrics[k] / epoch_samples, epoch)

![image|422x500](upload://1NET629OjbYs5ut4L1j5nETgYsi.png)</code></pre>

username: JuanFMontesinos
cooked: <p>You may want to use <code>add_scalars</code> rather than <code>add_scalar</code><br>
<a href="https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_scalars" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_scalars</a></p>