akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Performing normal or uniform initializations on Float tensors result in only zeros

username: Santosh-Gupta
cooked: <pre><code>import torch
pytorchGPUDirectCreateWEmpty = torch.empty(size=(20000000, 128), dtype=torch.float, device='cuda', requires_grad=False, pin_memory=False).uniform_(-1, 1)
pytorchGPUDirectCreateWEmpty
</code></pre>
<p>results in</p>
<pre><code>tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')
</code></pre>
<p>and</p>
<pre><code>import torch
torch.set_default_tensor_type('torch.cuda.FloatTensor')
u_embeddings = torch.nn.Embedding(20000000, 128, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False, _weight=None)
u_embeddings.weight.data.uniform_(-1, 1)
u_embeddings.weight.data
</code></pre>
<p>results in</p>
<pre><code>tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]])
</code></pre>
<p>If I initialize with <code>double</code> instead of <code>float</code> and the initialization works fine. I could convert to float later, but I am working with limited memory and unable to initialize a double first before converting.</p>
<p>Why is the initialization not working for float tensors?</p>

username: ptrblck
cooked: <p>This code seems to work for PyTorch <code>1.2.0</code> and CUDA10.0 (installed via conda binaries):</p>
<pre><code class="lang-python">pytorchGPUDirectCreateWEmpty = torch.empty(size=(20000000, 128), dtype=torch.float, device='cuda', requires_grad=False, pin_memory=False).uniform_(-1, 1)
print(pytorchGPUDirectCreateWEmpty.min())
&gt; tensor(-1., device='cuda:0')
print(pytorchGPUDirectCreateWEmpty.max())
&gt; tensor(1.0000, device='cuda:0')
print(pytorchGPUDirectCreateWEmpty.mean())
&gt; tensor(-1.4027e-05, device='cuda:0')
</code></pre>
<p>on a Titan V (driver 418.56).</p>
<p>Which setup are you using?</p>

username: Santosh-Gupta
cooked: <p>I am using Google Colab, so the GPU is a Tesla K80, Pytorch 1.10, Cuda ‘10.0.130’</p>
<p>I tried the embedding one this morning and somehow it works now. But torch empty still doesn’t at the shape I specified. I tried with a smaller shape, and it worked, so it may be an issue with memory.</p>
<p>For convenience, here’s a notebook of the code I ran this morning</p>
<p><a href="https://colab.research.google.com/drive/1wwHWF92TzRsdCqpG3V40N-CyDlpuaEIU" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1wwHWF92TzRsdCqpG3V40N-CyDlpuaEIU</a></p>
How to partially and alternatively update the model?

username: Nuex04
cooked: <p>I’ve designed a model with the weights in the network is supposed to be updated according to the input. I got confused when I tried to implement it with batch input. Since batches are processed in parrallel, I was expecting to process  each sample accordingly. For example, suppose there are two RNNs named 1 and 2, when an input is passed to the model, it will pass through both RNNs for further calculation, but only one of the RNN’s weights should be updated through BP (either RNN1 or RNN2 in this case). How can I implement with PyTorch? Any suggestion or simple example could be helpful. Thank you.</p>