akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
nn.Parameter contains nan when initializing

username: LinZehui
cooked: <p>Hi,<br>
When I define a Parameter like this:</p>
<pre><code class="lang-auto">from torch.nn import  Parameter
a=Parameter(torch.Tensor(10,10))

</code></pre>
<p>I found the parameter contains nan.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ef9b13e8fd2d3ba02398af5308d534e7703c4121.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ef9b13e8fd2d3ba02398af5308d534e7703c4121" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ef9b13e8fd2d3ba02398af5308d534e7703c4121_2_674x500.jpeg" alt="image" data-base62-sha1="ybEmOJaCBN9cS6yqUVXmdzEsM7L" width="674" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ef9b13e8fd2d3ba02398af5308d534e7703c4121_2_674x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ef9b13e8fd2d3ba02398af5308d534e7703c4121_2_1011x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ef9b13e8fd2d3ba02398af5308d534e7703c4121_2_1348x1000.jpeg 2x" data-dominant-color="696969"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1808×1340 617 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am wondering does Parameter have to do initialization manually to avoid getting nan? Or is my way of defining Parameter wrong?</p>

username: nutszebra
cooked: <p>NaN comes from <code>torch.Tensor</code>, not from <code>Parameter</code>.<br>
This topic is relevent and helpful for you: <a href="https://discuss.pytorch.org/t/nan-in-torch-tensor/8987" class="inline-onebox">Nan in torch.Tensor</a>.</p>
<p>If you would like to initialize with random value, then you can use <code>torch.rand</code> instead of <code>torch.Tensor</code>.</p>

username: LinZehui
cooked: <p>Thank you! That’s what I want!</p>
Understanding Getattribute call in class

username: Jaideep_Valani
cooked: <p>Hi i have below class. Please help ans below queries</p>
<pre><code class="lang-auto">class SplitData():
    def __init__(self, train, valid): 
      print('splid data')
      self.train,self.valid = train,valid
        
    def __getattr__(self,k): 
      print('__getattr',k)
      return getattr(self.train,k)
   def __repr__(self): return f'{self.__class__.__name__}\nTrain: {self.train}\nValid: {self.valid}\n'
</code></pre>
<ol>
<li>which  getattr  gets called  in return</li>
</ol>
<p>2)why do i get the below ouput  for print statement i have put inside <strong>getattr</strong><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>ipython_display</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_mimebundle</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_html</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_markdown</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_svg</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_png</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_pdf</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_jpeg</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_latex</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_json</em><br>
__getattr <em>ipython_canary_method_should_not_exist</em><br>
__getattr <em>repr_javascript</em></p>
<p>SplitData<br>
Train: ImageList (12894 items)…so on</p>
Memory leak/ Random memory Allocation in Pytorch - CPU

username: Koustubh_Sinhal
cooked: <p>This is pertaining to pytorch version - 1.1.0. I am doing a simple inference on images. On each iteration, it will take one image and forward it. But everytime I run the code, memory utilization is different. In some runs, memory just keeps on increasing with each iteration. While in other runs it becomes stable after few iterations(value it gets settled to is also random).<br>
The code is</p>
<pre><code class="lang-auto">import torch
#import torch.backends.cudnn as cudnn
#cudnn.enabled = False
#torch.backends.cudnn.deterministic = True
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import time
import os
import math
import numpy as np
import pickle
import psutil
process = psutil.Process(os.getpid())


class GaussianBlur(nn.Module):
    def __init__(self):
        super(GaussianBlur, self).__init__()
        self.pad = 4
        ##weight is proxy for gaussian kernel
        weight = torch.from_numpy(np.random.uniform(0.0,1.0,(1,1,9,9)).astype(np.float32))
        self.register_buffer('buf', weight)
        return
    def forward(self, x):
        w = Variable(self.buf)
        blurred = F.conv2d(F.pad(x, (self.pad,self.pad,self.pad,self.pad), 'replicate'), w, padding = 0)
        return blurred

class ScalePyramid(nn.Module):
    def __init__(self):
        super(ScalePyramid,self).__init__()
        return
    def forward(self,x):
        curr = GaussianBlur().eval()(x)
        return curr

with torch.no_grad():
    ScalePyrGen = ScalePyramid()
    while True:
        ### y is proxy for image
        y = np.random.uniform(0.0,255.0,(1,1,321, 512)).astype(np.float32)
        y = torch.from_numpy(y)
        ### Trimmed down the scale pyramid generation for the image
        temp = ScalePyrGen(y)
        ##Printing Memory Utilization
        print(process.memory_info().rss/(1024*1024))
</code></pre>
About torchvision.transforms for segmentation task

username: Cverlpeng
cooked: <p>when I use torchvison.transforms to Data Augmentation for segmentation task‘s input image and label,How can I guarantee that the two operations are the same?</p>
<p>image input</p>
<pre><code class="lang-python"> input_transform = transform.Compose([
            transform.RandomRotation(2),
            transform.ToTensor(),
            transform.Normalize([.485, .456, .406], [.229, .224, .225])])
</code></pre>
<p>label input</p>
<pre><code class="lang-python"> input_transform = transform.Compose([
            transform.RandomRotation(2)])
</code></pre>
<p>Make sure the rotation angle is the same？</p>

username: moskomule
cooked: <p>Hi, <a href="https://github.com/pytorch/vision/releases/tag/v0.2.0" rel="nofollow noopener">https://github.com/pytorch/vision/releases/tag/v0.2.0</a> will help you.</p>

username: MariosOreo
cooked: <p>Hello,</p>
<p>You could create a random number to control the transform synchronized.</p>
<pre><code class="lang-auto">import torchvision.transfroms.functional as TF
if random.random() &gt; 0.5:
    image = TF.rotate(image, angle)
    label = TF.rotate(image, angle)
</code></pre>
<p>And wraping it in you custom dataset, calling it in <code>__getitem__</code>.</p>

username: Cverlpeng
cooked: <p>angle How can get it? thanks</p>
Custom fairseq training input

username: keysmash97
cooked: <p>hi everyone, i’m working with pytorch/fairseq. I have a project that need to build a Encoder Decoder that can receive src language, target language and pos tag of src language. I’ve read some fairseq code and it kinda complicated. So how to customize fairseq that can make it read the file of src language and pos tag them. Then encoder can receive the tensor of training data consists of words_embedding and tags_embedding.</p>
Manual Seed Non-deterministic on Different GPUs

username: cy-xu
cooked: <p>Hello community,</p>
<p>I train on a cluster gpu-farm of Tesla V100s, the scheduling system will assign me to the next available gpu, I have no control over that.</p>
<p>One issue I realized recently is that by manual seeding both numpy and pytorch, I got consistent random numbers from numpy but different random numbers from pytorch across different GPUs. (correction: I just realized numpy random numbers are generated by CPUs.)</p>
<p>The manual seed to get matching random numbers passes on my local card and one particular V100 card that I could access. So I wonder if this behavior from pytorch is by design or a bug?</p>
<p>Thanks!</p>

username: colesbury
cooked: <blockquote>
<p>The manual seed to get matching random numbers passes on my local card and one particular V100 card that I could access. So I wonder if this behavior from pytorch is by design or a bug?</p>
</blockquote>
<p>You should get consistent random numbers if you’re using the same seed, PyTorch version, and CUDA version even if it’s run on a different physical GPU. For example:</p>
<pre><code class="lang-auto">python -c "import torch; torch.manual_seed(1); print(torch.randn(1, device='cuda'))"
</code></pre>
<p>The CPU and GPU random number generators are different and will generate different streams of numbers. Also the PyTorch CPU generator is different from the NumPy generator.</p>

username: cy-xu
cooked: <p>Thank you <a class="mention" href="/u/colesbury">@colesbury</a>, I posted a reply earlier saying I found torch.randn() produces consistent results but inconsistent initialization parameters inside modules.</p>
<p>Then I realized I was testing different module designs… Indeed everything is consistent across different hardware.</p>
<p>One other question though, previously I used the following line to check for consistent random number yet without specifying <code>device='cuda'</code> I got inconsistent resutls. What do you think is the reason?</p>
<pre><code class="lang-auto">print(f'testing torch random seed {torch.randint(1, 99, (5,))}')
</code></pre>
<p>Thanks again!</p>
Why cuda mode doesn't work?

username: malofleur
cooked: <p>This is a simple example for series prediction (sin function), when I set device as ‘cpu’, it works well, however, if I change to <code>device = 'cuda' if torch.cuda.is_available() else 'cpu'</code>, it shows</p>
<p><code>Process finished with exit code -1073740940 (0xC0000374)</code><br>
All of the codes are shown below.<br>
Anyone can help or provide any advice?<br>
Regards!</p>
<pre><code class="lang-auto">import numpy as np
import matplotlib.pyplot as plt
import torch.optim as optim
from visdom import Visdom
import torch
import torch.nn as nn
num_step = 50
input_size = 1
output_size = 1
hidden_size = 16
lr = 0.01
device = 'cuda' if torch.cuda.is_available() else 'cpu'


class RNNnet(nn.Module):

    def __init__(self):
        super(RNNnet, self).__init__()
        self.rnn = nn.RNN(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=1,
            batch_first=True
        )

        self.linear = nn.Linear(hidden_size, output_size)

    def forward(self, x, hidden):
        out, hidden = self.rnn(x, hidden)
        out = out.view(-1, hidden_size)
        out = self.linear(out)
        out = out.unsqueeze(dim=0)
        return out, hidden


if __name__ == '__main__':
    rnnnet = RNNnet().to(device)
    loss_func = nn.MSELoss()
    optimizer = optim.Adam(rnnnet.parameters(), lr=lr)
    hidden = torch.zeros(1, 1, hidden_size)

    for epoch in range(10000):
        start = np.random.randint(3, size=1)[0]
        time_steps = np.linspace(start, start+10, num_step)
        data = np.sin(time_steps)
        data = data.reshape(num_step, 1)
        x = torch.tensor(data[:-1]).float().view(1, num_step - 1, 1).to(device)
        y = torch.tensor(data[1:]).float().view(1, num_step - 1, 1).to(device)

        output, hidden = rnnnet(x, hidden)
        hidden = hidden.detach()
        loss = loss_func(output, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if epoch % 100 == 0:
            print("Iteration: {} loss: {}".format(epoch, loss.item()))
</code></pre>
Update weight values with a given std::vector

username: afshin67
cooked: <p>Hi,<br>
I need to asynchronously updates weights of several copies of one network in most of RL algorithms. I tried to write a class function in which an instance of <code>toch::nn::seqential</code> exists.<br>
Using <code>named_parameters()</code> I can access the parameters in a network. Now, the question is can I assign another tensor, with same shape, to <code>p.value()</code>?<br>
For example assume I have tensor <code>w</code> which has same characteristics as <code>p.value()</code> does. Does <code>p.value() = w</code> assigns values in <code>w</code> to <code>p.value()</code>?<br>
I tested this procedure as below and it did not work for me:</p>
<pre><code>torch::autograd::GradMode::set_enabled(false);
int m=0;
for (auto &amp;p : net-&gt;named_parameters()) {
    auto z = p.value(); // note that z is a Tensor, same as &amp;p : net-&gt;parameters
    auto w = torch::zeros_like(p.value());
    if (z.dim()==1){
        int first =  m;
        int last = m + z.size(0);
        m += z.size(0);
        auto v = slice(weights, first, last);
        w+= torch::tensor(v);//.to(cpu_device);
        p.value() = w;
    }
    else if (z.dim()==2){
        int first = m;
        int last = m + z.size(0)*z.size(1);
        m += z.size(0)*z.size(1);
        auto v = slice(weights, first, last);
        w += torch::reshape(torch::tensor(v), {z.size(0),z.size(1)});//.to(cpu_device);
        p.value() = w;
    }
}
</code></pre>
<p>in which <code>weights</code> is a <code>std::vector&lt;float&gt;</code>, and <code>slice</code> function returns the proper slice of the <code>weights</code> vector.</p>
<p>I also tried this code</p>
<pre><code>torch::autograd::GradMode::set_enabled(false);
int m=0;
for (auto &amp;p : layers-&gt;named_parameters()) {
    auto z = p.value(); // note that z is a Tensor, same as &amp;p : layers-&gt;parameters
    auto w = torch::zeros_like(p.value());
    if (z.dim()==1){
        int first =  m;
        int last = m + z.size(0);
        m += z.size(0);
        auto v = slice(weights, first, last);
        w+= torch::tensor(v);//.to(cpu_device);
        p-&gt;copy_(w);
    }
    else if (z.dim()==2){
        int first = m;
        int last = m + z.size(0)*z.size(1);
        m += z.size(0)*z.size(1);
        auto v = slice(weights, first, last);
        w += torch::reshape(torch::tensor(v), {z.size(0),z.size(1)});//.to(cpu_device);
        p-&gt;copy_(w);
    }
}
</code></pre>
<p>which updates the values fine, but it breaks the gradients link and after this step I loose all gradients.</p>
<p>Thanks,<br>
Afshin</p>