akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Ignore peak values with transforms.normalize

bkonk: <p>I am using transforms.Normalize to normalize chest x-rays before feeding into a DenseNet. I’ve noticed that some of these images have a large white square present somewhere in the image, and when this happens the normalized image ends up very dark. Aside from these random white squares the images should have a pretty normal distribution…so I’m assuming histograms of these have a normal distribution and then very high very narrow jump up around 255. Is there a way with transforms.Normalize to ignore these abnormal peak values?</p>

spanev: <p>Hi <a class="mention" href="/u/bkonk">@bkonk</a> ,</p>
<p>When using  <code>transforms.Normalize</code>, you are the one specifying the mean and std for each image. How do you compute it per sample?</p>

bkonk: <p>Thanks for the reply <a class="mention" href="/u/spanev">@spanev</a>,</p>
<p>This is branching off topic a bit, but I’m a little unclear on how to set the means and stds. I initially went with those used in Chexnet (densenet for a huge x-ray database) as I thought these would translate well, but the images I was viewing (with matplotlib.pyplot…maybe this distorts things a bit?) were looking very dark or very saturated. My images are 0 to 255 value .png files. I tried loading them, converting to a -1 to 1 range, and then grabbing the means and stds across all images, but this gave even worse looking results. I similarly opened the pngs as numpy arrays with cv2, and tried changing those white-box values (usually 1 or 255 depending on range) to whatever the mean value was of the other pixels and then re-saving. This seemed to help a bit when it came to previewing those white-box images (i.e., they weren’t super dark except for the box) but doesn’t seem like a great solution.</p>
Tensor image show

DonghunP: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3194ad545a6f045dc746fe34f4a04a79a4fa0420.png" data-download-href="https://discuss.pytorch.org/uploads/default/3194ad545a6f045dc746fe34f4a04a79a4fa0420" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3194ad545a6f045dc746fe34f4a04a79a4fa0420.png" alt="image" data-base62-sha1="74BVYUNLtBW0DjzzgmHztnVKjQY" width="551" height="500" data-dominant-color="BEA2A4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">641×581 30.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I show image tensor</p>
<p>I use</p>
<pre><code class="lang-auto">img = images.numpy().transpose(1, 2, 0)
plt.imshow(img)
plt.show()
</code></pre>
<p>But the color is wrong. Can anybody help me??</p>
<p>Thanks ahead.</p>

Pranavan_Theivendira: <aside class="quote no-group" data-username="DonghunP" data-post="1" data-topic="55644">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/donghunp/48/16163_2.png" class="avatar"> DonghunP:</div>
<blockquote>
<p>img = images.numpy().transpose(1, 2, 0)</p>
</blockquote>
</aside>
<p>Hi,</p>
<p>For imshow, if the image is 3 channel / RGB,  the pixel values should be [0-1] float or [0-255] int. I feel that the problem is coming from data type of pixel values.</p>
<p>If it does not fix your issue, please print some values in img.</p>
<p>Thanks</p>
Shaping 4d tensor into 3d

nlam: <p>my imshow function accepts a tensor of (1, 28, 28), but my dataloader is returning a tensor of (1000, 1, 28, 28). The “1000” seems to be the value of the batch_size that I specify in the loader. Is there anyway to fix this.</p>

ptrblck: <p>You could index the specific image tensor you would like to visualize:</p>
<pre><code class="lang-python">my_imshow(data[0])
</code></pre>

nlam: <p>that worked. thank you!</p>
Which one is correct to initialize separate layer?

John1231983: <p>I have a network likes</p>
<pre><code class="lang-auto">x--&gt;fc1(x)--&gt;outputx
y--&gt;fc2(y)--&gt;outputy
</code></pre>
<p>So, when I define the network, can I used the same name for fc1 and fc2 likes</p>
<pre><code class="lang-auto">class network(nn.Module):    
    def __init__(self):
       self.fc = nn.Linear(1,2)
   def forward(x,y)
      output1= self.fc(x)
      output2 = self.fc(y)
</code></pre>
<p>Or I have to seperate into two layer such as</p>
<pre><code class="lang-auto">class network(nn.Module):    
    def __init__(self):
       self.fc1 = nn.Linear(1,2)
       self.fc2 = nn.Linear(1,2)
   def forward(x,y)
      output1= self.fc1(x)
      output2 = self.fc2(y)
</code></pre>
<p>Thanks</p>

ptrblck: <p>Both models define valid forward methods (besides the missing <code>return</code> statement) and it depends on your use case, which one is correct.<br>
If you would like to apply the same parameters for both inputs (weight sharing), the first definition is correct, while the second model will use two different layers and parameter sets.</p>
<p>Based on the first pseudo code, it seems you would like to use two separate layers, so the second model definition should be right.</p>
Sampling from multinomial with extremely high values

Guillaume_Jeanneret: <p>Hello!</p>
<p>Is there a way to sample efficiently from a large collection of indexes (1M~) without repetition? I was using the function multinomial(torch.ones(length), num_samples, replacement=False) but it is taking too much time to generate the samples.</p>
<p>Thanks!</p>

LeviViana: <p><a href="https://github.com/pytorch/pytorch/pull/18624" rel="nofollow noopener">This PR</a> implements <code>torch.choice</code>, it accelerates sampling from multinomial distributions.</p>
<p>As a second option, <a href="https://github.com/LeviViana/torch_sampling" rel="nofollow noopener">this repo</a> does basically the same thing, but it is older than the PR.</p>
Batchsize issue

Megh_Bhalerao: <p>Hello,<br>
I am training a 3D CNN and I set batch_size = 1 in the dataloader object since a batch_size =2 gives cuda out of memory. So I thought about another way to increase the batch_size to 2. The crux of the code looks something like (in the trainer loop of the program) :</p>
<pre><code class="lang-auto">for batch_idx, (subject) in enumerate(train_loader):
        # Load the subject and its ground truth
        itr+=1
        image = subject['image']
        mask = subject['gt']
        # Loading images into the GPU and ignoring the affine
        image, mask = image.float().cuda(), mask.float().cuda()
        #Variable class is deprecated - parameteters to be given are the tensor, whether it requires grad and the function that created it
        optimizer.zero_grad()
        image, mask = Variable(image, requires_grad = True), Variable(mask, requires_grad = True)
        # Forward Propagation to get the output from the models
        output = model(image.float())
        # Computing the loss
        if itr%2==1 :
            loss_odd = loss_fn(output.double(), mask.double(), n_classes)
        if itr%2==0 :
            loss_even = loss_fn(output.double(), mask.double(), n_classes)
            loss = loss_odd + loss_even
            loss.backward()
            optimizer.step()
</code></pre>
<p>In the above dataloader I have set the batch_size =1 and I am trying to manually make the network “feel” as if the batch_size = 2 by accumulating the loss function and doing back prop after every 2 examples…<br>
Am I doing this right?<br>
Or am I missing something?<br>
Thanks a lot in advance</p>

ptrblck: <p><a class="mention" href="/u/alband">@albanD</a> explained some approaches <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">here</a> in a detailed way.<br>
If I’m not mistaken, you are currently using approach 3, so you might want to switch to the second one to save memory.</p>

Megh_Bhalerao: <p>Yes, thanks a lot!!<br>
It seems to work now. The main concept behind this was pytorch’s dynamic computational graph, if I’m not wrong…</p>
1*1 Conv2d functionality in Downsample of Resnet18 is different than other framework

Kothari_Gaurav: <p>In Pytorch <strong>Resnet</strong> class we have <strong>resnet18</strong> architecture which uses <strong>Basic block</strong> and In that <strong>Basic block</strong> we have a sequential object called <strong>Downsample</strong>.<br>
In Downsample pytorch is doing <strong>[1 * 1] conv2d</strong>  functionality.</p>
<p>My question is why I am getting different output for [1 * 1]convolution in Pytorch in comparison to other framework like Darknet for the same operation.<br>
Note - other [2  2] and [3  3] convolution are giving same output in other framework but only [1 * 1] convolution is giving very different output.</p>
<p>is there any different approach, which is Pytorch is using for the special case of [1 * 1]?<br>
code <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
downsample = nn.Sequential(<br>
conv1x1(self.inplanes, planes * block.expansion, stride),<br>
norm_layer(planes * block.expansion),<br>
)</p>
<p>usage <img src="https://discuss.pytorch.org/images/emoji/apple/yo-yo.png?v=9" title=":yo-yo:" class="emoji" alt=":yo-yo:"><br>
def forward(self, x):<br>
identity = x<br>
out = self.conv1(x)<br>
out = self.bn1(out)<br>
out = self.relu(out)<br>
out = self.conv2(out)<br>
out = self.bn2(out)<br>
out = self.relu(out)<br>
out = self.conv3(out)<br>
out = self.bn3(out)<br>
if self.downsample is not None:<br>
identity = self.downsample(x)<br>
out += identity<br>
out = self.relu(out)<br>
return out</p>

ptrblck: <p>You could create a dummy example with small sizes and compute the result manually:</p>
<pre><code class="lang-python">x = torch.randn(1, 2, 2, 2)
conv = nn.Conv2d(2, 1, 1)
output = conv(x)

print(x)
tensor([[[[-0.4150, -0.3450],
          [-0.2987,  1.7142]],

         [[ 1.2817, -0.7830],
          [-0.0724, -1.8714]]]])
print(conv.weight)
&gt; tensor([[[[ 0.6824]],

         [[-0.5454]]]], requires_grad=True)
print(conv.bias)
&gt; tensor([-0.5595], requires_grad=True)
print(output)
&gt; 
tensor([[[[-1.5418, -0.3679],
          [-0.7238,  1.6310]]]], grad_fn=&lt;MkldnnConvolutionBackward&gt;)

# manual calculation for first value
print((-0.415*0.6824) + (1.2817*-0.5454) + (-0.5595))
&gt; -1.5417351799999999
</code></pre>
<p>Try to create this small example in the other frameworks and try to check, how the result is calculated.</p>
Resizing the image is taking very long

Navneet_Sajwan: <p>I have 3662 images in my trainset, batchsize is 128. When I am performing iter(dataloader).next() it takes about 20 seconds for this batch size. I narrowed down on the problem and found that resizing the image is taking up most of the time. I am using torchvision.transforsms.Resize() to resize the images.</p>

InnovArul: <p>Have you tried increasing the <code>num_workers</code> parameter of data loader?</p>
<aside class="quote quote-modified" data-post="2" data-topic="813">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/guidelines-for-assigning-num-workers-to-dataloader/813/2">Guidelines for assigning num_workers to DataLoader</a> 
  </div>
  <blockquote>
    Having more workers will increase the memory usage and that’s the most serious overhead. I’d just experiment and launch approximately as many as are needed to saturate the training. It depends on the batch size, but I wouldn’t set it to the same number - each worker loads a single batch and returns it only once it’s ready. 
num_workers equal 0 means that it’s the main process that will do the data loading when needed, num_workers equal 1 is the same as any n, but you’ll only have a single worker…
  </blockquote>
</aside>


Navneet_Sajwan: <p>Yes, I did. It became slower.</p>
Implementing "Same" padding for 3d convolution in Pytorch

Victor_Butoi: <p>Hello.</p>
<p>I was wondering if there were any helpful implementations of a 3d convolution which maintains size for irregular dimensions (ie not square).</p>

albanD: <p>Hi,</p>
<p>You mean irregular dimensions of the kernel? If so, yes you can set a per-dimension padding that will match the kernel size in that dimension.</p>
Comprehensive guide to set up GPU for PyTorch?

Ankush: <p>I am shifting to using PyTorch from Keras and TensorFlow. For both of those, the setup on Anaconda is fairly simple.</p>
<pre><code class="lang-auto">conda install keras-gpu
</code></pre>
<p>One command does quick work of installing all libraries including cudatoolkit and keras recognizes my GPU.<br>
I installed pytorch and tried running Chatbot example by pytorch on my GPU (GTX 1050 ti) but it doesn’t seem to recognize my device.<br>
Can anyone illustrate the process I need to follow for this?<br>
Do i need to install CudaToolKit separately?<br>
Thank you.</p>

ptrblck: <p>You just need to install the GPU driver on your machine. The binaries ship with CUDA, cudnn and other libraries. Have a look at the <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">website</a> for install instructions.</p>

Ankush: <p>Solution worked for me, I was installing only PyTorch via Anaconda Navigator. Using command line and the command specified on the page fixed it for me. Thanks.</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
</code></pre>
<p>This one.</p>
Will function torch.tanh() cause backpropagated grad be nan?

Deng_Tony: <p>Hi,</p>
<p>I am working on learning an image filter, but while the training process, after some iterations, the loss and all learned parameters became nan values.</p>
<p>Here is the main part of my code:</p>
<pre><code class="lang-auto">class net(nn.Module): 
        def __init__(self):
        super(net, self).__init__()
        self.encoder = models.resnet18(pretrained=True)
        self.fc1 = nn.Linear(1000, 256)
        self.fc2 = nn.Linear(256, 128)
        self.tanh_params = nn.Linear(128, 1)
        self.relu = nn.ReLU()
    def forward(self, input):
        input = self.encoder(input)
        input = self.relu(self.fc2(self.relu(self.fc1(input))))
        tanh_params = torch.exp(self.tanh_params(input))
        image = torch.tanh(tanh_params * (input - torch.mean(input))
        image = image * input
        return image
</code></pre>
<p>The code seems well but the result is unsatisified, are there something wrong or missed in this code block, or maybe this error is caused by my datasets?</p>
<p>Thank you to whom reading this problem and want to give me a hand.</p>

albanD: <p>Hi,</p>
<p>No tanh cannot return nans as it’s gradient is well defined everywhere.<br>
If this happens after some iterations, you should make sure your loss is well behaved and is not just diverging to very very large values until it gets nan.</p>

Deng_Tony: <p>Hi,</p>
<p>Thank for your reply, and is there possible that if tanh_params is large and the (input - torch.mean(input) block is very close to zero, then it will return a very large gradient and finally cause gradient explosion to make the situation I came across?</p>
<p>Is this comment stated a good way to check and debug my problem, [<a href="https://discuss.pytorch.org/t/how-to-check-for-vanishing-exploding-gradients/9019/3">How to check for vanishing/exploding gradients</a>]</p>

albanD: <p>Hi,</p>
<p>If I’m not mistaken, the gradient for tanh is never large right? It goes from 0 to 1?</p>
<p>I would recommend to print some values first to make sure that they become nan become they are too big.</p>
Torch.cuda.empty_cache() will occupy extra memory on GPU 0

dreamfly: <p>Hi, anyone who cares.</p>
<p>I was aware of the functionality of torch.cuda.empty_cache() that calling this function can release the GPU memory which is no longer bound to a python variable but still in the memory pool. However, when I place the model in any GPU other than GPU 0 and call torch.cuda.empty_cache(), besides releasing memory on the specified GPU, about 700MB memory of <strong>GPU 0</strong> will be occupied. It confused me so much and I can’t figure out why. Does anyone know about the internal machenism of this function and can explain why the strage memory comes out?</p>
<p>FYI, the approach of placing model to a certain GPU was specifying device index in pytorch code, rather than setting environment variable ‘CUDA_VISIBLE_DEVICES’ outside.</p>

albanD: <p>Hi,</p>
<p>If this is a one time thing, then this is most likely because it initializes the cuda context on GPU 0.<br>
We discourage the use of this function unless you do have a very good reason to release that memory. Could you explain your usecase for needing to clear the cache and not use <code>CUDA_VISIBLE_DEVICES</code> ?</p>

dreamfly: <p>Hi,</p>
<p>Thanks for your quick reply and explaination! I agree with you that the extra memory occupation is due to the cuda context being initialized on GPU 0. However, I was wondering if there was a solution that allowed me to specify which GPU to initialize the cuda context.</p>
<p>The usecase of torch.cuda.empty_cache() is that I need to call a subprocess to achieve an exhaustive test after 10 epoch training. Releasing GPU cached memory allows this subprocess to utilize them and avoid OOM.</p>
<p>Using environment variables is very convenient. As for CUDA_VISIBLE_DEVICES, I’m sure that CUDA_VISIBLE_DEVICES can save me from such a circumstance for now. However, it will modify the environment variable, and I may need other processes to perform some deeplearning tasks on other GPUs with potentially different framework, like TensorFlow. As a result, I’d like to find a way than can constrain the GPU utility locally, trying not to mess up with the environment variables.</p>

albanD: <p>Hi,</p>
<p>This sounds like a completely valid use case for <code>empty_cache()</code> !</p>
<p>For the env variable, you can set it temporarily for a single command by doing:<br>
<code>CUDA_VISIBLE_DEVICES=2 python your_command.py</code><br>
This way, only this python command will have the env variable defined.</p>
How to get a pair of image from different dir

Kevinkevin189: <p>suppose I have 2 dirs,A contains 70 images and B contains 50 images. Now I need to randomly choose one from A and one from B and then operate them(such as dotwise multiplication), could someone give me a code snippet to show how to write the code?I can’t overwrite the <strong>len</strong> and <strong>getitem</strong> properly.</p>

ptrblck: <p>How would you like to sample from these datasets having different number of samples?<br>
Would you like to get every sample or <code>A</code> and sample the pair from <code>B</code> with repetition?</p>
<p>Once you have both <code>Datasets</code> defined, you could wrap them in another <code>Dataset</code> class and use your sampling logic inside the <code>__getitem__</code>.<br>
Here is a code example:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, datasetA, datasetB):
        self.datasetA = datasetA
        self.datasetB = datasetB
        
    def __getitem__(self, index):
        xA, yA = self.datasetA[index]
        
        # implement your sampling logic here
        indexB = torch.randint(0, len(self.datasetB), (1,))
        xB, yB = self.datasetB[indexB]
        
        # your operations on the datasets
        x = xA * xB
        
        return x
        
    def __len__(self):
        # use the larget dataset's length
        return len(self.datasetA)


# Create dummy datasets
datasetA = TensorDataset(torch.randn(70, 3, 224, 224), torch.randint(0, 10, (70,)))
datasetB = TensorDataset(torch.randn(50, 3, 224, 224), torch.randint(0, 10, (50,)))
dataset = MyDataset(datasetA, datasetB)
</code></pre>
<p>I’m not sure, how you would like to use the targets of the datasets (if available).</p>

Kevinkevin189: <p>I’d like to use a pure image and an arbitrary mask to mask the original image so that i can get a masked image.<br>
So it means that I can combine A and B and the true len of dataset is len(A)*len(B)<br>
but I reckon ur code example a good one,the len(dataset)=len(A), and mask is random sampled by the randint func.</p>
Introduction singleton distribution in Normal distribution

blackPython: <p>I am trying to introduce new singleton dimension to a Normal distribution. I know I can access loc, scale and change them to get the desired distribution. I could use expand() to introduce new dimension as first dimensions. But what do I do if I want to introduce new dimension else where?</p>
<p>example:<br>
Given Normal of size [10,10,10], I want a distribution with shape [10,10,1,10]</p>

phan_phan: <p>The simplest way to do this is at the creation of the distribution, call <code>.unsqueeze(2)</code> on the <code>loc</code> and the <code>scale</code> argument tensors.</p>
<p>If you don’t want to do this, you can always create another distribution from the first one:</p>
<pre><code class="lang-auto">dist2 = torch.distributions.Normal(loc=dist1.loc.unsqueeze(2), scale=dist1.scale.unsqueeze(2))
</code></pre>
<p>I guess you are looking for a <code>.unsqueeze()</code> method for distributions, that would work like <code>.expand()</code>, but it does not exist. You could submit a PR or an issue on github but I don’t know if such a method would be very useful.</p>

blackPython: <p>Thank you very much… I ultimately ended up doing the samething…</p>
Distributed.init_process_group failure

MehdiA: <p>Hello,</p>
<p>I am trying to get started with torch.distributed with the following toy example, on a multi-gpu cluster :</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/narumiruna/pytorch-distributed-example/blob/master/toy/main.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/narumiruna/pytorch-distributed-example/blob/master/toy/main.py" target="_blank" rel="nofollow noopener">narumiruna/pytorch-distributed-example/blob/master/toy/main.py</a></h4>
<pre><code class="lang-py">import argparse
from random import randint
from time import sleep

import torch
import torch.distributed as dist


def run(world_size, rank, steps):
    for step in range(1, steps + 1):
        # get random int
        value = randint(0, 10)

        # group all ranks
        ranks = list(range(world_size))
        group = dist.new_group(ranks=ranks)

        # compute reduced sum
        tensor = torch.tensor(value, dtype=torch.int)
        dist.all_reduce(tensor, op=dist.ReduceOp.SUM, group=group)
</code></pre>

  This file has been truncated. <a href="https://github.com/narumiruna/pytorch-distributed-example/blob/master/toy/main.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>After running the program with the following command :</p>
<pre><code class="lang-auto">python3 main.py --init-method tcp://127.0.0.1:23456 --rank 0 --world-size 2
</code></pre>
<p>The program gets stuck in an the  dist.init_process_group on line 42. I am not really sure about the reason as no message gets displayed.</p>
<p>Thanks,</p>

SimonW: <p>it’s waiting for both ranks to reach that line to actually initialize the proc group.</p>

pietern: <p>Also see the docs for the <a href="https://pytorch.org/docs/stable/distributed.html#launch-utility" rel="nofollow noopener"><code>torch.distributed.launch</code></a> tool.</p>

meilu_zhu: <p>I have launched all the node, but the program still gets stuck in the init_process_group.</p>

meilu_zhu: <p>have solved. it is the problem about communication between nodes.</p>

alchemi5t: <p>How did you solve this?</p>

pietern: <p><a class="mention" href="/u/alchemi5t">@alchemi5t</a> If you’re running processes on two machines, they won’t be able to talk if you’re using localhost (127.0.0.1) for the address of rank 0 in the initialization method. It must be an IP that’s reachable from all other ranks. In the example here, rank 1 was trying to connect to rank 0 over 127.0.0.1.</p>

alchemi5t: <p>Hi <a class="mention" href="/u/pietern">@pietern</a>,</p>
<p>I was running it on one machine with 4 cards in it( trying to train only on 2). I fixed my problem by installing and using nvidia Apex(apex.parallel.multiproc).</p>
<p>Not sure why I had to do this, because I’ve seen people use the same script without any hacks like this.</p>

pietern: <p>Very odd. Especially since Apex also uses torch.distributed under the hood.</p>
Pytorch C++ model for raspberry pi

Javier_Naranjo: <p>Hi everybody!</p>
<p>I want to run my Pytorch model in a raspberri py with an executable. I have followed the official manual to create the executable (<a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a>).</p>
<p>I have noticed that all the flags in the Makefile are for PC. Is there a way to create the executable but compiling it for raspberry pi OS?</p>
<p>Thanks in advance</p>

MadmanNero: <p>Could you provide a link to the Makefile you’re talking about. Don’t really know about any specific flags but as long ‘raspberry pi OS’ means some kind of linux I guess things should compile…</p>
<p>I’m not sure I’m getting it right. Are you compiling on pi for pi or are you cross compiling?</p>

Javier_Naranjo: <p>I am cross compiling. Do you mean that if I compile following the instructions on the pi, the executable would work?</p>

MadmanNero: <p>Well I never did that but I’d actually assume so. Guess comilation on pi might take a long time though…</p>
'OSError: cannot identify image file' for custom dataset

KBacsa: <p>I have made a custom dataset class based on the ImageFolder class. When I load a batch of the dataset using :</p>
<p>inputs, labels = next(iter(dataloaders[‘train’]))</p>
<p>The batch is loaded correctly. However, when I try to load the data in my training loop similarly to the Transfer Learning tutorials, i.e :</p>
<p>for data in dataloaders[phase]:</p>
<p>I get the following error :<br>
‘OSError: cannot identify image file &lt;_io.BufferedReader name=’/path/to/file/sample.png’&gt;</p>
<p>This is not my first custom ImageFolder class, yet it is the first time that I am encountering this type of error. Any ideas as to what might be going wrong ?</p>
<p>PS : I am using 0.3.1</p>

chenglu: <p>I think some of the pathes of the image are not right. Try to print every image path before load that image, and see what path encounter this error.</p>

KBacsa: <p>Good call, I had a look at the problematic paths. Turns out part of my dataset is corrupted, that’s why the files weren’t loading.</p>
<p>Thanks you for your help !</p>

error: <p>we also have a same problem during running the code after training the dataset.What to do now ?</p>
Exactly same loss curve for different LR on multi-gpus

Shangyin_Gao: <p>I designed a model for object detection and trained on one gpu, the loss is decreasing and got a good result.</p>
<p>But, when I was trying to train the same network on multi-gpus, the loss is not decreasing. And more strange thing is that, the loss curve is exactly the same not matter the lr is 0.05 or 0.0005.</p>
<p>The dataset and training pipeline are used for other models and working good on multi-gpus.</p>
<p>Anybody got an idea, what the potential reason could be?</p>
<p>If you need more details, just write on the comment.</p>

spanev: <p>Hi <a class="mention" href="/u/shangyin_gao">@Shangyin_Gao</a>,</p>
<p>What’s the single-GPU LR? Did you also try to scale it such as <code>LR = LR * num_gpus</code>?</p>

Shangyin_Gao: <p>for the single I use 0.02, and for 2 gpus I tried LR from 0.0005 to 0.05</p>
Socket Timeout when wrapping model parallel with DDP

CrazyRundong: <p>Hi guys,</p>
<p>I was trying to wrap my model with DistributedDataParallel. My model is separated into 2 parts, each parnt runs on one GPU. Thus I followed the <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#combine-ddp-with-model-parallelism" rel="nofollow noopener">Combine DDP with Model Parallelism</a> in official tutorial, but after that I encountered with <code>RuntimeError: Socket Timeout</code>.</p>
<p>My codebase is basically like this:</p>
<pre><code class="lang-python"># fire tasks on SLURM cluster...
os.environ["MASTER_PORT"] = str(port)
os.environ["MASTER_ADDR"] = str(master_ip)
os.environ["WORLD_SIZE"] = str(n_tasks)
os.environ["RANK"] = str(proc_id)
dist.init_process_group(backend=dist.Backend.NCCL, timeout=timedelta(seconds=30))

# ...
class MyModel(nn.Module)
    def __init__(self, ..., device0, device1):
        # ...
        self.part_1.to(device0)
        self.part_2.to(device1)

# task0 get GPU{0, 1}, task1 get GPU(2, 3)...
d0 = torch.device(f"cuda:{rank * 2}")
d1 = torch.device(f"cuda:{rank * 2 + 1}")
model = MyModel(..., d0, d1)
# not all parameters are used in each iteration
ddp_model = DistributedDataParallel(model, , find_unused_parameters=True)

# ...
</code></pre>
<p>Invoking DDP did not raise any error, however after the <code>timeout</code> (30s in my setting), I encountered with following error:</p>
<pre><code class="lang-bash">Traceback (most recent call last):
  File "../tools/train_val_classifier.py", line 332, in &lt;module&gt;
    main()
  File "../tools/train_val_classifier.py", line 103, in main
    model, model_without_ddp = get_ddp_model(model, devices=(fp_device, q_device))
  File ".../quant_prob/utils/distributed.py", line 120, in get_ddp_model
    ddp_model = DistributedDataParallel(model, device_ids=devices, find_unused_parameters=True)
  File "/envs/r0.3.0/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 286, in __init__
    self.broadcast_bucket_size)
  File "/envs/r0.3.0/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 410, in _dist_broadcast_coalesced
    dist._dist_broadcast_coalesced(self.process_group, tensors, buffer_size, False)
RuntimeError: Socket Timeout
</code></pre>
<p>Seems that this error came from DDP implementation. I denifitely sure that I followed the official tutorial, and GPUs assiged to each tasks did not overlap. How can I fix this? Thank you so much~</p>

pietern: <p>This is a dup of <a href="https://github.com/pytorch/pytorch/issues/25767" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/25767</a>. Cross linking for posterity.</p>
Gain value of kaiming_initialization

07hyx06: <p>I know that the default initialization of layers are <code>torch.nn.init.kaiming_uniform(tensor,a=sqrt(5))</code><br>
where a is the gain value of nonlinearity.</p>
<p>In my VGG, all my nonlinearity is <code>ReLU</code>. So according to the paper of <code>kaiming_initialization</code>, i should set a=0. When i use that initialization, loss fly to NAN.</p>
<p>But when i use the default initialization, i trained my net successfully.</p>
<p>What’s the problem with that?Why can pytorch set default gain equal to sqrt(5)?</p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/nn-linear-default-weight-initialisation-assumes-leaky-relu-activation/46712/2">this answer</a>.</p>

07hyx06: <p>Thanks for solving my problem:laughing:</p>
Distributed send/recv (CUDA, MPI backend)

spaceinvader: <p>We’ve built PyTorch from source and tried to call send/recv, but failed. Could you tell me what I am doing wrong? A toy program is here:</p>
<pre><code class="lang-auto">  1 import os
  2 import socket
  3 import torch
  4 import torch.distributed as dist
  5 from torch.multiprocessing import Process
  6 
  7 
  8 def run(rank, size, hostname):
  9     print(f"I am {rank} of {size} in {hostname}")
 10     tensor = torch.zeros(1, device=torch.device('cuda:{}'.format(rank)))
 11     if rank == 0:
 12         tensor += 1
 13         # Send the tensor to process 1
 14         dist.send(tensor=tensor, dst=1)
 15     else:
 16         # Receive tensor from process 0
 17         dist.recv(tensor=tensor, src=0)
 18     print('Rank ', rank, ' has data ', tensor[0])
 19     
 20     
 21 def init_processes(rank, size, hostname, fn, backend='tcp'):
 22     """ Initialize the distributed environment. """
 23     dist.init_process_group(backend, rank=rank, world_size=size)
 24     fn(rank, size, hostname)
 25     
 26     
 27 if __name__ == "__main__":
 28     world_size = int(os.environ['OMPI_COMM_WORLD_SIZE'])
 29     world_rank = int(os.environ['OMPI_COMM_WORLD_RANK'])
 30     hostname = socket.gethostname()
 31     init_processes(world_rank, world_size, hostname, run, backend='mpi')
</code></pre>
<p>The cluster which I use is managed using slurm. Here is a list of loaded modules:</p>
<pre><code class="lang-auto">  1) /gpu/cuda-10.0          2) /mpi/hpcx-v2.4.0        3) /python/python-3.6.8    4) /python/pytorch-1.3.0,
</code></pre>
<p>where pytorch-1.3.0 is installed from source.</p>
<p>And this is how I call this script:</p>
<pre><code class="lang-auto">mpirun -np 2 python3 pytorch_distributed.py
</code></pre>
<p>The error looks as follows</p>
<pre><code class="lang-auto">I am 1 of 2 in gn10.zhores
I am 0 of 2 in gn10.zhores
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 11502 on node gn10 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
</code></pre>

pietern: <p>I don’t know what’s wrong, but I do know it’s likely going wrong somewhere in the MPI code.</p>
<p>PyTorch performs a runtime check for CUDA awareness of the MPI distribution before running any collective (including send/recv) with a CUDA tensor. You’re passing CUDA tensors, and not seeing this error, so the runtime check must be successful. What happens beyond that, in the MPI code, is beyond my purview. To be sure there is nothing wrong with your code you can try running it with CPU side tensors. If that passes, and it only fails with CUDA tensors, I’d try and run MPI in some kind of debug mode. Good luck!</p>
CUDA: out of memory on Multi-GPUs

zzuczy: <p>When I was training my model on single GPU(cuda:0), it just worked with batch_size==4. However, I keep the batch_size == 4 and train my model on 4 GPUs, it raise warning : <code>RuntimeWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().</code><br>
and error:<code>RuntimeError: CUDA out of memory. Tried to allocate 98.38 MiB (GPU 1; 15.89 GiB total capacity; 14.56 GiB already allocated; 33.25 MiB free; 150.76 MiB cached</code><br>
Hope somebody helps me. It’s confusing。</p>

zzuczy: <p>Traceback:</p>
<pre><code class="lang-auto">File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 142, in forward
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
  File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 147, in replicate
    return replicate(module, device_ids)
  File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/replicate.py", line 13, in replicate
    param_copies = Broadcast.apply(devices, *params)
  File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 21, in forward
    outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
  File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/cuda/comm.py", line 40, in broadcast_coalesced
    return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
RuntimeError: CUDA out of memory. Tried to allocate 11.38 MiB (GPU 3; 15.89 GiB total capacity; 13.87 GiB already allocated; 7.25 MiB free; 408.58 MiB cached) (malloc at /opt/conda/conda-bld/pytorch_1544174967633/work/aten/src/THC/THCCachingAllocator.cpp:231)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x45 (0x7f59fa323cc5 in /home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/lib/libc10.so)
</code></pre>
Training with BatchNorm in pytorch

sachinruk: <p>I’m wondering if I need to do anything special when training with BatchNorm in pytorch. From my understanding the <code>gamma</code> and <code>beta</code> parameters are updated with gradients as would normally be done by an optimizer. However, the mean and variance of the batches are updated slowly using momentum.</p>
<ol>
<li>So do we need to specify to the optimizer when the mean and variance parameters are updated, or does pytorch automatically take care of this?</li>
<li>Is there a way to access the mean and variance of the BN layer so that I can make sure it was changing while I trained the model.</li>
</ol>
<p>If needed here is my model and training procedure:</p>
<pre><code class="lang-python">def bn_drop_lin(n_in:int, n_out:int, bn:bool=True, p:float=0.):
    "Sequence of batchnorm (if `bn`), dropout (with `p`) and linear (`n_in`,`n_out`) layers followed by `actn`."
    layers = [nn.BatchNorm1d(n_in)] if bn else []
    if p != 0: layers.append(nn.Dropout(p))
    layers.append(nn.Linear(n_in, n_out))

    return nn.Sequential(*layers)

class Model(nn.Module):
    def __init__(self, i, o, h=()):
        super().__init__()
        
        nodes = (i,) + h + (o,)
        self.layers = nn.ModuleList([bn_drop_lin(i,o, p=0.5) 
                                     for i, o in zip(nodes[:-1], nodes[1:])])
        
    def forward(self, x):
        x = x.view(x.shape[0], -1)
        for layer in self.layers[:-1]:
            x = F.relu(layer(x))
            
        return self.layers[-1](x)
</code></pre>
<p>Training:</p>
<pre><code class="lang-python">for i, data in enumerate(trainloader):
    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
</code></pre>
<p>This is a x-post from <a href="https://stackoverflow.com/questions/57865112/training-with-batchnorm-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/57865112/training-with-batchnorm-in-pytorch</a></p>

ptrblck: <ol>
<li>
<p>No, since the running estimates are buffers and do not require gradients, you don’t have to pass them to the optimizer.</p>
</li>
<li>
<p>You can access them via <code>my_bn_layer.running_mean</code> and <code>my_bn_layer.running_var</code>.</p>
</li>
</ol>
Debugging inside an LSTM Layer

barakb: <p>Hi, I’m trying to debug the results I’m getting inside a LSTM layer, I mean inside the cells and the intermediate h_i and c_i, someone knows if it’s even possible? or has any trick to check it?<br>
Just to give context I’m trying to write a network using lstm cells in “low level” , so I need reference to debug.</p>
<p>Thanks!</p>
Issues with torch.utils.data.random_split()

Ariv_Arasan: <p>Hi,<br>
I’m not getting split when I use torch.utils.data.random_split. Here’s my code:</p>
<p>class DeviceLoader(Dataset):</p>
<pre><code>def __init__(self, root_dir, train=True, transform=None):
    self.file_path = root_dir
    self.train = train
    self.transform = transform
    self.file_names = ['%s/%s'%(root,file) for root,_,files in os.walk(root_dir) for file in files]
    self.len = len(self.file_names)
    self.labels = {'BP_Raw_Images':0, 'DT_Raw_Images':1, 'GL_Raw_Images':2, 'PO_Raw_Images':3, 'WS_Raw_Images':4}
    
def __len__(self):
    return(len(self.file_names))

def __getitem__(self, idx):
    file_name = self.file_names[idx]
    device = file_name.split('/')[5]
    img = self.pil_loader(file_name)
    if(self.transform):
        img = self.transform(img)
    cat = self.labels[device]            
    if(self.train):
        return(img, cat)
    else:
        return(img, file_name)
</code></pre>
<p>full_data = DeviceLoader(root_dir=’/kaggle/input/devices/dataset/’, transform=transforms, train=True)<br>
train_size = int(0.7*len(full_data))<br>
val_size = len(full_data) - train_size<br>
train_data, val_data = torch.utils.data.random_split(full_data,[train_size,val_size])</p>
<p>I get correct numbers for train_size and val_size, but when I do random_split, both train_data and val_data get full_data. There is no split happening.</p>
<p>Please help me fix this issue.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Nikronic: <p>Hi,</p>
<p>It looks weird, because I just ran the code using random <code>TensorDataset</code> and it is ok when you can get the length of your dataset, trainset, and valset correctly as you have achieved this too.</p>
<pre><code class="lang-auto">import torch
from torch.utils.data.dataset import TensorDataset, random_split

init_dataset = TensorDataset(
    torch.randn(100, 3, 5, 5),
)

train_size = int(0.7*len(init_dataset))
val_size = len(init_dataset) - train_size
train_data, val_data = torch.utils.data.random_split(init_dataset,[train_size,val_size])
</code></pre>
<p>What is the length of your dataset? I would like to do some experiments with your defined class.</p>

Ariv_Arasan: <p>Total length is 3001 <a class="mention" href="/u/nikronic">@Nikronic</a></p>

Nikronic: <p>I copied your code and ran, it works fine. Could you show the version of your environment packages?</p>
<p>Here is a script you can use: <a href="https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py" rel="nofollow noopener">https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py</a></p>
<p>Just run the code using <code>python collect_env.py</code> and paste output here.</p>

Ariv_Arasan: <p>I’m sorry. I was checking train_data.dataset.len instead of len(train_data). Split is happening correctly. No issues.</p>
Training on Multi-GPUs out of Memory

freelv: <p>When I was training my model using multi-gpus, I set the batch_size=8, and used two gpus.  This setting cannot make the best of my machine(4 GPUs and each with memory of 12G), however, if I set the batch_size bigger or use more GPUs, it would raise exception(GPU out of memory). It seems like that, the first GPU takes more assignment, so when I set the batch_size bigger or used more GPUs, the first GPU was out of memory. How can I fix it?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cfecd5d6e4edf6189c7fd10ae17b8dd6aa578647.png" data-download-href="https://discuss.pytorch.org/uploads/default/cfecd5d6e4edf6189c7fd10ae17b8dd6aa578647" title="%E5%9B%BE%E5%83%8F%201"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cfecd5d6e4edf6189c7fd10ae17b8dd6aa578647.png" alt="%E5%9B%BE%E5%83%8F%201" data-base62-sha1="tFofJcD6ItipCHF3eMPrrBBV2Bx" width="690" height="382" data-dominant-color="17435A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E5%9B%BE%E5%83%8F%201</span><span class="informations">714×396 79.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The image show my GPU’s state. I used GPU 2 and 3.</p>

alex.veuthey: <p>This is commonly observed in multi-gpu setups, because some kind of aggregation has to be performed on one selected GPU (in the default case, <code>cuda:0</code>). See <a href="https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551/3">this answer</a>, which explains the problem in a bit more details, and check the answer just after it for a possible solution (i.e. split the loss on multiple GPUs as well, not just the network).</p>
A bug of pytorch about optim.sgd(weight_decay)

Mandy: <p>When I was looking into the source codes of <a href="https://pytorch.org/docs/stable/optim.html?highlight=torch%20sgd#torch.optim.SGD" rel="nofollow noopener">optim.sgd()</a>, I found that</p>
<pre><code class="lang-auto">for p in group['params']:
      if p.grad is None:
            continue
      d_p = p.grad.data
      if weight_decay != 0:
            d_p.add_(weight_decay, p.data)
</code></pre>
<p>where I think that <code>weight_decay</code> is used for <code>L1 penalty</code>(maybe I was wrong? There is only a add_ between weight_decay and weight-p). But official doc points out that <code>weight decay (L2 penalty)</code>, is this a bug?</p>

albanD: <p>Hi,</p>
<p>Weight decay is an l2 penalty on the loss function.<br>
So when you take the derivative, it becomes just the value of the weights (times 2).<br>
So we add that directly to the gradients.</p>

dsuess: <p>For non-adaptive optimizers without momentum, weight decay is the same (up to the factor of 2 mentioned by <a class="mention" href="/u/alband">@albanD</a>) as an additional L2-penalty added to the loss function. For optimizers such as Adam, the empirical evidence suggests that simple weight decay as implemented above outperforms a proper L2-penalty, but the interpretation isn’t as clear. For more details, have a look at <a href="https://openreview.net/forum?id=Bkg6RiCqY7" rel="nofollow noopener">these reviews</a> as well as the article.</p>

Mandy: <p><a class="mention" href="/u/alband">@albanD</a> Thank you for your explanation, that makes sense.</p>

Mandy: <p>Thank you <a class="mention" href="/u/dsuess">@dsuess</a> for that helpful link and your explanation</p>
Libtorch Register_hook how to use?

21fa417e3fb06e56040c: <p>Hi.<br>
How to the register_hook function in Libtorch? Or do similar functions exist?</p>

maris_ancans: <p>Looking forward to this too. Can’t find any documentation <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

yf225: <p><a class="mention" href="/u/21fa417e3fb06e56040c">@21fa417e3fb06e56040c</a> <a class="mention" href="/u/maris_ancans">@maris_ancans</a> Please see <a href="https://github.com/pytorch/pytorch/blob/115494b00bf31549aa5227068bd66a3da9de469b/test/cpp/api/autograd.cpp#L535-L612" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/115494b00bf31549aa5227068bd66a3da9de469b/test/cpp/api/autograd.cpp#L535-L612</a> for example usage of <code>Tensor.register_hook</code>. We will update the documentation on this soon.</p>

21fa417e3fb06e56040c: <p>thanks a lot ! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"><img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>