akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Customize data loader using .npy files

username: Yupjun
cooked: <p>Hello, I am new to PyTorch.<br>
I’m struggling data loader part in PyTorch.</p>
<p>Objective<br>
I want to bring a pair of images and a binary value written on test.npy file. It will look like this.</p>
<p>To do that, I prepared the location of the image pair and a binary value written on loc.npy file.<br>
It will look like this.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6b9c390cfd4b645946cc9f96dcf122e7ffd2bbed.png" alt="image" data-base62-sha1="flXNiYJXpn5x1a0i4ePPPmiIlT7" width="445" height="388"></p>
<p>(My account allowed to upload only one image. sorry)</p>
<p>I saw a similar question in PyTorch forum. <a href="https://discuss.pytorch.org/t/upload-a-customize-data-set-for-multi-regression-task/43413/2">[1]</a></p>
<p>Based on that I wrote a code like this.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, data_paths_1,data_paths_2, label_list, transform=None, target_transform=None):
        # length of data_paths_1 and data_paths_2 is equal 
        self.data_paths_1 = data_paths_1 # source1 image dataset links ex) ["./AA/a.jpg","./BB/b.jpg","./CC/c.jpg",...]
        self.data_paths_2 = data_paths_2  # source2 image dataset links ex) ["./BB/b.jpg","./CC/c.jpg","./DD/d.jpg",...]
        self.label_lists = label_list  # label_list ex) [0,1,1,...]
        self.transform = transforms
        
    def __getitem__(self, index):
        x1 = Image.open(self.data_paths_1[index])
        x2 = Image.open(self.data_paths_2[index])
        if self.transform:
            x1 = self.transform(x1)
            x2 = self.transform(x2)
        y = self.label_lists[index]
        return x1,x2, ys

    def __len__(self):
        return len(self.data_paths_1)
</code></pre>
<p>My hardware environment is not really good so I can’t run it right now.<br>
Can you check if I missed something on that code line?<br>
What I want is that check the overall architecture of the code. you can ignore minor problems.<br>
I have no one to check my code so I uploaded my code in here. Sorry for just uploading code line in here.</p>

username: Mazhar_Shaikh
cooked: <p>Hi Yupjun,<br>
I understand that you don’t have the computing resource right now to test your code.<br>
Fortunately, there are online services that seek to help people with out access.<br>
One such website is <a href="https://colab.research.google.com/notebooks/welcome.ipynb" rel="nofollow noopener">Google Colab</a>. This should enable you to carry out tests on you code in a gpu environment. There is also support for pytorch. An tutorial for the same can be found <a href="https://jovianlin.io/pytorch-with-gpu-in-google-colab/" rel="nofollow noopener">here.</a></p>
<p>Hope this helps!</p>
Get 0 loss and not change

username: hagsmand
cooked: <pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 128, 3, 1, 1)
        self.relu1 = nn.ReLU()
        self.pool1 = nn.MaxPool2d(2)
        self.conv2 = nn.Conv2d(128, 256, 3, 1, 1)
        self.relu2 = nn.ReLU()
        self.pool2 = nn.MaxPool2d(2)
        
        self.fc1 = nn.Linear(256 * 56 * 56, 20)
        
        self.fc2 = nn.Linear(20, 1)
        
    def forward(self, x):
        
        x = self.conv1(x)
        x = self.relu1(x)
        x = self.pool1(x)
        x = self.conv2(x)
        x = self.relu2(x)
        x = self.pool2(x)
        
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        
        return x
      
model = Net()

criterion = nn.CrossEntropyLoss()
learning_rate = 0.1
optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)
train_loader = torch.utils.data.DataLoader(dataset=load_dataset(), batch_size=200, shuffle=True)
validation_loader = torch.utils.data.DataLoader(dataset=load_validate(), batch_size=5000, shuffle=True)
</code></pre>
<p>Try to fixed follow this link but nothing changed.<br>
<a href="https://discuss.pytorch.org/t/always-output-of-0/21784/4">https://discuss.pytorch.org/t/always-output-of-0/21784/4</a></p>

username: Tony-Y
cooked: <p>If your problem is a binary classification, use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss" rel="nofollow noopener">BCELoss</a> or <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss" rel="nofollow noopener"> BCEWithLogitsLoss</a>.</p>
BN weight is exactly a same value . multiprocessing and DistributedDataParallel

username: sftb644517236
cooked: <p>I already find the reason.<br>
pytorch1.2.0, ubuntu16,cuda10.<br>
I use  multiprocessing  and DistributedDataParallel in my  code  rewrited  as <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a>.</p>
<p>When I get the pth file, I print some  BN layers’ weight , I found it is exactly a same value.<br>
That not happend when I use single gpu to train.</p>
<p>Someone can help me?? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
BN weight is exactly a same value when use mp+DistributedDataParallel

username: sftb644517236
cooked: <p>I already find the reason.<br>
When I use single gpu to train my code, the bn weight is normal.</p>
<p>But when I use torch.multiprocessing and torch.nn.parallel.DistributedDataParallel for 4 gpus, it’s wired that all BN weight in a layer is the same value.</p>
<p>I use pytorch1.2.0, ubuntu16,cuda10.</p>
<p>I use multiprocessing and DistributedDataParallel in my code like <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a>.</p>
<p>Someone can help me??</p>
Is this possible to give gradient clipping in a specific layer?

username: Uk_Jo
cooked: <p>I have been trying to implement <a href="https://arxiv.org/pdf/1507.06527.pdf" rel="nofollow noopener">DRQN</a>. In the last page, I can see “LSTM’s gradients were clipped to a value of ten to ensure learning stability” in  Appednix C : Experimental Details. Is this possible? Only thing I know is to give clipvalue as a parameter of optimizer.</p>

username: ptrblck
cooked: <p>Yes, <a href="https://pytorch.org/docs/stable/nn.html#clip-grad-norm" rel="nofollow noopener">torch.nn.utils.clip_grad_norm_</a> will clip the gradients of the passed parameters.</p>
Getting started with open source for Pytorch?

username: Victor_Butoi
cooked: <p>Hello all,</p>
<p>I’ve been working with Pytorch for a quite a time now and I’m interested in contributing to its open source. I was wondering if anyone had any tips for really getting involved or areas where development is pretty hot?</p>
<p>thanks!</p>

username: ptrblck
cooked: <p>Hi Victor!</p>
<p><a class="mention" href="/u/tom">@tom</a> published a <a href="https://lernapparat.de/fixing-your-first-pytorch-bug/" rel="nofollow noopener">blog post</a> about fixing your first bug in PyTorch recently (with a Video showing him fixing one bug <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ), which might be a good starter.</p>
<p>I would also suggest to have a look at the <a href="https://github.com/pytorch/pytorch/projects/5" rel="nofollow noopener">usability / simple-fixes category</a> in the GitHub issues.</p>
<p>Let us know, if you cannot decide where to start <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
CrossEntropyLoss value do not decrease

username: skyunyoo
cooked: <p>I’m now dealing with images with 0 and 3 pixel values.<br>
Values of 0 and 1 are unbalance, and often have only a value of 0 in one image.</p>
<p>I’m trying 2label(background+desired) segmentation with UNet.<br>
Data info as follow -<br>
X : numpy_images (dtype:uint8)<br>
Y : numpy_masks (dtype:uint8)</p>
<p>X_max --&gt; 255,  Y_max --&gt; 3<br>
X_shape --&gt; (480, 512, 512),  Y_shape --&gt; (480, 512, 512)</p>
<p>I converted X,Y to PIL using <code>x = x.fromarray(x)</code>for applying augmentation and also X,Y were converted to tensor <code>x = TF.to_tensor(x).float()</code>, <code>y = TF.to_tensor(y).long()</code><br>
X_tesnsor_shape --&gt; torch.Size([1, 1, 512, 512])<br>
Y_tesnsor_shape --&gt; torch.Size([1, 512, 512])<br>
output_tensor_shape --&gt; torch.Size([1, 2, 512, 512])</p>
<p>My training loop as follow:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.99)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)


def fit(epoch,model,data_loader,phase='train',volatile=False):
    if phase == 'train':
        exp_lr_scheduler.step()
        model.train()
    if phase == 'valid':
        model.eval()
    running_loss = 0.0
    for batch_idx , (data,target) in enumerate(data_loader):
        inputs,target = data.cpu(),target.cpu()
        if is_cuda:
            inputs,target = data.cuda(),target.cuda()
        inputs , target = Variable(inputs),Variable(target)
        if phase == 'train':
            optimizer.zero_grad()

        output = model(inputs)
        loss = criterion(output,target)
        
        running_loss += loss.data.item()
        
        if phase == 'train':
            loss.backward()
            optimizer.step()
    
    loss = running_loss/len(data_loader.dataset)
    
    print('{} Loss: {:.4f}'.format(
                phase, loss))
    return loss
</code></pre>
<h2>but, I got<br>
`Epoch 0/4</h2>
<p>train Loss: 0.1523<br>
valid Loss: 0.1041</p>
<h2>Epoch 1/4</h2>
<p>train Loss: 0.0815<br>
valid Loss: 0.0792</p>
<h2>Epoch 2/4</h2>
<p>train Loss: 0.0791<br>
valid Loss: 0.0792</p>
<h2>Epoch 3/4</h2>
<p>train Loss: 0.0791<br>
valid Loss: 0.0792</p>
<h2>Epoch 4/4</h2>
<p>train Loss: 0.0791<br>
valid Loss: 0.0792<br>
`<br>
The loss value is no longer decreased even if epoch is increased.</p>
<p>Is the loss no longer decreased because the data is unbalanced?.</p>

username: Prerna_Dhareshwar
cooked: <p>Your loss could have stagnated because your learning rate is too high, I suggest reducing your learning rate (use an LR scheduler to do it) and then continuing training.</p>
Is there a version of the ChatBot tutorial that uses dataloader class?

username: pinocchio
cooked: <p>I wanted to use the dataloader class for the ChatBot tutorial and use the best practices for padding, packing, variable size sequences with a real model (ideally using GPU as much as possible). Is there an example of how to use the dataloader class like that with the chatbot tutorial that is fully functional?</p>
<hr>
<p>cross posted: <a href="https://www.quora.com/unanswered/Is-there-a-version-of-the-ChatBot-tutorial-that-uses-dataloader-class" rel="nofollow noopener">https://www.quora.com/unanswered/Is-there-a-version-of-the-ChatBot-tutorial-that-uses-dataloader-class</a></p>
How does one padd a tensor of 3 dimensions?

username: pinocchio
cooked: <p>I was trying to use the built in padding function but it wasn’t padding things for me for some reason. This is my reproducible code:</p>
<pre><code class="lang-auto">import torch

def padding_batched_embedding_seq():
    ## 3 sequences with embedding of size 300
    a = torch.ones(1, 4, 5) # 25 seq len (so 25 tokens)
    b = torch.ones(1, 3, 5) # 22 seq len (so 22 tokens)
    c = torch.ones(1, 2, 5) # 15 seq len (so 15 tokens)
    ##
    sequences = [a, b, c]
    batch = torch.nn.utils.rnn.pad_sequence(sequences)

if __name__ == '__main__':
    padding_batched_embedding_seq()
</code></pre>
<p>error message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "padding.py", line 51, in &lt;module&gt;
    padding_batched_embedding_seq()
  File "padding.py", line 40, in padding_batched_embedding_seq
    batch = torch.nn.utils.rnn.pad_sequence(sequences)
  File "/Users/rene/miniconda3/envs/automl/lib/python3.7/site-packages/torch/nn/utils/rnn.py", line 376, in pad_sequence
    out_tensor[:length, i, ...] = tensor
RuntimeError: The expanded size of the tensor (4) must match the existing size (3) at non-singleton dimension 1.  Target sizes: [1, 4, 5].  Tensor sizes: [3, 5]
</code></pre>
<p>any idea?</p>
<hr>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/57116878/how-does-one-padd-a-tensor-of-3-dimensions-in-pytorch" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3167448/pinocchio" target="_blank" rel="nofollow noopener">
    <img alt="Pinocchio" src="https://www.gravatar.com/avatar/ba92064b084638cb903cbd271120be64?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/57116878/how-does-one-padd-a-tensor-of-3-dimensions-in-pytorch" target="_blank" rel="nofollow noopener">How does one padd a tensor of 3 dimensions in Pytorch?</a>
</h4>

<div class="tags">
  <strong>python, machine-learning</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/3167448/pinocchio" target="_blank" rel="nofollow noopener">
    Pinocchio
  </a>
  on <a href="https://stackoverflow.com/questions/57116878/how-does-one-padd-a-tensor-of-3-dimensions-in-pytorch" target="_blank" rel="nofollow noopener">05:06PM - 19 Jul 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: pinocchio
cooked: <p>do I really have to unsequeeze it first for it to work?</p>
Changing the convolution behaviour

username: Mafuyu_Aizawa
cooked: <p>Hello there.</p>
<p>I would like to ask if it is possible to change the convolution behaviour in pytorch with the following goal:</p>
<p>In convolutional operations, for specific numbers occured in multiplication in conv, replace their result with specific number, for example: if 5 * 3 is found, replace its result with 12 instead of 15. Basically I want to check each operation in convolution and change their result according to their value.</p>
<p>I first tried to use nn.functional.unfold and define a custom dot product function to replace conv2d but it is extremely slow (2000X slow-down for my platform).</p>
<p>I think I need to define a custom Function or Module by imitating the soure code of conv2d, but no luck on figuring out how conv2d works in the bottom layer (operation level). Is it possible to do this in python, or I have to dive into the C++ codes?</p>
<p>Thanks,</p>
Where is the source code for kl_div?

username: John_Deterious
cooked: <p>I’m looking @ <a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/functional.html</a></p>
<p>to see torch.nn.kl_div but it is just a class that wraps a function called torch.kl_div<br>
I can’t find that original function. I basically made my own function and it spits out different results from what Pytorch built-in is spitting so I’m wondering how does it look like.</p>
<p>For completition, here’s my code:</p>
<pre><code class="lang-auto">def my_kl(predicted, target):
    return -(target * t.log(predicted.clamp_min(1e-7))).sum(dim=1).mean() - \
           -1*(target.clamp(min=1e-7) * t.log(target.clamp(min=1e-7))).sum(dim=1).mean()

</code></pre>
<p>Which I believe is spitting out correct results LOL</p>

username: tom
cooked: <p>kl_div and the CPU backward are in <a href="https://github.com/pytorch/pytorch/blob/7cc029cb75c292e93d168e117e46a681ace02e79/aten/src/ATen/native/Loss.cpp#L69-L94" rel="nofollow noopener"><code>aten/src/ATen/native/Loss.cpp</code></a>, the cuda backward is in <code>aten/src/ATen/native/cuda/Loss.cu</code>.<br>
I once tried to write this up more generally:<br>
<a href="https://lernapparat.de/selective-excursion-into-pytorch-internals/" class="onebox" target="_blank" rel="nofollow noopener">https://lernapparat.de/selective-excursion-into-pytorch-internals/</a></p>
<p>Best regards</p>
<p>Thomas</p>
Convolutional - lstm batch dimensions question

username: pytorchnewbie
cooked: <p>Hi!</p>
<p>I want to pass in a sequence of images and translate them into another sequence of images using an LSTM. Each image is passed into a CNN layer which decreases the height and width to 1 and the channel to 128, we reshape that to a 1d 128 and pass the 128 1d vector into the lstm cell.</p>
<p>More concretely, I can pass in b x c x h x w to the convolutional layers/network, which then outputs (after squeezing) b x 128 x 1 x 1 which is effectively b x 128. 128 will be the lstm’s input_size. The shape that the lstm expects is  (seq_len, batch, input_size), but from the CNN output we only have batch and input_size, not seq_len. This is because the CNN doesn’t take multiple images at a time. Is the solution to write a for loop in the training code for the CNN seperately on each image, and then pass that input to the lstm model? I was hoping there would be a way to do it faster/better.</p>
<p>Thanks!</p>

username: Paulo_Mann
cooked: <p>My understanding is that your <code>batch_size</code><sub>CNN</sub> for the CNN is not the same thing as the <code>batch_size</code><sub>LSTM</sub> for the LSTM. I think, and I can be completely wrong, that in the context of LSTMs your <code>batch_size</code><sub>LSTM</sub> is 1 while the <code>seq_len</code> <code>== batch_size</code><sub>CNN</sub>.</p>
<p>Hope this helps!</p>