akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How could I set slurm parameters

coincheung: <p>Hi,</p>
<p>I am using some gpu cluster to train my model. I am using the <code>torch.distributed</code> module to manipulate the two gpus on one node. I set the dataloader like this:</p>
<pre><code class="lang-auto">ds = CityScapes(cfg, mode='train')                                     
 dl = DataLoader(ds,                                                    
     batch_size=8,                            
     shuffle=False,
     sampler=sampler,
     num_workers=4,
     pin_memory=True,
     drop_last=True)

</code></pre>
<p>This means that I will open 4 processes on each gpu to loader the data. Considering the process that is used to train the model, on this node with 2gpus, there would be 2(1+4)=10 processes in this node. So shall I start my training like this?</p>
<pre><code class="lang-auto">srun -p gpu --gres=gpu:2 --ntasks_per_node=10 python -m torch.distributed.launch --nproc_per_node=2 train.py
</code></pre>
<p>Is this the correct way to make it work please?</p>
How to reverse the operation of grid_sample?

huagl: <p>If we get two images, target image <code>tgt_img</code> and source image <code>src_img</code>. And, we also know the grid<br>
<code>grid</code> from source image to target image. Thus, we can use the <code>F.grid_sample</code> to generate target image from source image.</p>
<pre><code class="lang-auto">tgt_img = F.grid_sample(src_img, grid)
</code></pre>
<p>Is there any good idea to get the <code>inv_grid</code> that are from target image to source image. So that:</p>
<pre><code class="lang-auto">inv_grid = ?
src_img =  F.grid_sample(tgt_img, inv_grid)
</code></pre>
<p>I have tried to read the data in grid and generate a <code>inv_grid</code>, but I want to know if there any better and faster method.</p>

huagl: <p>any guys can help me? <img src="https://discuss.pytorch.org/images/emoji/apple/mask.png?v=7" title=":mask:" class="emoji" alt=":mask:"></p>

Liqian_Ma: <p><a class="mention" href="/u/huagl">@huagl</a> Have you got any solution for the inverse grid_sample?</p>

huagl: <p>Sorry, I haven’t checked the message for a long time. I still just read the data in <code>grid</code> and generate a <code>inv_grid</code> to solve this problem.<br>
(PS. your name is the same as my senior’s who is in our lab. hh~)</p>
How to load data efficiently in online learning

fabian: <p>I’m working in the reinforcement learning. In RL, we need to grow the replay buffer and draw samples in an online manner.</p>
<p>I can’t figure out a way to use the <code>DataLoader</code> and <code>Sampler</code> concepts in PyTorch. They seem to only support batch training.</p>

alexis-jacq: <p>You must make your own replay buffer by hand (because the quality of RL sometimes strongly depends on the quality of the buffer, you can’t impose a generic method). However, the most standard approach (not the most efficient) is to save the last 1e6 examples of transitions and to uniformly sample from them. You have one simple example here:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pranz24/pytorch-soft-actor-critic/blob/master/replay_memory.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pranz24/pytorch-soft-actor-critic/blob/master/replay_memory.py" target="_blank" rel="nofollow noopener">pranz24/pytorch-soft-actor-critic/blob/master/replay_memory.py</a></h4>
<pre><code class="lang-py">import random
import numpy as np

class ReplayMemory:
    def __init__(self, capacity):
        self.capacity = capacity
        self.buffer = []
        self.position = 0

    def push(self, state, action, reward, next_state, done):
        if len(self.buffer) &lt; self.capacity:
            self.buffer.append(None)
        self.buffer[self.position] = (state, action, reward, next_state, done)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        batch = random.sample(self.buffer, batch_size)
        state, action, reward, next_state, done = map(np.stack, zip(*batch))
        return state, action, reward, next_state, done

</code></pre>

  This file has been truncated. <a href="https://github.com/pranz24/pytorch-soft-actor-critic/blob/master/replay_memory.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Pytorch tensor inverse slower on GPU than CPU

satyakrishnagorti: <p>Hi I observe that inverse operation on GPU is slower than CPU</p>
<p>I am not sure if this is the right way to profile but here is what I have done</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import time
&gt;&gt;&gt; gpu_tensor = torch.randn(3,3).cuda()
&gt;&gt;&gt; cpu_tensor = torch.randn(3,3)

&gt;&gt;&gt; def test1():
        s = time.time()
        for i in range(50):
            torch.inverse(cpu_tensor)
        e = time.time()
        print(e - s)

&gt;&gt;&gt; def test2():
        s = time.time()
        for i in range(50):
            torch.inverse(gpu_tensor)
        e = time.time()
        print(e - s)

&gt;&gt;&gt; test1()
&gt;&gt;&gt; 0.000229120254517

&gt;&gt;&gt; test2()
&gt;&gt;&gt; 0.310909032822
</code></pre>
<p>Any idea why?</p>

ptrblck: <p>If you are timing cuda ops, you should add a <code>torch.cuda.synchronize()</code> before starting and stopping the timer.<br>
The first cuda call needs some time to init CUDA, thus your timing might measure this time as well.#<br>
Also CUDA ops are called asynchronously, so that your main thread can continue its execution while the GPU is busy.<br>
Could you add it to your code and run it again?</p>

satyakrishnagorti: <aside class="quote no-group quote-post-not-found" data-username="satyakrishnagorti" data-post="3" data-topic="20815">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/898d66/48.png" class="avatar"> satyakrishnagorti:</div>
<blockquote>
<p>Ok so I added torch.cuda.synchronize() before the timer.</p>
</blockquote>
</aside>
<p>Ok so I added torch.cuda.synchronize() before the timer.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import time
&gt;&gt;&gt; gpu_tensor = torch.randn(3,3).cuda()
&gt;&gt;&gt; torch.cuda.synchronize()
&gt;&gt;&gt; cpu_tensor = torch.randn(3,3)


&gt;&gt;&gt; def test1():
        s = time.time()
        for i in range(50):
            torch.inverse(cpu_tensor)
        e = time.time()
        print(e - s)

&gt;&gt;&gt; def test2():
        s = time.time()
        for i in range(50):
            torch.inverse(gpu_tensor)
        e = time.time()
        print(e - s)

&gt;&gt;&gt; test1()
&gt;&gt;&gt; 0.000277042388916

&gt;&gt;&gt; test2()
&gt;&gt;&gt; 0.325435876846
</code></pre>
<p>I still don’t see any improvement.</p>

SimonW: <p>[3, 3] is too small for gpu to be better</p>

deepmo: <p>I test [1024,1024] with range(5)， GPU is still slower than CPU.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; test1()
&gt;&gt;&gt;0.08186817169189453

&gt;&gt;&gt; test2()
&gt;&gt;&gt;0.9191479682922363
</code></pre>
<p>Any suggestion?.</p>
How to re-normalize an image tensor back to a given range?

XavierXiao: <p>Hi! I have images from a dataloader which is normalized by a specific setting of mean and std, for example:</p>
<pre><code class="lang-auto">transform = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize(mean=(0.485, 0.456, 0.406),
                       std=(0.229, 0.224, 0.225))
            ])
</code></pre>
<p>Since the normalization is not as simple as taking values from 0 to 1 and normalizing to -1 and 1, I don’t know how I can re-nornalize the image to 0 and 1. Say I have a batch of data from this data loader, which is a batch_size<em>3</em>H*W tensor, how do I normalize this tensor so that each image are now in range (0,1)?</p>
<p>Thanks！</p>

JuanFMontesinos: <p>Just delete the normalization part and they will be in that range.</p>
Clarification on `torch.autograd.grad`'s `only_inputs` boolean?

mattrobin: <p>I have a few questions about what <code>only_inputs</code> for <a href="http://pytorch.org/docs/master/autograd.html#torch.autograd.grad" rel="nofollow noopener"><code>torch.autograd.grad</code></a> really does. When <code>True</code> does this mean that gradients are computed for <em>no</em> other leaves? Or that no gradient flows <em>past</em> the inputs. Similarly, does <code>False</code> mean that gradient will flow <em>beyond</em> the inputs to leaves further in the network? Or does <code>False</code> mean it will flow to everything between inputs and outputs, but not past inputs?</p>
<p>In particular, what happens when using <code>allow_unused=True</code> where all inputs are “unused”? Can I pass <code>require_grad=False</code> variables as “unused” as input, then use <code>grad</code> to compute gradients on leaf nodes based on an output dependent on the unused inputs?</p>

Xingyi_Zhou: <p>Also interested in this, can anyone help?</p>
Use two imagefolders in dataset

Vali_Calinescu: <p>I am trying to create a dataset with an imagefolder as input and another imagefolder to use as labels. Is there any way I can have them both in the same DataLoader?</p>
<p>My code is as follows:</p>
<pre><code class="lang-auto">dataset = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
dataset_labels = torchvision.datasets.ImageFolder(root=TRAIN_LABELS_PATH, transform=TRANSFORM_IMG)
</code></pre>
<p>I would like to be able to integrate <code>dataset_labels</code> into <code>dataset</code>, so that I can then just create a DataLoader from <code>dataset</code>.</p>

AlbertZhang: <p>use <code>torch.utils.data.ConcatDataset</code></p>
Convert Full ImageNet Pre-trained Model from MXNet to PyTorch

ahkarami: <p>I think this link would be helpful (for deep learning researchers/Engineers):<br>
<a href="https://blog.paperspace.com/convert-full-imagenet-pre-trained-model-from-mxnet-to-pytorch/" rel="nofollow noopener"> Convert Full ImageNet Pre-trained Model from MXNet to PyTorch</a><br>
In this tutorial the MXNet ResNet-152 model which is trained on ~ <strong>14M images</strong> and <strong>11K Labels</strong> converted to PyTorch via MMdnn framework.<br>
I hope this tutorial would be helpful for PyTorch community.</p>
t-SNE for Pytorch

shivangi: <p>Is there some workaround to do t-sne visualization of my autoencoder latent space in pytorch itself without using sklearn as it is relatively slow</p>

Diego: <p>You can use <a href="https://github.com/CannyLab/tsne-cuda" rel="nofollow noopener">this implementation</a>. It uses CUDA to speed things up.  Or <a href="https://github.com/DmitryUlyanov/Multicore-TSNE" rel="nofollow noopener">this one</a> that allows multiprocessing. The speedup will be quite significant</p>
Backward propagation with custom denormalization layer

Lukasz_Kurowski: <p>Hi everyone, I am trying to port my dataset normalization and output normalization into a PyTorch graph. It normalizes input and denormalizes output based on the input.</p>
<pre><code class="lang-auto">import torch


class NormDenorm(torch.nn.Module):
    def __init__(self, module):
        super(NormDenorm, self).__init__()
        # Indices of channels for denormalization.
        self.indices = torch.tensor([0])
        self.add_module('capsule', module)

    def print(self, *args):
        if not self.training:
            print(*args)

    def forward(self, x):
        min_x, _ = torch.min(x, dim=-1, keepdim=True)
        max_x, _ = torch.max(x, dim=-1, keepdim=True)
        max_min_diff = torch.add(max_x - min_x, 1e-4)

        # Normalize input for computation.
        x_norm = torch.add(x - min_x, 1e-4)
        x_norm = x_norm / max_min_diff

        # Perform main module operations.
        y = self.capsule(x_norm)

        # Denormalize output based on selected input.
        min_x = torch.take(min_x, self.indices)
        max_x = torch.take(max_x, self.indices)
        max_min_diff = torch.take(max_min_diff, self.indices)
        y_denorm = y * max_min_diff + min_x
        return y_denorm

model = NormDenorm(MyModel())
</code></pre>
<p>It takes arbitrary numerical data and normalizes it to [0, 1] range.</p>
<p>Is it generally a bad idea and if not how should I approach writing <code>backward</code> function for this case?</p>
Kernel size compare

wonchulSon: <p>Let’s suppose to the Input image data size is <code>1x10</code> (WxH) in CNN.<br>
And we have two kernel size.<br>
One is <code>1x3</code> and the other one is <code>3x3</code>.<br>
With <code>zero-padding</code>, to get output size is as same as input size <code>1x10</code>.<br>
Then, the both performance must be <code>no different</code>?</p>
<p>Any opinion would be helpful for me.<br>
Thank you.</p>

tom: <p>You’ll see that the result does not depend on the 0th and 2nd slice of the 3x3 kernel, so effectively, the result is the same as if you use the 1x3 kernel given by <code>w[:, :, 1, :]</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

wonchulSon: <p><a class="mention" href="/u/tom">@tom</a><br>
I understood.<br>
Then what if input size is (32x32)x1 (W x H) x N and kernel size is 1x3<br>
compare to input size is (1x32)x32 and same kernel size 1x3.</p>
<p>In this case too, the performance would be similar??</p>

tom: <p>Sigh. Is that even a PyTorch question - it has TF-style WHN layout (and no C)?<br>
I’d recommend checking out one of the <a href="https://github.com/vdumoulin/conv_arithmetic" rel="nofollow noopener">excellent</a> <a href="http://colah.github.io/posts/2014-07-Understanding-Convolutions/" rel="nofollow noopener">tutorials</a> on how convolution works and think about what values in the kernel and in your tensor interact.</p>
<p>Best regards</p>
<p>Thomas</p>
How to load Model in C++?

RuFAI: <p>Hello world,</p>
<p>I want to run my Pytorch model written in python in C++. The model is serialized an this part works. When it comes to load the model in C++ I am very confused with the given tutorial to this topic.</p>
<p>Why does the main function take the arguments argc and argv?<br>
Why do they use cmake?<br>
Where is the path of the  model given?<br>
What is this thing actually doing?<br>
Don’t I understand Pytorch or do I have a problem with C++? I never dealt with it bevor and every solution I find causes 3 other problems.</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.

#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
    return -1;
  }

  // Deserialize the ScriptModule from a file using torch::jit::load().
  std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load(argv[1]);

  assert(module != nullptr);
  std::cout &lt;&lt; "ok\n";
}
</code></pre>
Tensor's reference vs name

dragen: <p>hi, all,<br>
for a simple instance：</p>
<pre><code class="lang-auto">class MyModule(nn.Module):

    def __init__(self):
        super(MyModule, self).__init__()

        self.register_parameter('t', nn.Parameter(torch.tensor([3.])))


net = MyModule().cuda()

print(net.t, net.t.name, net.t.device)
</code></pre>
<p>and the output is</p>
<pre><code class="lang-auto">tensor([3.], device='cuda:0', requires_grad=True) None cuda:0
</code></pre>
<p>I wonder the parameter <code>name</code> for function <code>register_parameter</code> is indeed a reference. So what does tensor’s property <code>name</code> actually means?</p>

Oli: <p>Hello <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
I believe that the name is just what you want to name the attribute within the module so that you could, in fact, do the <code>print(net.t)</code>. So instead of doing self.t = nn.Parameter(…) you can do it the way you did it. Not sure if there is any difference.</p>
<p>Or is that what you are asking?</p>
What kind of attention mechanism used in Seq2Seq tutorial

keysmash97: <p>Hi, i’ve read the seq2seq translation <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#the-decoder" rel="nofollow noopener">tutorial</a> and i still dont know what kind of attention mechanism author use in the tutorial. It kinda confused me about Bahdanau way and Luong way. Are there any better explaination about the code in tutorial ?</p>

vdw: <p><a href="https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb" rel="nofollow noopener">I find this Seq2Seq example more useful</a>. As far as I can tell, it expands on the linked tutorial by using different attention mechanisms (Bahdanau &amp; Luong).</p>
<p>The tutorial uses only the Bahdanau approach.</p>

keysmash97: <p>thks for your support <img src="/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Porting simple 0.4 CUDA/C extensions to 1.0

hinken: <p>I’ve tried to find a proper migration guide for porting pyTorch C-extensions to pyTorch 1.0+ but the only things I find are confused forum posts and comprehensive C+±extension guides of hundreds of pages for writing advanced ATen integrations… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a supersimple setup, just a few very simple CUDA kernels I need to call from pyTorch and have no interest in bells-and-whistles C++ integration with ATen, autograd or whatever, they worked with 10 rows of C and 10 rows of Python in 0.4 <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> So was hoping for someone else that went through this migration before and could suggest a good procedure.</p>
<p>Basically what I’ve done so far is I have a build-script in Python using torch’s create_extension:</p>
<p>from torch.utils.ffi import create_extension</p>
<p>ffi = create_extension(<br>
‘nc’,<br>
headers=[‘custom_c_stuff.h’],<br>
sources=[‘custom_c_stuff.c’],<br>
define_macros=[(‘WITH_CUDA’, None)],<br>
relative_to=<strong>file</strong>,<br>
with_cuda=True,<br>
extra_objects=[‘custom_cuda_stuff.so’,  ],<br>
include_dirs=[osp.join(abs_path, ‘.’)]<br>
)<br>
ffi.build()</p>
<p>and in the custom_stuff.c/h files I do something like this:</p>
<p><span class="hashtag">#include</span> &lt;THC/THC.h&gt;</p>
<p>int custom_c_function(THCudaTensor *in_tensor, THCudaTensor *out_tensor)<br>
{<br>
call_custom_cuda_stuff_function();<br>
}</p>
<p>What do you think, is this easily portable to 1.0? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Very grateful for hints <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Device of resulting tensor for addition

zhaopku: <p>I am using PyTorch 1.1 and I found the following problem:</p>
<blockquote>
<p>&gt;&gt;&gt; import torch</p>
<p>&gt;&gt;&gt; torch.cuda.is_available()</p>
<p>True</p>
<p>&gt;&gt;&gt; x = torch.tensor(1).cuda()</p>
<p>&gt;&gt;&gt; x</p>
<p>tensor(1, device=‘cuda:0’)</p>
<p>&gt;&gt;&gt; y = torch.tensor(2)</p>
<p>&gt;&gt;&gt; x+y</p>
<p>tensor(3, device=‘cuda:0’)</p>
<p>&gt;&gt;&gt; y+x</p>
<p>tensor(3)</p>
<p>&gt;&gt;&gt; (y+x).device</p>
<p>device(type=‘cpu’)</p>
<p>&gt;&gt;&gt; (x+y).device</p>
<p>device(type=‘cuda’, index=0)</p>
</blockquote>
<p>x is a cuda tensor while y is a cpu tensor. The device of the resulting tensor is different when the order of addition changes, is it an expected behavior? And when should the resulting tensor on gpu or on cpu?</p>
Torchscript C++ Extension List type

half-potato: <p>What is the type that torch considers as the proper list type for C++ when going from Torchscript to C++?</p>
<p>I am trying to implement a function in C++ for torchscript but I need a list type for strings and I can’t seem to figure out how to create one.</p>

half-potato: <p>It’s a std::vector&lt;std::string&gt;. Sometimes just try shit and hope that the programmers were nice.</p>
How can be the gradient values for each input of batch accessed?

Seungyoung_Park: <p>Hello,</p>
<p>I am working to get the gradient values for each input from the batch simultaneously.</p>
<p>More specifically, I need the mean of squared gradients of inputs from the batch.</p>
<p>I think it is possible to get it if the number of GPUs are the same as that of batch size, because each GPU calculates each input of the batch.</p>
<p>Here is the simple script.</p>
<pre><code>import numpy as np
import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.nn.functional as F

input = Variable(torch.randn(4, 10).cuda())
target = Variable(torch.ones(4).long().cuda())

class dp(nn.Module):
    def __init__(self):
        super(dp, self).__init__()
        self.n1 = nn.Linear(10,10)
        self.n2 = nn.Linear(10,2)
    def forward(self, x):
        x = self.n1(x)
        x = F.log_softmax(self.n2(x),dim=1)
        return x

dp = dp().cuda()
dp.zero_grad()
dp = torch.nn.DataParallel(dp, device_ids=[0, 1, 2, 3])
dp.eval()
output = dp(input)
loss = F.nll_loss(output, target, reduction='none')
torch.autograd.backward([element for element in loss])
</code></pre>
<p>As you can see, 4 GPUs are employed and the batch size is 4.<br>
So, each GPU evaluates the gradient of each input.</p>
<p>My question is how the gradient values of each input can be accessed.</p>
<p>Thanks in advance for your help.</p>

JuanFMontesinos: <p>Well, as no smarter person is replaying I will.</p>
<p>You can probably use register backward hooks. Each time a the hooked layer gets backwarded you will get an output.</p>
<p>The problem is it will be called as many times as gpus you have and I have not hints about how to deal with that.</p>

Seungyoung_Park: <p>Thanks for your reply.</p>
<p>I will try as you suggested.</p>
<aside class="quote no-group" data-username="JuanFMontesinos" data-post="2" data-topic="44162">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>no smarter person is replaying I will.</p>
<p>You can probably use register backward hooks. Each time a the hooked layer gets backwarded you will get an output.</p>
<p>The problem is it will be called as many times as gpus you have and I have not hints about how to deal with that.</p>
</blockquote>
</aside>

tom: <p>Still no smarter person than <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> replying, but I’m gray-haired enough to recommend <a href="https://discuss.pytorch.org/t/efficient-computation-of-per-sample-examples/18587" class="inline-onebox">Efficient computation of per-sample examples</a> and the discussion in issue <a href="https://github.com/pytorch/pytorch/issues/15359" rel="nofollow noopener">#15359</a> for more inspiration.</p>
<p>Best regards</p>
<p>Thomas</p>
Compute difference of each element in a tensor with each other

PySimon: <p>Hello,</p>
<p>I have a tensor with shape Nx2 that stores coordinates (x,y). I want to compute the x-distance and the y-distance between all coordinates. So if I have N coordinates, I will have <code>N*N</code> distances in the end. I could easily do this with two loops or with python’s itertools. However, it would be much faster to do it with PyTorch on the GPU.</p>
<p>Is there an easy possibility to combine each element in a tensor with every other element in this tensor?</p>
<p>The number of coordinates will be &lt; 100, so in the worst case there will be 10.000 distances to be computed. So it should not be a problem with GPU memory.</p>
<p>Best,<br>
Simon</p>

ptrblck: <p>I think broadcasting might work:</p>
<pre><code class="lang-python">x = torch.arange(10).view(-1, 1).expand(10, 2)
x[:, 0] - x[:, 1].view(-1, 1)
</code></pre>

PySimon: <p>Awesome, thank you!</p>
<p>The expand is not necessary (if I don’t miss anything). And your idea extended to a batch of coordinates looks like this:</p>
<pre><code class="lang-python"># y shape: BS x N_COORDS x 2
# dist_gt shape: BS x N_COORDS x N_COORDS x 2
dist_gt = torch.zeros(bs, n_lm, n_lm, 2)
dist_gt[:,:,:,0] = y[:,:,0].view(bs,1,-1) - y[:,:,0].view(bs,-1,1)
dist_gt[:,:,:,1] = y[:,:,1].view(bs,1,-1) - y[:,:,1].view(bs,-1,1)
</code></pre>
<p>It is much faster than looping over the tensor manually (~5s for the whole dataset compared to 12 min)</p>

ptrblck: <p>Good to hear it’s working fine! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I’ve just used <code>expand</code> to be on par with your input shapes, but you are right: also without <code>expand</code> it should work. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
K-nearest neighbor using index select

erikgaas: <p>I’m working on a fun problem where I am looking at nearest neighbors between two 3-dimensional point sets. So for every point in point cloud <code>p1</code> I’m finding the mean center of the k nearest points in opoint cloud <code>p2</code>. Here is working code that does just that:</p>
<pre><code class="lang-auto">def calc_k_nearest(p1, p2, k=10):
    adj = p1[:, None,...] - p2[None, ...]
    adj = adj.pow(2).sum(-1)
    top_k = adj.topk(k=k, dim=1, largest=False)[1]
    
    ls = []
    for i in range(top_k.shape[0]):
        tmp = torch.index_select(p2, dim=0, index=top_k[i])
        ls.append(tmp.mean(0))
    
    near_points = torch.stack(ls)
    mag = near_points - p1
    return near_points, mag
</code></pre>
<p>What I don’t like though is the for loop. It would be nice to have index_select return a two-dimensional matrix instead of iterating through every one. I’m not sure what sort of broadcasting logic would exist for something like this. Is there a way to do this with index_select or a different function?</p>
nn.Parameters keep cpu or gpu tensor reference when calling module.to() method?

dragen: <p>I wonder the inner mechnism when calling <code>mymodule.to(device)</code> for a typical module;</p>
<pre><code class="lang-auto">class MyModule(nn.Module):

  def __init__()
    super...
    self.w = nn.Parameter(torch.ones(1))
</code></pre>
<p>When we call <code>mymodule.cuda()</code> the returned tensor <code>w</code> from <code>nn.Parameters()</code> will pointer to <code>gpu tensor w</code>. However, our optimizer can not optimize <code>non-leaf tensor</code> if  gpu <code>w</code> is created by <code>gpu_w = cpu_w.cuda()</code>, so What does pytorch solve this problem?<br>
Thank you.</p>

JuanFMontesinos: <p>But the way you defined w it’s a leaf parameter <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=9" title=":expressionless:" class="emoji" alt=":expressionless:"></p>

dragen: <p><a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> But <code>gpu_w = cpu_w.cuda()</code>, in this case, <code>gpu_w</code> is not a leaf tensor.</p>

JuanFMontesinos: <p>It’s product of the way you define it.<br>
gpu_w itself is not leaf as it’s a copy of the cpu tensor. The cpu version is leaf in fact.</p>
<p>You can bypass this by calling<br>
<code>self.w = nn.Parameter(torch.ones(1).cuda())</code></p>
<p>Edit :<br>
It may be better you to define that parameter on cpu and then to allocate on gpu if you use it during forward function. That makes much more sense. In fact, you can also wrap it inside another nn.Module in order to allocate it into gpu inside init.</p>
A question about the generated data

YuLei: <p>If I use pytorch to generate a data<br>
………………<br>
25558290159175134183***432631828857***345418473182708965577129101516406472191919574089<br>
……<br>
if I want to limit the scope of***4326318288***in the conditions of the generated model, how should I limit it?E.g  3456239452&lt;<em><strong>4326318288</strong></em>&lt;5632845623</p>
I have results with GAN but the colors are shifted

Nilesh_Pandey1: <p>The colors from GAN are shifted in sense that yellow looks like organe, green shifts to dark green.<br>
Any suggestion?</p>
<p>Thank you</p>

ptrblck: <p>What is your use case? Did you somehow manipulate the output colors (normalized the output images) or processed the input image’s color space?</p>
Impact of Sorting on Packed Padded LSTM

PopHSKE: <p>As I understand it, in order to ‘mask’ the input to an RNN (e.g. values of zero which simply represent padding of unequal length text to be uniform), one has to sort the sequence from longest to shortest and then this new sorted stacked sequence is sent into the RNN in mini batches.</p>
<p>Credit to : <a href="https://github.com/HarshTrivedi/packing-unpacking-pytorch-minimal-tutorial" rel="nofollow noopener">https://github.com/HarshTrivedi/packing-unpacking-pytorch-minimal-tutorial</a></p>
<p>Doesn’t this preclude you from shuffling your mini batches between epochs?</p>
<p>I am presuming that the full training set is packed prior to training. I tried to shuffle this packed data during training and an exception was raised that the data needed to be sorted.</p>
<p>Perhaps a work around is to pack each minibatch separately in the training loop? Not sure the impact this has on efficiency.</p>

vdw: <p>Slightly off-topic: For everything except machine translation, that is, for classifiers, autoencoder, sequence tagger, I always use custom batch sampler that organizes the dataset into batches with sequences of equal length. This avoids the hassle with <code>PackedSequence</code> as well as padding in general.</p>
<p><a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/utils/data/text/dataset.py" rel="nofollow noopener">Here’s the code.</a> And that’s what the usage looks like:</p>
<pre><code>from pytorch.utils.data.text.dataset import BucketBatchSampler, BucketDataset

bucket_batch_sampler = BucketBatchSampler(X_train, batch_size)
bucket_dataset = BucketDataset(X_train, None) # None because example is for an autoencoder (otherwise, e.g., y_train for classifier)

X_train_iter = DataLoader(bucket_dataset, batch_sampler=bucket_batch_sampler, num_workers=4)
</code></pre>
<p>Maybe useful for you.</p>

PopHSKE: <p>Thanks!<br>
I am curious though, it seems like you must have uneven batch sizes and always the same observations in the same order still?</p>

vdw: <p>If you check the code, there’s a lot of shuffling going on before each iteration.</p>
<p>Yes, say your chosen batch size is 32, some batches might not have 32 samples. However, if your dataset is large, these cases are negligible. I did a test with a 300k sentences or so. 99.7% of batches were full, and the rest were almost full maybe with an occasional outlier. There’s no performance loss.</p>
RuntimeError: set_sizes_contiguous is not allowed on Tensor created from .data or .detach(), in Pytorch 1.1.0

isalirezag: <p>Im trying to to do this:<br>
<code>Img_.data.resize_(Img.size()).copy_(Img)</code><br>
and got this error:</p>
<pre><code class="lang-auto"> RuntimeError: set_sizes_contiguous is not allowed on Tensor created from .data or .detach()
</code></pre>
<p>Any suggestion why?<br>
It was working for pytorch 1.0.1 but now gives me this error in 1.1.0 version</p>
<p>Img has size torch.Size([1, 3, 256, 256]) abd Img_ has size torch.Size([1])</p>

smth: <p><code>.data.resize_</code> was an unsupported operation (infact using <code>.data</code> is being discouraged). It worked in 1.0.1 because we still didn’t finish part of a refactor.</p>
<p>You should now use:</p>
<pre><code class="lang-auto">with torch.no_grad():
    Img_.resize_(Img.size()).copy_(Img))
</code></pre>
Network not learning when using conv2d as output

William_Correa: <p>Hi,<br>
I’m training a senet50 with a conv2d layer as its output, and I’m computing the loss based on the cosine similarity of pairs of embeddings, conditional on whether they belong to the same class or not. The problem is that my network is not learning at all. I have even tested it with a small batch of 20 images to check whether it was able to overfit it, but it did not. The same loss function works well with other architectures (where the output is a fully connected layer). I was wondering whether I can make it work without replacing the last conv2d layer of my network.</p>
<p>This is how my conv2d is used:</p>
<p><strong>self.last_layer = self.conv2d(2, in_channels=2048, out_channels=256, kernel_size=(1,1), stride = (1,1), groups=1, bias=False)</strong></p>
<p>Then my loss function is as follows:</p>
<p><strong>def my_loss(margin, f_embed, s_embed, label):</strong><br>
**    f_embed = f_embed.view(f_embed.shape[0], f_embed.shape[1])**<br>
**    s_embed = s_embed.view(s_embed.shape[0], s_embed.shape[1])**<br>
**    sim = CosineSimilarity().forward(f_embed, s_embed)**<br>
**    loss = (label * (1 - sim)) + ( (1-label) * torch.clamp(sim-margin,min=0.0 )**<br>
**    return loss**</p>
<p>The reason why I change the shape of the embeddings is because without doing it their shapes would be [batch_size, 256, 1, 1], which makes me get weird results when I compute the loss. With the shape transformation, my embeddings have instead the shape [batch_size, 256].</p>

William_Correa: <p>As suggested in <a href="https://github.com/pytorch/pytorch/issues/1479" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/1479</a>, I tried doing</p>
<p><strong>f_embed = Variable(f_embed.view(f_embed.shape[0], f_embed.shape[1]), requires_grad=True)</strong><br>
<strong>s_embed = Variable(s_embed.view(s_embed.shape[0], s_embed.shape[1]), requires_grad=True)</strong></p>
<p>but still my network is not learning</p>

William_Correa: <p>In fact, the issue was solved by performing a deepcopy of my original model, as shown below.</p>
<pre><code>    old_model = MyModelClass.to(device)
    state_dict = torch.load(model_state_dict_path)
    old_model.load_state_dict(state_dict)

    model = copy.deepcopy(old_model)
</code></pre>
<p>Anyways, I don’t know yet why these steps were necessary in order to make my model change its weights and be able to learn</p>
torch.nn.ReflectionPad3d?

kazem: <p>I was wondering if you could help me define a torch.nn.ReflectionPad3d layer?<br>
I only found this post:<br>
<a href="https://discuss.pytorch.org/t/reflectionpad3d/20403">https://discuss.pytorch.org/t/reflectionpad3d/20403</a><br>
And the answer doesn’t make any sense. Thank you.</p>

kazem: <p>Even this answer doesn’t work correctly, since this <code>ReflectionPadNd</code> function does not work correctly.<br>
<a href="https://github.com/c22n/unet-pytorch/blob/master/models/custom_layers.py" rel="nofollow noopener">https://github.com/c22n/unet-pytorch/blob/master/models/custom_layers.py</a></p>

kazem: <p>Found the answer in the following link:<br>
<a href="https://discuss.pytorch.org/t/how-to-pad-one-side-in-pytorch/21212/2">https://discuss.pytorch.org/t/how-to-pad-one-side-in-pytorch/21212/2</a><br>
<code>torch.nn.functional.pad ( input, pad , mode='reflect' , value=(1, 1, 1, 1, 1, 1) )</code> will do the job. Can be rapped around a module as well:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

class CustomPad(torch.nn.module):
  def __init__(self, padding):
    self.padding = padding
  def forward(self, x):
    return F.pad(x. self.padding, mode='reflect')
</code></pre>
How should I reshape output of log_softmax so that I can use it in nn.NLLLoss()

coyote: <p>Hi,</p>
<p>I am trying to implement token level tagging model. I have two sentences with 14 and 9 words. I’ve padded the shorter one with <code> utils.rnn.pad_sequence</code> function and  I obtained log probabilities (with <code>nn.LogSoftmax()</code>) for each word:</p>
<p><code>pred_logits = self.log_softmax( feats)</code><br>
<code># print(pred_logits.shape) =&gt;  2x14x2  (bs x max_token_num x size_of_label_set)</code><br>
my target array is:<br>
<code>target = torch.tensor([[1,0,1,1,0,1,1,0,1,0,1,1,0,1],[1,1,1,0,1,1,1,0,1,1,1,1,1,1]])</code><br>
<code>mask = torch.tensor([[1,1,1,1,1,1,1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1,1,0,0,0,0,0]])</code></p>
<p>Now I want to calculate the loss by using pred_logits and target tensors as follows:</p>
<pre><code class="lang-auto"> loss = self.nll_loss(pred_logits, target)
 # average/reduce the loss according to the actual number of of predictions (i.e. one prediction per token).
 loss /= mask.float().sum()
 return loss
</code></pre>
<p>however, I am getting</p>
<blockquote>
<p>ValueError: Expected target size (2, 2), got torch.Size([2, 14]) error.</p>
</blockquote>
<p>How could I fix this problem ?</p>
<p>EDIT</p>
<p>I can do it as follows without getting any error but I wonder if there is a better way to do the same thing:</p>
<pre><code class="lang-auto">loss = 0
for i in range(0,target.shape[0]):  
    loss_tmp = self.nll_loss(pred_logits[i,:,:], target[i,:])
    loss += loss_tmp
loss /= mask.float().sum()
 return loss
</code></pre>
Mistake in Attention Decoder Tutorial

oneilllml: <p>In the <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">official Pytorch seq2seq tutorial</a>, there is code for an Attention Decoder that I cannot understand/think might contain a mistake.</p>
<p>It computes the attention weights at each time step by concatenating the output and the hidden state at this time, and then multiplying by a matrix to get a vector of size equal to the output sequence length. Note, these attention weights don’t depend on the encoder sequence (named encoder_outputs in the code), which I think it should.</p>
<p>Also, <a href="https://nlp.stanford.edu/pubs/emnlp15_attn.pdf" rel="nofollow noopener">the paper cited in the tutorial</a>, lists three different score functions that can be used to compute attention weights (section 3.1 in the paper). None of these functions is just concatenating and multiplying by a matrix.</p>
<p>So it seems to me that the tutorial is mistaken both in the function it applies and the arguments that are passed to this function. Am I missing something?</p>
RNNLearner predict causes mem leak in docker

zh0uquan: <p>I am trying to run a simple predict, but it keeps eating memory. I am not sure is it a fastai or torch issue and I have no clue why.<br>
[ENV]<br>
docker: docker Ubuntu18.04, python3.6 container<br>
pytorch: <a href="https://download.pytorch.org/whl/cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl" rel="nofollow noopener">https://download.pytorch.org/whl/cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl</a><br>
fastai: 1.0.46</p>
<pre><code class="lang-auto">from fastai.basic_train import load_learner
predictor = load_learner(path, "file.pkl")
pred_class, pred_idx, outputs = predictor.predict("string")
</code></pre>
<p>I cant reproduce the memory leak on my mac or linux machine, so I think it is docker related problem, that maybe some missing lib. Any thought?</p>
RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM

Or_Rimoch: <p>Hi!<br>
My input is a batch of size:  [10, 1, 257, 65]<br>
I might have some problem with the Conv2d dimensions.<br>
Here  is attached the model. Please help me solve it.<br>
Thanks.</p>
<p>class SimpleCNN(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(SimpleCNN, self).<strong>init</strong>()</p>
<pre><code>    #Input channels = 1, output channels = 16
    self.layer1 = nn.Sequential(
        nn.Conv2d(1, 16, kernel_size=3, stride=2),
        nn.ReLU(),
        nn.Conv2d(16, 32,kernel_size=3, stride=2),
        nn.ReLU(),
        nn.Conv2d(32, 32, kernel_size=3, stride=2),
        nn.ReLU(), 
        nn.Conv2d(32, 32, kernel_size=3, stride=3),
        nn.ReLU(), 
        nn.Conv2d(32, 64, kernel_size=3, stride=2),
        nn.ReLU(), 
        nn.Conv2d(64, 64, kernel_size=3, stride=2)
     )
        
    self.fc1 = nn.Linear(64, 166)
    self.sig = nn.Sigmoid()  

    
def forward(self, x):
    out = self.layer1(x)
    out = self.fc1(out)
    out = out.view(out.size(0), -1)
    out = self.sig(out)
    return out</code></pre>

Husain_Kapadia: <p>I have a similar issue, did you manage to solve this?</p>
Adding cuda gives error CUDNN_STATUS_BAD_PARAM

batsa003: <p>I have the following code.</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable
import torch.nn as nn

def conv3d(in_channels, out_channels, kernel_size = 4, stride = 2, padding = 1):
    return nn.Conv3d(in_channels, out_channels, kernel_size = kernel_size, stride = stride, padding = padding, bias = True)

class G_encode(nn.Module):
    def __init__(self):
        super(G_encode, self).__init__()
        self.model = nn.Sequential(
                conv3d(3,128),
                )
    def forward(self,x):
        print('G_encode Input =', x.size())
        out = self.model(x)
        print('G_encode Output =', out.size())
        return out

x = Variable(torch.rand([1,3,1,64,64])).cuda()
model = G_encode().cuda()
out = model(x)
~                
</code></pre>
<p>This code seems to be working fine when I remove .cuda(). However, it is giving RuntimeError: CUDNN_STATUS_BAD_PARAM with .cuda(). My cuda version is 8.0.61, and the nvidia driver version is 384.111.</p>

SimonW: <p>I can reproduce it. The cause is that your <code>kernel_size</code> is larger than <code>size + padding * 2</code>. I will submit a fix to improve the error message.</p>

Husain_Kapadia: <p>I have a similar issue…can’t seem to fix it. I have an input of dimension (batch_size=64, channels=1, x=80, y=10) and kernel size (17,1) with no padding. I want the size in the x-dimension to reduce after every conv2d operation and no change in the y dimension.</p>
<p>later I use convtranspose2d where it throws this error.</p>
Reshaping Panel Data for RNN/LSTM

Hannibal: <p>I work in consumer credit where we deal with a lot of panel data, e.g., a customer’s payment history.  Consequently, I will have customers with  multiple and varying number of observations.  I’m looking at exploring an LSTM architecture but having difficulty figuring out how the time series element of the data should be reshaped and padded to account for the varying lengths of the sequence for different customers.  This would be a one step classification problem where we try to predict whether a customer defaults or not in the month following the end of the sequence.</p>
<p>For a given customer, if for example that customer had 25 observations because he took out an account 25 months ago we would have 24 sequences for that customer the shortest with 1 observation and the longest with 24 observations.</p>
<p>A dataset might look like this for your reference:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9171ed5a1fc5f1fea9e77b79c7b8829d16bef9c0.png" alt="image" data-base62-sha1="kKFjU0xIX5030IXykMFKwNWluTu" width="466" height="403"></p>
Confuse about torch.tensor's indexing

rock_j: <p>Can anyone tell me, What’s the difference between mask_a and mask_b?<br>
thanks!</p>
<pre><code class="lang-python">import torch
torch.manual_seed(1)
data = torch.rand(3,3)
print('data',data,'\n')

mask_a = a[:,0]&gt;0.5
mask_a = mask_a.unsqueeze(-1)
mask_a = mask_a.expand_as(data)
print('mask_a',data[mask_a],'\n')

mask_b = torch.tensor([1,1,0])
mask_b = mask_b.unsqueeze(-1)
mask_b = mask_b.expand_as(data)
print('mask_b',data[mask_b],'\n')
</code></pre>
<hr>
<p>output</p>
<pre><code class="lang-auto">data tensor([[0.7576, 0.2793, 0.4031],
        [0.7347, 0.0293, 0.7999],
        [0.3971, 0.7544, 0.5695]]) 

mask_a tensor([0.7576, 0.2793, 0.4031, 0.7347, 0.0293, 0.7999]) 

mask_b tensor([[[0.7347, 0.0293, 0.7999],
         [0.7347, 0.0293, 0.7999],
         [0.7347, 0.0293, 0.7999]],
        [[0.7347, 0.0293, 0.7999],
         [0.7347, 0.0293, 0.7999],
         [0.7347, 0.0293, 0.7999]],
        [[0.7576, 0.2793, 0.4031],
         [0.7576, 0.2793, 0.4031],
         [0.7576, 0.2793, 0.4031]]]) 
</code></pre>

ptrblck: <p>The first example uses “boolean array indexing” while the second uses plain indexing as described in the <a href="https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#boolean-array-indexing" rel="nofollow noopener">numpy advanced indexing docs</a>.<br>
If you call <code>print('mask_b', data[mask_b.byte()])</code>, you’ll get the same result.</p>

rock_j: <p>thanks man. help a lot !</p>
Does all_reduce_multigpu work with shared list (created by multiprocess.Module.list())?

yangyang: <p>Hello.</p>
<p>I’m trying to use “all_reduce_multigpu” function with shared memory (list created by multiprocess module). However, it does not work with the shared list object (only working with a normal list).</p>
<p>I want to use the shared list to get return values from the child processes. Is there any alternative way?<br>
Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
TorchScript = (trained) model export or application export?

RuFAI: <p>Hello World,</p>
<p>after excessively reading the ‘Loading a PyTorch model in c++’ I have some questions left. Is exporting the plain model the way to go, without training and testing it in Python? To me it seems like that what makes me kind of nervous when it comes to preprocessing the data with the functionalities given by numpy, which are actuallly really helpfull. Is there a way to export the key functionality to C++ transforming the code to TorchScript, including the preprocessing procedures? What ist the best practice here? I am not that familiar with C++ and would like to have as less export trouble as possible.</p>
<p>Thanks for reading und hopefully answering:)</p>
How to call one layer in CNN

wxystudio: <p>I define a discriminator as follow:</p>
<p>class D(nn.Module):<br>
def <em>init</em>(self, ngpu):<br>
super(D,self).<em>init</em>()<br>
self.main = nn.Sequential(OrderedDict([<br>
(‘conv1’, nn.Conv2d()),<br>
(‘relu1’, nn.ReLU()),<br>
‘conv2’, nn.Conv2d())<br>
(‘out’, nn.Signoid())<br>
]))</p>
<p>and I want to use modify one layer by<br>
D.conv1.weight.data<br>
but it doesn’t work because “D” object has not attribute “conv1”</p>
<p>thank you</p>

Tony-Y: <p>Your code has no initializer <code>__init__</code>.  Try the following code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from collections import OrderedDict

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.net = nn.Sequential(OrderedDict([
          ('conv1', nn.Conv2d(1,20,5)),
          ('relu1', nn.ReLU()),
          ('conv2', nn.Conv2d(20,64,5)),
          ('relu2', nn.ReLU())
        ]))

    def forward(self, x):
        return self.net(x)

d = Discriminator()
print(d.net.conv1)
</code></pre>

wxystudio: <p>thank you! in fact I lost the “.net”</p>