akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.multiprocessing rebuild_cuda_tensor having trouble with bn.num_batches_tracked

username: bhpfelix
cooked: <p>I am trying to pass a set of gpu models to a pool of processes, so something like the example code below:<br>
(torch version <code>1.0.0.dev20181015</code>)</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
mp.set_start_method('spawn', force=True)
import torch
import torch.nn as nn
import time


def run(inpt):
    workers, worker_queue = inpt
    while worker_queue.empty():
        time.sleep(0.1)
    worker_id = worker_queue.get()
    model = workers[worker_id]
    data = torch.rand(5, 10)
    data = data.cuda()
    with torch.no_grad():
        result = model(data)
    worker_queue.put(worker_id)
    return result


def main():
    workers = []
    worker_queue = mp.Manager().Queue()  # mp.Queue()
    for worker_id in range(10):
        worker_model = nn.BatchNorm1d(10)
        worker_model.cuda()
        worker_model.eval()
        workers.append(worker_model)
        worker_queue.put(worker_id)

    pools = mp.Pool(10)

    jobs = [(workers, worker_queue) for _ in range(20)]  # jobs
    result = []
    for res_id, res in enumerate(pools.imap_unordered(run, jobs)):
        result.append(res)
    print(result)


if __name__=="__main__":
    main()
</code></pre>
<p>However, when there is BatchNorm, the following error is raised in <code>torch._utils.rebuild_cuda_tensor</code>:</p>
<pre><code class="lang-auto">RuntimeError: Expected object of data type 4 but got data type 6 for argument #2 'source'
</code></pre>
<p>It is probably caused by the attribute bn.num_batches_tracked, which is supposed to be type 4 (int64_t), but looks like the error message is reporting data type 6 (float). What is likely the cause of this issue? Is there any way to fix it?</p>

username: SimonW
cooked: <p>I posted an issue at <a href="https://github.com/pytorch/pytorch/issues/12798" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/12798</a></p>
Federated Learning support in PyTorch

username: kaper
cooked: <p>Hi,<br>
Wondering if there is a native support for federated learning in PyTorch ?</p>
<p>Any input is appreciated.<br>
Thanks!<br>
Kartik</p>

username: RicCu
cooked: <p>Try out <a href="https://github.com/OpenMined/PySyft" rel="nofollow noopener">PySyft</a>.</p>
Easiest way to check for a "connected" graph

username: kekday
cooked: <p>Hi,<br>
Is there an easy way to check if my entire sequence of operations is connected? Apart from manually debugging through the entire flow and checking if each intermediate variable has a grad_fn? I haven’t used Tensorflow much either, but what I am looking for is probably a way to print out the entire “graph” in PyTorch (though I understand in PyTorch it’s dynamically created)?</p>
<p>Sorry if that didn’t make much sense. I mean if I had something like:<br>
x = tensor which requires grad<br>
y=x^2+3<br>
z=y.mean()</p>
<p>Now I want something which tells me the entire sequence of grad_functions. The purpose is to troubleshoot a project where I have quite a long such chain of functions and hopefully I am not messing up somewhere.</p>

username: akernel
cooked: <p>Sorry this isn’t much (it’s kinda late here), but <a href="https://discuss.pytorch.org/t/check-gradient-flow-in-network/15063">this post</a> has a snippet that lets you examine the gradient flow, and <a class="mention" href="/u/apaszke">@apaszke</a> has a <a href="https://gist.github.com/apaszke/f93a377244be9bfcb96d3547b9bc424d" rel="nofollow noopener">GitHub Gist</a> that should let you see the network’s graph.  Hope that helps!</p>
nn.Parallel wraps the whole model?

username: yuqli
cooked: <p>I’m rather confused after reading both official tutorials on multi-GPU and data-parallelism. Can I wrap the whole model in <code>nn.Parallel</code>, instead of one layer at a time?</p>
<p>e.g. is the following code block legitimate?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class DataParallelModel(nn.Module):

    def __init__(self):
        super().__init__()
        self.block1 = nn.Linear(10, 20)
        self.block2 = nn.Linear(20, 20)
        self.block3 = nn.Linear(20, 20)

    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.block3(x)
        return x

model = DataParallelModel()
model = nn.DataParallel(model)
</code></pre>
<p>Thanks!</p>

username: akernel
cooked: <p>Hi <a class="mention" href="/u/yuqli">@yuqli</a>! Yep, running the code as you have it works perfectly.  If you print out your model, you can see that the whole thing is wrapped in a <code>DataParallel</code> wrapper.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; model

DataParallel(
  (module): DataParallelModel(
    (block1): Linear(in_features=10, out_features=20, bias=True)
    (block2): Linear(in_features=20, out_features=20, bias=True)
    (block3): Linear(in_features=20, out_features=20, bias=True)
  )
)
</code></pre>
<p>There’s some more info on <code>DataParallel</code> and how it works in <a href="https://discuss.pytorch.org/t/how-pytorchs-parallel-method-and-distributed-method-works/30349">this forum post</a>, where <a class="mention" href="/u/rasbt">@rasbt</a> gives a good diagram.</p>
<p>Hope that answers your question!</p>
<p>(If you’re curious as to how the inner workings of <code>DataParallel</code> function, <a class="mention" href="/u/fantasticfears">@fantasticfears</a> has a great <a href="https://erickguan.me/2019/pytorch-parallel-model" rel="nofollow noopener">post on their blog</a>).</p>
Plotting loss with LBFGS

username: kekday
cooked: <p>Hi,<br>
Looking at the example for how LBFGS needs a closure() function to be used (<a href="https://github.com/pytorch/examples/blob/master/time_sequence_prediction/train.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/time_sequence_prediction/train.py</a>), is there a way I could plot the loss as well? Simply appending the loss calculated inside closure() to a list initialized outside closure doesn’t seem to work.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="kekday" data-post="1" data-topic="50400">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kekday/48/11990_2.png" class="avatar"> kekday:</div>
<blockquote>
<p>Simply appending the loss calculated inside closure() to a list initialized outside closure doesn’t seem to work.</p>
</blockquote>
</aside>
<p>Do you get any error or why is it not working?<br>
Simply appending <code>loss.item()</code> to a list seems to work:</p>
<pre data-code-wrap="python"><code class="lang-python">losses = []
def closure():
    optimizer.zero_grad()
    out = seq(input)
    loss = criterion(out, target)
    print('loss:', loss.item())
    losses.append(loss.item())
    loss.backward()
    return loss
optimizer.step(closure)
</code></pre>

username: kekday
cooked: <p>My bad, it does seem to work indeed. Not quite sure what I missed last time since I tried the exact same thing and the list just stayed empty. Thanks a lot!</p>
Convert tf regularization to pytorch

username: raunaks
cooked: <p>I am converting a tf model to pytorch:</p>
<pre><code class="lang-auto">import tensorflow.contrib.slim as slim

def net(posenet_inputs):
    with slim.arg_scope([slim.conv2d],
                         normalizer_fn=slim.batch_norm,
                         normalizer_params=batch_norm_params,
                         weights_regularizer=slim.l2_regularizer(scale=0.0001),
                         activation_fn=tf.nn.relu):
        conv1  = slim.conv2d(posenet_inputs, 16,  7, 2)
        conv2  = slim.conv2d(conv1, 32,  5, 2)
        ...
</code></pre>
<p>I have written the basic network in pytorch.<br>
I wanted to implement the exact same regularization for weights in pytorch. I found that changing <code>weight_decay</code> in the <code>AdamOptimizer</code> can add regularization, but I don’t think that would be exactly recreating the TF implementation.</p>
<p>Is there any other way that would similarly use the <code>scale</code> parameter?</p>
GPU cache free Pytorch

username: adarsh-kr
cooked: <p>Is it recommemded to use <code>torch.cuda.empty_cache()</code> before every training for every batch? Is it going to affect the performance ?</p>
<p>My model leads to OOM error for some batches becasue of its dynamic nature. So I am trying to <code>emtpy_cache</code> and garbage collect before every batch of data.</p>

username: ptrblck
cooked: <p>You won’t be able to use more memory, as you are just deleting the cache.<br>
I haven’t profiled it, but I would assume the performance might be affected in a negative way, since the memory has to be reallocated in order to be used again.</p>
<p>If you are seeing an OOM error, you could try to lower e.g. the batch size.<br>
If that’s not an option, have a look at <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="nofollow noopener">checkpointing</a> to trade compute for memory.</p>
How the derivative of round, ceil, and floor function defined in pytorch?

username: dem123456789
cooked: <p>How the derivative of round, ceil, and floor function defined in pytorch?</p>

username: gslaller
cooked: <p>you can write test to determine there evaluation.<br>
Mathematically one would assume that derivative of ceil, round and floor is not determined for integers, but everywhere else the derivative will be zero, since these functions convert any function to a step function of a sort. The torch derivative of these function is just zero for the whole function.</p>
<p>def test(x, f):<br>
\t p = torch.tensor(x).float().requires_grad_();<br>
\t f( p ).backward()<br>
\t return float(p.grad())</p>
<p>Some samples:<br>
test(-1.2, torch.round) = 0.0<br>
test(-1.2, torch.ceil) = 0</p>
Newbie learning to use COCO keypoints to dataloader

username: emmanuelol
cooked: <p>Hi on the past days I post <a href="https://discuss.pytorch.org/t/using-layers-of-pretrained-model-and-concatenate-some-additional-layers/49837" class="inline-onebox">Using layers of Pretrained model and Concatenate some additional layers</a>, with your help and reading pytorch doc, I create my model, my goal is to implement " Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields" by Zhe Chaon et al,  as a mention before I’m newbie my only experience with PyTorch was train two image classification models, so any guidance would be awesome because I kind loss and I don’t where or how to start with two topics the use COCO dataset keypoints (pose estimation) and how to mount this on a dataloader, sorry for the long post.<br>
The model  " Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields" use the first ten layers of VGG19 that’s done and also I add the next layers, this model have two outputs of size (1,19,46,46) and (1,38,46,46), on the GitHub of this work are a way to transfer the dataset to LMDB to use this as input for training but on PyTorch  I have not idea, I just found that Pytorch have a module for COCO dataset but is only for object detection and image caption, I was looking for something on GitHub but honestly I feel kind of loss, so any guidance, tip is more than welcome, really thanks for your time and consideration.</p>
Autograd_output in a simple custom linear layer

username: zahra
cooked: <p>Dear Experts</p>
<p>I try to generate a simple custom linear layer as follows, but the prediction of the network is incorrect <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
I tried hard for more than 2 weeks but I could not solve it. I hope someone help me.</p>
<pre><code class="lang-auto">class linearZ(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input, weight):

        ctx.save_for_backward(input, weight)
        l2IN = input
        l2 = l2IN * weight
        return l2

    @staticmethod
    def backward(ctx, grad_output):

        learnign_rate = 0.01
        input, weight = ctx.saved_tensors
        grad_weight = input * grad_output
        weight = weight - learnign_rate * grad_weight
        net.linearZ.weight.data = weight
        return grad_weight, None


class MyLinearZ(nn.Module):
    def __init__(self):
        super(MyLinearZ, self).__init__()
        self.fn = linearZ.apply
        self.weight = nn.Parameter(torch.Tensor([[np.random.randn(1, 1) * 5.66]]))

    def forward(self, x):
        x = self.fn(x, self.weight)
        return x


class Net(nn.Module):
    def __init__(self, conv_weight):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 1, 2)
        self.pool = nn.MaxPool2d((1, 3))
        self.linearZ = MyLinearZ()
        self.conv1.weight = nn.Parameter(conv_weight)



    def forward(self, x):
        x = torch.atan(self.conv1(x))
        x = self.pool(x)
        x = torch.atan(self.linearZ(x))
        return x
</code></pre>
<p>Thanks before all</p>

username: ptrblck
cooked: <p>I’m not sure how <code>linearZ</code> is supposed to work, as it seems like you would like to calculate the gradients in <code>backward</code> as well as manipulate some model parameters.<br>
Could you explain the use case a bit as I think both steps (gradient calculation and parameter manipulation) should be done separately.<br>
In case you would just like to reimplement a simple linear function, this code should work:</p>
<pre><code class="lang-python">class linearZ(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, weight):
        ctx.save_for_backward(input, weight)
        output = input.mm(weight.t())
        return output

    @staticmethod
    def backward(ctx, grad_output):
        input, weight = ctx.saved_tensors
        grad_input = grad_output.mm(weight)
        grad_weight = grad_output.t().mm(input)
        return grad_input, grad_weight


class MyLinearZ(nn.Module):
    def __init__(self):
        super(MyLinearZ, self).__init__()
        self.fn = linearZ.apply
        self.weight = nn.Parameter(torch.randn(1, 1) * 5.66)

    def forward(self, x):
        x = self.fn(x, self.weight)
        return x
</code></pre>

username: zahra
cooked: <p>Thanks ptrblk, you are the best <img src="https://discuss.pytorch.org/images/emoji/apple/rose.png?v=9" title=":rose:" class="emoji" alt=":rose:"></p>
Customized convolution in high dimension matrices

username: zahra
cooked: <p>Hi,</p>
<p>I have about 500000 matrix by dimension of 36*500 each time and I customized convolution layer by myself and use it in a network by 2 layers (a convolution layer and a linear layer) in pytorch. I think run time should be higher than it could be handled. In your idea, what should I do since I forced not to use convolution of pytorch or even convolve2d of numpy?</p>
<p>Many thanks,<br>
Best Regards</p>
Using target size that is different to the input size

username: chaslie
cooked: <p>Afternoon,</p>
<p>I have a Autoencoder working now, but at the start i get the following warning:</p>
<p><em>UserWarning: Using a target size (torch.Size([16, 2048])) that is different to the input size (torch.Size([16, 1, 2048])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.</em></p>
<p>16 is my batch size and the vector length is 2048, so i would expect to see 16,1,2048. I cannot understand where 16,2048 comes from…</p>
<p>how do I reshape the target size to 16,1,2048 to solve the probelm so that the warning goes away?</p>
<p>It looks like the input vector size is 16,2048, and using data=data.unsqueeze(1) changes this to 16,1,2048 but i still have the warning…</p>
<p>thanks,</p>
<p>Chaslie</p>

username: Tahir
cooked: <p>If you are already changing the tensor shape in target then you shouldnt be getting the warning. Can you you post your code?</p>

username: chaslie
cooked: <p>hi Tahir,</p>
<p>I get the warning after the loss function below:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    for data, target in train_loader:
        print("data=",data.shape)# this gives the size as [16,2048]
        data = data.cuda()
        z_loc, z_scale = model.Encoder(data)
        z = model.reparam(z_loc, z_scale)
        out = model.Decoder(z)
        loss = loss_fn(out, data, z_loc, z_scale)
        optimizer.zero_grad()
</code></pre>
<p>Which is after I have used the unsqueeze here:</p>
<pre><code class="lang-auto">    def forward(self, data):
        data=data.unsqueeze(1)
       print("data_mod=",data.shape) # this reshapes to [16,1,2048]

</code></pre>
Cublas runtime error for torch.bmm on RTX 2080Ti

username: Tal_Mezheritsky
cooked: <p>Hi, I am trying to run a script from github <a href="https://github.com/vsitzmann/deepvoxels" rel="nofollow noopener">https://github.com/vsitzmann/deepvoxels</a>. I have a RTX 2060 GPU on my laptop and a RTX 2080Ti GPU on a server. I use the exact same conda environment on both machines, both have CUDA version 10.1. I am able to run the code without a problem on my laptop, however I can’t make it work on the server.</p>
<p>Initially, the code didn’t work on either machine. I simply updated the version of pytorch on my laptop and it solved the problem. I ran this command: <code>conda install pytorch torchvision cudatoolkit=10.0 -c pytorch</code> on the server , it updated pytorch and torchvision as asked however the same problem remains. Here is the ouput</p>
<pre><code class="lang-auto">Begin training...
Traceback (most recent call last):
  File "run_deepvoxels.py", line 395, in &lt;module&gt;
    main()
  File "run_deepvoxels.py", line 386, in main
    train()
  File "run_deepvoxels.py", line 183, in train
    grid2world=grid_origin)
  File "/store/usagers/tamez/deepvoxels/projection.py", line 90, in comp_lifting_idcs
    voxel_bounds_min, voxel_bounds_max, _ = self.compute_frustum_bounds(camera_to_world, world2grid)
  File "/store/usagers/tamez/deepvoxels/projection.py", line 73, in compute_frustum_bounds
    p = torch.bmm(camera_to_world.repeat(8, 1, 1), corner_points)
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:450
</code></pre>
<p>This is my first post here so if I did not provide some information please let me know.</p>
<p>Thanks in advance.</p>

username: Tal_Mezheritsky
cooked: <p>After reinstalling pytorch multiple times through conda I could not make the script work.</p>
<p>I identified the problem by running <code>torch.version.cuda</code> in the python console on the server and realising python was using CUDA 9 instead of CUDA 10. The command <code>torch.__file__</code> then showed that it was not even using torch from my conda environment, it was using another torch install on the server. Once I got rid of the other pytorch install <code>torch.version.cuda</code> showed CUDA 10 and everything works now.</p>
Differentiable quantization

username: Santanu_Rathod
cooked: <p>Hello all,<br>
So I have a <strong>class Quant_conv2d</strong>, and the class <strong>Quantizer</strong><br>
returns differentiable quantized weights:</p>
<h3></h3>
<p>class Quant_conv2d(nn.Module):</p>
<pre><code>def __init__(self, levels, sigma=1):
    super(Quant_conv2d, self).__init__()
    # 1 input image channel, 6 output channels, 3x3 square convolution
    # kernel
    self.levels = levels
    self.sigma = sigma
    self.conv1 = nn.Conv2d(1, 1, 1)
    self.new_weight= Quantizer(x= self.conv1.weight, levels= self.levels).forward() 
def forward(self, input):
    return F.conv2d(input= input.cuda(), weight= self.new_weight, bias= torch.tensor([0.0]).cuda()) 
</code></pre>
<h3></h3>
<p><strong>Then I do</strong> :</p>
<h3></h3>
<p>qdata_interim1= Quant_conv2d(levels= torch.tensor([-1.0,0.0,1.0]))<br>
qdata_out1= (qdata_interim1.forward(input= qdata_inp1))<br>
(torch.sum(qdata_out1)).backward()</p>
<h3></h3>
<p><strong>And then</strong> :</p>
<h3></h3>
<p>for name, value in qdata_interim1.named_parameters():<br>
print(‘grad for Quant_conv2d is:’, value.grad)</p>
<h3></h3>
<p>I get the grad as: None.<br>
But I expect it to be something wrt Quant_conv2d.conv1.weight</p>
<p>Can anyone tell me how to get around this?<br>
And where exactly am I wrong?</p>
Load an ONNX model in pytorch

username: Erotemic
cooked: <p>Is it possible to load a onnx model as a pytorch model directly? Or can you only load it into caffe2?</p>

username: houseroad
cooked: <p>Now, PyTorch does not support loading ONNX model. Yes, you can load ONNX model into Caffe2.</p>

username: Erotemic
cooked: <p>Is there any plan to implement the ability to load an ONNX model into pytorch? It seems like an exchange format should have that capability.</p>

username: houseroad
cooked: <p>It is on our roadmap, but it is not high priority. Any help is welcome. <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ModarTensai
cooked: <p>Is there anybody working on this? If yes, the GitHub issue link is much appreciated!</p>

username: Niv_Nayman
cooked: <p>Anything new with this capability ? Is it being supported ?</p>
Why copy data to gpu so slow?

username: k3833653
cooked: <p>In training phase, I send batch tensor to main gpu cost about 0.07s, however,  in evaluation phase, it only cost 0.005s, the data size is almost same, I don’t know why…</p>

username: ptrblck
cooked: <p>Could you post the code you’ve used to time these operations?<br>
Note that CUDA operations are asynchronous in PyTorch, so that you should synchronize before starting and stopping the timer:</p>
<pre><code class="lang-python">torch.cuda.synchronize()
t0 = time.time()
# your operation
torch.cuda.synchronize()
t1 = time.time()
</code></pre>
<p>If you didn’t add these sync points, you might just time the initial CUDA context creation etc.</p>
ImportError: cannot import name 'get_image_backend'

username: Mactarvish
cooked: <p>Hi, I’m trying to import get_image_backend by</p>
<pre><code class="lang-auto">from torchvision import get_image_backend
</code></pre>
<p>But I get the error:</p>
<pre><code class="lang-auto">ImportError: cannot import name 'get_image_backend'
</code></pre>
<p>My pytorch version is 1.0.1.post2, does this error due to my pytorch version or something? Thanks in advance.</p>

username: ptrblck
cooked: <p>Which <code>torchvision</code> version did you install?</p>
Differentiable communication - Distributed Model Parallel

username: Andreas_Georgiou
cooked: <p>Hello everyone, I created this pip package that includes differentiable versions of scatter/gather/send/recv so that pytorch’s autograd can backpropagate through those. I thought I should share. I haven’t thoroughly tested it so apologies if something breaks. Contributions are welcome!</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/ag14774/diffdist" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/10788242?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/ag14774/diffdist" target="_blank" rel="nofollow noopener">ag14774/diffdist</a></h3>

<p>Contribute to ag14774/diffdist development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>There is some example code here: <a href="https://github.com/ag14774/diffdist/blob/master/diffdist/testing.py" rel="nofollow noopener">https://github.com/ag14774/diffdist/blob/master/diffdist/testing.py</a></p>

username: pietern
cooked: <p>Very nice, thank you for sharing!</p>
Extra argument in F.relu

username: DuttaAbhigyan
cooked: <p>So I was experimenting with my NN when while change <code>F.leaky_relu(x,0.2)</code> to <code>F,relu(x)</code> I accidentally changed it to<code>F.relu(x, 0.2)</code>. The training progressed fine, but is there any significance of the 0.2?</p>

username: ptrblck
cooked: <p>The <code>0.2</code> in <code>F.relu</code> will be treated as the <code>inplace</code> argument, which will be interpreted as <code>inplace=True</code>.</p>
Two different networks, combined loss, back propagation and optimizer step

username: Abhinav_Ravi
cooked: <p>Hi all,<br>
So here is my network architecture as shown in below image.<br>
I’ve two separate networks and <code>loss_1</code>  and  <code>loss_2</code>  are coming from two separate nn.Modules networks and  <code>final_loss = loss_1 + loss_2</code> .</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0e3432696780c5f982cfb361e1a6e91579e3754f.png" alt="Untitled%20Diagram" data-base62-sha1="21EvZoULNdPdF9OxKHqRsKS6NUb" width="608" height="291"></p>
<p>Still only one  <code>final_loss.backward()</code>  would calculate grad in both networks?<br>
I do have two separate optimizers to update grads in two separate networks respectively.</p>
<p>If not so, then what is the correct way to do back propagation in these individual networks on combined loss value.</p>

username: Filos92
cooked: <p>Yes, you need two Optimizers.<br>
The Network structure you have choosen is pretty similar to so called “GAN” Networks. There are a lot of them in the Link below<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/eriklindernoren/PyTorch-GAN#gan" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/7977505?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/eriklindernoren/PyTorch-GAN#gan" target="_blank" rel="nofollow noopener">eriklindernoren/PyTorch-GAN</a></h3>

<p>PyTorch implementations of Generative Adversarial Networks. - eriklindernoren/PyTorch-GAN</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Greetings <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: caonv
cooked: <p>Yes, calling <code>backward()</code> on the the combined loss computes gradient for both of the networks.<br>
However, in your case, there is no relationship between the networks, then you can call the backward separately without any problem.</p>
Suboptimal GPU utilisation

username: kgr
cooked: <p>I’ve a simple shallow neural network. When I train this model, GPU utilisation is just ~ 35%. Could anyone suggest what can cause this under-utilisation of resources? Training examples are fed to the model from 32 processes using the <code>torch.utils.data.DataLoader</code> API with pinned memory. I use <code>torch.nn.Embedding</code> with sparse gradients as input layer. The model in trained on Tesla P100 in GCP. The host machine has 16 vCPUs. According to <em>cProfile</em>, the <code>run_backward</code> method takes the most time, and the CPU -&gt; GPU transfer (the <code>cuda</code> method) is second. According to the autograd profiler, <code>pin_memory</code> takes the most time. Matrix multiplication (<code>bmm</code>) is second.</p>
<p>I would appreciate any suggestions for improving GPU utilisation.</p>
Vector of torch tensors to torch tensor

username: Navneet_M_Kumar
cooked: <p>I have a vector of torch tensors and wanted to know how to convert this to a torch tensor.<br>
The vector represents a batch of torch tensors.</p>
<pre><code class="lang-auto">std::vector&lt;std::tuple&lt;torch::Tensor, torch::Tensor, torch::Tensor, torch::Tensor, torch::Tensor&gt;&gt; batch =
                buffer.sample_queue(batch_size);

        std::vector&lt;torch::Tensor&gt; states;
        std::vector&lt;torch::Tensor&gt; new_states;
        std::vector&lt;torch::Tensor&gt; actions;
        std::vector&lt;torch::Tensor&gt; rewards;
        std::vector&lt;torch::Tensor&gt; dones;

        for (auto i : batch){
            states.push_back(std::get&lt;0&gt;(i));
            new_states.push_back(std::get&lt;1&gt;(i));
            actions.push_back(std::get&lt;2&gt;(i));
            rewards.push_back(std::get&lt;3&gt;(i));
            dones.push_back(std::get&lt;4&gt;(i));
        }
</code></pre>
<p>Thanks!</p>

username: tom
cooked: <p>Just like you would in Python, use <code>torch::cat</code> to concatenate along a single axis and <code>torch::stack</code> to create a new axis. Note that dimensions (except possible the <code>cat</code> dimension) must match for this to succeed.</p>
<p>Best regards</p>
<p>Thomas</p>
How to change torch.Tensor into C++ version

username: selinablck
cooked: <p>Hi, is it possible to modify a 4d tensor in python so that it can be called and used in C++ file?</p>
<p>Say I have a C++ function which is called and used in another file in python language. I have a 4d torch.Tensor in my python file and I have also called my function from C++ file to my python file. I want to pass that tensor to my function so that my function can do max pooling on the tensor.</p>
<p>How can I pass the tensor by reference to my C++ file?</p>

username: tom
cooked: <p>The usual way of using C++ functions from Python is in the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">C++ extension tutorial</a>. As you see there, Python <code>torch.Tensor</code>s are automatically converted when you declare your function to take <code>torch::Tensor &amp;</code> (optionally with <code>const</code>).</p>
<p>Best regards</p>
<p>Thomas</p>
Unsupported comparison operator: Transformer Model Export (MultiHeadAttention Layer)

username: sukuya
cooked: <p>I am trying to export the Transformer Model to Torch Script. In creating a module list from MultiHeadAttention Layer, the following error is generated. Accompanying code is also attached.</p>
<pre><code class="lang-auto">File "/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 505, in build_Compare
    raise NotSupportedError(err_range, "unsupported comparison operator: " + op.__name__)
torch.jit.frontend.NotSupportedError: unsupported comparison operator: In
    kv_same = key.data_ptr() == value.data_ptr()

    tgt_len, bsz, embed_dim = query.size()
    assert embed_dim == self.embed_dim
    assert list(query.size()) == [tgt_len, bsz, embed_dim]
    assert key.size() == value.size()

    if incremental_state is not None:
        saved_state = self._get_input_buffer(incremental_state)
        if 'prev_key' in saved_state:
            ~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>CODE:</p>
<pre><code class="lang-auto">__constants__ = ['attentions', 'causal', 'layers_module']

def __init__(self, &lt;parameters&gt;)
        att_modules = []
        for _ in range(num_layers):
            att_modules.append(nn.MultiheadAttention(embed_dim, num_heads, dropout=dropout))
        self.attentions = nn.ModuleList(att_modules)
</code></pre>
<p>If I create an empty ModuleList and then append the MultiHeadAttention Layer it gives no error in no script mode, but  ModuleList has to be Const in Script Mode, that route is blocked as well. Error is generated in constructor itself as confirmed while debugging, not in forward method.</p>

username: driazati
cooked: <p>Support for <code>in</code> was recently added, could you try using <code>pytorch-nightly</code> and see if that fixes this issue?</p>

username: sukuya
cooked: <p><a class="mention" href="/u/driazati">@driazati</a>  Yup, that issue is resolved but it fails now at this stage:</p>
<pre><code class="lang-auto">if hasattr(self, '_qkv_same_embed_dim') and self._qkv_same_embed_dim is False:
           ~~~~~~~ &lt;--- HERE
            return F.multi_head_attention_forward(
                query, key, value, self.embed_dim, self.num_heads,
                self.in_proj_weight, self.in_proj_bias,
                self.bias_k, self.bias_v, self.add_zero_attn,
                self.dropout, self.out_proj.weight, self.out_proj.bias, 
                training=self.training,
                key_padding_mask=key_padding_mask, need_weights=need_weights, 
                attn_mask=attn_mask, use_separate_proj_weight=True,
                q_proj_weight=self.q_proj_weight, k_proj_weight=self.k_proj_weight,
</code></pre>
<p>Is it because <code>hasattr</code>  or <code>and</code> is not supported yet ?</p>
Positioning of output image

username: Filos92
cooked: <p>Hey Everyone,<br>
i train a generative network to produce pictures from a few input variables.<br>
right now it works suprisingly well. All generated Pictures have a shape which are not distingquishable to original.<br>
My Problem is, that the Position of the Object in the Picture is more important to me than the shape. but every generated Object is is located in or close to the center.<br>
Do you have any idea how to train a Network, which is more sensitive to the Position of the Object?</p>
How to encode categorical data that have variable length so could be fetched to nn.Embedding in PyTorch

username: jangbi
cooked: <p>Let’s say i have a data field named movie_genre for each sample  <code>movie</code> , it is selected from the following genres:</p>
<pre><code class="lang-auto">Action
Adventure
Animation
Comedy
...
</code></pre>
<p>And for each  <code>movie</code> , it might contain multiple genres:</p>
<pre><code class="lang-auto">mid    genres
1      Action | Adventure
2      Animation
3      Comedy | Adventure | Action
</code></pre>
<p>which means, the movie’s genres is a variable list.</p>
<p>If i use one hot vector to encode the  <code>genre</code> , Action can be encoded as (1, 0, 0, 0), Adventure can be encoded as(0, 1, 0, 0), and so on.</p>
<p>So movie with mid1 can be encoded as (1, 1, 0, 0), mid2’s genre can be encoded as (0, 0, 1, 0), and so on.</p>
<p>However, the pytorch embedding layer  <code>nn.Embedding</code>  takes tensor containing the indices as input, but not one-hot vector. So how should i encode the data so that it can be fetched into the embedding layer?</p>
<p>here’s the relative link in stackoverflow<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/56972512/how-to-encode-categorical-data-that-have-variable-length-so-could-be-fetched-to" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/8510613/user8510613" target="_blank" rel="nofollow noopener">
    <img alt="user8510613" src="https://www.gravatar.com/avatar/a2f16dd862c8e1016f5ae3db1b224043?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/56972512/how-to-encode-categorical-data-that-have-variable-length-so-could-be-fetched-to" target="_blank" rel="nofollow noopener">How to encode categorical data that have variable length so could be fetched to nn.Embedding in PyTorch</a>
</h4>

<div class="tags">
  <strong>machine-learning, pytorch, data-mining, encode</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/8510613/user8510613" target="_blank" rel="nofollow noopener">
    user8510613
  </a>
  on <a href="https://stackoverflow.com/questions/56972512/how-to-encode-categorical-data-that-have-variable-length-so-could-be-fetched-to" target="_blank" rel="nofollow noopener">02:06PM - 10 Jul 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Discriminator Network instance creation generates an error

username: Achakma
cooked: <p>Hello,</p>
<p>I have two classifiers and two associated discriminator networks. While I can create the instances of Classifier networks but I can’t in the case of discriminator networks. It generates an “<em>AttributeError: ‘torch.device’ object has no attribute '_apply</em>’”. I’ve seen several related posts and I also checked my PyTorch version which is 1.1.0. I don’t know how to debug this issue. I’ve attached a screenshot of the Jupyter notebook and the Discriminator class. If needed I can provide more detailed information. Can you please help me out?</p>
<p>FYI: The main difference between Classifier and Discriminator class is that the Discriminator network has CNN layers.<br>
Thank you.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/775ec4003eb09b42656aff16e4d212f3cdcb8ba1.png" data-download-href="https://discuss.pytorch.org/uploads/default/775ec4003eb09b42656aff16e4d212f3cdcb8ba1" title="PyTorch%20Error"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/775ec4003eb09b42656aff16e4d212f3cdcb8ba1_2_690x191.png" alt="PyTorch%20Error" data-base62-sha1="h1ZQUuGYkayixzt025gg6z35nO1" width="690" height="191" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/775ec4003eb09b42656aff16e4d212f3cdcb8ba1_2_690x191.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/775ec4003eb09b42656aff16e4d212f3cdcb8ba1_2_1035x286.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/7/775ec4003eb09b42656aff16e4d212f3cdcb8ba1_2_1380x382.png 2x" data-dominant-color="A1A1A1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PyTorch%20Error</span><span class="informations">2910×807 236 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You are not creating an instance of the discriminator, i.e. you forgot the parentheses.<br>
Try to call <code>Discriminator().to(DEVICE)</code> instead.</p>
<p>PS: It’s better to post the code directly by wrapping it in three backticks ``` instead of screenshots. This will allow the search to index it and allows us to copy/paste it in case we need to debug it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Achakma
cooked: <p>Ah! Thank you for point me out the mistake that I did.</p>
<p>Next time I’ll the three back-ticks to wrap up the codes. Thanks again for the suggestion.</p>
How do I enforce two features to be symmetric?

username: lonelykid
cooked: <p>Say we have two features x1 and x2.<br>
I want the neural network to give the same results when I switch the two features.<br>
A unstrict way would be  argument the data by switching two columns. But this won’t guarantee symmetry.<br>
This neural network could be multi-layers and there could be other features which I don’t want to enforce any constraints.</p>
<p>Edit: My guessing is I could generate a layer purely for the two features, e.g, a Linear(1,10), using this to act on both of them. And use another linear layer to act on other features. I could then concat or add the next layer. Give me suggestions if there are better practice. Thanks！</p>

username: Prerna_Dhareshwar
cooked: <p>I agree, I think the best way to go about this is to create a linear layer to act on just these two features and create a separate linear layer for the remaining features and then concatenate the output.</p>

username: lonelykid
cooked: <p>This seems not working, the order I concatenate them still makes a difference.</p>

username: Prerna_Dhareshwar
cooked: <p>What is the final output you want? It is dependent on that as well. If you apply the same linear layer to the two features, there would be symmetry if the ordering of the output did not matter. What is the output that you are expecting from your model?</p>

username: lonelykid
cooked: <p>After I concatenate them, they will be passed to a few more layers to yield an output. The two feature are not really symmetric in this case.</p>
<p>x1–a few layers–&gt;f(x1)<br>
x2–a few layers–&gt;f(x2)<br>
(f(x1),f(x2),x3,x4)—subsequent layers—&gt;output</p>
<p>So when I switch x1 and x2, it becomes (f(x2),f(x1),x3,x4) which is different and will yield different final results.</p>

username: Prerna_Dhareshwar
cooked: <p>Yes, if you have more linear layers after the initial(‘symmetric’)  linear layer you will lose symmetry. You will likely have to constantly apply the same linear layer to the two features, and only concatenate the outputs at the absolute end.</p>
TTA with probabilities?

username: peter_wills
cooked: <p>Hi,</p>
<p>I’m working on an image classification problem, with 12 classes. I’d like to use test time augmentation by averaging the probabilities of each class over the augmented test images but I am uncertain as to the correct formula.</p>
<p>Currently, I just take the output of the final fc layer, treating it as unnormalized output, and apply a softmax across classes, interpreting the result as the probability of the class. This is what I average over, taking argmax after averaging.</p>
<p>The loss function is cross-entropy, which involves log(softmax) and so my question is: am I interpreting properly the probabilities in my scheme above, or should I do something different.</p>
<p>Grateful for any advice.</p>

username: rwightman
cooked: <p>The geometric mean of the ‘probabilities’ from the softmax layer provide the best result for this. If you’ve got the output of log-softmax, you can just take the (arithmetic) mean as it’s equivalent to the geometric of the non-log values. Then yes, take the argmax of either of those…</p>

username: tom
cooked: <aside class="quote no-group" data-username="rwightman" data-post="2" data-topic="49854">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/b9e5f3/48.png" class="avatar"> rwightman:</div>
<blockquote>
<p>The geometric mean of the ‘probabilities’ from the softmax layer provide the best result for this.</p>
</blockquote>
</aside>
<p>Hi Ross,</p>
<p>do you know if there is a fundamental argument for using the Geometric mean rather than the arithmetic or is it more empirical?<br>
If I assume that the prediction is a noisy estimate of the “true PD” with sufficiently nice noise, my intuition would lead me to an arithmetic estimate. Also, the geometric mean of probabilities doesn’t sum to 1 (in general), so you’d need to re-normalize:</p>
<pre><code class="lang-auto">a = torch.randn(2,4)
p = a.softmax(1)
p_gm = (p[0]**0.5*p[1]**0.5)
print(p_gm.sum())
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: rwightman
cooked: <p><a class="mention" href="/u/tom">@tom</a> My usage is based on empirical results, geometric mean of the probabilities for ensembling predictions from a NN has produced better results for me on many occasions.</p>
<p>I searched for theory once, as the intuition didn’t sit well, especially when thinking about the case when one ensemble member has a near 0 output. In practice though, it works. I suppose a ‘try both’ answer might be more appriopriate.</p>
<p>In my theory search, the papers that provided some insight in the context of NN were about dropout</p>
<ul>
<li><a href="https://arxiv.org/abs/1312.6197" rel="nofollow noopener">https://arxiv.org/abs/1312.6197</a></li>
<li><a href="http://papers.nips.cc/paper/4878-understanding-dropout.pdf" rel="nofollow noopener">http://papers.nips.cc/paper/4878-understanding-dropout.pdf</a></li>
</ul>

username: peter_wills
cooked: <p>Thanks for the answers. Very helpful.</p>
Multi output resent + foolbox.models.PyTorchModel

username: Soheil_Esmaeilzadeh
cooked: <p>In my main ResNet block as below:</p>
<pre><code class="lang-auto">class ResNet(nn.Module):
    def __init__(self, block, num_blocks, num_classes=10):
        super(ResNet, self).__init__()
        self.in_planes = 64

        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)
        self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
        self.linear = nn.Linear(512*block.expansion, num_classes)

    def _make_layer(self, block, planes, num_blocks, stride):
        strides = [stride] + [1]*(num_blocks-1)
        layers = []
        for stride in strides:
            layers.append(block(self.in_planes, planes, stride))
            self.in_planes = planes * block.expansion
        return nn.Sequential(*layers)

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = F.avg_pool2d(out, 4)
        out0 = out.view(out.size(0), -1)
        out = self.linear(out0)
        return out, out0
</code></pre>
<p>I want to output two last layers, i.e. out &amp; out0 here.<br>
In my main code I try to use foolbox for say fgsm attack. So just giving a few lines I have:</p>
<pre><code class="lang-auto">fmodel_source = foolbox.models.PyTorchModel(model, bounds=(0, 1), num_classes=10, preprocessing=(0, 1))
attack = foolbox.attacks.GradientSignAttack(model=fmodel_source, criterion=attack_criteria)
</code></pre>
<p>Then trying to execute the line<br>
<code>adversarial = attack(inputs.astype(np.float32), targets, max_epsilon=config['epsilon'])</code></p>
<p>I get error that<br>
File “anaconda36/lib/python3.6/site-packages/foolbox/models/pytorch.py”, line 94, in batch_predictions<br>
predictions = predictions.to(“cpu”)<br>
AttributeError: ‘tuple’ object has no attribute ‘to’</p>
<p>Is there a way I get around this and still have the last two layers outputs from the ResNet but don’t change the foolbox under pytorch.py?</p>
<p>Any suggestions?</p>

username: ptrblck
cooked: <p>I’m not familiar with <code>foolbox</code>, but it seems as if the attack expects your model to output a single tensor, while your model outputs two.<br>
If you don’t need <code>out0</code> for the training (or something else), just remove it from the <code>return</code> statement.</p>
Train model with new Data for a few times

username: deepfailure
cooked: <p>I have a problem where I train a fully connected NN and once trained I gather new data and train again (with both old and new data) for a few times.</p>
<p>Do I need to reinitialize something in order the model to learn correctly? Optimizer or the model weights. I guess if I use <code>lr_scheduler</code> I should reinitialize but I do not use it yet.</p>
<p>I use ADAM optimizer.</p>
<p>Thanks in advance!</p>
<p>(New to Pytorch)</p>

username: Prerna_Dhareshwar
cooked: <p>Can you give some more information about the model, the data, and why you believe the model is not training properly?</p>
How to access parameters using model's attributes' name

username: Jingchi_Zhang
cooked: <p>I am using for loop to modify the parameters in the model. I use named_parameters to check the names of the attributes and using for loop to record them.</p>
<p>weight_data = []<br>
bias_data=[]<br>
weight_key =[]<br>
bias_key = []</p>
<p>for key,value in net.named_parameters():<br>
if (torch.tensor(value.size()).size()==torch.Size([3])):<br>
weight_key.append(key)<br>
weight_data.append(value)<br>
elif …:<br>
bias_data.append(value)<br>
bias_key.append(key)</p>
<p>What I want to do is modify the parameters in weight_data, I now I can use:<br>
net.fc1.conv.data = [] to do that, but I need to use for loop.<br>
So how can I use keys in weight_key to do that? for example:<br>
net.‘weight_key[0]’.data = […] (which is not right, just need too use the value in ‘weight_key’)</p>
<p>I would appreciate your help!</p>

username: ptrblck
cooked: <p>You could use <code>getattr</code> to get the submodules using their names.<br>
Here is a small example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        
    def forward(self, x):
        x = self.fc1(x)
        return x

model = MyModel()

keys = []
for name, value in model.named_parameters():
    keys.append(name)

with torch.no_grad():
    getattr(model, keys[0].split('.')[0]).weight.fill_(0.)  # split key to only get 'fc1'
</code></pre>