akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to count macs and parameters during forwarding in models or layers by pytorch?

username: Maxwell_Albert
cooked: <p>This question is quite complicated. I give an example.</p>
<pre><code class="lang-python">class model(nn.Module)
    def __init__()
        self.l1 = nn.linear(10,10)
        self.l2 = nn.linear(10,10)
    def forward(x)
        y1 = self.l1(x)
        y2 = self.l2(x)
        y3 = y1 + y2
        return y2 + y2 + y3*y1
</code></pre>
<p>In this example, I could use forward_hook functions to trace two linear layers and their parameters.fn is hook function.\</p>
<pre><code class="lang-auto">m.register_forward_hook(fn)
</code></pre>
<p>However, <code>y3</code> is not counted as a parameter and the macs of <code>y2 + y2 + y3*y1</code> is not counted in macs, too.<br>
How can I solve this?<br>
While “macs” is a way of measuring layers’ complexity. For example, y1 *(y2 + y3) is one macs, if y1, y2, y3 are floats.</p>
Is nn.Module ``stream-safe''?

username: Lynx_Commando
cooked: <p>Is it safe to share a module among different cuda streams during training?<br>
More specifically, will module instance ``a’’ suffer from race condition in the following code?</p>
<pre><code class="lang-auto">form torch import nn
import torch

class A(nn.module):
    ...
class B(nn.module):
    ...
class C(nn.module):
    ...

class AB:
    def __init__(this,a,b):
        this.a=a;
        this.b=b;
        this.stream=torch.cuda.Stream();
    def involke(x):
        with torch.cuda.stream(this.stream):
            r=this.b(this.a(x));
    def wait():
        this.stream.synchronize();


class AC:
    def __init__(this,a,c):
        this.a=a;
        this.c=c;
        this.stream=torch.cuda.Stream();
    def involke(x):
        with torch.cuda.stream(this.stream):
            r=this.c(this.a(x));
    def wait():
        this.stream.synchronize();


def main():
    a,b,c=A(),B(),C()
    fab=AB(a,b);
    fac=AC(a,c);
    ...
    x=loaddata();
    y_ab=fab.involke(x);
    y_ac=fac.involke(x);
    fab.wait();
    fac.wait();
    ...
    
</code></pre>
AMD GPU not recognized by torch

username: jonkatz6
cooked: <p>I am using an AMD R9 390.<br>
I Installed pytorch given the instructions from the following suggestions:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ffde978ab63fce5e1334351bca6a44a08a0a437e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e_2_690x311.png" alt="image" data-base62-sha1="AvwEFOIod0vOQf8QCsGi8o4tS9w" width="690" height="311" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e_2_690x311.png, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e.png 2x" data-dominant-color="F4E0DD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">843×380 33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However in python <code>torch.cuda.is_availible()</code> returns false. As I understood, this ROCm version is meant to use the amd gpu as cuda.</p>
<p>If there are additional steps I need to take to utilize the GPU, please let me know. Thank you.</p>
.h5 weights to pth weights

username: Oguzhan
cooked: <p>I have parameters of retinanet in the format of h5 and I want to convert it .pth file because when I try to load parameters as h5 file to pytorch, I get this error : UnpicklingError: invalid load key, ‘H’.</p>
<p>What can I do ?</p>

username: ptrblck
cooked: <p>I don’t think you can unpickle a <code>hdf5</code> file, so you would need read it e.g. via <code>h5py.File('mytestfile.hdf5', 'r')</code> before trying to convert it to any other format.</p>

username: Oguzhan
cooked: <p>I did lots of things but none of them works. I think pytorch needs to read h5 file like pth. I moved the whole project from torch to keras. It is kind of strange solution but works <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
Loss.backward() error-loss(my_func(output),label)

username: dlfma_tjd
cooked: <p>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn<br>
I got this RuntimeError</p>
<pre><code class="lang-auto">def generate_3d_dipole_kernel(data_shape, voxel_size, b_vec):
    #fov = np.array(data_shape) * np.array(voxel_size)
    fov = torch.tensor(data_shape)*torch.tensor(voxel_size)

    ry, rx, rz = torch.meshgrid(torch.arange(-data_shape[1] // 2, data_shape[1] // 2),
                             torch.arange(-data_shape[0] // 2, data_shape[0] // 2),
                             torch.arange(-data_shape[2] // 2, data_shape[2] // 2))

    rx, ry, rz = rx / fov[0], ry / fov[1], rz / fov[2]

    sq_dist = rx ** 2 + ry ** 2 + rz ** 2
    sq_dist[sq_dist == 0] = 1e-6
    d2 = ((b_vec[0] * rx + b_vec[1] * ry + b_vec[2] * rz) ** 2) / sq_dist
    kernel = (1 / 3 - d2)

    return kernel
</code></pre>
<pre><code class="lang-auto">def forward_convolution_1(chi_sample):
    scaling = np.sqrt(chi_sample.shape[0]*chi_sample.shape[1]*chi_sample.shape[2])
    chi_fft = torch.fft.fftshift(torch.fft.fftn(torch.fft.fftshift(chi_sample))) / scaling
    
    chi_fft_t_kernel = chi_fft * generate_3d_dipole_kernel(chi_sample.shape, voxel_size=1, b_vec=[0, 0, 1])
   
    tissue_phase = torch.fft.fftshift(torch.fft.ifftn(torch.fft.fftshift(chi_fft_t_kernel)))
    tissue_phase = torch.real(tissue_phase * scaling)

    return tissue_phase
</code></pre>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(p, lr=0.01)
num_iter=100
for j in range(num_iter):
    optimizer.zero_grad()
    out = net(net_input)
    out1=out.clone().detach().cpu()
    outt = forward_convolution_1(out1[0][0]).cuda()
    total_loss = mse(outt, img_noisy_torch)
    total_loss.backward()
    optimizer.step()
</code></pre>
<p>I have to put the output from model into forward_convolution(self-made function).<br>
But it continues making error of no gradient…<br>
How can I solve it?</p>

username: gphilip
cooked: <p>You could try doing the following:</p>
<ul>
<li>Include the steps of <code>forward_convolution_1</code> in the <code>forward</code> function of <code>net</code>, so that <code>net(net_input)</code> does not require any further processing before you pass it to <code>mse</code>.</li>
<li>Use <code>torch.sqrt</code> (or some combination of PyTorch functions) instead of <code>np.sqrt</code>.</li>
</ul>

username: dlfma_tjd
cooked: <p>Thank you for your advice but torch.sqrt input should be tensor…<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=10" title=":joy:" class="emoji" alt=":joy:"></p>

username: gphilip
cooked: <p>As far as I understand, PyTorch won’t be able to track gradients if you compute your output using non-PyTorch functions such as <code>numpy.sqrt</code>. I could be wrong, though.</p>

username: ptrblck
cooked: <p>You are manually detaching the model output in this line of code:</p>
<pre><code class="lang-python">out1=out.clone().detach().cpu()
</code></pre>
<p>so the error message is expected, since the detached tensor is not attached to any computation graph.</p>
[Beginner Question] Trouble with Torch.load()

username: machina
cooked: <p>Here is the start and end the .th file that my weights are stored in  :</p>
<pre><code class="lang-auto">PATH = './pytorch_resnet_cifar10/pretrained_models/resnet32-d509ac18.th'
net = resnet32().to(device)
print(torch.load(PATH))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">{'state_dict': OrderedDict([('module.conv1.weight', tensor([[[[ 4.0355e-01, -5.4608e-01, -6.8499e-01],
          [-1.0668e-01, -9.9667e-01, -1.0748e+00],
          [ 1.0510e-01, -7.8233e-01, -4.1935e-01]],

         [[ 3.7818e-01,  7.3026e-01,  2.7383e-01],
          [ 2.3540e-01,  1.0729e+00,  6.1863e-01],
          [ 1.7275e-01,  2.7025e-01,  2.5061e-01]],
                             ...

 [-0.6881, -0.7202, -0.4475, -0.3835,  0.2615,  0.6608, -0.0996, -0.0243,
         -0.3856, -0.0648, -0.2151, -0.2414, -0.1699, -0.0557, -0.2486,  1.1742,
         -0.6238, -0.2589, -0.4804, -0.3014, -0.4141, -0.1574, -0.3830, -0.1402,
          0.3194, -0.3767,  0.0905,  0.4900, -0.2189, -0.0690, -0.2826,  0.3625,
          0.8595, -0.0683, -0.1282, -0.2713, -0.3498, -0.5063, -0.0771,  0.5548,
          0.2186, -0.1179,  0.4577, -0.2268,  0.1091,  0.4121,  0.7449, -0.1425,
         -0.4718, -0.0647,  1.3254,  0.9775,  0.7825, -0.3761, -0.3092, -0.5736,
          0.6093, -0.2387,  0.1829, -0.0349,  0.3537, -0.2790,  0.9631, -0.3342]],
       device='cuda:0')), ('module.linear.bias', tensor([ 0.1763, -0.3967,  0.0932,  0.1306,  0.2050, -0.1249,  0.1557, -0.1295,
        -0.0172, -0.0863], device='cuda:0'))]), 'best_prec1': 92.78000144958496}
</code></pre>
<p>You can see at the beginning and end it has ‘state_dict’ and ‘best_prec1’, both of which are mentioned in the below error thrown when trying to load the weights via: <code>net.load_state_dict(torch.load(PATH))</code></p>
<p>output:</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for ResNet:
	Missing key(s) in state_dict: "conv1.weight", "bn1.weight", "bn1.bias", "bn1.running_mean", "bn1.running_var", "layer1.0.conv1.weight", "layer1.0.bn1.weight", "layer1.0.bn1.bias", "layer1.0.bn1.running_mean", "layer1.0.bn1.running_var", "layer1.0.conv2.weight", 

                               ...
"layer3.4.conv2.weight", "layer3.4.bn2.weight", "layer3.4.bn2.bias", "layer3.4.bn2.running_mean", "layer3.4.bn2.running_var", "linear.weight", "linear.bias". 
	Unexpected key(s) in state_dict: "state_dict", "best_prec1". 
</code></pre>
<p>Any ideas on why PyTorch isn’t liking this state_dict?</p>

username: ptrblck
cooked: <p><code>net.load_state_dict</code> expects the actual <code>state_dict</code> as its input while you are passing your custom <code>dict</code> to it.<br>
Try to use:</p>
<pre><code class="lang-python">net.load_state_dict(torch.load(PATH)['state_dict'])
</code></pre>
<p>and it should work.</p>
Training Instance Segmentation model on torchvision.datasets.Cityscapes

username: AntMorais
cooked: <p>Hello everyone,</p>
<p>I want to use the code in the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">Object Detection Finetuning Tutorial</a> to train and test a Mask-RCNN model on <a href="https://pytorch.org/vision/stable/_modules/torchvision/datasets/cityscapes.html" rel="noopener nofollow ugc">torchvision.datasets.Cityscapes</a>.</p>
<p>I am having trouble with the <em>train_one_epoch</em> function, similarly to what is happening in this issue <a href="https://github.com/pytorch/vision/issues/1699" class="inline-onebox" rel="noopener nofollow ugc">'train_one_epoch' gives error while using COCO annotations · Issue #1699 · pytorch/vision · GitHub</a>. The solution is that <em>targets</em> should be a list of dictionaries, but I don’t know how to go from the type of target that we have in the Cityscapes class (PIL image) to a list of dictionaries like that.</p>
<p>Is there an easier way to train a model on Cityscapes without having to do this?<br>
Thanks a lot,<br>
AntMorais</p>

username: ptrblck
cooked: <p>It seems that the targets are expected in the “COCO format”, so <a href="https://github.com/TillBeemelmanns/cityscapes-to-coco-conversion">this repository</a> might be useful, which seems to convert Cityscapes to COCO.</p>
How to combine losses of individual dataset elements to perform mini-batch gradient descent

username: metro
cooked: <p>Hello there!<br>
I was wondering what the correct way is to combine individual losses of elements of a dataset to perform mini-batch gradient descent.</p>
<p>I have an unsupervised loss function myLossFunction that can only process one element of the dataset at a time. So I would like to loop over the predictions and calculate the losses one by one.</p>
<ol>
<li>But how do I then combine the losses? Just torch.mean them?</li>
<li>And is it required to call loss.backward() after calculating each individual loss or is it fine where it is in the example code?</li>
</ol>
<pre><code class="lang-auto">dataloader = DataLoader(dataset, batch_size=10)

for batch in dataloader:
    predictions = model(batch)
    losses = []
    for predictions in predictions:
        losses.append(myLossFunction(prediction))

    # what do I do here to combine the losses?

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>Cheers!</p>

username: eqy
cooked: <p>Calling mean should just work, although it seems surprising that the loss function cannot work on a batch of data. There might be some slowdown because of for loop overhead rather than computing the loss in a batched way.</p>
About the behavior of Variable()

username: phantom90
cooked: <p>Hi there,</p>
<p>I am refactoring some legacy code. There is an expression:</p>
<pre><code class="lang-auto">x = tensor.zeros(10, 10)
y = Variable(x).cuda()
</code></pre>
<p>How to write the second line in a ‘modern’ way (compatible with pytorch &gt; 1.0)?</p>
<p>Thanks!</p>

username: zetyquickly
cooked: <p>Hi <a class="mention" href="/u/phantom90">@phantom90</a></p>
<p>I suppose you can go with <code>y = x.cuda()</code> or <code>y = x.to(device='cuda')</code>.</p>
<p>Please consider this <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/" rel="noopener nofollow ugc">migration guide</a></p>
Fastest way to divide a tensor for topk

username: Zoran_Peric
cooked: <p>I currently use this code to calculate top k of whole batch together rather than do separate topk for each prediction.</p>
<p>Is it possible to do this faster than this code. predictions is [batch_size,sentence_lenght,30522], and before I done  topk of predictions[batch_index,mask_index].</p>
<pre><code>            the_big_tensor = torch.zeros([len(input_data),30522], dtype=torch.float16).to('cuda')
            for in_batch_index in range(0,len(input_data)):
                masked_index = masked_indexes[in_batch_index]
                the_big_tensor[in_batch_index] = predictions[in_batch_index, masked_index]
            top_prediction2 = torch.topk(the_big_tensor,30522)
</code></pre>
<p>I suppose it is possible to transform tensor predictions[batch_index,sentence_lenght, mask_index] to tensor the_big_tensor, given that I have tensor  masked_indexes[sentence_lenght] with some tensor multiplication.</p>
Workaround for jit export error for - built-in method apply of FunctionMeta

username: Pratibha_Tripathi
cooked: <pre><code class="lang-auto">from torch.autograd import Function

class SubMConvFunction(Function):
    @staticmethod
    def forward(ctx, features, filters, indice_pairs, indice_pair_num,
                num_activate_out, algo):
        ctx.save_for_backward(indice_pairs, indice_pair_num, features, filters)
        ctx.algo = algo
        return ops.indice_conv(features,
                               filters,
                               indice_pairs,
                               indice_pair_num,
                               num_activate_out,
                               False,
                               True,
                               algo=algo)

    @staticmethod
    def backward(ctx, grad_output):
        indice_pairs, indice_pair_num, features, filters = ctx.saved_tensors
        input_bp, filters_bp = ops.indice_conv_backward(features,
                                                        filters,
                                                        grad_output,
                                                        indice_pairs,
                                                        indice_pair_num,
                                                        False,
                                                        True,
                                                        algo=ctx.algo)

        return input_bp, filters_bp, None, None, None, None

indice_subm_conv = SubMConvFunction.apply
</code></pre>
<p>This giving error that</p>
<pre><code class="lang-auto">Python builtin &lt;built-in method apply of FunctionMeta object at 0x56a2498&gt; is currently not supported in Torchscript:
if self.subm:
                out_features = Fsp.indice_subm_conv(features, self.weight,
                               ~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
                                                    indice_pairs.to(device),
                                                    indice_pair_num,

</code></pre>
<p>I know that the autograd is currently not supported in the torch jit export. My model training is done, so I’m only interested in inference and exporting. Is there a way to replace this whole thing with just a simple forward function which is supported by jit.</p>
How to bring in new_tensor into cuda kernel

username: Seungsu
cooked: <p>Hi.</p>
<p>I study about EmbeddingBag execution and i want to convey new tensor into ATen/native/cuda kernel.</p>
<p>I proceeded by adding parameters in related functions while following the method of passing weights.</p>
<p>But i got this error.</p>
<p>RuntimeError: isDifferentiableType(variable.scalar_type()) INTERNAL ASSERT FAILED at “torch/csrc/autograd/functions/utils.h”</p>
<p>Do you give me a advice when this error occurs exactly??</p>
[Possible BUG] Strange Behavior of backward

username: phantom90
cooked: <p>Hi there,</p>
<p>I have run these pieces of code, but the results are strange:</p>
<pre><code class="lang-auto">x = torch.zeros(10, 300, requires_grad=True)
loss = torch.max(x)
loss.backward(retain_graph=True)
x.grad
</code></pre>
<p>Here, x.grad is correct.</p>
<p>Another script.</p>
<pre><code class="lang-auto">x = torch.zeros(10, 300, requires_grad=True).to(device)
loss = torch.max(x)
loss.backward(retain_graph=True)
x.grad
</code></pre>
<p>x.grad becomes None.</p>
<p>Why? Thanks!</p>

username: gphilip
cooked: <aside class="quote no-group" data-username="phantom90" data-post="1" data-topic="131535">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phantom90/48/12129_2.png" class="avatar"> phantom90:</div>
<blockquote>
<p>Here, x.grad is correct.</p>
</blockquote>
</aside>
<p>I am unable to run the first script, because <code>x.backward</code> throws an error saying “grad can be implicitly created only for scalar outputs”</p>

username: phantom90
cooked: <p>My bad. I have missed something. Could you please check it again?</p>

username: phantom90
cooked: <p>Please just refresh the webpage.</p>

username: gphilip
cooked: <p>In the first example <code>x</code> is a leaf tensor, since it is a tensor that you directly created. By default, gradients are computed only with respect to leaf tensors, so <code>x.grad</code> has actual numbers in it.</p>
<p>In the second example <code>x</code> is <em>not</em> a leaf tensor, since it is <em>not</em> a tensor that you directly created: it is a tensor that you obtained by calling a function (<code>to</code>) on another tensor.  By default, gradients are computed only with respect to leaf tensors, so <code>x.grad</code> is <code>None</code>.</p>
<p>Here is some code that may help make this distinction clearer:</p>
<pre><code class="lang-auto">x = torch.zeros(10, 300, requires_grad=True)
print(x.grad)
y = x.to(device)
loss = torch.max(y)
loss.backward(retain_graph=True)
print(y.grad)
print(x.grad)
</code></pre>
<p>You can make PyTorch compute gradients with respect to a non-leaf tensor, by calling <code>retain_grad()</code> on that tensor. Here is the same example as above, with an extra call to <code>y.retain_grad()</code>:</p>
<pre><code class="lang-auto">x = torch.zeros(10, 300, requires_grad=True)
print(x.grad)
y = x.to(device)
y.retain_grad()
loss = torch.max(y)
loss.backward(retain_graph=True)
print(y.grad)
print(x.grad)
</code></pre>

username: phantom90
cooked: <p>Thank you for the clarification. Now I know what happens in .to() method.</p>
Creating a torchscript wrapper? (vol2)

username: Sele
cooked: <p>Based on this question-answer: <a href="https://discuss.pytorch.org/t/creating-a-torchscript-wrapper/93137">https://discuss.pytorch.org/t/creating-a-torchscript-wrapper/93137</a>, I wonder how can I use this class Wrapper if I need to have an audio file as input and the preprocessing function to be the one that extracts the feature tensor.</p>
Formatting Dataset and labels (Dataframes). Please Help:)

username: Benjamin_Spiegel
cooked: <p>Hi,<br>
I am new to PyTorch and it is unclear to me how I should format my dataset before feeding it into a model. In the videos and most of the postings I have found, datasets used were images, I am working with numerical dataframes to make an RNN. Currently I have a list of numpy arrays representing each seperate dataframe and a corresponding sequenced list of targets/ labels, each target corresponds to one dataframe. <strong>What is the best way to turn this data into a dataset for pytorch ?</strong>. Should I use a <strong>dictionary</strong> with keys of targets and values of arrays?  Or Should I use <strong>lists</strong> as I currently have the data arranged?  If I do use lists is there a specifc function I will want to call? Also will I want to turn all of the dataset into one tensor? will I want to turn each dataframe into a seperate dataframe? Or will I want to create batches? Lastly, if I turn each dataframe into it’s own tensor, what should I do with the corresponding target? I believe it takes more than one number to make a tensor. I imagine that this question is for beginners, but I have looked around for answers to this questions extensively and I don’t think there is much in the way of detailed explanations for this.  I would be very appreciative of any answers you could give. Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=10" title=":pray:" class="emoji" alt=":pray:"></p>

username: andreys42
cooked: <p>What kind of information is in each numpy array or dataframe?  Why You chose RNN for that? If you have classification problem I would recommend to test class ML algorithms like decision trees, forest, gradient boosting over algorithms</p>
How can I add simple modification for the forward method of ResNet50?

username: NBu
cooked: <p>Hi all,</p>
<p>How can I add one statement to the forward method before the FC layer?</p>
<p>I can achieve this if I’ve defined my own CNN model, but I am unsure how to do it with a pre-trained model (e.g., ResNet50).</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    """CNN."""
    def __init__(self):
        """CNN Builder."""
        super(CNN, self).__init__()
       
        self.newLayer = newLayer(s, True)
        
        self.conv_layer = nn.Sequential(

            ...
            ...
            ...
            
        )

        self.fc_layer = nn.Sequential(
            ...
            ...
            ...
        )

    def forward(self, x):
        # conv layers
        x = self.conv_layer(x)
        
        x = self.newLayer(x)
        
        # flatten
        x = x.view(x.size(0), -1)
        
        # fc layer
        x = self.fc_layer(x)

        return x

</code></pre>
<p>When I checked the source code for Resnet50, I found two links, I am not sure about the correct approach:<br>
<a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py" rel="noopener nofollow ugc">https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py</a><br>
<a href="https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/torchvision_models.py" rel="noopener nofollow ugc">https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/torchvision_models.py</a></p>
<p>My attempt:</p>
<pre><code class="lang-auto">class MyResNet50(ResNet):
    def __init__(self):
        super(MyResNet50, self).__init__(BasicBlock, [3, 4, 6, 3])
        self.newLayer = newLayer(s, True)

    def forward(self, input):
        x = self.features(input)

         x = self.newLayer(x)

        x = self.logits(x)
        return x
model = MyResNet50()
#load pretrained weights
model.load_state_dict(models.resnet50(pretrained=True).state_dict())
</code></pre>
<p>The error:<br>
RuntimeError: Error(s) in loading state_dict for MyResNet50:<br>
Unexpected key(s) in state_dict:</p>

username: NBu
cooked: <p>I’ve done it by following instructions in this post: <a href="https://discuss.pytorch.org/t/how-can-i-replace-the-forward-method-of-a-predefined-torchvision-model-with-my-customized-forward-function/54224/8" class="inline-onebox">How can I replace the forward method of a predefined torchvision model with my customized forward function? - #8 by Shisho_Sama</a></p>
Merging multiple ops by Pytorch C++/CUDA extension leads to slower Backward speed

username: a53153fb407732ef019a
cooked: <p>Hey guys, I want to implement this cyclic function:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6e9bc4638297702c4fc754e3e86f931e7adc0e2.png" alt="example" data-base62-sha1="snFjwZhrMxLePmIk3dJy7OmdCQW" width="524" height="393"></p>
<p>Specifically, I first implement it by the combination of five Pytorch ops.</p>
<pre data-code-wrap="python"><code class="lang-python">def origin_op(x):
    x = torch.add(x, -1)
    x = torch.remainder(x, 4)
    x = torch.add(x, -2)
    x = torch.abs(x)
    x = torch.add(x, -1)
    return x
</code></pre>
<p>However, it is rather slow so I re-implement it in CUDA.</p>
<pre data-code-wrap="c++"><code class="lang-c++">template &lt;typename scalar_t&gt;
__device__ __forceinline__ scalar_t forward1(scalar_t z) {
  auto res = fmod(z-1,(scalar_t)4);
  res += res&lt;0?(scalar_t)2:-2;
  return res; // will be cached for backward
}

template &lt;typename scalar_t&gt;
__device__ __forceinline__ scalar_t forward2(scalar_t z) {
  const auto res = fabs(z) - 1;
  return res;
}

template &lt;typename scalar_t&gt;
__device__ __forceinline__ scalar_t backward(scalar_t tmp, scalar_t d_output) {
    const auto res = tmp&gt;0? d_output:-d_output;
  return res;
}
</code></pre>
<p>Then I test the speed of each op and get<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0aa83a67df77e39d12ea15b8e800fdff4a3cb070.png" alt="speed" data-base62-sha1="1whcvU4A5HBzQaiquiPuMXLHxUA" width="381" height="158"></p>
<p>I am confused:</p>
<ol>
<li>Why the speed of backward is even SLOWER than the original implementation?</li>
<li>Why torch.sin is as fast as relu?</li>
</ol>
<p>Looking forward to somebody’s help~</p>

username: googlebot
cooked: <p>I guess you’re testing with a large tensor and experiencing memory stalls (nvidia’s profiler can report this). Maybe there is also some overhead for attaching “custom” backward into autograd, not sure here.</p>
Iterable Dataset from multiple different sequences

username: nivesh_gadipudi
cooked: <p>I am trying to figure out how to write custom iterable dataloader with multiple folders of different sequences. The issue i am currently facing is to dis-jointly return different sequences.</p>
<p>For example: Lets take two sequences (folders) with number of images 340 and 500, with a batch size of 32. when iterating over first my last batch should have 20 images similar to drop_last and continue with the next folder.</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import DataLoader, IterableDataset

class myiter(IterableDataset):

  def __init__(self, data_dir, sequences):
    self.data_dir = data_dir
    self.seqs = sequences
    self.img_paths = []
    for seq in self.seqs:
        self.img_paths.append(sorted(os.listdir(os.path.join(self.img_dir, seq)))

  def process_data(self):
    for seq in self.img_paths:
      for image in seq:
         yield img

  def __iter__(self):
    return self.process_data()

</code></pre>
<p>from the above dummy script i am able to iterate continuously but the sequences are different from each other.</p>

username: nivesh_gadipudi
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> cloud you provide me any leads on this. Thank you.</p>

username: ptrblck
cooked: <p>I think writing a custom <code>BatchSampler</code> using the “split logic” might be the easiest approach to be able to yield batches from both datasets but make sure that the samples are not overlapping.</p>

username: nivesh_gadipudi
cooked: <p>Noted <a class="mention" href="/u/ptrblck">@ptrblck</a>. I will get back to you if I need any further assistance. Thank you.</p>
CyclicLR doesn't work with Adam

username: boris
cooked: <pre><code class="lang-auto">line 60, in __init__
    cycle_momentum=False)
  File "/Users/boris/conda/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 593, in __init__
    self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))
  File "/Users/boris/conda/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 593, in &lt;lambda&gt;
    self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))
KeyError: 'momentum'
</code></pre>
<p>That’s because Adam doesn’t have momentum like SGD does. Setting cycle_momentum=False is supposed to solve this problem but it does not. Either I am doing something wrong or there’s a bug in the code.</p>

username: Time0o
cooked: <p>I’ve run into this as well, it seems like this is a bug and will be fixed in the near future, see this <a href="https://github.com/pytorch/pytorch/issues/19003" rel="nofollow noopener">issue</a>. I’m currently trying to figure out if this can be circumvented by monkey patching the Adam class somehow.</p>

username: Yiru_Shen
cooked: <p>Any updates from this thread? Have the same problem using CyclicLR with Adam</p>

username: alexandre-blanc
cooked: <p>You need to pass cycle_momentum=False when you initialize the CyclicLR instance.</p>
How to interpret the return value of torchvision box_iou

username: duddal
cooked: <p>Hello,</p>
<p>I want to calculate the iou of the predicted and ground-truth bounding boxes.</p>
<p>The pred_boxes has shape:<code> torch.Size([9, 4])</code><br>
The gt_boxes has shape: <code>torch.Size([8, 4])</code></p>
<p>So I used the <code>box_iou</code> function from <code>torchvision.ops.boxes</code></p>
<pre><code class="lang-auto">import torchvision.ops.boxes as bops
iou = bops.box_iou(gt_boxes, pred_boxes)
</code></pre>
<p>I get the following output:</p>
<pre><code class="lang-auto">tensor([[0.0716, 0.8204, 0.0000, 0.0837, 0.3265, 0.2179, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0479, 0.0000, 0.0000, 0.0000, 0.7538, 0.1388, 0.0000],
        [0.0000, 0.2521, 0.0000, 0.3208, 0.1010, 0.2012, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0222, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.1546, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.2477],
        [0.0000, 0.3131, 0.0000, 0.0000, 0.7549, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.2785, 0.4618, 0.0000, 0.0000, 0.3540, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.3132, 0.0000, 0.0322, 0.0000, 0.0000, 0.0000]])
</code></pre>
<p>How do I interpret this tensor?<br>
And how do I find the gt_boxes (yes, ground truth boxes) for which the iou score between gt and pred boxes is more than 0.5?</p>
Average Precision (AP) from scratch for bounding boxes

username: _joker
cooked: <p>I am trying to evaluate average precision for a face detection algorithm. I have created a CSV file with all the predicted bounding boxes. I have another CSV in a similar format with all the ground-truth bounding boxes. I found a lot of simple codes online to calculate bounding box overlap and calculating AP but they don’t work for a variable number of predictions given ground-truth i.e. if an image has 8 bounding box predictions whereas ground truths are 10 then how do I calculate the average precision?</p>
<p>I tried to mimic one of the algorithms and create a CSV file in the format below:</p>
<pre><code class="lang-auto"># Filename-1.jpg
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
# Filename-2.jpg
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
# Filename-3.jpg
[ x1, y1, x2, y2 ]
[ x1, y1, x2, y2 ]
...
...
...
</code></pre>
<p>I am also sharing what I tried so far:</p>
<pre><code class="lang-auto">def get_max_iou(pred_boxes, gt_box):
    """
    calculate the iou multiple pred_boxes and 1 gt_box (the same one)
    pred_boxes: multiple predict  boxes coordinate
    gt_box: ground truth bounding  box coordinate
    return: the max overlaps about pred_boxes and gt_box
    """
    # 1. calculate the inters coordinate
    if pred_boxes.shape[0] &gt; 0:
        ixmin = np.maximum(pred_boxes[:, 0], gt_box[:, 0])
        ixmax = np.minimum(pred_boxes[:, 2], gt_box[:, 2])
        iymin = np.maximum(pred_boxes[:, 1], gt_box[:, 1])
        iymax = np.minimum(pred_boxes[:, 3], gt_box[:, 3])

        iw = np.maximum(ixmax - ixmin + 1., 0.)
        ih = np.maximum(iymax - iymin + 1., 0.)

    # 2.calculate the area of inters
        inters = iw * ih

    # 3.calculate the area of union
        uni = ((pred_boxes[:, 2] - pred_boxes[:, 0] + 1.) * (pred_boxes[:, 3] - pred_boxes[:, 1] + 1.) +
               (gt_box[2] - gt_box[0] + 1.) * (gt_box[3] - gt_box[1] + 1.) -
               inters)

    # 4.calculate the overlaps and find the max overlap ,the max overlaps index for pred_box
        iou = inters / uni
        iou_max = np.max(iou)
        nmax = np.argmax(iou)
        return iou, iou_max, nmax
</code></pre>

username: duddal
cooked: <p><a class="mention" href="/u/_joker">@_joker</a> Have you managed to solve this?</p>
[Possible Bug] Forcing 3D attn_mask to have wrong shape in MultiHeadAttention

username: Fed03
cooked: <p>Hi,<br>
As you can see in the following snippet of pytorch code</p>
<p><a href="https://github.com/pytorch/pytorch/blob/4533e76e7c3df68697a526e2827d661754ab3326/torch/nn/functional.py#L5009-L5020" rel="noopener nofollow ugc">MultiHeadAttention (pytorch/torch/nn/functional.py)</a></p>
<p>when feeding a 3D attn_mask, the expected shape is (bsz * num_heads, tgt_len, src_len) and not simply (bsz, tgt_len, src_len).<br>
Can I ask why?</p>
<p>It seems to me that we have a different way to treat 2D and 3D masks: since the former gets a new leading dimension meaning the same mask will be used for every batch, I would expected a 3D mask to have a leading dimension equal to the batch_size, regardless of heads number.</p>
<p>It is possible that I don’t see the real meaning behind this behavior…<br>
Thanks</p>
JIT autograd.grad with gradient_outputs

username: leiterrl
cooked: <p>Hi! I’m trying to use autograd.grad inside a jit decorated function but am having problems with mismatching type hints that I’m unable to resolve. My problem is similar to <a href="https://github.com/pytorch/pytorch/issues/46483" class="inline-onebox" rel="noopener nofollow ugc">Error when using TorchScript with autograd · Issue #46483 · pytorch/pytorch · GitHub</a> but I also need to use the grad_outputs argument which is not covered there.</p>
<p>Running the following ( with pytorch 1.9.0 )</p>
<pre><code class="lang-auto">@torch.script.jit
def gradient(y, x):
    # grad_outputs = [torch.ones_like(y)]
    grad_outputs = torch.jit.annotate(Optional[Tensor], torch.ones_like(y))
    # grad_outputs = torch.jit.annotate(Optional[List[Optional[Tensor]]], [torch.ones_like(y)]) -&gt; "Expected a List type hint but instead got Optional[List[Optional[Tensor]]]"
    grad = torch.autograd.grad(
        [y], [x], [grad_outputs], create_graph=True, retain_graph=True
    )[0]
    return grad
</code></pre>
<p>yields the error</p>
<pre><code class="lang-auto">RuntimeError: 

aten::grad(Tensor[] outputs, Tensor[] inputs, Tensor?[]? grad_outputs=None, bool? retain_graph=None, bool create_graph=False, bool allow_unused=False) -&gt; (Tensor?[]):
Expected a value of type 'Optional[List[Optional[Tensor]]]' for argument 'grad_outputs' but instead found type 'List[Tensor]'.
Empty lists default to List[Tensor]. Add a variable annotation to the assignment to create an empty list of another type (torch.jit.annotate(List[T, []]) where T is the type of elements in the list for Python 2)
</code></pre>
<p>Any help would be greatly appreciated!</p>
<p>Cheers<br>
Raphael</p>

username: ptrblck
cooked: <p>Would you mind creating an issue on GitHub as well, so that we could track and fix it, please?</p>

username: leiterrl
cooked: <p>Thanks for the quick reply. Created the issue here: <a href="https://github.com/pytorch/pytorch/issues/64728" class="inline-onebox" rel="noopener nofollow ugc">[JIT] autograd.grad with gradient_outputs argument results in type error · Issue #64728 · pytorch/pytorch · GitHub</a></p>
Saving some feature maps to be feed to another model

username: OBouldjedri
cooked: <p>Hi, I am trying to use a pretrained model (Resnet for example, by removing the last 2 layers and adding a 1*1  conv to get the desired shaped output ) this output will be used as input for another model.</p>
<p>How can I get the  output of the new model.</p>
<p>This is the original model:</p>
<p>model = models.resnet152(pretrained=True)</p>
<p>the new model is</p>
<p>newmodel = torch.nn.Sequential(*(list(model.children())[:-2]))</p>
<p>newmodel.add_module(“convlast”,nn.Conv2d(2048, 64, 1))</p>

username: ptrblck
cooked: <p>I’m not sure, if you are seeing any errors, but wouldn’t a vanilla forward pass work?</p>
<pre><code class="lang-python">output = newmodel(input)

output = other_model(output)
</code></pre>
How to do an efficient complex index_select operation using torch?

username: 111562
cooked: <p>How to do an efficient index_select operation over  an input tensor of size <code>[1, 32, 32, 12]</code> and an index tensor of size <code>[1, 5, 2]</code> ?</p>
<p>Here is my Python code:</p>
<pre><code class="lang-auto">ret = []
for batch_idx in range(N):
    actions = []
    for agent in range(self.n_agents):
        agent_idx = obs[batch_idx][agent]
        act = pi[batch_idx, agent_idx[0], agent_idx[1]]
        actions.append(act)
    actions = torch.stack(actions)
    ret.append(actions)
ret = torch.stack(ret)

</code></pre>
<p>batch_idx = 1 # or 32 in training phase<br>
self.n_agents = 5<br>
<code>obs</code> is a  tensor index of shape <code>[1, 5, 2]</code>, which refers to the batch_size, num_of_agents, and the location coordination x and y respectively.<br>
<code>pi</code> is a policy map, whose shape is <code>[1, 32, 32, 12]</code>, which refers the batch_size, map_max_x, map_max_y, and feature_dim respectively.<br>
the <code>ret</code> is a tensor of shape <code>[1, 5, 12]</code></p>
<p>This code is two time-wasting when the batch size become larger from 1 to 32.<br>
How can I write an efficient torch style Code?</p>

username: 111562
cooked: <p>Ok, the problem is solved.<br>
Using a reshape operation before index_select is more efficient.</p>
<pre><code class="lang-python"># the example code
        pi = pi.reshape(-1, self.act_dim)
        obs = obs.reshape(N, -1)

        selected_agents_idx = []
        for batch_idx in range(N):
            batch_select_agent_idx = obs[batch_idx] + (batch_idx * self.map_size * self.map_size)
            selected_agents_idx.append(batch_select_agent_idx)
        selected_agents_idx = torch.cat(selected_agents_idx)
        
        action_selected = pi.index_select(0, selected_agents_idx)
        action_selected = action_selected.reshape(N, self.n_agents, -1)
</code></pre>
High learning rate vs

username: 111242
cooked: <p>I am aware that some previous posts discuss similar issues, but I didn’t find a solution.</p>
<p>In my situation, when I have a learning rate that is too high, such as 0.001 (I know this is not high for other instances but I guess it is for mine), I get NaNs as the output of my NN model after a few iterations. If I lower the lr to something like 0.00001, the NN outputs reasonable numbers, but the updates after each epoch are negligible (essentially seeing no updates). What is a good way to fix this?</p>

username: ptrblck
cooked: <p>Manually finding a sweet spot for the learning rate might be tricky and I believe that FastAI has written a “learning rate finder”, which tries to find a learning rate yielding the largest loss decrease, so you might want to search for their implementation.</p>
Convert Keras model to pytorch

username: Aash
cooked: <p>Hello, Can anyone help me to convert this Keras model to pytorch.</p>
<pre><code>model_in =Input(shape=(None,1024))

x = Dense(512, kernel_initializer='glorot_uniform', bias_initializer='zeros')(model_in)

x = Activation('sigmoid')(x)

x = Dropout(0.8)(x)

x = Dense(1, kernel_initializer='glorot_uniform', bias_initializer='zeros')(x)

att_weights = Activation('sigmoid')(x) 

att_mull = Multiply()([model_in,att_weights])

mean = Lambda(lambda x:K.mean(x,axis=1))(att_mull)</code></pre>

username: ptrblck
cooked: <p>I would recommend to start with <a href="https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_module.html">this tutorial</a> and try to replace the Keras layers with their PyTorch equivalents:</p>
<ul>
<li>
<code>Dense</code> → <code>nn.Linear</code>
</li>
<li>
<code>Actiation('sigmoid')</code> → <code>nn.Sigmoid</code> (or <code>torch.sigmoid</code> in the <code>forward</code> method)</li>
<li>
<code>Dropout</code> → <code>nn.Dropout</code>
</li>
<li>
<code>Multiply</code> → <code>x = x * att_weights</code> in the <code>forward</code> method (or a custom <code>nn.Module</code>, which doesn’t seem to be necessary)</li>
<li>
<code>Lambda</code> → I guess you could just use <code>x = x.mean(dim=1)</code> in the <code>forward</code> method</li>
</ul>
RuntimeError: multi-target not supported at C:/cb/pytorch_1000000000000/work/aten/src\THCUNN/generic/ClassNLLCriterion.cu:15

username: 111558
cooked: <p>this is my input shape<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/530405c49c3fb809f23b9862d56722fbe79b9ab9.png" alt="image" data-base62-sha1="bQofoogArVGbCJ1dDmqaIxYBjgl" width="212" height="101"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c03d49e3608d92af5a36b0767c3765711745540.png" alt="image" data-base62-sha1="3ZPzIomx50zGWOgBFGRte4NBn68" width="385" height="50"></p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss"><code>nn.CrossEntropyLoss</code></a> expects the model output to have the shape <code>[batch_size, nb_classes]</code> and a target in the shape <code>[batch_size]</code> containing class indices in the range <code>[0, nb_classes-1]</code> for a multi-class classification use case.<br>
Based on your screenshot it seems you are one-hot encoding the target, which is wrong.<br>
In that case, use <code>target = torch.argmax(target, dim=1)</code> to create the expected target tensor.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>
Is there a problem with using hook and fx together？

username: MartinZhang
cooked: <p>Is there a problem with using hook and fx together？</p>
<p>when I use the torch.fx to get quantizated-module, but as the same as with hook to get output’s param, it return the Proxy type, why.</p>
<p>Thanks</p>

username: MartinZhang
cooked: <p>Is there any way to change Proxy to Tensor？</p>

username: MartinZhang
cooked: <pre><code class="lang-auto">    def __init__(self, node: Node, tracer: 'Optional[TracerBase]' = None):
        if tracer is None:
            # This allows you to create a Proxy object around a raw Node
            tracer = GraphAppendingTracer(node.graph)
        self.tracer = tracer
        self.node = node
</code></pre>
Symbolic trace with *args

username: Arseny
cooked: <p>Hi all.</p>
<p>I’m trying to trace model with positional arguments <code>*args</code> in <code>forward</code>.<br>
The simplified problem is <code>Container</code> that forwards positional arguments to its <code>Module</code>.<br>
When I pass <code>Container</code> to <code>fx.symbolic_trace</code> I get the following error:</p>
<p><code>torch.fx.proxy.TraceError: Proxy object cannot be iterated. This can be attempted when used in a for loop or as a *args or **kwargs function argument.</code></p>
<p>This happened because all arguments with prefix <code>*</code> were replaced by <code>Proxy</code>.</p>
<p>What can be done to solve this problem?</p>
<pre><code class="lang-python">from torch import Tensor
import torch.nn as nn
import torch.fx as fx

class Module(nn.Module):
    def forward(self, x):
        return x

class Container(nn.Module):
    def __init__(self):
        super().__init__()
        self._module = Module()

    def forward(self, *args):
        return self._module(*args)

model = Container()
model(Tensor())

trace = fx.symbolic_trace(model)  # Error.
</code></pre>
<p>Thanks.</p>

username: MartinZhang
cooked: <pre><code class="lang-auto">from torch import Tensor
import torch.nn as nn
import torch.fx as fx

class M(nn.Module):
    def forward(self, x):
        return x

class Container(nn.Module):
    def __init__(self):
        super().__init__()
        self._module = M()

    def forward(self, *args):
        return self._module(args)  # set a variable to store

model = Container()
model(Tensor())

trace = fx.symbolic_trace(model)
</code></pre>

username: MartinZhang
cooked: <pre><code class="lang-auto">    ``Proxy`` objects cannot be iterated. In other words, the symbolic
    tracer will throw an error if a ``Proxy`` is used in a loop or as
    an ``*args``/``**kwargs`` function argument.
</code></pre>
Calculating F.cross_entropy for each class separately

username: meshghi
cooked: <p>Hello, I’m implementing a research paper, and I need to calculate the F.cross_entropy for each class separately as I will be using them later separately before joining them as the final loss (like the attached image). I was wondering how I could do it? Thanks.<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3ddadaa635bd0b5f8e1f362d4081a507518ebbf3.png" alt="Screen Shot 2021-09-07 at 12.50.49 PM" data-base62-sha1="8Pc18T4kOBvTcIrQLOUWcmKhpgn" width="590" height="102"></p>

username: ptrblck
cooked: <p>I’m not entirely sure, but you could try to use the unreduced loss via <code>reduction='none'</code>, apply your custom operations, and reduce it afterwards.</p>

username: meshghi
cooked: <p>Thank you. As you said, it gave me the individual terms (per observation cross_entropy) that I separated out in order to create each of the two summation terms.</p>
Help me with this piece of code

username: saurav23
cooked: <p>import torch</p>
<p>x = torch.tensor([[1., -1.], [1., 1.]], requires_grad=True)<br>
out = x.pow(2).sum()<br>
out.backward()<br>
x.grad</p>

username: ptrblck
cooked: <p>What kind of issues are you seeing with this code?</p>

username: MartinZhang
cooked: <pre><code class="lang-auto">import torch

x = torch.tensor([[1., -1.], [1., 1.]], requires_grad=True)
out = x.pow(2).sum()
out.backward()
print(x.grad)
</code></pre>