akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Hessian vector product of each gradient separately

username: ITs
cooked: <p>I trying to compute the Hessian vector product with an unused vector in a customized optimizer. What is optimized is Logistic Regression where the output is a `{Tensor: (10, 784) }’, that is a 10 class problem with an input of dimension  784. For simplicity lets say that we do not have a bias term. To implement the optimizer I am using the PyTorch optimizer interface with closure to get the loss.  I have tried some examples like this</p>
<pre><code>_grads = grad(loss, params, grad_outputs=None, only_inputs=True, retain_graph=True, create_graph=True)
</code></pre>
<p>and this</p>
<pre><code>v = tuple([torch.ones_like(_, requires_grad=True) for _ in group['params']])
v_hess = grad(_grads, params, grad_outputs=v, only_inputs=True, retain_graph=True,create_graph=True)
</code></pre>
<p>and seems to work. However, what I need in my case is to compute the Hessian vector product for a specific gradient in <code>_grads</code> with a  predefined vector (<code>grad_outputs</code>) like this</p>
<pre><code>for p in group['params']:
if p.grad is not None:
   params.append(p)
   groups.append(group)
   grads.append(p.grad)

for grad_x, param_x in zip(grads_x, params_x):
   dummy_vector = torch.ones_like(grad_x, requires_grad=True)
   Hdv = grad(grad_1_out_of_10, param_1_out_of_10, torch.ones_like(grad_1_out_of_10, requires_grad=True), allow_unused=True)
</code></pre>
<p>but <code>Hdv</code> is <code>None</code>. As an example of my problem think of that when you want to compute the conjugate directions where the output vector is initialized with the -grad_1_out_of_10 and Hessian vector products are needed in each iteration. Is it possible or meaningful to compute something like that? Could you please give an example?</p>

username: ITs
cooked: <p>I found some useful posts like <a href="https://github.com/rlworkgroup/garage/blob/master/src/garage/torch/optimizers/conjugate_gradient_optimizer.py" rel="noopener nofollow ugc">this</a> that solved my question.</p>
AdaptiveAvgPool2d gives TypeError: 'tuple' object is not callable

username: laro
cooked: <p>I want to check the results of AdaptiveAvgPool2d.<br>
I built a simple test:</p>
<blockquote>
<p>import torch<br>
import torch.nn as nn</p>
<p>total = 256 * 500 * 4 * 4<br>
x     = torch.arange(0, total).view(256, 500, 4, 4).float()<br>
p     = nn.AdaptiveAvgPool2d(1),<br>
res   = p(x)</p>
</blockquote>
<p>I’m getting error:</p>
<blockquote>
<p>—&gt; 7 res   = p(x)</p>
<p>TypeError: ‘tuple’ object is not callable</p>
</blockquote>
<ol>
<li>Why I’m getting this error ?</li>
<li>How can I run AdaptiveAvgPool2d on my created tensor and check the results ?</li>
</ol>

username: mMagmer
cooked: <aside class="quote no-group" data-username="laro" data-post="1" data-topic="139542">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laro/48/47125_2.png" class="avatar"> laro:</div>
<blockquote>
<p>p = nn.AdaptiveAvgPool2d(1),</p>
</blockquote>
</aside>
<p>there is a typo, change it to <code>p = nn.AdaptiveAvgPool2d(1)</code> (remove ‘,’)</p>
Is there any paper or any existing models tries to train weights for weighted sum like this?

username: EJHyun0214
cooked: <p>Hi<br>
I know it’s possible to train weights for weighted sum<br>
for example,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/204c22bda3c5033416c8bf74699ed025e864ae01.png" alt="image" data-base62-sha1="4BIoJXosnj7Nhkh4W810FAmkFIR" width="569" height="63"><br>
like this.</p>
<p>But i’m wondering<br>
Is there any paper or any existing models<br>
tries to train weights for weighted sum like this?</p>
<p>please let me know<br>
Thanks</p>

username: mMagmer
cooked: <p>There is <a href="https://pytorch.org/docs/stable/generated/torch.lerp.html" rel="noopener nofollow ugc">this</a> nn module in PyTorch that does a linear interpolation of two tensors.<br>
And probably there is a use case for it.</p>
Asterisk when passing input to nn.Module

username: nilakshan_kunananth1
cooked: <p>What is the use of asterisk when parsing the inputs to nn.Module?</p>
<pre><code class="lang-auto">x = torch.ones((1,4,5,6))

m = nn.Identity()

print(x.shape, m(x).shape,m(*x).shape)

</code></pre>
<p>This gave me</p>
<blockquote>
<p>torch.Size([1, 4, 5, 6]) torch.Size([1, 4, 5, 6]) torch.Size([4, 5, 6])</p>
</blockquote>
<p>Why does it squeeze batch dimension in 3d case?</p>

username: mMagmer
cooked: <p>Torch tensor have some same behavior as python lists.<br>
*tensor will unpack first dim of tensor into list of tensors (in your case list of one tensor with shape [4,5,6])<br>
your code raises an error for first dimension grater than 1.<br>
You can fix it as follows,</p>
<pre><code class="lang-auto">x = torch.ones((2,4,5,6))

m = torch.nn.Identity()

print(x.shape, m(x).shape,[out.shape for out in m([*x])])
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt;
torch.Size([2, 4, 5, 6]) torch.Size([2, 4, 5, 6]) [torch.Size([4, 5, 6]), torch.Size([4, 5, 6])]
</code></pre>
Multiprocessing shared memory

username: andrei-rusu
cooked: <p>Hi all,</p>
<p>Due to a known memory limitation that causes errors on Windows when importing torch as multiple processes get spawned (refer to <a href="https://stackoverflow.com/questions/64837376/how-to-efficiently-run-multiple-pytorch-processes-models-at-once-traceback" class="inline-onebox" rel="noopener nofollow ugc">python - How to efficiently run multiple Pytorch Processes / Models at once ? Traceback: The paging file is too small for this operation to complete - Stack Overflow</a>), I would like to avoid any unnecessary memory usage when training my distributed model. As such, I switched to torch.multiprocessing in the hope that at least the model parameters will be shared among processes.</p>
<p>Now, I would like to understand what and how are Tensors actually shared among child mp.Processes because it seems to me that the data within Tensors remains consistent throughout multiprocessing when these get passed to ‘args’ (I haven’t used mp.Queue in my tests yet), even when I’m simply importing the default ‘multiprocessing’ package, and despite omitting the ‘share_memory_()’ calls altogether? This makes me suspect the memory is not actually shared, but copied around a lot, just like using a Manager.dict() or so.</p>
<p>Another ‘strange’ thing that I observed is that an ‘args’ Tensor’s ‘is_shared()’ method does return False outside the mp.Process startup when no calls to ‘share_memory_()’ are being made, but returns True within the child processes to which it was sent no matter what (again, even if torch.multiprocessing is never imported). What is more, despite my best efforts in inspecting ‘data_ptr’ or the backend ‘storage’ objects, I was not able to confirm whether my Tensor data is ever truly shared or not.</p>
<p>These aspects also beg my final question: How would I check whether the parameters of my model are actually living in a true shared memory space or not?</p>
<p>Thanks for your time!</p>
In optimizer.zero_grad(), set p.grad = None?

username: marmelad
cooked: <p>Hi, I have been looking into the source code of the optimizer, zero_grad() function in particular.</p>
<pre><code class="lang-auto">    def zero_grad(self):
        r"""Clears the gradients of all optimized :class:`torch.Tensor` s."""
        for group in self.param_groups:
            for p in group['params']:
                if p.grad is not None:
                    p.grad.detach_()
                    p.grad.zero_()
</code></pre>
<p>and I was wondering if one could just exchange</p>
<pre><code class="lang-auto">p.grad.detach_()
p.grad.zero_()
</code></pre>
<p>with<br>
<code>p.grad = None</code></p>
<p>In what cases these two options would make difference?</p>

username: albanD
cooked: <p>Hi,</p>
<p>There is one main difference:<br>
The main difference is that the Tensor containing the gradients will not be reallocated at every backward pass. Since memory allocation is quite expensive (especially on GPU), this is much more efficient.</p>
<p>There are other subtle differences between the two like some optimizers that behave differently if a gradient is 0 or None. It am sure there are other places that behave like that.</p>

username: marmelad
cooked: <p>Thank you very much for the response!</p>

username: SofiaCP
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>.</p>
<p>Will setting the grads to none (instead of zero) avoid that an optimizer’s internal states (momentum, weight decay etc) update the model in those parameters?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/how-to-freeze-or-fix-the-specific-subset-partial-weight-in-convolution-filter/76050/7">here</a>.</p>
default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.PngImagePlugin.PngImageFile'>

username: abhishekstha98
cooked: <pre><code class="lang-auto">train_transforms = transforms.Compose(
    [
        transforms.Resize((224, 224)),
        transforms.ToTensor(),        
    ]
)

train_data = datasets.VOCSegmentation(root='./data', image_set ='train', download=False, transform=train_transforms)

train_loader = DataLoader(dataset = train_data, batch_size=batch_size, shuffle=True)

for data, label in train_loader:
        # data = data.PILToTensor()
    print(data.shape)
</code></pre>
<blockquote>
<p>Blockquote File “C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 86, in default_collate<br>
raise TypeError(default_collate_err_msg_format.format(elem_type))</p>
</blockquote>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.PngImagePlugin.PngImageFile’&gt;</p>
Profiling computations at the operation level

username: Mafuyu_Aizawa
cooked: <p>I am trying to profile all the addition or multiplication computations completed at the very bottom operation level (two input operands and the result, for example, 2 + 3 = 5) during the inference of a model.</p>
<p>Is it possible to achieve this in python (such as the pytorch profiler) or I need to look into the C++ back end? (Will also need some hints if need to go deep into the C++)</p>
<p>Thank you!</p>

username: hemeng
cooked: <p>Based on my experience.</p>
<p>If you only want to calculate the execution time of some operation. <a href="https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html" rel="noopener nofollow ugc">Tensor Broad</a> could help you. No matter your code is running on CPU or GPU.</p>
<p>If you want to dig into the GPU kernel level for more detailed information like SM utilization or memory workload, you can use Nsight Compute to profile your program. It can provide detailed information of GPU utilization.</p>

username: Mafuyu_Aizawa
cooked: <p>Thank you for your suggestions! Actually what I am trying to do is to replace the multiplications and additions in the convolutions with my custom operations (like myadd, mymult) for some purposes so I am trying to find a way to do so… (I only need to replace during forward pass though)</p>

username: tom
cooked: <p>The <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="noopener nofollow ugc">PyTorch profiler</a> does exactly that for PyTorch functions and has good analysis options.</p>

username: hemeng
cooked: <p>Then you may get some help <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">here</a>.</p>
(FileAdapter at ..\caffe2\serialize\file_adapter.cc:11) (no backtrace available)

username: wsy
cooked: <p>(FileAdapter at …\caffe2\serialize\file_adapter.cc:11)(no backtrace available)<br>
when I load a .pt file by ‘torch.load’,It will report this error.I would be very grateful if you can help me solve it</p>
How to copy specific layer strucutres and weights to a new model

username: Olii
cooked: <p>Hi, how can I copy specific layer structures and weights to a new model? I need to use Resnet but with additional layers and operations. What would be the best practice?</p>
<p>here’s my sample code</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, args1, args2):
        super().__init__()

        resnet = models.resnet50(pretrained=True)
        # conv1, bn, relu, maxpool
        self.stem = resnet.stem
        # resnet block 1,2
        self.layer1 = resnet.layer1
        self.layer2 = resnet.layer2
        # some new layers
        self.layer3 =  MultiInputBlock(args)
    def forward(self, x, y):
        x = self.stem(x)
        x = self.layer1(x)
        x = self.layer2(x)
        if some_conditions:
           x = self.layer3(x, y)
        return x
</code></pre>

username: ptrblck
cooked: <p>Your code looks alright and should work assuming you fix the attribute names.<br>
Alternatively, you could also create a custom model by deriving from a resnet implementation in case that’s easier or cleaner.</p>
Array Index operation

username: cbd
cooked: <p>In below code “tensor_a” is one dimensional tensor with all value 1. From “tensor_b” i want to find out the index for which all the channel values are greater than 100 and “3rd channel” value should be  greater than value of “1st channel” and “2nd channel”. That index value in “tensor_a” should be replace with 0.<br>
In below code, “tensor_a” first value satisfy that condition(as first value of each channel is 160,185,210 respectively) .so b[0,0,0,0] should be 0.</p>
<pre><code class="lang-auto">b=torch.ones(1,1,3,3)
a=torch.FloatTensor([[[[160, 56, 20],
          [54, 6, 97],
          [65, 119, 56]],

         [[185, 13, 90],
          [10, 6, 220],
          [67, 88, 4]],

         [[210, 92, 74],
          [48, 217, 86],
          [42, 12, 24]]]])</code></pre>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">a = torch.tensor([[[[160, 56, 20],
                    [54, 6, 97],
                    [65, 119, 56]],
                 
                   [[185, 13, 90],
                    [10, 6, 220],
                    [67, 88, 4]],

                   [[210, 92, 74],
                    [48, 217, 86],
                    [42, 12, 24]]]]).float()

idx = (a[:, 2] &gt; a[:, 0]) &amp; (a[:, 2] &gt; a[:, 1]) &amp; (a[:, 2] &gt; 100.)
result = idx.float()
print(result)
&gt; tensor([[[1., 0., 0.],
           [0., 1., 0.],
           [0., 0., 0.]]])
</code></pre>
When the program attempt to conduct 'from torch._C import *', it would get stuck

username: zhenshan_wang
cooked: <p>I wanted to conduct the code following<br>
‘from torch._C import *,’<br>
at the first line in my python file,<br>
but when I was doing this, the program would stay there for a long time and do nothing<br>
how can I handle it<br>
thanks!!</p>

username: ptrblck
cooked: <p>Could you describe your setup a bit more, please?<br>
I.e. in particular:</p>
<ul>
<li>how did you install PyTorch</li>
<li>what exactly are you trying to import</li>
<li>is <code>import torch</code> also taking a long time</li>
<li>is this a new issue, was it working before, or always hanging/slow?</li>
</ul>

username: zhenshan_wang
cooked: <p>Thanks for you reply!<br>
I installed pytorch with the comand recommend  like this</p>
<pre><code class="lang-auto">conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.1 -c pytorch
</code></pre>
<p>Actually I did intend to import torch,  but it failed, and we found it was the following code</p>
<pre><code class="lang-auto">from torch._C import *
</code></pre>
<p>in torch’s <strong>init</strong>()  caused this issue. And when I attemped to run a python file test.py which only contains the above code in Xshell using the comand :  python3 test.py, it would work with a quite short time.<br>
I hope these imformation can help you!<br>
Thaks again.</p>

username: ptrblck
cooked: <p>Thanks for the update!</p>
<aside class="quote no-group" data-username="zhenshan_wang" data-post="3" data-topic="139365">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhenshan_wang/48/44917_2.png" class="avatar"> zhenshan_wang:</div>
<blockquote>
<p>in Xshell using the comand : python3 test.py, it would work with a quite short time.</p>
</blockquote>
</aside>
<p>Since it’s working fine in <code>Xshell</code>, what’s the setup causing the slow execution/hang and how do these setups differ? It sounds as if your environments might be different (e.g. re you using conda/pip envs?) and one of them might be broken.</p>

username: zhenshan_wang
cooked: <p>Yes，you are right!<br>
After I decided to remove my conda env and recreate  the env,  then reinstall torch,  it worked.<br>
I should realize this cause a bit earlier.<br>
Finally, thaks for your reply with your patience.</p>
How to copy a predefined nn.Sequential properly and make it trainable?

username: aktgpt
cooked: <p>I want to copy a pre-defined <code>nn.Sequential()</code> like the <code>torchvision</code> ResNet <code>layer1</code> and make a new module that is trainable, without going through the pain of using <code>_make_layer</code> function. Is there an easy and safe way to do this?</p>
<p>Something like this:</p>
<pre><code class="lang-auto">class MyModel():
    def __init__(self):
        model = torchvision.models.resnet18(pretrained=True)
        self.layer1 = model.layer1
        self.layer11 = safe_copy_way(self.layer1)

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer11(x)
        return x 
</code></pre>

username: ptrblck
cooked: <p>Yes, you can reassign modules as seen in your code snippet. You would have to change the attribute names of course to valid ones and you could use <code>copy.deecopy</code> for <code>safe_copy_way</code>.</p>
How to multiply tensors of different size without using `repeat`?

username: SamRoss
cooked: <p>I would like to do an in-place multiplication of tensors as follows:<br>
<code>Q</code> is a tensor of size <code>bxnxn</code> (We have <code>b</code> batches of different <code>Q</code>'s)<br>
<code>T</code> is a tensor of size <code>nxn</code><br>
For each batch, I would like to perform <code>T*Q</code>. My current method is to first do <code>T = T.repeat(b,1,1)</code> and then do <code>T*Q</code>. This seems to be wasteful as the copies of <code>T</code> take up valuable space in GPU memory. Is there a better way to do this?</p>

username: ptrblck
cooked: <p>You might want to avoid explicitly calling <code>repeat</code> on <code>T</code> and let PyTorch use broadcasting as it would lower the memory usage as seen here:</p>
<pre><code class="lang-python">import torch

print(torch.cuda.memory_allocated()/1024**2) # 0
print(torch.cuda.max_memory_allocated()/1024**2) # 0


b, n = 1024, 1024

Q = torch.randn(b, n, n, device='cuda') # 4096 GB
T = torch.randn(1, n, n, device='cuda') # 4 MB

print(torch.cuda.memory_allocated()/1024**2) # 4100 MB
print(torch.cuda.max_memory_allocated()/1024**2) # 4100 MB

#T = T.repeat(b, 1, 1) # 4GB
#print(torch.cuda.memory_allocated()/1024**2) # 8192 MB
#print(torch.cuda.max_memory_allocated()/1024**2) # 8196 MB

y = T*Q # 4096 MB

print(torch.cuda.memory_allocated()/1024**2) # 8196 MB / 12288 MB with repeat
print(torch.cuda.max_memory_allocated()/1024**2) # 8196 MB / 12288 MB with repeat
</code></pre>
<p>You can reuse this code to check the memory usage for your use case and compare both approaches.</p>
Extract gradients of a model like model.state_dict()

username: Bruce_zhuang
cooked: <p>Hi, I understand that to extract a model’s parameters’ state can be done through<br>
`state = model.state_dict()’. Is it possible to get the model’s gradients with something like ‘gradients = model.gradient_dict()’ so that I can do ‘model.load_gradient_dict(gradients)’ back to the model? Any help will be much appreciated!</p>

username: LeviViana
cooked: <p>The following snippet allows you to get a sort of <code>gradient_dict</code>:</p>
<pre><code class="lang-python">import torch
net = torch.nn.Linear(2, 3)
x = torch.rand(4, 2).requires_grad_(True)
loss = net(x).sum() ** 2
loss.backward()
grad_dict = {k:v.grad for k, v in zip(net.state_dict(), net.parameters())}
print(grad_dict)
</code></pre>

username: BruceDai003
cooked: <p>Just want to mention that one should be cautious when using this line</p>
<pre><code class="lang-auto">grad_dict = {k:v.grad for k, v in zip(net.state_dict(), net.parameters())}
</code></pre>
<p>as <code>net.state_dict()</code> 's keys is not guranteed to be in the same order as in <code>net.parameters()</code>, as I tried this in my SSD model. So I would sugget to use:</p>
<pre><code class="lang-auto">grad_dict = {k:v.grad for k,v in net.named_parameters()}
</code></pre>
<p>(Maybe in that time(2 years ago), we don’t have this issue or <code>named_parameters</code> is not available, I’m not sure, but I think it’s better to use <code>named_parameters</code> nowadays)</p>
MultiHeadAttention identical outputs in Eval mode

username: bglick13
cooked: <p>I’m using the TransformerEncoder module and have noticed that in eval mode, I get the same output sequence regardless of input. I’ve tracked this down to the MultiHeadAttention module, specifically line 3112:</p>
<pre><code class="lang-auto">q, k, v = linear(query, in_proj_weight, in_proj_bias).chunk(3, dim=-1)
</code></pre>
<p>where q gets squashed to the same values for every position in the input sequence. This doe not happen when the dropout is active (i.e., during training). I suppose this could just be due to overfitting, but I was wondering if anyone had encountered a similar problem or had ideas for what else could be causing it.</p>

username: idontlikelongname
cooked: <p>encounter the same issues</p>
How to debug ‘Loss is NaN’ in Vision Transformer when using DistributedDataParallel?

username: Longday0923
cooked: <p>Hello Everyone,<br>
I am new to pyotorch and learning quickly.</p>
<p>I’m trying to train a <a href="https://arxiv.org/abs/2012.12877" rel="noopener nofollow ugc">DeiT</a> from scratch. Obeying the strategy in the original paper, I use a batch size of 256 and 8Gpus(2080Ti) parallel training.</p>
<p>According to <a href="https://arxiv.org/abs/2111.09883" rel="noopener nofollow ugc">SwinV2</a>, I try to move LayerNorm beind the residual module(attention or MLP) in the original DeiT, as showed below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edb4fe5a2c0ff5ee3efa5ec46eb3614bb2585a70.png" data-download-href="https://discuss.pytorch.org/uploads/default/edb4fe5a2c0ff5ee3efa5ec46eb3614bb2585a70" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edb4fe5a2c0ff5ee3efa5ec46eb3614bb2585a70_2_690x299.png" alt="image" data-base62-sha1="xUQW7qS0zj3d0xmgPsdTtfgnaxy" width="690" height="299" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edb4fe5a2c0ff5ee3efa5ec46eb3614bb2585a70_2_690x299.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edb4fe5a2c0ff5ee3efa5ec46eb3614bb2585a70_2_1035x448.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edb4fe5a2c0ff5ee3efa5ec46eb3614bb2585a70_2_1380x598.png 2x" data-dominant-color="F6F3F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1417×616 89.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Unfortunately I didn’t get a more stable training as SwinV2 said, but resulted in ‘Loss is NaN’, after about 3 epochs of training. I’ve set the warmup and learning rate same as DeiT.</p>
<p>I don’t know how to debug in DistributedDataParallel mode. Could somebody give some suggestions? Or could somebody give some insights in training Vision Transformer?</p>
<p>Thanks a lot in advance for help.</p>
Why can't I outperform tensorflow in pytorch?

username: Yasar_Niyazoglu
cooked: <p>Hi. I did a study on private data with pytorch. but I didn’t get a good result and my training period was too long. I got a very very short training time and a very high accuracy rate when I used tensorflow to work the same, what could be the reason. note: the same model and the same optimization algorithm in both</p>

username: tom
cooked: <p>Typically, people with this problem don’t actually have as much “the same model and the same optimization algorithm” as they think. That is because if you had, you would get comparable results.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Yasar_Niyazoglu
cooked: <p>So how do we know if it’s exactly the same? I wrote the code for both of them myself. everything looks the same</p>

username: tom
cooked: <p>What I do is that I check that same input produces same output, then same grads then same optimizer step (with “same” up to numerical accuracy). Many operations have subtle difference between different frameworks, e.g. whether a “mean” is taken or a “sum” or similar.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Yasar_Niyazoglu
cooked: <p>Can you take a look if I send you the codes?</p>

username: mMagmer
cooked: <p>There are differences in how some modules are implementing in TensorFlow vs PyTorch.<br>
i.e. batch norm momentum, .1 in PyTorch works same as .9 in TensorFlow.<br>
Paddings are slightly different too.</p>

username: tom
cooked: <p>I’m fully booked this year, sorry.</p>

username: Yasar_Niyazoglu
cooked: <p>thank you again for everything</p>

username: lola
cooked: <p>I agree with Tom that you cannot just think your code look that same and the results have to be the same.<br>
If you want to make sure everything is the same, layer by layer, you need to have testing for that, from layer to sequential of layers between TensorFlow and Pytorch.</p>
<p>Just for reference: <a href="https://github.com/jalola/compare-tensorflow-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jalola/compare-tensorflow-pytorch: Compare outputs between layers written in Tensorflow and layers written in Pytorch</a></p>
How can I solve this IO problem

username: zeming_hou
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d222006e882bdc29a3a81687311fc0dcbed84bb1.png" alt="image" data-base62-sha1="tYV7g1uJX5SXDgwKEWXEq3MXK6Z" width="613" height="395"></p>
<p>As you can see I need to use this kind of data during my training process, so IO has been a huge problem.<br>
I also use DDP, which makes me tougher to adjust my code.<br>
Is there a way in pytorch can handle this problem? Help me, please.</p>

username: ptrblck
cooked: <p>What kind of IO problem are you seeing?</p>

username: zeming_hou
cooked: <p>Loading data for training might take 10 seconds+，which makes my GPU -util 0% at the most of time.</p>

username: ptrblck
cooked: <p>You could try to increase the number of workers so that the loading latency might be hidden in the background while the model is training. Also, if you are using an older HDD, try to move the data to a faster SSD to speed up the loading time.</p>

username: zeming_hou
cooked: <p>I have already tried to increase the number of workers, and I found 4 is the best for my code.<br>
The size of training data is about 400G, I don’t have SSD as that big.<br>
But my HHD should be fast enough, almost 6GB/s.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50db008f8a3ea3c583c51eb11094ceb96e78c07e.png" alt="image" data-base62-sha1="bxhph3Sr0eBKwAlohKdJKG5qQ5U" width="538" height="181"></p>

username: ptrblck
cooked: <p>I don’t think your HDD can achieve this bandwidth and based on e.g. <a href="https://www.newegg.com/hgst-hus728t8tale6l4-8tb/p/1Z4-001J-004Z9">these newegg specs</a> you could get ~255MB/s:</p>
<blockquote>
<p>this 3.5-inch drive features 8TB capacity, 7200RPM spinning speed, SATA 6 Gbps host interface and 256 MB Cache, and delivers sustained transfer rate of up to Up to 255 MB/s (for reference only)</p>
</blockquote>
<p>You could also profile the read speed to check the “for reference only” claim and might see a lower performance.</p>

username: zeming_hou
cooked: <p>I think the main problem might not be the speed of my HDD but the way I load files.<br>
Every training I have to load 64 files of the size of 1.7M.<br>
The operation to load files of this size might cause cpu jam.</p>

username: zeming_hou
cooked: <p>Is there any way I can build a big file to save the data stocking in this file?<br>
Loading a big file should be faster than a bunch of files of not small size.</p>

username: ptrblck
cooked: <p>Yes, you could load multiple numpy files in a script, concatenate the arrays (assuming their shape allows it), and save the newly created (bigger) numpy array again.</p>

username: zeming_hou
cooked: <p>Maybe LMDB is a feasible way to solve my problem, what do you think?</p>
Torch function runtime dependent on scipy call?

username: wwen_waabi
cooked: <p>I’ve noticed that certain PyTorch functions run slower when I make calls to <code>scipy.linalg.solve_triangular</code>.</p>
<p>I have managed to reproduce this issue with the following script:</p>
<pre><code class="lang-auto">import time

import numpy as np
import torch
import scipy.linalg

def torch_fn(input_size):
    with torch.no_grad():
        a = torch.randn(input_size, 4, 4, dtype=torch.float64)
        b = torch.randn(input_size, 4, 5, dtype=torch.float64)
        begin = time.perf_counter()
        c = torch.bmm(a, b)
        return time.perf_counter() - begin


def scipy_fn(input_size):
    a = np.asarray([[1, 0], [1, 1]], dtype=np.float64)
    b = np.random.randn(input_size, 2).astype(np.float64)
    begin = time.perf_counter()
    c = scipy.linalg.solve_triangular(a, b.T, lower=True).T
    return time.perf_counter() - begin


def avg_time(torch_input_size, scipy_input_size):
    tot = 0
    for _ in range(100):
        tot += torch_fn(torch_input_size)
        if scipy_input_size &gt; 0:
            scipy_fn(scipy_input_size)
    return tot / 100


if __name__ == '__main__':
    print(avg_time(5, 0))
    print(avg_time(5, 3))
    print(avg_time(5, 75))
    print(avg_time(30, 0))
    print(avg_time(30, 3))
    print(avg_time(30, 75))
</code></pre>
<p>The output I get is</p>
<pre><code class="lang-auto">0.0002978778630495071
8.901087567210197e-06
0.006054648053832352
0.0008917823247611523
8.631939999759198e-06
0.004081738111563027
</code></pre>
<p>So it seems like making a <code>scipy.linalg.solve_triangular</code> call with sufficiently large inputs results in <code>torch.bmm</code> running much slower. The runtime difference may not seem too significant, but in our codebase, we call other torch functions and we observe much larger differences (e.g. 30ms vs. &lt; 0.5ms if we call/do not call scipy).</p>
<p>I suspect the issue may have to do with the underlying linear algebra library that scipy calls (in the above case, the <code>trtrs</code> function in lapack/blas). Beyond this, I can’t explain why this runtime discrepancy occurs.</p>
<p>I’m wondering if someone might have an explanation for this?</p>

username: ptrblck
cooked: <p>Another interesting thing is that the order also seems to matter, as the results are different when this is executed (at least in my setup):</p>
<pre><code class="lang-python">if __name__ == '__main__':
    print(avg_time(30, 75))
    print(avg_time(30, 3))
    print(avg_time(30, 0))
    print(avg_time(5, 75))
    print(avg_time(5, 3))
    print(avg_time(5, 0))
</code></pre>
<p>Would you mind creating an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we could track and check it?</p>
Pytorch for vfx

username: VFX
cooked: <p>Hi all,</p>
<p>In new to this forum so forgive me if I ask any stupid questions. I come from the film visual effects industry and recently one of the main vfx softwares - Nuke from the Foundry added the ability to use pytorch files. There isn’t much further information on this at present so im just reaseaching and trying to find out any pytorch models that could be useful for visual effects and image manipulation. If you have any recommendations please post below</p>
<p>thank you!</p>

username: ptrblck
cooked: <p>I’m not familiar with this software stack, but some ideas which models might be useful:</p>
<ul>
<li>segmentation models e.g. to blur content/remove it</li>
<li>super-resolution models</li>
<li>any kind of “filters” e.g. changing the appearance of the scene/person (e.g. different color of the clothing, hair etc.)</li>
<li>super-slowmotion</li>
<li>image/video stabilization</li>
<li>Generative models to create new scenes</li>
<li>denoising or general image enhancement (e.g. “night mode” enhancement)</li>
</ul>

username: VFX
cooked: <p>Slow motion retimes would be very useful, depth maps, any face manipulation would be great. If you know of anything that you thibk could be of interest please paste links im still new and finding it difficult to find where to look</p>

username: ptrblck
cooked: <p>I would probably just search for (updated) GitHub repositories and check if you could use these models. E.g. for SuperSlomo I’ve found <a href="https://github.com/avinashpaliwal/Super-SloMo">this repo</a>, which is quite outdated by now so I would expect that some code changes are needed.</p>
Error while using 16-bit floats (.half())

username: blade
cooked: <p>I’m trying to run my code using 16-nit floats. I convert the model and the data to 16-bit with no problem, but when I want to compute the loss, I get the following error:</p>
<blockquote>
<p>return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)<br>
RuntimeError: “nll_loss_forward_reduce_cuda_kernel_2d_index” not implemented for ‘Half’</p>
</blockquote>
<p>I’m wondering if there is something that I have to do or simply my choice of loss function does not support 16-bit floats. Thanks!</p>

username: ptrblck
cooked: <p><code>nn.NLLLoss</code> and thus also <code>nn.CrossEntropyLoss</code> don’t support <code>float16</code> tensors on the CPU, if I’m not mistaken, so you could either use the GPU or transform the model output tensor to <code>float32</code> before calculating the loss.<br>
Also, in case you don’t want to manually transform your tensors, use the mixed-precision utility via <code>torch.cuda.amp/torch.autocast</code>.</p>

username: blade
cooked: <p>Thanks. I took your advice <a href="https://discuss.pytorch.org/t/training-with-half-precision/11815/17">here</a> and used <code>torch.autocast</code>, as following:</p>
<pre><code class="lang-auto">with torch.autocast('cuda'):
    ... (my code) ...
</code></pre>
Run out of memory when subtracting medium sized tensors?

username: keviny2
cooked: <p>I’m trying to subtract two tensors, which I don’t think should be prohibitively large…</p>
<pre><code class="lang-auto">uv.shape = torch.Size([256, 256, 2])
xy.shape = torch.Size([40, 28672, 3])</code></pre>
<p>When I try to perform a simple subtraction, my kernel restarts by itself. I’m guessing it’s a memory issue, but don’t see why my computer can’t handle it as I have 128GB RAM. What could be triggering this problem?</p>
<pre><code class="lang-auto">uv[None,None,:,:,0]-xy[:, :,None,None,0]</code></pre>

username: ptrblck
cooked: <p>Based on the posted operation your output would have the shape <code>[40, 28672, 256, 256]</code>, thus <code>75161927680</code> elements.<br>
If you are using <code>float32</code> (the default <code>dtype</code>) this would use <code>40 * 28672 * 256 * 256 * 4 / 1024**3 = 280GB</code>.</p>

username: keviny2
cooked: <p>Thanks, that makes sense. Do you have any suggestions on how to perform simple math operations on large tensors?</p>

username: ptrblck
cooked: <p>Your code would work assuming your system has enough memory.<br>
If that’s not the case you might need to reduce the input shapes or check if this output shape is even expected.</p>
TypeError: load_state_dict() takes from 2 to 3 positional arguments but 4 were given

username: cui
cooked: <p>TypeError: load_state_dict() takes from 2 to 3 positional arguments but 4 were given<br>
The source code is as follows<br>
def generate(self):<br>
<a href="http://self.net" rel="noopener nofollow ugc">self.net</a> = DeepLab(num_classes=self.num_classes, backbone=self.backbone, downsample_factor=self.downsample_factor, pretrained=False)</p>
<pre><code>    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    self.net.load_state_dict(torch.load(self.model_path, map_location=device), ['state_dict'], False)
    self.net    = self.net.eval()
    print('{} model, and classes loaded.'.format(self.model_path))
    
    if self.cuda:
        self.net = nn.DataParallel(self.net)
        self.net = self.net.cuda()</code></pre>

username: ptrblck
cooked: <p>It seems the <code>['state_dict']</code> indexing is wrong and I guess you want to apply it on the loaded object?<br>
If so, remove the <code>,</code> and use<br>
<code>torch.load(self.model_path, map_location=device)['state_dict']</code>.</p>

username: cui
cooked: <p>self.net.load_state_dict([‘state_dict’], torch.load(self.model_path, map_location=device)[‘state_dict’])</p>
<p>A keyError appears after modification</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9d03152c46fac6b7a3c49af2c562c29c532d3f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9d03152c46fac6b7a3c49af2c562c29c532d3f6" title="cuowu"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9d03152c46fac6b7a3c49af2c562c29c532d3f6.png" alt="cuowu" data-base62-sha1="zDWWkJESEYBd27BJG8TcsTTR34a" width="690" height="149" data-dominant-color="323235"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cuowu</span><span class="informations">996×216 26 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I’m unsure why you’ve passed the <code>['state_dict']</code> list now as the path argument.<br>
In any case, <code>load_state_dict</code> expects a <code>state_dict</code> as the first and <code>strict</code> as its second argument as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=load_state_dict#torch.nn.Module.load_state_dict">docs</a>.<br>
Check what kind of object you’ve saved previously, index it ,if needed, to get the <code>state_dict</code> of the loaded object, and pass it to <code>load_state_dict()</code>.</p>
Slicing mobilenet V3 matrix shape mistmatch

username: Tanuj
cooked: <p>I have been trying to obtain the output of all but last linear layer of Mobilenet v3. The last module in small mobilnetv3 is a Sequential one:</p>
<pre><code class="lang-auto">print(mobilenet_v3_small.classifier)
Sequential(
  (0): Linear(in_features=576, out_features=1024, bias=True)
  (1): Hardswish()
  (2): Dropout(p=0.2, inplace=True)
  (3): Linear(in_features=1024, out_features=1000, bias=True)
)
</code></pre>
<p>I would like only the first Linear layer and all the preceding network. Here’s my attempt:</p>
<pre><code class="lang-auto">class mobilenetv3(nn.Module):
    def __init__(self):
        super().__init__()
        mobilenet_v3_small = models.mobilenet_v3_small(pretrained=True)
        self.mobilenet_preclassifier = nn.Sequential(mobilenet_v3_small.features, 
                                                     mobilenet_v3_small.avgpool)
        self.mobilenet_linear = nn.Sequential(mobilenet_v3_small.classifier[:1])
        
    def forward(self, x):
        x = self.mobilenet_preclassifier(x)
        x = self.mobilenet_linear(x)
        return x
</code></pre>
<p>But I keep getting the following error:</p>
<p><code>RuntimeError: mat1 and mat2 shapes cannot be multiplied (576x1 and 576x1024)</code>.</p>
<p>Even if I simply assemble the complete mobilenetv3 module by module (without slicing the classifier layer), the error remains the same. So, same error is seen  if I do:</p>
<pre><code class="lang-auto">class mobilenetv3(nn.Module):
    def __init__(self):
        super().__init__()
        self.m = nn.Sequential(mobilenet_v3_small.features, 
                               mobilenet_v3_small.avgpool, 
                               mobilenet_v3_small.classifier)
        
    def forward(self, x):
        x = self.m(x)
        return x
</code></pre>
<p>This points to the fact that I’m somehow assembling the modules incorrectly leading to the issue. Would be grateful if someone can point to the appropriate way to do it and maybe explain the problem with my approach. Thanks in advance.</p>

username: ptrblck
cooked: <p>Wrapping submodules into <code>nn.Sequential</code> containers often fails, as you might be missing functional API calls used in the <code>forward</code> method.<br>
In your case, <a href="https://github.com/pytorch/vision/blob/9f48c6274f66f71dd0cabe6a48bde84c8f239abe/torchvision/models/mobilenetv3.py#L227">this <code>torch.flatten</code></a> operation is missing and will yield the shape mismatch.<br>
You could either add it as an <code>nn.Flatten</code> to your <code>nn.Sequential</code> model or derive a custom class and override the <code>forward</code> method.</p>

username: Tanuj
cooked: <p>Thanks. That solved the issue.</p>
Unexpected grad_sample type: <class 'NoneType'>

username: amugesu
cooked: <p>I was trying to use opacus with DCGAN,while i am training Discriminator with fake images,<br>
In the line of “opt_D.step()”,such value error happened,i can’t figure out what’s going wrong?PLEASE HELP<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f508181a76d08e0515d4b9ee30ee6a703c93a451.png" data-download-href="https://discuss.pytorch.org/uploads/default/f508181a76d08e0515d4b9ee30ee6a703c93a451" title="162604"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f508181a76d08e0515d4b9ee30ee6a703c93a451.png" alt="162604" data-base62-sha1="yXEjXoScEpTsA3xlADZq1dpeHZv" width="690" height="283" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">162604</span><span class="informations">723×297 21.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: amugesu
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a16617d5f29afefbd7fa3b9e972fc3902cadc94.png" data-download-href="https://discuss.pytorch.org/uploads/default/0a16617d5f29afefbd7fa3b9e972fc3902cadc94" title="162543"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a16617d5f29afefbd7fa3b9e972fc3902cadc94.png" alt="162543" data-base62-sha1="1reIYoq9Ja2GHaXOSFX2hRfSyd6" width="690" height="321" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">162543</span><span class="informations">730×340 16.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: alexandresablayrolle
cooked: <p>May I ask which version of Opacus you are using? Also, if you can provide a minimal reproducing code it would be easier for us to help you.</p>
<p>I just tried the dcgan example (<a href="https://github.com/pytorch/opacus/blob/main/examples/dcgan.py" class="inline-onebox" rel="noopener nofollow ugc">opacus/dcgan.py at main · pytorch/opacus · GitHub</a>) on my end and it seems to work well with Opacus 1.0</p>

username: amugesu
cooked: <p>Thanks for your reply and test <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=10" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:"></p>
A question about applying a neural network on a tensor's specific dimension

username: Federico_Ottomano
cooked: <p>Hello I have a little question that might be straightforward:</p>
<p>Suppose we have a <code>torch.tensor</code> <code>x</code> with shape <code>(4,5,1)</code>.</p>
<p>What I’d like to do is applying a neural network on this input data, but on the <strong>last dimension</strong> only. Let’s say we decide some output features having dimension 64, then my output’s shape would be something like <code>(4,5,64)</code>.</p>
<p>Trying with the <em>standard</em> procedure, I have noticed that the input tensor gets automatically <em>flattened</em> and the input seeks to receive an input of dimensions <code>(20,1)</code></p>
<p>Many thanks,</p>
<p>Federico</p>

username: AlphaBetaGamma96
cooked: <p>Hi</p>
<p>This can be done but it’ll depend on the network’s architecture, so for example for a simple network with 1 <code>nn.Linear</code> layer. Both the batch dimensions are outputted. For example,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.randn(4,5,64)
&gt;&gt;&gt; fc = torch.nn.Linear(64, 1)
&gt;&gt;&gt; y = fc(x)
&gt;&gt;&gt; y.shape #torch.Size([4, 5, 1])
</code></pre>
<p>Can you share some more about the model?</p>
How to export an onnx model with buffers changeable during inference?

username: lawlict_L
cooked: <p>Hi, I try to create a first-in-first-out queue as a pytorch model. The queue, with a limited size, updates every time when a new input comes, and returns the updated queue. Codes are very simple:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class WavBuffer(nn.Module):
    def __init__(self, size=10):
        super().__init__()
        self.size = size
        wavbuf = torch.zeros(size)
        self.register_buffer('wavbuf', wavbuf)

    def forward(self, x):
        self.wavbuf = torch.cat([self.wavbuf, x])[-self.size:]
        return self.wavbuf

model = WavBuffer(10)
x = torch.ones(5)
for i in range(2):
    wavbuf = model(x)
    print(wavbuf)
</code></pre>
<p>As expected, the outputs are:</p>
<pre><code class="lang-auto">tensor([0., 0., 0., 0., 0., 1., 1., 1., 1., 1.])
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
</code></pre>
<p>Then I export the model to onnx format and infer with onnxruntime:</p>
<pre><code class="lang-auto">torch.onnx.export(
    model, torch.zeros(5), 'model.onnx', verbose=False, input_names=['wav'],
    output_names=['wavbuf'], opset_version=11
)

import numpy as np
import onnxruntime

model = onnxruntime.InferenceSession('model.onnx')
x = np.ones(5, dtype=np.float32)
inputs = {model.get_inputs()[0].name: x}
for i in range(2):
    outputs = model.run(None, inputs)
    wavbuf = outputs[0]
    print(wavbuf)
</code></pre>
<p>However, now the outputs are:</p>
<pre><code class="lang-auto">[0. 0. 0. 0. 0. 1. 1. 1. 1. 1.]
[0. 0. 0. 0. 0. 1. 1. 1. 1. 1.]
</code></pre>
<p>I guess that weights in onnx models are not changeable, but is there any solution to create writable buffers during model design and change the buffers in onnx inference? An available example is LSTM, where the hidden states update for each time step. However, it is too difficult for me to its implementation.</p>