akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to access hidden states computed on a different device when using DataParallel in PyTorch?

LILAKKU: <p>If my batch size is set to 16 and I have 4 devices, PyTorch will divide them into 4 batches of size 4, then send them off to different devices to make the forward call. But I want to access all hidden states of the 16 instances in the forward call, is it possible to do that?</p>

ptrblck: <p>You might be able to create a “global” <code>dict</code> and store the forward activations via a key which uses the layer name as well as the device id. Note that we generally recommend using <code>DistributedDataParallel</code> for a better performance, where one process per device would be used and would allow you to directly access the corresponding forward activations.</p>

LILAKKU: <p>Thanks! Does DataParallel scatter and distribute the input data in order? Or will the data end up being shuffled?</p>

ptrblck: <p>The <code>DataParallel</code> wrapper will slice the data (in order) and snd each chunk to the corresponding device. The shuffling is done before by the <code>sampler</code> in the <code>DataLoader</code>.</p>

LILAKKU: <p>I tried adding a <code>dict</code> as a model attribute. The forward call would involve the following:</p>
<ol>
<li>Compute hidden states using a transformer model, save these hidden states to that <code>dict</code> with current device index as the key.</li>
<li>Gather all hidden states computed on all devices from the <code>dict</code>, perform some further computations.</li>
</ol>
<p>But when I get to the second step, the <code>dict</code> doesn’t contain all device entries, most likely because some devices would finish before others. However, it seems like the devices don’t execute in order. Sometimes accessing the <code>dict</code> on device 0 will return more than one entry. Is there a way to either regulate the order in which devices run, or synchronize across all devices before going into step 2? Many thanks!</p>

ptrblck: <p>Yes, use <code>DDP</code> as it will allow you for a manual communication via <code>all_reduce</code>, <code>barrier</code>, etc. which might all be needed for your custom approach.</p>

LILAKKU: <p>Thanks, I will look into that.</p>
Need old version pytorch 0.3.0 for windows

Alexey_Jurin: <p>Hi. Need old version pytorch 0.3.0 for windows install via conda for using my GPU gt 750m - help me pls… Old topics via internet not work today, need fresh help</p>

ptrblck: <p>Based on the release notes <a href="https://github.com/pytorch/pytorch/releases/tag/v0.4.0">PyTorch 0.4.0</a> introduced Windows support.</p>

Alexey_Jurin: <p>Thx , is this 0.4.0 release support CUDA computation at my old Geforce GT 750m ?</p>

ptrblck: <p>I guess so, but don’t know as <code>0.4.0</code> is ~4.5 years old, so you could try it out.<br>
Obviously I would not recommend using such an old version as it’s older now than the lifetime of other frameworks <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

KFrank: <p>Hi Alexey!</p>
<aside class="quote no-group" data-username="Alexey_Jurin" data-post="1" data-topic="168817" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexey_jurin/48/55885_2.png" class="avatar"> Alexey_Jurin:</div>
<blockquote>
<p>Hi. Need old version pytorch 0.3.0 for windows</p>
</blockquote>
</aside>
<p>I had a similar question some years back.  i was referred to some unofficial<br>
“legacy builds” and was able to install version 0.3.0 on my windows machine<br>
and run it on its k1100m gpu.  See the following post and the links it contains:</p>
<aside class="quote quote-modified" data-post="4" data-topic="128875">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-0-3-0-or-0-3-1-not-available-for-windows-now/128875/4">Pytorch 0.3.0 or 0.3.1 not available for windows now</a> <a class="badge-category__wrapper " href="/c/windows/26"><span data-category-id="26" data-drop-close="true" class="badge-category " title="This category is focused on PyTorch on Windows related issues."><span class="badge-category__name">windows</span></span></a>
  </div>
  <blockquote>
    Hi Shuai! 

I was able to install and run pytorch 0.3.0 on windows 10.  (My 
motivation was to run pytorch on an old laptop gpu, and 0.3.0 
was the newest version of pytorch I could find that was pre-built 
with support for my old gpu.) 
See this post from <a class="mention" href="/u/ptrblck">@ptrblck</a> where he gives a link to some “legacy” 
binaries built by <a class="mention" href="/u/peterjc123">@peterjc123</a> (but do read through the entire thread 
for context): 


Please be aware, however, that newer versions of pytorch are much 
better – they have numerous new and impr…
  </blockquote>
</aside>

<p>However, 0.3.0 is very old now – many more bugs and missing features than<br>
recent versions.  You will almost certainly be better off figuring out some sort<br>
of solution that permits you to use an up-to-date version of pytorch.</p>
<p>Best.</p>
<p>K. Frank</p>
4d tensor mean by columns

Rane90: <p>Hi,</p>
<p>I have a 4d tensor with a shape batch_data = <code>(b, c, h, w)</code>.</p>
<p>I want to center each sample by subtracting the mean of each column (<code>w</code>) for every channel.  That means I need to apply <code>batch_data - batch_data.mean(dim=(???)).unsqueez(???)</code> such that the shape of the mean data will be <code>(b, c, h, w)</code>, and each sample’s channel will have an <code>(h, w)</code> tensor where, for each <code>h</code>, all <code>w</code>’s are the same.</p>
<p>I hope this question is clear <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Thank you</p>

KFrank: <p>Hi Rane!</p>
<aside class="quote no-group" data-username="Rane90" data-post="1" data-topic="168824" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rane90/48/47118_2.png" class="avatar"> Rane90:</div>
<blockquote>
<p>the shape of the mean data will be <code>(b, c, h, w)</code>, and each sample’s channel will have an <code>(h, w)</code> tensor where, for each <code>h</code>, all <code>w</code>’s are the same.</p>
</blockquote>
</aside>
<p>If I understand you correctly, you wish to adjust your original tensor so that<br>
for any <code>b</code> and <code>c</code>, the mean over the <code>w</code> dimension will be independent of <code>h</code><br>
(but will still depend on <code>b</code> and <code>c</code>).</p>
<p>You could try something like:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; t = torch.rand (2, 2, 3, 5)
&gt;&gt;&gt; u = t - t.mean (dim = 3, keepdim = True) + t.mean (dim = (2, 3), keepdim = True)
&gt;&gt;&gt; u.mean (dim = 3)
tensor([[[0.5798, 0.5798, 0.5798],
         [0.4724, 0.4724, 0.4724]],

        [[0.5582, 0.5582, 0.5582],
         [0.5791, 0.5791, 0.5791]]])
&gt;&gt;&gt; torch.allclose (u - u.mean (dim = 3, keepdim = True), t - t.mean (dim = 3, keepdim = True))
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Order of backward(), step() and zero_grad()

gfotedar: <p>In most codes the order I see is</p>
<pre><code class="lang-auto">training loop:
    # forward pass and calculate loss
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>If I change it to:</p>
<pre><code class="lang-auto">training loop:
    # forward pass and calculate loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>is it still ok?</p>

googlebot: <p>Yes, it is clear-fill-use vs fill-use-clear</p>

rad: <p>Yes, but you’ll backpropagate n-1 times only, where n is the number of epochs. Why would you not want to follow the “standard” order of the 3?</p>

googlebot: <p>there are subtle differences in memory management, esp. when zero_grad(set_to_none=True) is used</p>

rozlana-g: <p>Why backpropagation will take place n-1 times only in that case?</p>

nwn: <p>Hello Gaurav,<br>
yes, both should work as long as your training loop does not contain another loss that is backwarded in advance to your posted training loop, e.g. in case of having a more complex architecture that consists of several ANNs and loss functions that all modify the same network during their backward and step pass.<br>
Important is only that your gradients are set to zero before you call the backward command. I does not matter whether you do it at the beginning of a loop or at the end of the previous one.</p>
<p><strong>But calling zero_grad just before the backward pass is certainly more readible. Therefor, I would stick to using it.</strong></p>

IdoAmit198: <p>Hi, can you please share more details about the differences?</p>

ptrblck: <p><code>.zero_grad(set_to_none=True)</code> will delete the <code>.grad</code> attributes and will thus free memory.<br>
Calling it early might thus be beneficial assuming you don’t need the gradients anymore.</p>
Inverse of torch.tensor.unfold

Yuv: <p>Hello,</p>
<p>I want to use .unfold on some tensor, for example:</p>
<p>x = torch.rand(1,3,6,32,32)</p>
<p>and because I want to use unfold on the 3,4 dimensions I would use .unfold as following</p>
<p>x = x.unfold(3,5,1).unfold(4,5,1)<br>
and gets a shape of [1,3,6,28,28,5,5]</p>
<p>and I want to get back to [1,3,6,32,32] is there any inverse of the .unfold kind of method?</p>
<p>I don’t want to use nn.Unfold if possible</p>
RuntimeError: Calculated padded input size per channel: (2 x 130 x 130). Kernel size: (3 x 3 x 3). Kernel size can't be greater than actual input size

shadow: <p>Hello,</p>
<p>I am using unet to reconstruct an image from 6 features, the input tensor in (batch size, channels=6, x, y, z). z is 1,2 or 3. x, y &lt; 64.</p>
<p>You can find a summary of my model summary attached<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aafe0a0693c02d07e71b069822ebcb8f368b8e4c.png" data-download-href="https://discuss.pytorch.org/uploads/default/aafe0a0693c02d07e71b069822ebcb8f368b8e4c" title="modelsum"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aafe0a0693c02d07e71b069822ebcb8f368b8e4c_2_690x211.png" alt="modelsum" data-base62-sha1="ooFrFR5mqjHM2boZQqEa2kb0w0s" width="690" height="211" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aafe0a0693c02d07e71b069822ebcb8f368b8e4c_2_690x211.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aafe0a0693c02d07e71b069822ebcb8f368b8e4c_2_1035x316.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/a/aafe0a0693c02d07e71b069822ebcb8f368b8e4c.png 2x" data-dominant-color="292929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">modelsum</span><span class="informations">1182×362 85.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
:<br>
while trying to train I receive the padding error, any help?<br>
thanks in advance</p>

ptrblck: <p>The input activation to a conv layer is too small since it’s <code>[depth=2, height=130, width=130]</code> while the kernel size of <code>[3, 3, 3]</code>. Make sure the <code>depth</code>, i.e. <code>x</code> in your case, is large enough for the model.</p>
Questions about soft parameter sharing with loss function design

ChineseBest: <p>Hi all, I intend to implement parameter sharing method here to add it as a regularization approach:</p>
<pre><code class="lang-auto">def weight_assign_loss(module, graph1, graph2):
    loss = 0
    
    dict_l2_g1 = module.convl2[graph1].state_dict()
    dict_l2_g2 = module.convl2[graph2].state_dict()
    
    for i in list(dict_l2_g1.keys()):
        loss += F.mse_loss(dict_l2_g1[i], dict_l2_g2[i])
        
    dict_l3_g1 = module.convl3[graph1].state_dict()
    dict_l3_g2 = module.convl3[graph2].state_dict()
    
    for i in list(dict_l3_g1.keys()):
        loss += F.mse_loss(dict_l3_g1[i], dict_l3_g2[i])
        
    return loss

</code></pre>
<p>Here is my function, which refers from this page:<br>
<a href="https://avivnavon.github.io/blog/parameter-sharing-in-deep-learning/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://avivnavon.github.io/blog/parameter-sharing-in-deep-learning/</a><br>
However, if I add this loss to the original loss, this part will never decrease.</p>
<p>If I do not add it to the original loss, I will receive this error:<br>
element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>Could anyone please help me figure out why? Thanks.</p>
GPU in PiNN training

robinho: <p>Suppose there’s the following code for PiNN training:<br>
…<br>
for each epoch:<br>
x=torch.linspace(0,1,batchsize)<br>
loss.backward()<br>
optimize<br>
epoch++<br>
…<br>
How to use GPU?</p>

khushi-411: <p>Hi, You can add the <code>device='cuda'</code> parameter in linspace. It’ll resolve your issue. You can refer to the <a href="https://pytorch.org/docs/stable/generated/torch.linspace.html" rel="noopener nofollow ugc">documentation</a> for more details. Thanks!</p>

robinho: <p>thanks khushi! It seems using either takes the same time on my pc. What data size will make a difference?</p>

robinho: <p>thanks khushi! It seems using either takes the same time on my pc. What data size will make a difference?</p>

khushi-411: <p>Hi! Thanks for pinging me; I somehow missed the notification earlier. Answer to your question: The larger the data points, the more difference it creates. From my experience, I usually see the difference in speeds from 1000-5000 data points. You can iterate it to a larger number to see a significant difference.</p>
<p>PS: I haven’t found the exact number anywhere.</p>
Cumsum_cuda_kernel determinism

Sergei_Vorobev: <p>Hello!</p>
<p>This code works fine in PyTorch 1.11.0 and 1.12.0</p>
<pre><code class="lang-python">import torch

print(torch.__version__)

torch.use_deterministic_algorithms(True)
t = torch.tensor(range(10), dtype=float)
t = t.cuda()
x = torch.cumsum(t, 0)
print(x)
</code></pre>
<p>On PyTorch 1.13 it fails with error</p>
<pre><code class="lang-auto">RuntimeError: cumsum_cuda_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
</code></pre>
<p>I found an open issue about it <a href="https://github.com/pytorch/pytorch/issues/89492" class="inline-onebox" rel="noopener nofollow ugc">Feature Request: deterministic CUDA cumsum · Issue #89492 · pytorch/pytorch · GitHub</a></p>
<p>We run our models with determinism and without it – both modes are used wildly on the same models.</p>
<p>I tried to write a simple workaround like this</p>
<pre><code class="lang-python">def cumsum(*args: Any, **kwargs: Any) -&gt; Any:
    # `cumsum_cuda_kernel` does not have a deterministic implementation
    # we turn off determinism enforcement just for this one operation, otherwise it error
    # in the deterministic mode.
    is_deterministic = torch.are_deterministic_algorithms_enabled()
    is_warn_only = False
    try:
        if is_deterministic:
            is_warn_only = torch.is_deterministic_algorithms_warn_only_enabled()
            torch.use_deterministic_algorithms(False)
        result = torch.cumsum(*args, **kwargs)
    finally:
        if is_deterministic:
            torch.use_deterministic_algorithms(True, warn_only=is_warn_only)
    return result
</code></pre>
<p>But then when we are trying to TorchScript the model we run into</p>
<pre><code class="lang-auto">E       RuntimeError: 
E       Unknown builtin op: aten::are_deterministic_algorithms_enabled.
E       Could not find any similar ops to aten::are_deterministic_algorithms_enabled. This op may not exist or may not be currently supported in TorchScript.
</code></pre>
<p>Is there any recommended way to go about it?</p>

ptrblck: <p>Also tracked <a href="https://github.com/pytorch/pytorch/issues/89492">here</a>.</p>
How to save params with modified out channels

anguoyang: <p>Here is one module inside my model:</p>
<pre><code class="lang-auto">class ConstantInput(nn.Module):
    def __init__(self, channel, size=4):
        super().__init__()

        self.input = nn.Parameter(torch.randn(1, channel, size, size))

    def forward(self, input):
        batch = input.shape[0]
        out = self.input.repeat(batch, 1, 1, 1)

        if hasattr(self, 'first_k_oup') and self.first_k_oup is not None:  # support dynamic channel
            assert self.first_k_oup &lt;= out.shape[1]
            return out[:, :self.first_k_oup]
        else:
            return out
</code></pre>
<p>I will modify the first_k_oup outside, but need to save the modified param instead of the original(before modification), how could I do it?</p>
<pre><code class="lang-auto">.............
        self.model.generator = Generator(......
        pretrained_dict_generator = torch.load('./weights/ckpt-best.pt', map_location=torch.device('cpu'))['g_ema']
        self.model.generator.load_state_dict(pretrained_dict_generator)

        set_uniform_channel_ratio(self.model.generator, 0.25) #this line will modify the output channels, like first_k_oup in above class definition
..............
</code></pre>
<p>after I modify the output channels with set_uniform_channel_ratio, I want to save the model like:<br>
<code>torch.save(self.model.generator.state_dict(), 'gen0.25.pth')</code></p>
<p>but unfortunately, the model size is same with the one before modification.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>

huahuanZ: <p>Not sure whether I understand correctly</p>
<pre><code class="lang-auto">
class MyModel(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.param = nn.Parameter(torch.zeros(10, 100))

    def forward(self, x):
        return x

model = MyModel()
print("#params before: {}".format(sum(x.numel() for x in model.parameters())))
# modify parameters
model.param.data = model.param.data[:, :50]
print("#params after: {}".format(sum(x.numel() for x in model.parameters())))

# output:
#params before: 1000
#params after: 500
</code></pre>

anguoyang: <p>thank you! let me try with it</p>
Check two tensor is hardware level same

LTTc: <p>I want to check if two tensors are exactly the same. You need to check that the nature of the tensors is the same, not allclose the same way.<br>
Does a function <code>foo</code> exist that works like this? <code>foo</code> must not change the state of its input tensors.</p>
<pre><code class="lang-auto">X = Tensor(...)
X2 = X.copy()
Y = nn.Parameter(X)
foo(X, X) == True
foo(X, Y) == True
foo(Y, Y) == True
foo(X2, X) == False
foo(X2, Y) == False
</code></pre>

ptrblck: <p>I guess you want to check the identity of the underlying data, so using <code>.data_ptr()</code> should work:</p>
<pre><code class="lang-python">def foo(a, b):
    if isinstance(a, nn.Parameter):
        a = a.data
    if isinstance(b, nn.Parameter):
        b = b.data
    return a.data_ptr() == b.data_ptr()
    
X = torch.randn(1)
X2 = X.clone()
Y = nn.Parameter(X)

foo(X, X) == True
foo(X, Y) == True
foo(Y, Y) == True
foo(X2, X) == False
foo(X2, Y) == False
</code></pre>

LTTc: <p>thanks for the reply!</p>
Question about matrix multiplication precisions

Yi_wen_Chen: <p>During my training, I found the below calculations don’t equal. I can’t figure out the reason. Could you please help me?</p>
<pre><code class="lang-auto">    a=torch.tensor([[0.0641, 0.0434, 0.7166, 1.0000],
            [0.0642, 0.0434, 0.7170, 1.0000]])
    T=torch.tensor([[9.3945e-01, 0.0000e+00, -3.4204e-01, 0.0000e+00],
            [-2.9890e-08, 1.0000e+00, -8.2131e-08, -3.5527e-15],
            [3.4204e-01, 8.7428e-08, 9.3945e-01, 7.0020e-01],
            [0.0000e+00, 0.0000e+00, 0.0000e+00, 1.0000e+00]])

    res_1 = (T @ a[...,None]).reshape(2,4)
    res_2 = (T @ a.T).T
    print(res_1-res_2) # not zero
</code></pre>

ptrblck: <p>These small differences are caused by a different order of operation and the limited floating point precision. <a href="https://pytorch.org/docs/stable/notes/numerical_accuracy.html#batched-computations-or-slice-computations">These docs</a> explain it in more detail.</p>

Yi_wen_Chen: <p>Thanks!  I understand</p>
Tensor(-inf, inf, inf....inf) obtained on GPU

Prakhar_Pradhan1: <p>Hi Everyone<br>
Recently I came across a bizarre issue.</p>
<p>The output of the convolutional layer is different on the CPU and GPU. CPU gives correct conv output, whereas some values in the output of the conv layer are becoming inf or -inf on GPU</p>

KFrank: <p>Hi Prakhar!</p>
<aside class="quote no-group" data-username="Prakhar_Pradhan1" data-post="1" data-topic="168757" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prakhar_pradhan1/48/53366_2.png" class="avatar"> Prakhar_Pradhan1:</div>
<blockquote>
<p>The output of the convolutional layer is different on the CPU and GPU.</p>
</blockquote>
</aside>
<p>If you are using an Ampere-architecture gpu (e.g., an A100), then you<br>
might be seeing the effects of the <a href="https://pytorch.org/docs/1.13/notes/numerical_accuracy.html#tensorfloat-32-tf32-on-nvidia-ampere-devices" rel="noopener nofollow ugc">dreaded TensorFloat32</a> floating-point<br>
format.  If so, you could try disabling TF32, as discussed in the linked<br>
documentation, with <code>torch.backends.cudnn.allow_tf32 = False</code>.</p>
<p>(If this is not your issue, please post a simplified, fully-self-contained,<br>
runnable script that reproduces your issue, together with its output and<br>
your gpu model and pytorch and cuda versions.)</p>
<p>Best.</p>
<p>K. Frank</p>
Is there significant performance issues when running PyTorch for Linux on Windows with Docker and WSL compared to running a native Linux version?

Thibaut: <p>Hi,</p>
<p>We have a containerized PyTorch that runs neural networks.</p>
<p>When we run this on Linux PC it is quite fast.</p>
<p>But when we switch to Windows PC that runs HyperV, that runs WSL, that runs Docker, that runs Ubuntu, that runs PyTorch, it is quite slower. Maybe 4 to 10 time slower.</p>
<p>There are several potential sources for this. Do you know if there are known overhead when running PyTorch with this type of configuration?</p>
<p>Of course this seems something quite stupid to do but there are complex constraints due to the rest of the project.</p>
<p>Thanks</p>

ptrblck: <p>I don’t know what might be causing the slowdown, but you could profile your workload (e.g. via Nsight Systems) and check which part of the code slows down and where the bottleneck might be.</p>
Use gradient of hidden layer wrt network inputs in the calculation of the loss function

caaf: <p>Hi,</p>
<p>I’d like to use the gradient of the output  of a hidden layer wrt the network input as one of the terms in the calculation of the network loss. Any suggestion of how this could be done?</p>

soulitzer: <p>You could take the grad of hidden_out wrt the input with grad_input = torch.autograd.grad(hidden_out, (input,)), and that use that however you’d like in the final loss.</p>

caaf: <aside class="quote no-group" data-username="soulitzer" data-post="2" data-topic="168091">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>hidden_out</p>
</blockquote>
</aside>
<p>And where may I find such <em>hidden_out</em> ?</p>

srishti-git1110: <p>That further depends on your network and use case.<br>
One way to do that -</p>
<pre><code class="lang-python">class LinearModel(nn.Module):
  def __init__(self, input_size, output_size, hidden_size):
     super(LinearModel, self).__init__()
     self.fc1 = nn.Linear(input_size, hidden_size)
     self.hidden = nn.Linear(hidden_size, hidden_size)
     self.fc2 = nn.Linear(hidden_size, output_size)

  # def forward(self, x):
    # body

lm = LinearModel(input_size=8, output_size=2, hidden_size=4)
inp_x = torch.randn(1, 8, requires_grad=True)
hidden_out = lm.hidden(F.relu(lm.fc1(inp_x)))

grad_input = torch.autograd.grad(hidden_out.sum(), (inp_x)) # (tensor([[-0.0315,  0.0286,  0.0313,  0.0286, -0.0108, -0.0141,  0.0190, -0.0012]]),)
</code></pre>

caaf: <p>Aha, got the point. Thanks for the help!</p>
Pytorch 2.0 C++

Peter_Featherstone: <p>It seems the only big thing in pytorch 2.0 is <code>torch.compile</code> which is just a “make faster” function like <code>torch.mobile_optimizer.optimize_for_mobile</code>.<br>
That’s great but my worry from the motivation behind it is that they’re giving up on first class support for C++. I don’t really care about training in C++ but inference is of massive importance to me and i’m sure to others. I hope they won’t relax the support for ONNX export. Ideally, it would be great if there was a “libtorch-lite.a” for inferring torchscript or <code>torch.compile</code>-ed models using a very minimal library. That would be awesome. At the moment i have to use onnxruntime, which is fine, but it would be great if i could infer torch models directly in C++ without having to use the massively bloated, and impossible to cross compile, libtorch.so.</p>
<p>Anybody else worrying that using pytorch models in C++ will slowly become harder and harder to do?</p>

marksaroufim: <p>The export functionality is in the works, keep an eye out on Raziel’s and Suo’s talks at Dev con to learn more <a href="https://www.youtube.com/@PyTorch/videos" rel="noopener nofollow ugc">https://www.youtube.com/@PyTorch/videos</a></p>

swilson314: <p>Yes, I’m worrying about that as well. And I actually do care about training in C++.</p>
<p>It would be nice if the repercussions of moving pytorch “back into python” was recognized as an issue for c++ developers and addressed in a comprehensive discussion.</p>
How to train models that have a many-to-one relation

Larsvbl: <p>I’m currently working on a task wherein I need to combine a large number of images into a single image. I do this in two steps: first passing X images into N segmentation models each, and then passing the N results into another segmentation model to create the final image. My current issues are how to create the N intermediate images, and how to deal with missing data.</p>
<p>The code fragment below shows what I currently have working. In this example x is a list of tensors, where each tensor represents X images. Each tensor contains a different set of X images. Due to the nature of the dataset, it is not always possible to create N tensors. This code works, but only when I set the batch size to 1 and replace missing images with tensors filled with zeros.</p>
<pre><code class="lang-auto">    def training_step(self, batch, batch_idx):
        x, y = batch

        segmented_images_list = []

        for current_image in x:
            intermediate_result = self.model_1(current_band)
            segmented_bands_list.append(result)

        intermediate_tensor = torch.cat(segmented_images_list , 0).permute(1, 0, 2, 3)

        y_hat = self.model_2(intermediate_tensor )
        loss = self.loss_function(y_hat, y)
        self.log("train/loss", loss)

        return loss
</code></pre>
<p>I have two questions:</p>
<ol>
<li>Is there a more elegant solution than iterating over tensors, ideally one that would let me use a higher batch size than 1.</li>
<li>How should I handle missing data? The second segmenter has a fixed number of input channels, meaning I must always supply N tensors.</li>
</ol>
<p>Thanks for reading <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Tips on an unusual form of pipeline parallelism

geomlyd: <p>Hi there,<br>
I want to run a scenario that features an unusual sort of pipeline parallelism. In short, the idea is this:</p>
<p>There are N devices, d1, d2, … dN and in each of those “resides” part of a neural network. This can be either a single layer, or a more complicated part, but in any case it’s expressed as a function f1, f2, … fN for each device respectively. Each f_i receives as input the output of the previous function/device. It also has a set of parameters. As such, f_i are instances of subclasses of torch.nn.Module.</p>
<p>The catch is this: each device maintains a buffer that should only hold the <strong>last</strong> input it processed. When the final device/function computes an output, the loss is computed and the gradient wrt. to the input of fN is sent to the previous device. Each device should then backpropagate the “pseudogradient” that results from computing its “local” derivative on the input it <strong>currently</strong> has in its buffer and combining it with the backpropagated gradient it received.</p>
<p>This is a kind of “delayed” backpropagation that aims on avoiding the storage of activations until a backward pass is computed.</p>
<p>It’s not entirely clear to me how something along these lines can be done in PyTorch. How can I make sure that each time an f_i runs on a new buffer input it overwrites the computational graph of its previous run? I’m fairly sure that I will need calls to detach() for this, in order to avoid backward passes propagating too “far back”. However, by themselves they don’t seem to give me all of the fine-grain control this needs.</p>
<p>Any help is appreciated!</p>
IndexError: index out of range in self in GCN - GAE

atharva_s: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f788721cdd38a65ed7deff142b83d2b628c224b0.png" data-download-href="https://discuss.pytorch.org/uploads/default/f788721cdd38a65ed7deff142b83d2b628c224b0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f788721cdd38a65ed7deff142b83d2b628c224b0_2_690x305.png" alt="image" data-base62-sha1="zjMgFzeKXgtyE5ycajG1c9hNhDi" width="690" height="305" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f788721cdd38a65ed7deff142b83d2b628c224b0_2_690x305.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f788721cdd38a65ed7deff142b83d2b628c224b0_2_1035x457.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f788721cdd38a65ed7deff142b83d2b628c224b0_2_1380x610.png 2x" data-dominant-color="F3F4ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1555×688 119 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Vmap a nn.Module

111470: <p>Hi, I found the new functorch.vmap feature to be very useful，I am trying to use vmap to parallelize multiple decoders with the same input and output shapes.  There are some examples of using vmap to parallelize nn.Module in the doc, but none of them deal with the use of vmap within classes. This is my initial idea of implementing vmap inside a class, can you help me with it? Thanks！</p>
<p>Without vmap it looks like this, which is very time-consuming when num_mod is large.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.num_mod=6
        
        self.encoder=nn.RNN()
        decoder=[nn.Linear(10,1) for i in range(self.num_mod)]
        self.decoder = nn.ModuleList(decoder)
        
    def forward(self,x):
        x=self.encoder(x)
        
        x=x.repeat(self.num_mod,1)
        out=[]
        for i in range(self.num_mod):
            out.append(self.decoder[i](x[i]))
            
        return torch.stach(out,dim=1)      
</code></pre>
<p>When I know that vmap can also apply to nn.Module , I try to achieve something like this:   can you help me with it</p>
<pre><code class="lang-auto">from functorch import make_functional,vmap

class Net(torch.nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.num_mod=6
        
        self.encoder=nn.RNN()
        
        func, params = make_functional(nn.Linear(10,1))
        self.decoder=vmap(func)
        
    def forward(self,x):
        x=self.encoder(x)
        x=x.repeat(self.num_mod,1)
            
        return self.decoder(x)     
</code></pre>
Transpose CNN weights

andreasceid: <p>Hello,</p>
<p>I am trying to automate the process of transposing a CNN. This basically means that I am trying to transfer the weights from a CNN to a NN with ConvTranspose layers (instead of Conv2d). Is there any automation or “hacky” way to do it?</p>

ptrblck: <p>You could <code>permute</code> the <code>weight</code> tensor and <code>copy_</code> it to the transposed conv layer:</p>
<pre><code class="lang-python">conv = nn.Conv2d(16, 32, (3, 4))
print(conv.weight.shape)
# torch.Size([32, 16, 3, 4])

conv_trans = nn.ConvTranspose2d(16, 32, (3, 4))
print(conv_trans.weight.shape)
# torch.Size([16, 32, 3, 4])

with torch.no_grad():
    conv_trans.weight.copy_(conv.weight.permute(1, 0, 2, 3))
    conv_trans.bias.copy_(conv.bias)
</code></pre>

andreasceid: <p>I think that will do the job <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Setting PyTorch to `train` or `eval` modes globally instead of at nn.Module level

abdulmuneer: <p>At PyTorch <code>model</code> that inherit from <code>torch.nn.Module</code> can be set to <code>train</code> or <code>eval</code> modes using <code>model.train()</code> or <code>model.eval()</code>. Is there a way to do this globally for the present PyTorch session, so that it applies to any op that is invoked, without having to update different models?</p>
<p>This could be useful in running PyTorch scripts in training or inference context as desired.</p>
__IN_ONNX_EXPORT variable in torch.onnx.export

lkdci: <p>I’ve noticed the global variable <code>__IN_ONNX_EXPORT</code> in <code>torch.onnx.utils</code> that is supposed to prevent calling the torch.onnx.export again while the previous is still in progress. I’ve tried to find documentation about why it should be prevented, with no success. Does that means that the torch.onnx.export can’t be distributed across threads?</p>
Torch.cuda.is_available() returns False. Why?

Dev2150: <ul>
<li>I have NVIDIA GeForce GTX 950M</li>
<li>I have installed CUDA 12 (cannot install older ones)</li>
<li>I have installed cuDNN</li>
<li>I have PyTorch 1.13.1+cpu</li>
<li>
<ul>
<li>“I ran pip3 install torch torchvision torchaudio --extra-index-url <a href="https://download.pytorch.org/whl/cu116" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu116</a>”</li>
</ul>
</li>
</ul>

ptrblck: <aside class="quote no-group" data-username="Dev2150" data-post="1" data-topic="168743">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dev2150/48/55862_2.png" class="avatar"> Dev2150:</div>
<blockquote>
<p>I have PyTorch 1.13.1+cpu</p>
</blockquote>
</aside>
<p>You’ve installed the CPU-only binaries, which is why <code>torch.cuda.is_available()</code> returns <code>False</code>.</p>

Dev2150: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/869b95f61acb7db73cb9a75a4ab30cee610747da.png" data-download-href="https://discuss.pytorch.org/uploads/default/869b95f61acb7db73cb9a75a4ab30cee610747da" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/869b95f61acb7db73cb9a75a4ab30cee610747da.png" alt="image" data-base62-sha1="jcNjXXDaptiTrwiFPWSOxbTcBT4" width="690" height="344" data-dominant-color="F1CDC8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">747×373 16.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>" With CUDA</p>
<p>To install PyTorch via pip, and do have a <a href="https://developer.nvidia.com/cuda-zone" rel="noopener nofollow ugc">CUDA-capable</a> system, in the above selector, choose OS: Windows, Package: Pip and the CUDA version suited to your machine. Often, the latest CUDA version is better. Then, run the command that is presented to you"</p>
`tensor.contiguous()` overhead on already contiguous tensor?

kevalmorabia97: <p>As per my understanding, calling <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.contiguous.html" rel="noopener nofollow ugc">tensor.contiguous()</a> would not have any effect if the tensor is already contiguous.</p>
<p>But the later case is 10x slower as shown below:</p>
<pre><code class="lang-python">import torch
x = torch.randn(128, 256, 3, 3)

assert x[:128].is_contiguous()

# 87 ns ± 0.222 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
_ = x.contiguous()

# 920 ns ± 16.3 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
_ = x[:128].contiguous()

# 73.5 ns ± 1.35 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
_ = x.is_contiguous()

# 892 ns ± 16.6 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
if not x[:128].is_contiguous():
    _ = x[:128].contiguous()
</code></pre>

ptrblck: <p>Based on your code you are timing the indexing. Remove it and you will see the same results:</p>
<pre><code class="lang-python">y = x[:128]
%timeit _ = y.contiguous()
</code></pre>
Border Artifacts when generating RGB images using a UNIT encoder-decoder architecture

usmancheema89: <p>I am using the UNIT architecture (<a href="https://github.com/milesial/Pytorch-UNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - milesial/Pytorch-UNet: PyTorch implementation of the U-Net for image semantic segmentation with high quality images</a>) as an encoder-decoder for 3 RGB output images. The issue is that the borders of the output images have color issues as can be seen in the attached image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d02ff83da242e46dd0cdd7b407034bfbd782365.png" data-download-href="https://discuss.pytorch.org/uploads/default/1d02ff83da242e46dd0cdd7b407034bfbd782365" title="Screenshot from 2022-12-14 10-42-44"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d02ff83da242e46dd0cdd7b407034bfbd782365_2_250x500.png" alt="Screenshot from 2022-12-14 10-42-44" data-base62-sha1="48EgK5BSUWIs1ZGTCaYrFkwtSWF" width="250" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d02ff83da242e46dd0cdd7b407034bfbd782365_2_250x500.png, https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d02ff83da242e46dd0cdd7b407034bfbd782365.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d02ff83da242e46dd0cdd7b407034bfbd782365.png 2x" data-dominant-color="8B8D7E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-12-14 10-42-44</span><span class="informations">346×690 280 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am using a 6-channel input (two images concatenated together) to generate a 3-channel output.</p>

nwn: <p>Hello,<br>
do you crop your images in any way?</p>

usmancheema89: <p>I have not cropped the images, the norm for face images is that they are aligned so that face features can be correlated. I will try out random cropping and see how it affects the results. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

nwn: <p>Pls let me know whether this solves the issue. Do you need to do image segmentation? If not or if you are open to different approaches, you can try this: <a href="https://openaccess.thecvf.com/content_CVPR_2020/html/Pidhorskyi_Adversarial_Latent_Autoencoders_CVPR_2020_paper.html" rel="noopener nofollow ugc">StyleALAE</a>. If you pick the right size of the latent space, you can rather easily do image segmentation, too. If you are keen, I can send my/our implementation of the above presented paper with a little add on so that the latent space is more disentangled.</p>

usmancheema89: <p>Thanks <a class="mention" href="/u/nwn">@nwn</a><br>
I’ll try out the method sometime this week. I’ll take a look at the paper and let you know if I can adapt the method for my gains.<br>
Cheers</p>
Untrained parameters affect final model performance

Jay_Li: <p>I wrote some DL models in PyTorch, and fixed random seeds. Some unused functions affect the final model performance. For example, this is part of my code.</p>
<p>…<br>
self.linear_1 = nn.Linear(27, d_model//2)<br>
self.linear_2 = nn.Linear(d_model//2, d_model)<br>
…</p>
<p>Even if I wrote self.linear_1 &amp; self.linear_2, they are not under forward command. However, I’ll get different results if I remove them. I feel it’s weird since they are not in the model. Can anyone explain it?</p>

ptrblck: <p>That’s expected since the order of calls into the pseudo-random number generator (PRNG) differs if you are randomly initializing these layers even if they are never used in the <code>forward</code> method.<br>
Seeding will guarantee that the same sequence of PRNG calls will generate the same random number sequence.</p>

Jay_Li: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="168696">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>sequence</p>
</blockquote>
</aside>
<p>Thanks for the kind explanations. The performance decreased after I remove those unused functions, and it seems that there is no way to fix it.</p>

ptrblck: <p>Yes, you are right that there is no proper way of “fixing” the expected noise by changing random seeds.<br>
You should expect to see the same behavior in your final model accuracy by simply using another initial seed. If the <code>stddev</code> of the final accuracy is too large, you would have to check your training routine and try to stabilize the training.</p>
Pass input data to independent modules of nn.ModuleList concurrently

Sohrabbeig: <p>Hi,</p>
<p>I have a simple module list like as follows:</p>
<pre><code class="lang-auto">self.Linear = nn.ModuleList()
for i in range(self.channels):
    self.Linear.append(nn.Linear(self.seq_len, self.pred_len))

</code></pre>
<p>In the forward pass, I pass the input data to this module list sequentially, which is not efficient. It is worth mentioning that these modules are independent of each other, and ideally we should be able to pass the data to them concurrently. Is there a way to do that?</p>
<p>Here is what I have in the forward method:</p>
<pre><code class="lang-auto">for i in range(self.channels):
    output[:,:,i] = self.Linear[i](x[:,:,i])
</code></pre>

Sohrabbeig: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you help me with this?</p>

ptrblck: <p>You could check <code>vmap</code>’s ensemble approach described in <a href="https://pytorch.org/functorch/nightly/notebooks/ensembling.html">this tutorial</a>.</p>
If I create an commercial app, must I credit torchvision

Jamil_Karabaev: <p>I am using a torchvision model, deeplabv3, not pretrained, must I mention that I took this model from torchvision in my app</p>

tom: <p>If you want lawyer-grade advice, you should get a lawyer (which I am not), but TorchVision’s <a href="https://github.com/pytorch/vision/blob/main/LICENSE" rel="noopener nofollow ugc">LICENSE</a> file says that it’s 3-clause-BSD, which has attribution requirements of sorts.</p>
<p>Best regards</p>
<p>Thomas</p>
How to count frozen parameters in CNN model?

Tahsir_Ahmed_Munna: <p>I have frozen one layer and now I want to count the number of the frozen parameters in my CNN model. How do I count this?</p>
<pre><code class="lang-auto">model_parameters = filter(lambda p: p.requires_grad, model.parameters())
params = sum([np.prod(p.size()) for p in model_parameters])
print(f"Number of Parameter: {params/1000000:.2f}M")
</code></pre>
<p>In this script, I count the all parameters that I have but I need only the number of my frozen parameters. When using p.requires_grad==False, It gives 0.</p>
<p>Thank you</p>

ptrblck: <p>Your code seems to work fine:</p>
<pre><code class="lang-python">model = models.resnet50()
model.fc.weight.requires_grad = False
print(model.fc.weight.nelement())
# 2048000

model_parameters = filter(lambda p: p.requires_grad, model.parameters())
params = sum([np.prod(p.size()) for p in model_parameters])
print(f"Number of Parameter: {params/1000000:.2f}M")
# Number of Parameter: 23.51M

model_parameters = filter(lambda p: p.requires_grad==False, model.parameters())
params = sum([np.prod(p.size()) for p in model_parameters])
print(f"Number of Parameter: {params/1000000:.2f}M")
# Number of Parameter: 2.05M
</code></pre>
Get the latest pytorch with pip

ambodi: <p>I wonder why all the discussions on upgrading pytorch points to the conda version, but not the pip version?</p>
<p>Can we not get the latest pytorch through pip upgrade?</p>

ptrblck: <p>PyTorch is now available for more or less a week on PyPI, so I guess it’s too new to see a lot of threads <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

arogozhnikov: <p>Doesn’t look its on PyPI. I’m using pip version from <a href="http://pytorch.org" rel="nofollow noopener">pytorch.org</a> custom PyPI without any problems.</p>
<pre><code class="lang-bash">pip install pytorch==
Collecting pytorch==
  Could not find a version that satisfies the requirement pytorch== (from versions: 0.1.2)
No matching distribution found for pytorch==
</code></pre>

ptrblck: <p>Try</p>
<pre><code class="lang-auto">pip install torch
</code></pre>

arogozhnikov: <p>oh, indeed. As I see from <a href="http://pytorch.org" rel="nofollow noopener">pytorch.org</a>, this is cuda 8.0 build. Not really what I expected</p>

NewUser: <p>This helped me on ubuntu 16, python3.6</p>
<p><code>pip install --upgrade torch torchvision</code></p>

traits: <p>This should be on <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">pytorchs install site</a>. Caused some head scratching here too, because the example command there simply throws “Requirement already satisfied”, doing nothing else.</p>

FamousCarl: <p>this works after  upgrade numpy.<br>
pip install --upgrade numpy<br>
Version: 1.24.0</p>