akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Adding Dropout in DenseNet201 Pretrained

username: bryan123
cooked: <p>How can I add dropout layers after every convolution layer in DenseNet201 pretrained if I want to keep the value of its parameters (weights/Biases)? (FYI, I wanted to add dropout layers between the convolutional layers in order to quantify MC-Dropout uncertainty during prediction).</p>

username: ptrblck
cooked: <p>You could try to replace each original conv layer with a new <code>nn.Sequential</code> layer containing the new dropout layer as well as the pre-trained conv layer.<br>
Something like this should work:</p>
<pre><code class="lang-python">model = models.densenet201()

# replace conv layer
old_conv = model.features[0]

model.features[0] = nn.Sequential(
    nn.Dropout(),
    old_conv)
</code></pre>
<p>Note that you are changing the model architecture and thus won’t be able to load a pre-trained <code>state_dict</code> to this manipulated model anymore so make sure to load the <code>state_dict</code> before adding the dropout layers or store the new <code>state_dict</code> afterwards.</p>

username: bryan123
cooked: <p>Shouldn’t dropout be added to the dense layer as well? Would it be better to put dropout before or after Conv2d?</p>
<p>Could you also show me how to load the state_dict before adding dropout layers?</p>

username: ptrblck
cooked: <p>You can add the dropout layers wherever you think it would work and my code snippet is just one example how to add it before one conv layer.</p>
<p>If you want to use the pretrained model from <code>torchvision</code> use:</p>
<pre><code class="lang-python">model = models.densenet201(pretrained=True)
</code></pre>
<p>otherwise if you want to load a <code>state_dict</code> that you’ve stored after training the model load it via:</p>
<pre><code class="lang-python">model = models.densenet201()
model.load_state_dict(torch.load(path_to_state_dict))
</code></pre>
<p>before changing the architecture.</p>
Gradient of X is NoneType in second iteration

username: samlk
cooked: <p>Hi, I’m trying to make images which will fool model, but I have some problem with this code. In the second iteration I get <code>TypeError: unsupported operand type(s) for -: 'Tensor' and 'NoneType' </code><br>
Why is the grad NoneType even if it’s working for the first time?</p>
<pre><code class="lang-auto">X_fooling = X.clone()
X_fooling.requires_grad_()
loss_f = torch.nn.MSELoss()

for i in range(1000):
      score = model(X_fooling)
      y = torch.zeros(1000)
      y[target_y] = 1
      loss = loss_f(score, y)
      print(loss)
      loss.backward()
        
      if target_y == torch.argmax(score):
        break

      X_fooling = X_fooling - X_fooling.grad
</code></pre>

username: ptrblck
cooked: <p>You are replacing the leaf variable:</p>
<pre><code class="lang-python">X_fooling = X.clone()
X_fooling.requires_grad_()
</code></pre>
<p>with a non-leaf:</p>
<pre><code class="lang-python">X_fooling = X_fooling - X_fooling.grad
</code></pre>
<p>I’m unsure if you want to recreate the leaf variable, but you could use:</p>
<pre><code class="lang-python">X_fooling.detach().requires_grad_()
</code></pre>
<p>to recreate the leaf variable.</p>

username: samlk
cooked: <p>Thank you, I solved the problem with usage of optimizer. I haven’t thought about replacing of the tensor</p>
Pytorch tensor.unfold

username: gpas
cooked: <p>Hi,</p>
<p>I had to use tensorrt but the tensor unfold operation was not supported and the output of the converted model was rubbish.<br>
So , I wrote the unfold with simple operations (<a href="https://discuss.pytorch.org/t/how-nn-unfold-works/137349">forum thread</a>) but it is extremely slow.<br>
Do you think python’s for loops is causing this?</p>

username: ptrblck
cooked: <p>Yes, a nested loop could create a large overhead compared to a single operation.</p>

username: gpas
cooked: <p>OK thank you , I suppose it is possible to replace unfold with conv2d then and retrain the model then?</p>

username: ptrblck
cooked: <p>Yes, if you were using <code>unfold</code> and a <code>matmul</code> to execute a convolution, you could directly use a conv layer instead.</p>
Support for distributed dataset?

username: Luosuu
cooked: <p>Greetings! I think currently DDP is assuming all the processes share a same dataset no matter copying them on each node or stored in a NFS? Do we have any support for the case like the following picture?<br>
If so, how to ensure the global shuffling?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76209986e3ced03318319a8d7beea53e1000a1bc.png" alt="image" data-base62-sha1="gR0bxhSKByvbCcu6h7lvB45O5nm" width="553" height="276"></p>
Is F.layernorm equal to normalize tensor?

username: Anikily
cooked: <p>a = torch.rand(32, 100)<br>
layer_norm = nn.LayerNorm(100, elementwise_affine=False)<br>
b = layer_norm(a)<br>
c = (a - a.mean(1).unsqueeze(1).repeat(1,100)) / a.std(1).unsqueeze(1).repeat(1,100)<br>
In my understanding, b should be equal to c in every element, however, i obtained slightly different results, can anyone help me?</p>

username: ptrblck
cooked: <p>As explained in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html#torch.nn.LayerNorm">docs</a>, the stddev is calculated via the biased estimator.<br>
This should work:</p>
<pre><code class="lang-python">c = (a - a.mean(1).unsqueeze(1).repeat(1,100)) / torch.sqrt(a.var(1, unbiased=False).unsqueeze(1).repeat(1,100) + 1e-5)
</code></pre>
Rearrange Tensor by Index

username: Fst_Msn
cooked: <p>Given a 2D tensor A of arbitrary shape (e,g (8, 2)):</p>
<pre><code class="lang-auto">A = torch.tensor([[0., 0.],[0., 1.],[0., 2.],[0., 3.],[0., 4.],[0., 5.],[0., 6.],[0., 7.]])
</code></pre>
<p>and an index tensor B of the same length as the first dimension of A:</p>
<pre><code class="lang-auto">B = torch.IntTensor([0, 1, 0, 1, 2, 3, 2, 3])
</code></pre>
<p>I would like to rearrange A according to the indices of B,<br>
In this case, I need A to be of shape (4, 2, 2):</p>
<pre><code class="lang-auto">[
[[0., 0.], [0., 2.]],
[[0., 1.], [0., 3.]],
[[0., 4.], [0., 6.]],
[[0., 5.], [0., 7.]]
]
</code></pre>
<p>Is there any way to do it, without a for loop on B?</p>
<p>Thanks in advance</p>

username: tom
cooked: <p>I’m not sure how you would get 4, 2 from 8 (maybe just view(…), if you know in advance that there are two each?) but the indexing part to a 8x2 tensor could be <code>idx = torch.argsort(B); Asorted = A[idx]</code>.<br>
Note that <code>argsort</code> might not be stable (I think that that is not guaranteed, in particular for CUDA), you could also do <code>idx = idx.view(4, 2).sort(1).values</code> or so to sort it again and then index into A.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: Don’t use <code>torch.IntTensor</code>, use <code>torch.tensor</code> there, too, if needed with <code>dtype=torch.int32</code>.</p>

username: Fst_Msn
cooked: <p>Thank you,<br>
since I know the indices in B go from 0 to n, I just solved it like:</p>
<pre><code class="lang-auto">idx = torch.argsort(B)
dim_0 = torch.max(B) + 1
dim_1 = torch.div(idx.shape[0], dim_0, rounding_mode='trunc')
A = A[idx].view(dim_0, dim_1, -1)
</code></pre>

username: tom
cooked: <p>I would probably use <code>dim_0 = torch.max(B).item() + 1</code> and <code>idx.shape[0] //  dim_0</code> instead if you don’t know the number in advance to reduce mixed gpu/cpu ops.</p>
<p>If you do know in advance, you can avoid the sync point (explicit in my <code>.item()</code> implicit in your use of <code>dim_0</code> and <code>dim_1</code> as arguments to <code>.view()</code>).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Fst_Msn
cooked: <p>Ok, thanks for the tip!</p>
Execute multi-task branches in parallel

username: bask0
cooked: <p>I am working on a model for multi-task learning that has a shared part and multiple independent branches for the different tasks.</p>
<p>This is a MWE of such a model:</p>
<pre><code class="lang-auto">import torch
from torch import nn

class MT(nn.Module):
    def __init__(self):
        super(MT, self).__init__()
        # Shared layer.
        self.shared = nn.Linear(in_features=10, out_features=10)
        # Branch for task 0.
        self.branch0 = nn.Linear(in_features=10, out_features=1)
        # Branch for task 1.
        self.branch1 = nn.Linear(in_features=10, out_features=1)
    def forward(self, x):
        out = self.shared(x)
        out_0 = self.branch0(out)
        out_1 = self.branch1(out)
        return out_0, out_1
</code></pre>
<p>I have the suspicion that the independent branches are causing the GPU to starve as the shared model is more complex than the branches. Thus, I wonder if PyTorch understands that the two branches <code>self.branch0(out)</code> and <code>self.branch1(out)</code> are independent and that the forward and backward pass can be computed in parallel. If not, is there a way to achieve parallel execution of the branches?</p>

username: Kathy1
cooked: <p>Hello, I am faced with the same problem. May I kindly ask if you have figured it out? Thanks.</p>

username: bask0
cooked: <p>I think the bottleneck was somewhere else in my case. If the branches are not huge models, I think this is not essential. If performance is key, <a href="https://pytorch.org/tutorials/advanced/torch-script-parallelism.html" rel="noopener nofollow ugc">TorchScript</a> could be an option, I guess the branches could be ran in parallel then.</p>
Torch.onnx.export can use multiprocessing.pool?

username: lzcstar
cooked: <h1>
<a name="hi-everyone-i-have-a-question-when-i-use-multiprocessingpool-to-torchonnxexport-it-is-slower-than-before-has-anyone-ever-met-this-question-whats-the-solution-1" class="anchor" href="#hi-everyone-i-have-a-question-when-i-use-multiprocessingpool-to-torchonnxexport-it-is-slower-than-before-has-anyone-ever-met-this-question-whats-the-solution-1"></a>Hi, everyone<br>
I have a question.<br>
When I use multiprocessing.Pool to torch.onnx.export, it is slower than before.<br>
Has anyone ever met this question? What’s the solution?</h1>
<p>The code example is as follows:<br>
for coding_value in list1:<br>
pool.apply_async(process_one,args=(coding_value,))<br>
pool.close()<br>
pool.join()</p>
<h1>
<a name="the-process_one-includes-torchonnxexportxxxcodingcoding_value-input-onnx_path-verbosefalse-input_namesplaceholder-output_namesoutput-opset_version11-2" class="anchor" href="#the-process_one-includes-torchonnxexportxxxcodingcoding_value-input-onnx_path-verbosefalse-input_namesplaceholder-output_namesoutput-opset_version11-2"></a>the &lt;process_one&gt; includes<br>
torch.onnx.export(xxx(coding=coding_value), (input), onnx_path, verbose=False, input_names=[‘Placeholder’],  output_names=[‘output’], opset_version=11)</h1>
ONNX export if submodule has extra state

username: cherrywoods
cooked: <p>How may I export a module with a submodule that implements <code>get_extra_state()</code> but doesn’t return a tensor (returns a <code>dict</code> maybe)?<br>
For example, when I run the following code:</p>
<pre><code class="lang-python">from typing import Any

import torch
from torch import nn


class TestModule1(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = nn.Linear(10, 10)

    def get_extra_state(self) -&gt; Any:
        return {"value1": 42, "value2": "very important"}

    def set_extra_state(self, state: Any):
        pass

    def forward(inputs):
        return self.layer(inputs)

class TestModule2(nn.Module):
    def __init__(self):
        super().__init__()
        self.test = TestModule1()
        self.relu = nn.ReLU()

    def forward(inputs):
        x = self.test(inputs)
        return self.relu(x)


module = TestModule2()
pretty_string = torch.onnx.export_to_pretty_string(module, torch.ones((1, 10)))
print(pretty_string)
</code></pre>
<p>it fails with the error:</p>
<pre><code class="lang-auto">  File ".../lib/python3.10/site-packages/torch/onnx/__init__.py", line 388, in export_to_pretty_string
    return utils.export_to_pretty_string(*args, **kwargs)
  File ".../lib/python3.10/site-packages/torch/onnx/utils.py", line 852, in export_to_pretty_string
    graph, params_dict, torch_out = _model_to_graph(
  File ".../lib/python3.10/site-packages/torch/onnx/utils.py", line 727, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File ".../lib/python3.10/site-packages/torch/onnx/utils.py", line 602, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File ".../lib/python3.10/site-packages/torch/onnx/utils.py", line 515, in _trace_and_get_graph_from_model
    orig_state_dict_keys = torch.jit._unique_state_dict(model).keys()
  File ".../lib/python3.10/site-packages/torch/jit/_trace.py", line 71, in _unique_state_dict
    filtered_dict[k] = v.detach()
AttributeError: 'dict' object has no attribute 'detach'
</code></pre>
<p>I would appreciate any help!</p>
Reduction the dimension of input using Conv 1x1

username: S_M
cooked: <p>Hi,<br>
I have an input x by dimension 128x512x1x1 and I need to reduce it to 10x10x1x1, how can I do that using Conv 1x1 in Pytorch?<br>
any help would be appreciated.</p>

username: eqy
cooked: <p>I’m assuming the given dimensions are in y, x, c, n or y, x, n, c (which seems to be an unusual ordering) as otherwise it would imply a reduction in the batch dimension which seems unintended.</p>
<p>A simple method would be to simply increase the stride:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.randn(1, 1, 128, 512)
&gt;&gt;&gt; c = torch.nn.Conv2d(1, 1, 1, (13, 52))
&gt;&gt;&gt; o = c(a)
&gt;&gt;&gt; o.shape
torch.Size([1, 1, 10, 10])
</code></pre>
<p>However, you may wish to downsample in a “smoother” way, so you might want to take a look at e.g., AdaptiveAvgPool2D:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html</a></p>

username: S_M
cooked: <p>thank you very much.</p>
How to understand aten::lsm

username: MartinZhang
cooked: <p>Hi , I can’t understand about the args of aten::lstm function</p>
<p>How to match the args of aten::lstm and torch.nn.LSTM()</p>
<p>Thanks.</p>
<p>ps:I am not found the api doucment about aten::lstm.</p>
<p><a href="https://pytorch.org/cppdocs/api/library_root.html" rel="noopener nofollow ugc">Library API — PyTorch master documentation</a></p>

username: ptrblck
cooked: <p>You can specify the options via <code>LSTMOptions</code> and create an LSTM layer via e.g.:</p>
<pre><code class="lang-auto">auto lstm_opt = LSTMOptions(1, 1).num_layers(1).batch_first(false).bidirectional(false);
LSTM lstm{lstm_opt);
</code></pre>

username: MartinZhang
cooked: <p>oh, awsome, thanks very much</p>
PyTorch v1.9.0 is not compatible with CUDA 11.3?

username: Dee2
cooked: <p>Hi,</p>
<p>I have followed the installation instructions from the website: <a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">https://pytorch.org/get-started/previous-versions/</a> to install PyTorch v1.9.0 with CUDAtoolkit of v11.3.0 on Windows11. However, when running the code, it always shown that ‘<strong>Torch is not compatible with CUDA enabled</strong>’. I have already installed CUDA v11.3 on Windows11 as well. My device is: Nvidia RTX 3090Ti, Windows11 OS.</p>
<p>Can I know if PyTorch v1.9.0 is compatible with CUDA 11.3? How can I solve this issue?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>I don’t remember with CUDA version was used for the <code>1.9.0</code> release, but what did the install log show?<br>
Did it show the proper wheel name with the <code>cu113</code> tag in it or, in case you are using conda, did it install <code>cudatoolkit=11.3</code>?</p>

username: Dee2
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thanks for your reply. According to the official installation instruction (<a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">https://pytorch.org/get-started/previous-versions/</a>), <code>1.9.0</code> should be compatible with CUDA 10.2 and 11.3. I used <code>conda</code> to install it using the official installation command: <code>conda install pytorch==1.9.0 torchvision==0.10.0 torchaudio==0.9.0 cudatoolkit=11.3 -c pytorch -c conda-forge</code>.</p>
<p>Do you think if I need to use <code>pip</code> to install it instead?</p>
<p>Thank you!</p>
Does Pytorch support to export model with deformable convolution to ONNX?

username: hitbuyi
cooked: <p>The original deformable convolution  is described <a href="https://arxiv.org/abs/1703.06211v3" rel="noopener nofollow ugc">here</a>,  It is said in paddlepaddle, the deformable convolution is not supported, how about it in  Pytorch? if not, what’s the remedy?</p>
How to train cascaded networks iteratively?

username: medvisioner
cooked: <p>I hope to train two cascaded networks, e.g. X-&gt;Z-&gt;Y, Z=net1(X), Y=net2(Z).<br>
I hope to optimize the parameters of these two networks iteratively, i.e., for a fixed parameter of net1, firstly train parameters of net2 using MSE(predY,Y) loss util convergence; then, use the converged MSE loss to train a iteration of net1, etc.<br>
So, I define two optimizers for each networks respectively. My training code is below:</p>
<pre><code class="lang-auto">net1 = SimpleLinearF()
opt1 = torch.optim.Adam(net1.parameters(), lr=0.01)
loss_func = nn.MSELoss()

for itera1 in range(num_iters1 + 1):
    predZ = net1(X)
        
    net2 = SimpleLinearF()
    opt2 = torch.optim.Adam(net2.parameters(), lr=0.01)
    for itera2 in range(num_iters2 + 1):
        predY = net2(predZ)
        loss = loss_func(predY,Y)
        if itera2 % (num_iters2 // 2) == 0:
            print('iteration: {:d}, loss: {:.7f}'.format(int(itera2), float(loss)))
        loss.backward(retain_graph=True)
        opt2.step()
        opt2.zero_grad()
    
    loss.backward()
    opt1.step()
    opt1.zero_grad()
</code></pre>
<p>However, I encounter the following mistake:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an 
inplace operation: [torch.FloatTensor [1, 1]], which is output 0 of AsStridedBackward0, is at
version 502; expected version 501 instead. Hint: enable anomaly detection to find the 
operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Does anyone know why this error occurs? How should I solve this problem. Many Thanks.</p>
How to measure time of each function(aten::cudnn_convolution)?

username: yuri123
cooked: <p>Hello,</p>
<p>I want to measure the execution time of each function on CUDA<br>
I used torch profiler with activities=[torch.profiler…], but I can only see the total cuda time, self cuda time and time avg. What I want to know is the time of the fisrt cudnn_convolution, second cudnn_convolution…, and the last one. Is there no way to know the execution time of each function separately?<br>
do I need to measure the kernel execution time on tensorboard one by one?<br>
Thank you.</p>

username: ptrblck
cooked: <p>You could use a visual profiler, such as Nsight Systems, which shows the kernel runtimes etc. in the timeline directly, alternatively you could also check the summary of the different kernels via <code>nsys nvprof python script.py</code>, or you could also manually measure the runtimes via a timer and synchronizations.</p>

username: yuri123
cooked: <p>Thank you for your reply.</p>
<p>I already used Nsight systems, Nsight compute. but pytorch profiler with tensorboard also shows kernel runtime(wall duration), right? and each “cudnn_convolution” launches multiple kernel so I need integrated kernel runtime, not only one. I just measured each runtime with tensorboard(end time of last kernel - start time of fisrt kernel).<br>
and I think you mean that using time.time() and torch.cuda.synchronize() but I thought it is better to use tensorboard so i used it. Thank you.</p>
Torch.optim.lbfgs nan

username: UestcJay
cooked: <p>The task is to reconstruct the 3D face of a single photo. The similar code is <a href="https://github.com/HavenFeng/photometric_optimization/blob/master/photometric_fitting.py" rel="noopener nofollow ugc">here</a>. When I use adam to optimize as written in the code, it is very smooth, but I rewrite it as lbfgs optimization, and the loss always becomes nan after a period of time. For a week, can anyone help me</p>

username: listar0810
cooked: <p>Hello,I also met this problem,have u solved it?thx</p>

username: 111267
cooked: <p>no, <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=10" title=":sob:" class="emoji" alt=":sob:"> if you solve the problem, please let me know</p>

username: listar0810
cooked: <p>torch.optim.LBFGS doesn’t work for mini-batches,I see <a href="https://github.com/hjmshi/PyTorch-LBFGS" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hjmshi/PyTorch-LBFGS: A PyTorch implementation of L-BFGS.</a> here which support multi-batch,but I cant understand the example shows about multi-batch.</p>

username: jahanifar
cooked: <p>Is there any solution to this issue? i am getting nan</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b541693252c9ac8a8b9b2d0c4d1891674350afa4.png" data-download-href="https://discuss.pytorch.org/uploads/default/b541693252c9ac8a8b9b2d0c4d1891674350afa4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b541693252c9ac8a8b9b2d0c4d1891674350afa4_2_690x236.png" alt="image" data-base62-sha1="pRsyZAG6p5PQ9awNB0wMUMnoCyw" width="690" height="236" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b541693252c9ac8a8b9b2d0c4d1891674350afa4_2_690x236.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b541693252c9ac8a8b9b2d0c4d1891674350afa4_2_1035x354.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b541693252c9ac8a8b9b2d0c4d1891674350afa4_2_1380x472.png 2x" data-dominant-color="F1F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1852×634 69.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Keep getting cannot import name 'multi30k' from 'torchtext.datasets' when trying to run the transformers tutorial example

username: EssamWisam
cooked: <p>I’m trying to run the tutorial in here: <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" class="inline-onebox" rel="noopener nofollow ugc">Language Translation with nn.Transformer and torchtext — PyTorch Tutorials 1.12.1+cu102 documentation</a> but I keep getting the error</p>
<p>“cannot import name ‘multi30k’ from ‘torchtext.datasets’ (/opt/anaconda3/envs/arm/lib/python3.9/site-packages/torchtext/datasets/<strong>init</strong>.py)”</p>
<p>Not sure if it’s a mistake in the code or what.</p>
How to compute the mount of data transferred and the time during the training?

username: Taghreed
cooked: <p>In Federated learning during the training, I want to measure the communication time for transferring data from clients to the server. How to compute the amount of data transferred (byte) and the time it takes in each round during the training?</p>
<p>I use tqdm() (documentation here <a href="https://pypi.org/project/tqdm/3.8.0/#table-of-contents" class="inline-onebox" rel="noopener nofollow ugc">tqdm · PyPI</a>) in order to show the progress bar with time and unit as:</p>
<pre><code class="lang-auto">for i in tqdm((range(num_selected)), unit='B', unit_divisor=1000, unit_scale=True):
        loss += client_update(client_models[i], opt[i], train_loader[client_idx[i]], epoch=epochs)
</code></pre>
<p>The output that I got is:</p>
<pre><code class="lang-auto">100%|██████████| 30.0/30.0 [00:45&lt;00:00, 1.50s/B]
0-th round
average train loss 1.78 | test loss 0.0279 | test acc: 0.278
100%|██████████| 30.0/30.0 [00:44&lt;00:00, 1.48s/B]
1-th round
</code></pre>
<p>t is not clear to me if the data is transferring 1.50 Byte every second or Byte every 1.50 second?</p>
<p>even when I change <code>unit='kB'</code> the output is the same except the letter B is changed to kB with the same amount of the data!</p>
<p><strong>How to use tqdm() to show the real amount of transferred byte in each round?</strong></p>
Deep model over predicting heavily for new users

username: A_G1
cooked: <p>Hi,</p>
<p>I am working to improve the quality of a user model. The changes that I am making makes the results worth for new users.</p>
<p>We already are over predicting for new users and currently don’t have a solution.</p>
<p>My main question is regarding these new users, how can  I makes the prediction better for those, or at least push it towards under prediction ( we pay more out of pocket when we over predict).</p>
<p>Model is 5 layer relus with last layer being sigmoid.</p>
<p>Not sure if this is the right forum. Thanks.</p>

username: parthshah231
cooked: <p>Hello,</p>
<p>Could you elaborate more about the use case (business requirement)? type of data? What EDA steps have you used?<br>
And since you have just mentioned about over-fitting? Have you tried either decreasing your model complexity or some regularization techniques?</p>

username: A_G1
cooked: <p>Hi,</p>
<p>I tried to keep some of the business information intentionally.</p>
<p>The goal is to have better prediction for our customers. For example assume the prediction is the number of clicks need to be predicted.</p>
<p>The model is already well established and I haven’t done much EDA other than seeing that when slice by the age of my customer on my website (let’s say users that are 5 days on my website compare to users that have a lot of activities in the past 5 years). I see that the prediction for these users is really bad. We do use user IDs in our models and other identifying information.</p>
<p>Some people call this a cold start problem but it is mostly in ads space I am not sure how to apply it here.</p>
<p>Re over fitting, I don’t think my model is over fitting. Rather when I look at the calibration graphs, we are consistently over predicting for new users compared to the real number clicks they make.</p>
Memory efficent way to sum over a sequence of tensors

username: Hao_Tang1
cooked: <p>Hi, I want to implement a simple summation over a sequence of tensors. Previously, I implemented it using stack &amp; sum as</p>
<pre><code class="lang-auto">torch.sum(torch.stack(seq, dim=0), dim=0)
</code></pre>
<p>However, this implementation will cause OOM for large tensors. For-loop implementation may also cause OOM since gradients are needed.</p>
<p>May I ask if there is any memory-efficient method to implement the summation over a sequence? Intuitively, I feel like this should just be a bigger sum, but I cannot find a solution. I also found this new wild <a href="https://pytorch.org/docs/master/nested.html" rel="noopener nofollow ugc">nested Tensor</a> feature. Would it support summation / mean in the future? Thanks!</p>
Loading resnet pretrained model

username: Tm4
cooked: <p>I want to use resnet50 pretrained model using PyTorch and I am using the following code for loading it:</p>
<pre><code>    import torch
    model = torch.hub.load("pytorch/vision", "resnet50", weights="IMAGENET1K_V2")
</code></pre>
<p>Although I upgrade the torchvision but I receive the following error:</p>
<p><a href="https://i.stack.imgur.com/r22Vv.jpg" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/r22Vv.jpg" alt="" width="690" height="161"></a></p>
<p>Any idea?</p>

username: ptrblck
cooked: <p>Which version are you using, as it seems to be already fixed as described <a href="https://discuss.pytorch.org/t/import-error-while-using-torch-hub-load/159690/2">here</a>?</p>
Converting feedforward NN to CNN

username: JessM
cooked: <p>Hi everyone,</p>
<p>I’m training a basic feedforward neural network to make amino acid predictions based on protein structure information.</p>
<p>Here is how the NN architecture is set up:</p>
<pre><code class="lang-auto">class NeuralNetwork(nn.Module):
    def __init__(self, input_nodes):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(input_nodes, 64),             # input 
            nn.ReLU(),
            nn.Dropout(p=0.5),

            nn.Linear(64, 64),                      # hidden layer 1 
            nn.ReLU(),
            nn.Dropout(p=0.5),

            nn.Linear(64, 64),                      # hidden layer 2 
            nn.ReLU(),
            nn.Dropout(p=0.5),

            nn.Linear(64, 64),                      # hidden layer 3 
            nn.ReLU(),
            nn.Dropout(p=0.5),

            nn.Linear(64, 20)                       # output
        )
</code></pre>
<p>I’m wondering if its possible to make modifications to the set-up above in order to convert it to a convolutional NN?<br>
I’m using a non-image dataset, so I’m wondering if a CNN would even be possible?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="JessM" data-post="1" data-topic="160792">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jessm/48/52510_2.png" class="avatar"> JessM:</div>
<blockquote>
<p>I’m wondering if its possible to make modifications to the set-up above in order to convert it to a convolutional NN?</p>
</blockquote>
</aside>
<p>Yes, you could e.g. replace the <code>nn.Linear</code> layers with <code>nn.ConvXd</code>.</p>
<aside class="quote no-group" data-username="JessM" data-post="1" data-topic="160792">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jessm/48/52510_2.png" class="avatar"> JessM:</div>
<blockquote>
<p>I’m using a non-image dataset, so I’m wondering if a CNN would even be possible?</p>
</blockquote>
</aside>
<p>Yes, it would be possible as long as the inputs are in the expected shape, e.g. <code>[batch_size, channels, height, width]</code> for <code>nn.Conv2d</code>. It depends on your use case and how you are handling the data to claim it “makes sense”. If you think your input data has a correlation between neighboring “pixels” it sounds like a valid idea to try out a CNN on this data.</p>
DeprecationWarning: elementwise comparison failed; this will raise an error in the future

username: ArshadIram
cooked: <pre><code class="lang-auto">idx = np.where(label.cpu().numpy() == np.array([0]*data.cpu().shape[0]))[0]
</code></pre>
<p>I am tyring to perform elementwise comparison. However, getting the error.</p>
<pre><code class="lang-auto"> DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
  idx = np.where(label.cpu().numpy() == np.array([0]*data.cpu().shape[0]))[0]

&lt;__array_function__ internals&gt;:180: DeprecationWarning: Calling nonzero on 0d arrays is deprecated, as it behaves surprisingly. Use `atleast_1d(cond).nonzero()` if the old behavior was intended. If the context of this warning is o
f the form `arr[nonzero(cond)]`, just use `arr[cond]`.
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 8.00 GiB total capacity; 6.85 GiB already allocated; 0 bytes free; 7.30 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting
max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

</code></pre>
<p>Any help would be appreciated.</p>

username: ptrblck
cooked: <p>The <code>DeprecationWarning</code> is raised by <code>numpy</code> not PyTorch while the “Out of memory” error is raised by PyTorch if your GPU is running out of memory.<br>
Try to decrease the memory usage by e.g. reducing the batch size.</p>
torch.optim.SparseAdam requires dense parameter

username: Max_Unhold
cooked: <p>Hello,<br>
when I use the <a href="https://pytorch.org/docs/stable/generated/torch.optim.SparseAdam.html" rel="noopener nofollow ugc">SparseAdam — PyTorch 1.12 documentation</a> I get the error</p>
<pre><code class="lang-python">ValueError: Sparse params at indices [0, 1]: SparseAdam requires dense parameter tensors
</code></pre>
<p>when I initialize my <code>nn.Module</code>-Layer with a sparse weight matrix (as a <code>nn.Parameter</code>).</p>
<p>I wonder wether this is an error or indeed wanted. The second would lead me to the question why you would need a SparseAdam optimizer if it is not working with sparse weights.</p>

username: ptrblck
cooked: <p>That’s expected since <code>SparseAdam</code> expects sparse gradients (e.g. from <code>nn.Embedding(..., sparse=True)</code>).</p>

username: Max_Unhold
cooked: <p>inputs,outputs and weights of the model are initialized as <code>torch.sparse.FloatTensor</code> and the only model layer is a simple sparse matrix multiplication. Where do I have to change the gradients?</p>
<p>Do I have to define the backward and the forward by myself with an autograd function?</p>

username: ptrblck
cooked: <p>Yes, this might be a possible approach and you could then use <code>torch.sparse.addmm()</code> to create the sparse gradients. However, a custom <code>nn.Module</code> might be sufficient as the operation should be differentiable and thus you wouldn’t need to implement the <code>backward</code> as well. <a href="https://pytorch.org/docs/stable/sparse.html#supported-linear-algebra-operations">This list</a> shows the supported operations and if the resulting gradient will be sparse.</p>
How to determine where do the gradients come from

username: itsme_andras
cooked: <p>I am wondering if there is a standard way to know where do the gradients come from, when they add up to make the gradient for my <code>torch.nn.Parameter</code> during <code>backward()</code>. I have a custom recurrent network where the same parameters are reused inside the temporal sequence of steps I take during <code>forward()</code>, thus when calculating the gradients, those come from multiple items in the sequence and if I understand correctly, they are summed up to give me my <code>parameter.grad</code> after <code>backward()</code>. Is it possible to determine where do those contributions come from?</p>

username: srishti-git1110
cooked: <p>Could you please let me know if I understood you correctly when I say:<br>
You are trying to calculate the gradient of the loss with respect to, let’s say, each token in an input sentence separately.</p>
<p>I have worked on such a problem previously using backward hooks but while working with a BERT based model, not RNNs. So maybe, backward hooks could help if that’s what you are trying to do.</p>
Can custom autograd function handle list of tensors in backward

username: EdwinYam
cooked: <p>I’m wondering if <strong>list of tensors</strong> can backward in custom autograd function? Below is my sample code.</p>
<pre><code class="lang-auto">class ReversibleFunction(Function):
    @staticmethod
    def forward(
        ctx: FunctionCtx,
        x, blocks, reverse, layer_state_flags: List[bool],
    ) -&gt; Tuple[Tensor, List[Tensor]]:

        # layer_state_flags: indicate the outputs from
        # which layers are used for intermediate loss calculation
        ctx.layer_state_flags = layer_state_flags
        blocks = blocks[::-1] if reverse else blocks

        states = [x] if layer_state_flags[0] else []
        for i, block in enumerate(blocks):
            x = block(x)
            if layer_state_flags[i + 1]:
                states.append(x)

        ctx.blocks = blocks
        ctx.save_for_backward(x.detach())
        return x, states

 
    @staticmethod
    @once_differentiable
    def backward(
        ctx: FunctionCtx, dy: Tensor, dys: List[Tensor]
    ):
        # dys: list of grads from the intermediate loss of each selected layer
        y = ctx.saved_tensors

        layer_state_flags = ctx.layer_state_flags[::-1]
        for i, block in enumerate(ctx.blocks[::-1]):
            if layer_state_flags[i]:
                dy += dys[i]

            y, dy = block.backward(y, dy)

        return dy, None, None, None, None, None

</code></pre>
<p>Since my blocks are reversible, so I don’t need to store <code>states</code> intermediate outputs from each selected layer. However, I want to use those intermediate outputs <code>states</code> to calculate intermediate losses and backward then. I wonder if grads input <code>dys</code> for backward will also come in the form of a list</p>
None type return while trying to find gradient of model's parameters

username: bsg_ssg
cooked: <p>I am trying to calculate gradients wrt model’s parameters. But it is returning None value. What should I do ?<br>
Thanks<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85740c4a3dbbd4ed068ccde0ad0b951893f9cebd.png" alt="image" data-base62-sha1="j2A8o0Y73WbuQJFyG9c3RYqsOkt" width="583" height="398"></p>

username: srishti-git1110
cooked: <p>Hi <a class="mention" href="/u/bsg_ssg">@bsg_ssg</a></p>
<p>None is returned when the grad attribute isn’t populated during a backward call because of <code>requires_grad</code> being False for those tensors (parameters in your case).</p>
<p>Going by your code, I would say the for loop where you set <code>params.requires_grad = True</code> should be placed before any loss calculation or backward propagation is done.</p>
<p>It’d be best if you just place the <code>params.requires_grad = True</code> for loop outside the outermost for loop which is -<br>
<code>for i, j in train_loader_clean_ewc:</code></p>
<p>Let me know if you still face any errors after doing this.</p>

username: bsg_ssg
cooked: <p>Hi <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a><br>
If I place params.requires_grad = True before the outermost loop then also it is throwing None. However, if I place it inside the second loop where params are defined , then it is working. Is it right way? Please let me know if I am making any wrong steps.<br>
Thanks<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79b662fa365146880697229d85c2ed9493e1058b.png" alt="Capture" data-base62-sha1="hmIxcnmUGG0rns96WwLLvSCyKcr" width="645" height="439"></p>

username: srishti-git1110
cooked: <p>Your current code looks fine to me.</p>
<p>And this is what I actually meant in my previous reply. I asked you to shift “the loop” containing <code>requires_grad = True</code> statement completely and not just that particular statement.</p>
<p>Now, since you’ve put it before doing any loss related calculations, it should be fine.</p>
<p>We basically need to set the <code>requires_grad</code> attribute to <code>True</code> for all the params, that’s why a <code>for</code> looping over params is needed.</p>
<p>[As a side note, it’d be best if you just place “the whole loop” setting the attribute <code>requires_grad = True</code> for all the params outside the outer for.]</p>

username: bsg_ssg
cooked: <p>Hi <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a> ,<br>
Sorry for the misreading. It is working fine now. Thanks for your help. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
RuntimeError: the derivative for 'unique_dim' is not implemented.

username: wenqian_liang
cooked: <p>When I added a small part of my own processing based on other people’s code, this error occurred (it seems to be an error in the process of loss_backward()), which part should I modify?<br>
The complete error is as follows:</p>
<p><code>Traceback (most recent call last):   File "D:\PyCharm 2020.1.1\plugins\python\helpers\pydev\pydevd.py", line 1438, in _exec     pydev_imports.execfile(file, globals, locals)  # execute the script   File "D:\PyCharm 2020.1.1\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile     exec(compile(contents+"\n", file, 'exec'), glob, loc)   File "F:/杂七杂八的代码/EntityMatcher-master/EntityMatcher-master/run.py", line 74, in &lt;module&gt;     run_experiment(model_name, dataset_dir, embedding_dir)   File "F:/杂七杂八的代码/EntityMatcher-master/EntityMatcher-master/run.py", line 32, in run_experiment     model.run_train(train,   File "D:\anaconda3\envs\dm\lib\site-packages\deepmatcher\models\core.py", line 183, in run_train     return Runner.train(self, *args, **kwargs)   File "D:\anaconda3\envs\dm\lib\site-packages\deepmatcher\runner.py", line 338, in train     Runner._run(   File "D:\anaconda3\envs\dm\lib\site-packages\deepmatcher\runner.py", line 249, in _run     loss.backward()   File "D:\anaconda3\envs\dm\lib\site-packages\torch\_tensor.py", line 255, in backward     torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)   File "D:\anaconda3\envs\dm\lib\site-packages\torch\autograd\__init__.py", line 147, in backward     Variable._execution_engine.run_backward( RuntimeError: the derivative for 'unique_dim' is not implemented.</code></p>

username: ptrblck
cooked: <p>Based on the error message it seems you are calling <code>torch.unique(input, dim)</code> in your code and expect to backpropagate through it. Since this operation is not differentiable it will fail:</p>
<pre><code class="lang-python">x = torch.randn(10, 10, requires_grad=True)
out = torch.unique(x, dim=1)
out.mean().backward()
# NotImplementedError: the derivative for 'unique_dim' is not implemented.
</code></pre>

username: wenqian_liang
cooked: <p>Thanks for the answer my problem was solved!</p>
Weight decay together with L2 regularization baked into loss function

username: Filipe_Silva
cooked: <p>Based on what I have been reading here, one can get L2 regularization by providing a value other than 0 to the optimizer through the argument <strong>weigh_decay</strong>.</p>
<p>Yet, one may implement a custom loss function like this one where the L2 regularization is already taken into account:</p>
<pre><code class="lang-auto">class AutoRec_Loss(torch.nn.Module):

    def __init__(self):
        super(AutoRec_Loss,self).__init__()

    def forward(self,predicted_ratings, real_ratings, weights, reg_strength):
        ratings_loss = torch.norm(real_ratings - predicted_ratings)
        # L2 regularization
        weights_regularization = (reg_strength/2)*torch.norm(weight)
        return ratings_loss + weights_regularization
</code></pre>
<p>What would happen if I set a value other than 0 to the underlying optimizer given this loss function?</p>

username: albanD
cooked: <p>Hi,</p>
<p>It will just increase even more the l2 regularization. You loss would be <code>loss = ratings_loss + weight_regularization + weight_decay * weight_norm</code>.</p>

username: Filipe_Silva
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Makes sense! Thank you for your clarification <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

username: JINGHAN_YANG
cooked: <p>May I ask in the weight decay, it is l2 norm for <code>weight_norm</code> in your formula?</p>
TorchScript register backward C++ functions

username: ralphmao
cooked: <p>I just read this tutorial and it does not touch the backward part.<br>
<a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html</a></p>
<p>My understanding is that to get autograd work, we will also need to register the corresponding backward function. However, I could not find how this works. I checked PyTorch source code (torch/csrc/autograd/generated/VariableTypeEverything.cpp) and could not find where backward functions are registered either.</p>
<p>Help will be appreciated!</p>

username: driazati
cooked: <p>You implement it in C++ similarly to in Python via <code>autograd.Function</code>. You then have to register an op that uses your autograd function. See this example</p>
<pre><code class="lang-cpp">#include &lt;torch/script.h&gt;
#include &lt;torch/all.h&gt;

#include &lt;iostream&gt;
#include &lt;memory&gt;

using namespace at;
using torch::Tensor;
using torch::autograd::AutogradContext;
using torch::autograd::Variable;
using torch::autograd::variable_list;

// computes f(x) = 2x
class MyDouble : public torch::autograd::Function&lt;MyDouble&gt; {
 public:
  static variable_list forward(
      AutogradContext* ctx,
      Variable input) {
    return {input + input};
  }

  static variable_list backward(
      AutogradContext* ctx,
      variable_list grad_output) {
    return {torch::ones({2, 2}) + 1};
  }
};

Tensor double_op(const Tensor&amp; input) {
  return MyDouble::apply(input)[0];
}

static auto registry =
  torch::RegisterOperators("my_ops::double_op", &amp;double_op);
</code></pre>
<p>and in Python</p>
<pre><code class="lang-python">import torch

torch.ops.load_library("build/libmy_custom_op.so")

@torch.jit.script
def fn(x):
    return torch.ops.my_ops.double_op(x)

x = torch.randn(2, 2, requires_grad=True)
print(fn(x))
x.backward(torch.ones(2, 2))
</code></pre>

username: copium
cooked: <p>Hi, saw this thread and thought i’d post an interesting issue - perhaps you could see whats wrong?</p>
<p>If you modify your python script to:</p>
<pre><code class="lang-python">import torch

torch.ops.load_library("build/libmy_custom_op.so")

@torch.jit.script
def fn(x):
    xxx = torch.pow(x, 2)
    return xxx * torch.ops.my_ops.double_op(x)

x1 = torch.tensor([[0.3, 0.5], [2.0, 0.7]], requires_grad=True)
x2 = torch.tensor([[0.3, 0.5], [2.0, 0.7]], requires_grad=True)

print(fn(x1))
x1.backward(torch.ones(2, 2))

print (x1.grad)

torch_only = torch.sum(torch.pow(x2, 2) * 2 * x2)
print (torch_only)

torch_only.backward(torch.ones(2, 2))
print (x2.grad)
</code></pre>
<p>then you’ll notice that the gradients between the torch-only function and the composite torchscript + pytorch are different. The torch-only function has the correct gradients.</p>
How do i load tensorrt model using torch.hub.load?

username: jojowarez
cooked: <p>hey all. i’ve exported a pt file to an engine file for better inference speed to use it on yolov5<br>
but after loading the model, i get this error msg when trying to inference.<br>
can anyone tell me what im missing and what should i do :S? (i’d also appreciate it if you could give me an easy example to follow!)</p>
<pre><code class="lang-auto">import torch

# Model
#model = torch.hub.load('ultralytics/yolov5', 'yolov5s')  # or yolov5m, yolov5l, yolov5x, custom
model = torch.hub.load('./', 'custom', path='./best.engine', source='local')

# Images
img = 'https://ultralytics.com/images/zidane.jpg'  # or file, Path, PIL, OpenCV, numpy, list

# Inference
results = model(img)

# Results
results.print()  # or .show(), .save(), .crop(), .pandas(), etc.
</code></pre>
<p>this is the code and heres the error msg</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac03591204f28ae941dd56c0f18073898a4c45bd.png" data-download-href="https://discuss.pytorch.org/uploads/default/ac03591204f28ae941dd56c0f18073898a4c45bd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac03591204f28ae941dd56c0f18073898a4c45bd.png" alt="image" data-base62-sha1="oxHisQsfOyzRzPqKVmnZ2A2k46V" width="689" height="418" data-dominant-color="1D1D1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1293×785 23.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Based on the error message an illegal memory access was encountered (probably in TensorRT).<br>
You could either try to debug it via e.g. <code>cuda-gdb</code> or forward this issue to TensorRT so that the devs could take a look at it.</p>

username: mostafa_said
cooked: <p>hello</p>
<p>did you get any solution for it</p>