akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
FX: insert a graph instead of a node

username: nscotto
cooked: <p>the fx api has methods for inserting nodes, it is flexible enough so that the node I insert can be a fully featured model with several layers, however I am facing a problem in which I need to insert each nodes from the submodel instead of the submodel as one node and I am not sure how to achieve that.</p>
<p>The context is that I am writing a backbone replacer, it automatically detects backbones and can replace them. It is fine to just remove all nodes from the old backbone and insert the new backbone as a single node in most cases, but in some cases there are skip connections between the backbone and the head and I would like to be able to handle that case. That means I need to insert the replacing backbone layer by layer so I can remap the previous skip connections to layers of the replacing backbone.</p>
<p>I hope that made sense!<br>
Thank you in advance.</p>

username: nscotto
cooked: <p>What I would somewhat like to do is:</p>
<pre><code class="lang-auto">model_nodes = list(model.graph.nodes)
block_nodes = list(model.graph.nodes)
block_nodes[0].replace_all_uses_with(model_nodes[insert_idx])
model_nodes[insert_idx].replace_all_uses_with(block_nodes[-2])
</code></pre>
<p>However I don’t think this works, the block nodes are referring to modules not present in the model. I can add the block as a module to the model, but I think <code>replace_all_uses_with</code> is only intended to work with nodes already added to the graph if that makes sense.</p>
<p>I would be really thankful to anyone willing to share some help.</p>

username: Chillee
cooked: <p>So, I think there are a couple of different questions here:</p>
<ol>
<li>
<p>The first one is that you want to insert a module in op by op instead of the entire module at once. I think the easiest way of doing this is through creating proxies and implicitly creating your graph. You would use the <code>inserting_before</code>, and then initialize the proxies and call your module: <a href="https://github.com/pytorch/examples/blob/master/fx/proxy_based_graph_creation.py" class="inline-onebox" rel="noopener nofollow ugc">examples/proxy_based_graph_creation.py at master · pytorch/examples · GitHub</a></p>
</li>
<li>
<p>The other issue is that the new graph is referring to modules that don’t currently exist in your graph. You currently need to resolve this yourself, by inserting your modules into your GraphModule.</p>
</li>
</ol>

username: nscotto
cooked: <p>Thanks for the answer.<br>
For the second point, how can I get a fresh instance of the module I want to insert if it is already in a model?<br>
Would inserting a copy of it just work?</p>
How to get the real conv2d function definition in Python

username: ilovepytorch
cooked: <p>I am trying to understand how conv2d is actually invoked, e.g.  the exact code for passing the arguments from Python to C++. I know that the source code for Conv2d is implemented in C++ here:</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp</a></h4>
<pre><code class="lang-cpp">#include &lt;ATen/ATen.h&gt;
#include &lt;ATen/Parallel.h&gt;
#include &lt;ATen/native/ConvUtils.h&gt;
#include &lt;ATen/native/Pool.h&gt;
#include &lt;ATen/native/cpu/DepthwiseConvKernel.h&gt;
#include &lt;ATen/native/utils/ParamUtils.h&gt;
#include &lt;ATen/native/xnnpack/Engine.h&gt;
#include &lt;ATen/NativeFunctions.h&gt;
#include &lt;c10/util/accumulate.h&gt;
#include &lt;c10/util/irange.h&gt;

#include &lt;ATen/Config.h&gt;
#include &lt;c10/macros/Macros.h&gt;

#include &lt;limits&gt;

#if AT_NNPACK_ENABLED()
#include &lt;nnpack.h&gt;
#endif

</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Also, I assume that the torch.nn.functional.conv2d is should be defined somewhere in Python. However, by searching “def conv2d(” doesn’t give me satisfactory results:<br>
(1) torch/nn/quantized/functional.py, which is actually the quantization version of conv2d.<br>
(2) torch/onnx/symbolic_opset9.py doesn’t not seem to be correct either.<br>
(3) torch/onnx/symbolic_caffe2.py seems to be for onnx/caffe.<br>
Can anyone help me find the real definition in Python code for conv2d(not Module/Class, but function), and explains the stack trace for running conv2d, from Python to C++? Much appreciated!</p>
Pytorch uninstall and reinstall (CUDA problem)

username: I-want-to-use-my-GPU
cooked: <p>Hi.<br>
I’ve been willing to use the GPU of my nvidia GeForce GTX 1050 on Linux for a will now.<br>
But I never managed to install the CUDA and drivers properly.<br>
I’m on Linux Mint 20 Ulyana.</p>
<p>I had CUDA 11.1 installed and launched the</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia
</code></pre>
<p>And realized a little too late that it was launching another installation of CUDA over my first install. At this point I started thinking it was not good…</p>
<p>result : the <code>torch.cuda.is_available()</code> was <strong>false</strong></p>
<p>So I tried to uninstall all CUDA and PyTorch.</p>
<p>I seems that the uninstall only works partially, reinstalling ends  very quickly compared to the first install.</p>
<p>and  <code>torch.cuda.is_available()</code> is now <strong>false</strong> every times.</p>
<p>How can I completely uninstall everything ?<br>
Which NVIDIA driver is recommended with the CUDA 11.1 version of PyTorch ?</p>
<p>Thank you for your time.</p>

username: eqy
cooked: <p>Is your GPU visible (e.g., with <code>nvidia-smi</code>)? I have not used Linux Mint myself, but if it is based on Ubuntu you can try one of the installers <a href="https://developer.nvidia.com/cuda-downloads?target_os=Linux&amp;target_arch=x86_64" rel="noopener nofollow ugc">here</a>.</p>

username: I-want-to-use-my-GPU
cooked: <p>Hi eqy, thanx for the answer.</p>
<p>No I made some <code>sudo apt-get purge *nvidia*</code><br>
and some <code>apt-get remove *nividia*</code><br>
there doesn’t seem to be any nvidia programs installed anymore.</p>
<p>for your suggestion : yes I used these installer at first (the CUDA 11.1.1).<br>
The problem is that the command line<br>
<code>conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia</code><br>
re-installed another version of CUDA over my first one (something like CUDA 11.1.75 if I read correctly ).<br>
And it seems that at some point it broke something.</p>
<p>Now I want to uninstall everything, to try again.<br>
until i succeed.</p>
<p>I tried the commands at this page:<br>
<a href="https://stackoverflow.com/questions/43664444/how-can-l-uninstall-pytorch" rel="noopener nofollow ugc">https://stackoverflow.com/questions/43664444/how-can-l-uninstall-pytorch</a></p>
<p>And now the uninstall command says PyTorch is uninstalled. But I can still make a <code>import torch</code> in my python3 environment.</p>
<p>And reinstall with the PyTorch command above is very short and doesn’t seem to go through the whole process again.</p>
<p>Anyway thank you for your time, i will try to get it done.</p>

username: I-want-to-use-my-GPU
cooked: <p>Hi I have some news:<br>
I Made a “system backup saving” a few days ago with Timeshift application on Linux Mint.<br>
(when my CUDA 11.1 installation was successful)<br>
Just restored that backup this morning</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
True

</code></pre>
<p>everything works fine for me now !</p>
<p>have a wonderful day.</p>
One-hot encoding of multi-class mask

username: banikr
cooked: <p>Hello,<br>
I am creating one-hot encoded labels from 3D Mask (with <code>HxWxD</code> shape) by the following code:</p>
<pre><code class="lang-auto">seg_3d = nib.load(seg_file)
seg = seg_3d.get_data() # fixme
labels =np.unique(seg) # [ 0  1  2  3  4  8 10 11 56]
num_labels = len(labels) # 9
segD = np.zeros((num_labels, seg.shape[0], seg.shape[1], seg.shape[2]))
for i in range(1, num_labels):  # this loop starts from label 1 to ignore background 0
    segD[i, :, :, :] = seg == labels[i]
</code></pre>
<p>Are there better and efficient ways to do that?<br>
I tried <code>torch.scatter_</code> method in PyTorch but couldn’t work it out. Moreover, I need the one-hot encoding in NumPy type, <code>.scatter_</code> from PyTorch takes tensor type.<br>
Any help?</p>

username: tom
cooked: <p>These days there is a <a href="https://pytorch.org/docs/master/generated/torch.nn.functional.one_hot.html" rel="noopener nofollow ugc"><code>torch.nn.functional.one_hot</code></a> that does what you need except that the label dimension is last, but one_hot + permute will get you what you need.</p>
<p>You can (almost) always use torch.from_numpy(array) and tensor.numpy() to get from and to numpy efficiently.</p>
<p>Best regards</p>
<p>Thomas</p>

username: banikr
cooked: <p>Hello <a class="mention" href="/u/tom">@tom</a></p>
<pre><code class="lang-auto">from torch.nn.functional import one_hot
seg = seg_3d.get_data() # 3D numpy array with 9 different values
num_labels = len(np.unique(seg)) # 9
seg = torch.tensor(seg).to(torch.int64)
seg_hot = one_hot(seg, num_labels)
print(seg_hot.shape, torch.unique(seg_hot))
</code></pre>
<p>with the codes I get the following error:</p>
<pre><code class="lang-auto">    seg_hot = one_hot(seg, num_labels)
RuntimeError: Class values must be smaller than num_classes.
</code></pre>
<p>Looks like they discussed the issue here: <a href="https://github.com/pytorch/pytorch/issues/45352" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.one_hot should gracefully skip negative and out-of-range indices · Issue #45352 · pytorch/pytorch · GitHub</a><br>
But found no alternates or solutions.<br>
I think some of the class value e.g. 56, 11 or 10 are higher than the number of classes(9), that results in the error.</p>

username: tom
cooked: <p>I think the problem you are seeing is from the relabeling. You probably would want to use np.unique with return_inverse and pass the output of that to one_hot.</p>
How to convert non nn.module based code into nn.module based code in pytorch?

username: mobassir94
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
i am trying to use this one : <a href="https://github.com/ying09/TextFuseNet/blob/master/detectron2/modeling/roi_heads/roi_heads.py#L469" class="inline-onebox" rel="noopener nofollow ugc">TextFuseNet/roi_heads.py at master · ying09/TextFuseNet · GitHub</a></p>
<p>into this one : <a href="https://github.com/facebookresearch/detectron2/blob/master/detectron2/modeling/roi_heads/roi_heads.py#L536" class="inline-onebox" rel="noopener nofollow ugc">detectron2/roi_heads.py at master · facebookresearch/detectron2 · GitHub</a></p>
<p>you can see the latest version of detectron2 is written with lot of nn.module based code</p>
<p>the <strong>init</strong> function in my first link attached above is different than the 2nd link’s function,i am not understanding how do i pass cfg there?<br>
i tried code conversion like this : <a href="https://pastebin.com/3QCex9xZ" class="inline-onebox" rel="noopener nofollow ugc">det - Pastebin.com</a><br>
but i get this error :</p>
<p>roi_heads.py", line 593, in <strong>init</strong><br>
self.mutil_path_fuse_on = cfg.MODEL.TEXTFUSENET_MUTIL_PATH_FUSE_ON<br>
NameError: name ‘cfg’ is not defined</p>
Trading from checkpoint seems to be same as training from the 0 epoch

username: Captaindoggo
cooked: <p>So I trained GAN model for 400 epochs and saved G and D and both optimizers as state_dict, after loading model from this checkpoint I evaluated it and it showed same score as during first training session (~ 0.77, the closer to 0.5 the better), however when if I continue training from this checkpoint losses values appear to be in completely different range than during the end of the first training session, as if the model is training from random initialisation, and so does the score, back to ~ 0,97. Also I checked the learning rates and the values are exactly how they need to be.</p>

username: ptrblck
cooked: <p>These issues are often created either by a failure in the model saving or a change in the data loading.<br>
To check the first aspect, you could compare the outputs of the model (in <code>.eval()</code>) for a constant input (e.g. <code>torch.ones</code>) before saving and after loading. If the outputs have a larger error than the expected floating point precision, then the saving of the parameters and buffers fails.<br>
On the other hand, if the outputs are equal, you could check the data loading pipeline and make sure the data is processed in the same way.</p>
Training an ensemble model with and without saperately

username: Ashish_Gupta1
cooked: <p>I have such model. How to train such model because of if condion. I want to optimizer.step with  modelA, modelB and modelA+ModelB(MyEnsemble(). Its creating batch issue.</p>
<pre><code class="lang-auto">class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        
    def forward(self, x1):
        output = self.modelA(x1)
        print(output)
        
        if output[0]&gt;0.5:
            out = self.modelB(x1)
        return out
</code></pre>

username: ptrblck
cooked: <p>What kind of issue are you seeing? Could you post the complete error message and, if possible, extend the code snippet to reproduce this issue?</p>
Converting 3D label to one-hot encoding

username: se7en
cooked: <p>Hi, I have a 3d label, and I would like to convert it to one-hot encoding, for example: (C, D, H, W), C=1, after one-hot encoding, converting to (2, D, H, W). Is there any quick solution to this?</p>

username: banikr
cooked: <pre><code class="lang-auto">labels = np.unique(img)
one_hot = np.zeros((2, D, H, W))
for i in range(0, num_labels):
     seg_ =  img == labels[i]
     one_hot[i, :, :, :] = seg_[:, 0:img.shape[1], 0:img.shape[2], 0:img.shape[3]]
</code></pre>
How can I get returns from a function in distributed data parallel?

username: minsikseo
cooked: <p>Hi, I’m working on a model with distributed data parallel. Referring to <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">this</a>, I got</p>
<pre><code class="lang-auto">import os
import torch
import torch.nn as nn
import torch.multiprocessing as mp
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

def main_worker(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12345'
    dist.init_process_group('nccl', rank=rank, world_size=world_size)
    ...
    model = DDP(model, device_ids=[rank])
    ...
    loss = criterion(y_batch, y_pred)
    dist.destroy_process_group()

if __name__ == "__main__":
    mp.spawn(main_worker, args=(ngpus,), nprocs=ngpus, join=True)
</code></pre>
<p>And I’d like to get return <code>loss</code> from <code>main_worker</code>.<br>
I tried <a href="https://discuss.pytorch.org/t/return-from-mp-spawn/94302">this</a>, but it does not work for me.</p>
<p>How can I get  returns from a function using distributed data parallel?</p>

username: seungjun
cooked: <p>Hi, you can use</p>
<ol>
<li>
<p><code>torch.multiprocessing.SimpleQueue</code> to let the child processes to put the results in the queue.</p>
</li>
<li>
<p><a href="https://pytorch.org/docs/stable/distributed.html#point-to-point-communication" rel="noopener nofollow ugc">point-to-point communication</a> functions to send tensors between different distributed processes.</p>
</li>
</ol>
<p>You may want to refer to <a href="https://discuss.pytorch.org/t/access-data-of-a-gpu-process-from-main/95529/3">this thread</a> for more explanation.</p>

username: minsikseo
cooked: <p>Thx! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
In my case, however, I solved using <code>torch.multiprocessing.Pipe</code> as below:</p>
<pre><code class="lang-auto">def main_worker(rank, world_size, conn):
    ...
    conn.send(loss)

if __name__ == "__main__":
    parent_conn, child_conn = mp.Pipe()
    mp.spawn(main_worker, args=(ngpus, child_conn,), nprocs=ngpus, join=True)
    losses = []
    while parent_conn.poll():
        losses.append(parent_conn.recv())
</code></pre>
<p>Then I can gather all losses from every worker.</p>
Creating custom dataset while applying patching on the images

username: ali.maleky7997
cooked: <p>I am trying to write a custom dataset and data loader for a large dataset(SIDD-Full). The dataset contains 200 scenes and there are 150 different image captures of each of these 150 scenes, and each of these captures has a noisy and clean version. for more details, please refer to the <a href="https://www.eecs.yorku.ca/~kamel/sidd/dataset.php" rel="noopener nofollow ugc">dataset website here</a>.<br>
In each epoch, I want to sample one index from the 150 different captures for every 200 different instances and load them to the code, and then, I want to extract <code>n</code> patches from each image with size <code>(32, 32)</code>. I don’t know how to handle this in <code>__getitem__(self, idx)</code> function since I want to return instances of size <code>(1, 4, 32, 32)</code> (the raw images have 4 channels), but if I do patching inside <code>__getitem__()</code> function, I will have a tensor of size <code>(n, 1, 4, 32, 32)</code>, and I don’ know how can I handle that when creating batches out of the dataset with <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc"><code>torch.utils.data.DataLoader</code></a>.</p>
<p>I also have no idea of how to iterate on the patches inside the <code>__getitem__()</code> function. Can anybody help me with this?</p>
Interpreting loss/metric curves

username: banikr
cooked: <p>Hello All,<br>
I would like to get some feedbacks on interpreting the following dice curve. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29a25ffc5d2001966fcfe7b3262d8ac863f0c621.png" data-download-href="https://discuss.pytorch.org/uploads/default/29a25ffc5d2001966fcfe7b3262d8ac863f0c621" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29a25ffc5d2001966fcfe7b3262d8ac863f0c621_2_517x254.png" alt="image" data-base62-sha1="5WjsHMtYAG24qSAwOD0M6UehhsZ" width="517" height="254" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29a25ffc5d2001966fcfe7b3262d8ac863f0c621_2_517x254.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29a25ffc5d2001966fcfe7b3262d8ac863f0c621_2_775x381.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29a25ffc5d2001966fcfe7b3262d8ac863f0c621.png 2x" data-dominant-color="E5EEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">877×432 75.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Trained for 100 epochs, 100-20 train-validation split between the number of subjects. (the lower X-axis is 100 subjects per epoch).<br>
I have used 3D UNet and patch size of <code>48x64x64</code> for multi-label segmentation.<br>
The validation score reaches 74% avg and after that doesn’t increase at all(5 epochs saving criteria).<br>
Is it legit to think that the model may be underfitting?<br>
Any suggestion or feedback appreciated.<br>
Thanks.</p>
Binary Segmentation Using DeepV3

username: joe_dimaggio
cooked: <p>Hello,</p>
<p>I am trying to do a binary segmentation task on the coco dataset using RestNet18 as encoder and DeepLabV3 as the decoder. Looking around i found that BCEWithLogitsLoss is the recommended criterion but i cannot get my training to work. Can you please help me proceed in the right direction ?</p>
<p>I am trying the following in the train method :<br>
outputs = model(inputs)<br>
loss = criterion(torch.nn.Sigmoid(outputs), labels)</p>
<p>The input to my network is torch.Size([32, 3, 256, 256])<br>
The Output is torch.Size([32, 2, 256, 256])<br>
and the labels are torch.Size([32, 1, 256, 256])</p>
<p>I cannot get the sigmoid to work, earlier i tried the CrossEntropyLoss with 2 classes but i doubt if it even worked. I am mostly stressed about the shapes.</p>
<p>Will be grateful for any help, thanks !</p>

username: ptrblck
cooked: <p><code>nn.BCEWithLogitsLoss</code> expects the model output to contain logits, so you would have to remove the <code>sigmoid</code> (it’ll be applied internally in a more numerically stable way).<br>
However, this criterion expects the model output and the target to have the same shape, which doesn’t seem to be the case in your workflow.<br>
For a binary segmentation use case, the model should return an output in <code>[batch_size, 1, height, width]</code>. Based on the current shape I assume you are setting the number of output classes to 2 in a multi-class segmentation. If that’s the case, you would have to either change the last layer so that it outputs one class channel or use <code>nn.CrossEntropyLoss</code> alternatively. In the latter case, you would have to remove the channel dimension from the target so that is has the shape <code>[batch_size, height, width]</code> and contains class indices in the range <code>[0, nb_classes-1]</code>.</p>

username: joe_dimaggio
cooked: <p>Many thanks for your reply. I am sure it’s some detail i just dont understand.</p>
<p>Alternatively, I did try this the way you say i.e (using the softmax on my output(Shape - [32, 2, 256,256])) followed by the Cross Entropy loss and doing squeeze(1) on my target(Shape - [32, 256, 256])<br>
It works but it doesn’t learn anything, the first output of the segmentation mask before training is already good since i use a transfer learning pretrained model however, after running first iteration and calculating loss, the output has nothing besides the training loss goes down for a few batches and plateaus.</p>
<p>I am quite certain my loss and shapes are just not right, it’ll be awesome if you could help out. I wasted a bright sunny day getting nowehere.</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects raw logits, so you would have to remove the softmax and rerun the script.</p>
Gradient calculation for torch.nn.Embedding

username: kevs
cooked: <p>How is the gradient for <code>torch.nn.Embedding</code> calculated? The weight is simply a lookup table - is the gradient being propagated only for the certain indices?</p>
<p>I also have a side question if anyone is knows anything about fine-tuning the BERT model. Are the embedding layers weights adjusted when fine-tuning? I assume they are since the paper states:</p>
<blockquote>
<p>… all of the parameters are fine-tuned using labeled data from the<br>
downstream tasks.</p>
</blockquote>
<p>But wouldn’t changing the embedding parameters (for example the positional encoding) distort the model? I am not sure.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="kevs" data-post="1" data-topic="120071">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevs/48/33660_2.png" class="avatar"> kevs:</div>
<blockquote>
<p>How is the gradient for <code>torch.nn.Embedding</code> calculated? The weight is simply a lookup table - is the gradient being propagated only for the certain indices?</p>
</blockquote>
</aside>
<p>Yes, that’s the case:</p>
<pre data-code-wrap="python"><code class="lang-python">emb = nn.Embedding(num_embeddings=4, embedding_dim=10)
x = torch.tensor([[1]])
out = emb(x)
out.mean().backward()
print(emb.weight.grad)
&gt; tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
           0.0000],
          [0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000,
           0.1000],
          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
           0.0000],
          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
           0.0000]])
</code></pre>
<p>I don’t know the answer to the BERT-related question.</p>

username: kevs
cooked: <p>Thanks a lot, how would it handle duplicate indices? Is the gradient accumulated for duplicate in this case? For example if <code>emb.weight</code> has the shape <code>(4, 10)</code> and <code>x = tensor([[0, 0, 0, 1, 1, 1, 1]])</code> with the shape <code>(1,7)</code>.</p>

username: kevs
cooked: <p>Ok, just tested it and that seems to be the case <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Data-dependent custom dropout

username: kyong
cooked: <p>I am new to ML and pytorch, so probably this is a dumb question.</p>
<p>I saw some posts to make a custom mask for dropout, but in my case, the mask depends on the input: drop out the data in a certain range, drop out max value, or drop out min value, etc. just to mention a few examples.</p>
<p>Since the mask-making process is not differentiable in general, I do not want the mask generation be a part of the backpropagation. So in forward pass, I do the mask generation based on the inputs and apply the mask. And in backprop, I only do the backprop for the mask apply without mask generation. Can I do this in pytorch? If so, how can I do this?</p>

username: eqy
cooked: <p>Do you mean something like:</p>
<pre><code class="lang-auto">    with torch.no_grad():
        mask = ... # mask generation code
</code></pre>
<p>You can then use the mask however you want. I’m actually curious if you need to explicitly use <code>no_grad</code> here if your mask generation code doesn’t rely on any weights…</p>

username: kyong
cooked: <p>Thanks! Will try this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Regarding no_grad, IIUC it is required, otherwise the gradient from the output does not propagate to the input (but I am not sure though).</p>
Best practice for uneven dataset sizes with DistributedDataParallel

username: jdvalencia
cooked: <p>I am wondering about the recommended approach to balancing dataset sizes across different devices while training with DDP. I have split my dataset across four GPUs, but one of them receives a single extra batch, which causes training to hang and wait indefinitely for gradient synchronization with the other devices. I have thought of a few fixes but each seems like it has a drawback:</p>
<p>1.Throw out the final batch to guarantee equal number of iterations<br>
2. Use torch.cuda.no_sync() decorator on the final batch. This will cause one device to have different model weights.<br>
3. Proceed to the next epoch on the other devices and allow the first batch of epoch 2 to synchronize with this final batch from epoch 1.</p>
<p>I appreciate any suggestions you can give!</p>

username: mrshenli
cooked: <p>I saw people doing option 1.</p>
<p>People reporting this issue was usually because applications do not know how many batches each process will take prior to training. It seems in your case, you deterministically know what processes will take one more batch? In that case, I think we might be able to do better. For example,</p>
<p>option 1. randomly skipping one batch in each of the processes that takes one more input batch<br>
option 2. using no_sync on the <strong>first</strong> batch in each of the processes that takes one more input batch. no_sync won’t lead to parameter disparities, it will just accumulate the grad in param.grad. As long as you don’t run <code>optimizer.step()</code> in that iteration, it should be fine. The next forward-backward pass out of the no_sync context will accumulate more grad to param.grad and consume them together.</p>

username: mrshenli
cooked: <p>Would sth like <a href="https://github.com/pytorch/pytorch/issues/33148#issuecomment-584400677" rel="nofollow noopener">this</a> work for you? This can be implemented in the application using allreduce.</p>

username: jdvalencia
cooked: <p>I think that would accomplish it but I basically adopted approach 3 and it has been working fine.</p>

username: rvarm1
cooked: <p>Just to follow up, a feature to support uneven dataset sizes has been added natively to DDP. You can find the docs for the feature here: <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html?highlight=join#torch.nn.parallel.DistributedDataParallel.join" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch 1.8.1 documentation</a></p>
How to put the tensors belonging to the model into cuda

username: ADONAI_TZEVAOT
cooked: <p>I have a model and I have put the model into cuda depending on the availability of gpu. If gpu is available, I put the model and the inputs x into cuda but the tensors in class A which are part of the model are getting loaded into cpu. How to put all the tensors used in class A into cuda if gpu is available? Please help!</p>
<blockquote>
<p>class A(nn.Module):</p>
<pre><code>def __init__(self, dim):

    super().__init__()

    self.dim = dim



    Q = torch.nn.init.orthogonal_(torch.randn(dim, dim))

    P, L, U = torch.lu_unpack(*Q.lu())

    self.P = P # remains fixed during optimization

    self.L = nn.Parameter(L) # lower triangular portion

    self.S = nn.Parameter(U.diag()) # "crop out" the diagonal to its own parameter

    self.U = nn.Parameter(torch.triu(U, diagonal=1)) # "crop out" diagonal, stored in S

def _assemble_W(self):

    """ assemble W from its pieces (P, L, U, S) """

    L = torch.tril(self.L, diagonal=-1) + torch.diag(torch.ones(self.dim))

    U = torch.triu(self.U, diagonal=1)

    W = self.P @ L @ (U + torch.diag(self.S))

    return W

def forward(self, x):

    W = self._assemble_W()

    z = x @ W

    log_det = torch.sum(torch.log(torch.abs(self.S)))

    return z, log_det
</code></pre>
</blockquote>

username: ptrblck
cooked: <p>All <code>nn.Parameters</code> and registered buffers will be moved to the device in <code>.cuda()/.cpu()</code> and <code>.to()</code> operations. This is also the case for <code>self.L, S, U</code>, however, <code>self.P</code> will remain on the CPU, since it’s neither a parameter nor buffer.<br>
If you don’t want to train this tensor, use <code>self.register_buffer("P", P)</code> to register the tensor as a buffer.</p>
How to implement the gradient penalty of WGAN?

username: Mactarvish
cooked: <p>I’m trying to implement the WGAN using PyTorch.</p>
<p>I’ve found there is a way to do that:</p>
<pre><code class="lang-auto">        prob =self.D(input_image)
        # calculate ∂D(input_image) / ∂input_image
        grad = torch.autograd.grad(outputs=prob , inputs=input_image,
                                               grad_outputs=torch.ones(prob .size()).cuda(),
                                               create_graph=True, retain_graph=True)[0]
</code></pre>
<p>But the code version is very old and I’m not sure if it’s the correct way to do it. Anyone knows how to implement the GP elegantly using latest version? Thanks in advance.</p>

username: ptrblck
cooked: <p>I don’t think that <code>torch.autograd.grad</code> was changed recently, so the call still looks alright.<br>
Are you seeing any issues with it?</p>

username: Mactarvish
cooked: <p>Not yet. But I’ve noticed that the doc (<a href="https://pytorch.org/docs/stable/autograd.html" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html</a>) says about the parameter <code>create_graph</code> “Note that in nearly all cases setting this option to  <code>True</code>  is not needed and often can be worked around in a much more efficient way”, so I’m wandering if there’s a better way to do it. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: ptrblck
cooked: <p>Do you see an error, if you don’t create it?<br>
This is usually needed to work with higher derivatives, so it might not be necessary for your use case.</p>
Embedding implementation

username: kevs
cooked: <p>I trying to understand how the <code>torch.functional.F.embedding</code> works with <code>norm_type</code>. I have looked at the source code: <a href="https://github.com/pytorch/pytorch/blob/46a32e075c0bdb54e3e2a08a370d785ee5c5a6cd/torch/nn/functional.py#L1930" class="inline-onebox" rel="noopener nofollow ugc">pytorch/functional.py at 46a32e075c0bdb54e3e2a08a370d785ee5c5a6cd · pytorch/pytorch · GitHub</a></p>
<p>Where does <code>torch.embedding</code> go on line 2024?</p>
<p><strong>EDIT</strong></p>
<p>Found it: <a href="https://github.com/pytorch/pytorch/blob/5b03ff0a09d43d721067e39da10aa23edc6997cd/aten/src/ATen/native/Embedding.cpp#L14-L29" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Embedding.cpp at 5b03ff0a09d43d721067e39da10aa23edc6997cd · pytorch/pytorch · GitHub</a></p>
Dynamic quantize timm models

username: 111520
cooked: <p>I try to quantize my pretrained model from timm library. But it not work, so the question is why it not work and how to make timm models being quantized?</p>
<pre><code class="lang-auto">import timm 

model = timm.create_model('mobilenetv2_120d', pretrained=True)

model_int8 = torch.quantization.quantize_dynamic(
    model,  # the original model
    {torch.nn.Linear, torch.nn.Conv2d, torch.nn.ReLU, torch.nn.BatchNorm2d}, 
    dtype=torch.qint8) 

print_model_size(model)
print_model_size(model_int8)
</code></pre>
<pre><code class="lang-auto">23.74 MB
23.74 MB
</code></pre>

username: supriyar
cooked: <p>We currently only support dynamic quantization of Linear operations from the list you’ve specified. Can you print the quantized model to check how many layers were actually quantized?</p>

username: 111520
cooked: <p>Yes it truly work only for Linear.</p>
<pre><code class="lang-auto">(conv_head): Conv2d(384, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn2): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (act2): ReLU6(inplace=True)
    (global_pool): SelectAdaptivePool2d (pool_type=avg, flatten=True)
    (classifier): DynamicQuantizedLinear(in_features=1280, out_features=5, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
</code></pre>
<p>But my goal is to measure evaluation time of quantized model and compare this time with float32 model. For that i try to static quantization:</p>
<pre><code class="lang-auto">model_sigmoid.qconfig = torch.quantization.get_default_qconfig('qnnpack')

# insert observers
torch.quantization.prepare(model_sigmoid, inplace=True)
# Calibrate the model and collect statistics

# convert to quantized version
torch.quantization.convert(model_sigmoid, inplace=True)
</code></pre>
<p>This code quantize all the layer. But i cant run this quantized model,because of that:</p>
<pre><code class="lang-auto">start_time = time.time()

with torch.no_grad():
# with torch.autograd.set_detect_anomaly(True):
    pred = model_sigmoid(torch_img)

print('Time = ', time.time() - start_time)
</code></pre>
<pre><code class="lang-auto">RuntimeError: Could not run 'quantized::conv2d.new' with arguments from the 'CPU' backend. 'quantized::conv2d.new' is only available for these backends: [QuantizedCPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].
</code></pre>
<p>I understand that this backend not support cpu and cuda, so question is it possible to run this static quantized model on windows 10 (x64)? And it will be cool if you compare each backed in RuntimeError with device on which it can evaluete.</p>

username: supriyar
cooked: <p>For static quantization, in addition to using the qconfig you also need to add Quant/Dequant Stubs around the modules you want quantized.</p>
<p>The tutorial <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.8.1+cu102 documentation</a> has more details on how to do so.</p>
How to load part of state_dict to device?

username: ap579
cooked: <p>When I call model.to(‘cpu’) on a model that’s currently in gpu, does it do anything aside from move the individual tensors in the state_dict to cpu? I’m trying to move the tensors individually because I need a portion of the parameters to remain in gpu because they’re shared by another model that’s still running. I’ve tried the following to edit the state_dict and then load it back:</p>
<p>updated = model2.state_dict()<br>
for i in updated.keys():<br>
print(f’before: {updated[i]}’)<br>
updated[i] = updated[i].to(‘cpu’)<br>
print(f’after: {updated[i]}’)<br>
model2.load_state_dict(updated)<br>
print(model2.state_dict())</p>
<p>I can see that the tensors in “updated” are moving from cuda to cpu in the before and after calls, but when I load it back into model2, all elements of the state_dict still appear to be in cuda. How do I move them individually so that I can then filter some keys out?</p>
<p>Thanks!</p>
How to get back the words from nn.Embedding?

username: ADONAI_TZEVAOT
cooked: <p>I have a sentence x = ‘I love to play football’ , here I=1 , love = 2, to = 3, play = 4, football = 5;</p>
<blockquote>
<p>x = [1 2 3 4 5]<br>
self.vocab_size = 5<br>
self.enc_word_dim = 3<br>
self.enc_word_vecs = nn.Embedding(self.vocab_size, self.enc_word_dim)<br>
word_vecs = self.enc_word_vecs(x)</p>
</blockquote>
<p>How to get back ‘I love to play football’  using  <code>word_vecs</code>  ? Please help!</p>

username: pascal_notsawo
cooked: <p>A naive approach if you have an already predefined vocabulary (and of relatively small size), would be to store all the representations of the words of this vocabulary. Once you have a representation, you look for the word whose representation is closer (or equal, in the sense of a distance like the Euclidean distance) to that one.</p>
<p>For example if I have a function <code>get_word(model : nn.Module, word : str) -&gt; Tensor</code> that takes a word and returns its representation and <code>word_to_idx</code> the dictionary containing my words and their index in the vocabulary, the following function will do the trick (with Euclidean distance):</p>
<pre><code class="lang-auto">import torch
def closest(model, vec, word_to_idx, n = 10):
    """
    finds the closest words (n) for a given vector
    """
    all_dists = [(w, torch.dist(vec, get_word(model, w))) for w in word_to_ix]
    return sorted(all_dists, key=lambda t: t[1])[:n]
</code></pre>
<p>I’ve done this once with Glove, but it’s definitely not a good approach.</p>
Tensors with physical Quantities

username: Alex_Finkelshtein
cooked: <p>Is there a standard off-the-shelf way to introduce physical quantities to torch tensors while keeping them differential?<br>
I mostly mean a cosmetic wrapper that can let me know what is the quantity at hand throughout the computation graph. A bonus would be taking care of conversions and “translation” to other quantities when relevant.<br>
any help appreciated, thanks</p>
Some questions in parameters of LSTM layer in Pytorch

username: mhxin
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/746281e30b1000486159cfc1dfd15297aa626590.png" data-download-href="https://discuss.pytorch.org/uploads/default/746281e30b1000486159cfc1dfd15297aa626590" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/746281e30b1000486159cfc1dfd15297aa626590.png" alt="image" data-base62-sha1="gBAr8hX0GjVnqZlHqsQbSf16ppu" width="690" height="47" data-dominant-color="1B1818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1769×121 6.82 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In above figure, I Create a LSTM layer with input_size of 8, hidden_dim of 4 and num_layer of 1. when i print the parameters of this layer, I just find two weight and two bias. but generally, LSTM should have 4 weights and biases, and them have same size.</p>
<p>In addition, I try to calculate the parameters number  with following fomula.<br>
Param count = 4 * [(input_size + hidden_size) * hidden_size + hidden_size ] = 4 * [(8 + 4) * 4 + 4 ] = 208<br>
this result is not equal to 224.</p>
<p>So i just wonder why that happened, Is anything wrong in my above understading.</p>
<p>I will very grateful if you help me.</p>

username: ptrblck
cooked: <p>The parameters of the <code>nn.LSTM</code> module are explained in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM">docs</a>, where also the “packaging” of these parameters is explained.</p>

username: mhxin
cooked: <p>I got it, thank you so much.</p>
How to replace a model's head?

username: algol1
cooked: <p>I’m trying to implement the simCLR algorithm (<a href="https://arxiv.org/abs/2002.05709" class="inline-onebox" rel="noopener nofollow ugc">[2002.05709] A Simple Framework for Contrastive Learning of Visual Representations</a>) following some steps. One of them is: Create a resnet18 instance (using torchvision) as the base encoder, and replace the classifier head with a projection head. The projection head architecture is as follows: A fully connected layer with 100 units, a relu activation, another fully connected layer with 40 units, a relu activation and finally an output layer with 20 units (so the final features will have dimention=20).</p>
<p>But I got lost with how can I do that in pytorch and would appreciate help. Thanks.</p>
<p>My code:</p>
<pre><code class="lang-auto">def create_simclr_model():
  # create new resnet18 as the base encoder
  simclr_model = torchvision.models.resnet18(pretrained=False)
  # change its classifier to a projection head
  # I don't know what to do here.  
  return simclr_model
</code></pre>

username: ptrblck
cooked: <p>You can replace the last linear layer with an <code>nn.Sequential</code> module containing the described layers:</p>
<pre><code class="lang-python">model = torchvision.models.resnet18(pretrained=False)
model.fc = nn.Sequential(
    nn.Linear(512, 100),
    nn.ReLU(),
    nn.Linear(100, 40),
    nn.ReLU(),
    nn.Linear(40, 20))
out = model(torch.randn(2, 3, 224, 224))
print(out.shape)
&gt; torch.Size([2, 20])
</code></pre>

username: ariG23498
cooked: <p>Hey <a class="mention" href="/u/algol1">@algol1</a><br>
I did something like this to help distinguish between the <code>encoder</code> and the <code>projector</code>.</p>
<pre><code class="lang-python">resnet18 = models.resnet18(
    pretrained=False,
    num_classes=resnet_dim)
encoder = resnet18
projection = nn.Sequential(
    nn.Linear(resnet_dim, hidden_dim),
    nn.ReLU(),
    nn.Linear(hidden_dim, out_dim)
)
</code></pre>
<p>This will help you do the evaluation on the SimCLR representation properly.</p>
Best practice for slow customized nonlinearity

username: wasabi
cooked: <p>Hi guys. I have a customized nonlinearity f(x) that runs very slowly, and for each training/inference step I have to run f on a list of tensors (x_1,…,x_n). The good thing is x_1…x_n are disjoint so there is no tensor sharing complication and I think the best practice is to use multiprocessing here. From <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch 1.8.1 documentation</a> it seems that whenever the process_worker is called, a new process will be created, and I don’t think it suits my situation. Any suggestion for the best way to use multiprocess? Thanks.</p>
Using already calculated values in `autograd.functional.jacobian`

username: VRehnberg
cooked: <p>Say that I want to calculate the output of a function and its Jacobian. One way to do it would be to do the following</p>
<pre><code class="lang-auto">import torch
x = torch.rand(5, requires_grad=True)
net = torch.nn.Linear(5, 2)
y = net(x)

jac = torch.autograd.functional.jacobian(net, x)
</code></pre>
<p>however, this means that I’ll have to do two forward passes. It feels like there should be a way to do the following</p>
<pre><code class="lang-auto">jac = torch.autograd.functional.jacobian(lambda x: y, x)
</code></pre>
<p>but then <code>jac</code> is all zeros. Is there a way to do this?</p>
<p>The end goal is to use the Jacobian with respect to the network’s activations in a regularization term, but this is the step I’m bothered about.</p>

username: pascal_notsawo
cooked: <p>I already made <a href="https://discuss.pytorch.org/t/how-to-compute-jacobian-matrix-in-pytorch/14968/14">a post</a> about the jacobian computation with pytorch (as well as the gradient, divergence and laplacian).<br>
Have a look at it, it might be useful</p>

username: VRehnberg
cooked: <p>Thanks, your solution works fine as long as one makes sure that input and output is 1D. I got an order of magnitude speed-up in the Jacobian calculation with your way.</p>
<pre><code class="lang-auto">import torch


def gradient(y, x, grad_outputs=None):
    """Compute dy/dx @ grad_outputs"""
    if grad_outputs is None:
        grad_outputs = torch.ones_like(y)
    grad = torch.autograd.grad(y, [x], grad_outputs = grad_outputs, create_graph=True)[0]
    return grad


def jacobian(y, x):
    """Compute dy/dx = dy/dx @ grad_outputs; 
    for grad_outputs in [1, 0, ..., 0], [0, 1, 0, ..., 0], ...., [0, ..., 0, 1]"""
    jac = torch.zeros(y.shape[0], x.shape[0])
    grad_outputs = torch.zeros_like(y)
    for i in range(y.shape[0]):
        grad_outputs[i] = 1
        jac[i] = gradient(y, x, grad_outputs = grad_outputs)
        grad_outputs[i] = 0
    return jac


x = torch.rand(3, requires_grad=True)
net = torch.nn.Linear(3, 2)
y = net(x)

jac1 = torch.autograd.functional.jacobian(net, x, create_graph=True)
jac2 = jacobian(y, x)

print(torch.allclose(jac1, jac2)) # True
</code></pre>
The counterpart of "strict" flag for loading the optimizer state dict

username: Kamil_Adamczewski
cooked: <p>If we load a saved checkpoint to a model, but the parameters in the model do not exactly correspond to the checkpoint’s parameters, we can use a <code>strict=False</code> flag to allow for this behavior. However, in the case of optimizer I don’t see such a flag, and and loading the save optimizer <code>state_dict</code> results in an error. How can I load the optimizer state_dict when the parameters are different?</p>
<pre><code>model.load_state_dict(checkpoint['model'], strict=False)
optimizer.load_state_dict(checkpoint['optimizer'])

ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group</code></pre>

username: dlsf
cooked: <p>I have the same problem. How can we load the optimizer states when additional parameters are added to the models?<br>
There is an attempt to do this (<a href="https://github.com/pytorch/pytorch/issues/34660" class="inline-onebox" rel="noopener nofollow ugc">Add "strict" flag to ignore missing parameters in Optimizer.load_state_dict · Issue #34660 · pytorch/pytorch · GitHub</a>) , however it its still not implemented, anyone has some updates on this?</p>