akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is the faster-rcnn-resnet50 pretrained on COCO from scratch?

Louis_FOUQUET: <p>Hello,<br>
I’m using the fasterrcnn_resnet50_fpn and there are two parameters pretrained (full model pretrained on COCO) and pretrained_backbone (only backbone pretrained on ImageNet).<br>
So I was wondering for the COCO model if this model was also trained using the ImageNet backbone or was it completely from scratch ?</p>
Difference between torch.mean(torch.square(Tensor)) and Tensor.pow_(2).sum()/D

Siladittya_Manna: <p>I am using two version of my loss function</p>
<p>In version 1, I am using <code>torch.mean(torch.square(Tensor))</code> to calculate the loss</p>
<p>In version 2, I am using <code>Tensor.pow_(2).sum()/D</code> to clculate the loss</p>
<p>But the behavior of the loss in the two versions are completely different. The version 2 works fine, but using version 1 I get horrible results.</p>
<p>Both the experiments have the exactly same hyper-parameter configuration.</p>
<p>Does not version 1 and 2, represent the same thing?</p>

InnovArul: <aside class="quote no-group" data-username="Siladittya_Manna" data-post="1" data-topic="148789">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siladittya_manna/48/26984_2.png" class="avatar"> Siladittya_Manna:</div>
<blockquote>
<p>torch.mean(torch.square(Tensor))</p>
</blockquote>
</aside>
<p>Note that <code>torch.mean()</code> will divide the sum by the count of all the dimensions. i.e., if the tensor is 4D (B,D,H,W), the division factor is (B*D*H*W). If the tensor is 2D (B,D), the division factor is (B*D).<br>
I am not sure about the tensor dimension in your case. Check it maybe.</p>

Siladittya_Manna: <p>Yes. That was the issue. Thanks for pointing it out.</p>
[Mixed precision] nn.Parameter autocast problem

111114: <p>Hello I’m trying mixed precision training with pretrained model.</p>
<p>But when I load pretrained_model[1], nn.Parameter module[2] in WavLM(pretrained model) makes trouble.</p>
<pre><code class="lang-auto">### Error Message
  File "/data/leecho/xi-stt/xi-stt/model/WavLM.py", line 286, in apply_mask
    x[mask_indices] = self.mask_emb
RuntimeError: Index put requires the source and destination dtypes match, got Half for the destination and Float for the source.
</code></pre>
<p>Is there any way to fix this without breaking loaded data?</p>
<pre><code class="lang-auto">### trainer.py
model = ASRModel(...)

with autocast(enabled=config.fp16_run):
    predictions = model(inputs, apply_mask=True)

### ASRmodel.py
class ASRModel(nn.Module):
    def __init__(
        self, cfg: ASRConfig, ...) -&gt; None:
        super().__init__()
        
        self.cfg = cfg
        
        ### [1] load pretrained_model
        if load_from_pretrained_model:
            checkpoint = torch.load(cfg.pretrained_model_path)
            wavlm_cfg = WavLMConfig(checkpoint['cfg'])
            wavlm_cfg.feature_grad_mult = 0.
            self.acoustic_model = WavLM(wavlm_cfg)
            self.acoustic_model.load_state_dict(checkpoint['model'])
                    
            self.cfg.wavlm_cfg = wavlm_cfg


### WavLM.py, 
### [2] self.mask_emb
self.mask_emb = nn.Parameter(
    torch.FloatTensor(cfg.encoder_embed_dim).uniform_()
)

def apply_mask(self, x, padding_mask):
    B, T, C = x.shape
    ....
        mask_indices = torch.from_numpy(mask_indices).to(x.device)
        x[mask_indices] = self.mask_emb
    else:
        mask_indices = None

</code></pre>

ptrblck: <p>It seems you are trying to call <code>index_put_</code> with mixed dtypes, which creates the error. Cast e.g. the value tensor to the source dtype and it should work:</p>
<pre><code class="lang-python">lin = nn.Linear(1, 1).cuda()
x = torch.randn(1, 1).cuda()

with torch.cuda.amp.autocast():
    out = lin(x)
    
print(out.dtype)
# torch.float16

val = torch.randn(1, dtype=torch.float32, device='cuda')

out.index_put_((torch.tensor(0),), val)
# RuntimeError: Index put requires the source and destination dtypes match, got Half for the destination and Float for the source.

out.index_put_((torch.tensor(0),), val.to(out.dtype)) # works
</code></pre>

111114: <p>Thank you for your reply.</p>
<p>The problem solved by your suggestion.</p>
<p>Thank you.</p>
Cross entropy in batches

Kelvin_Ting: <pre><code class="lang-auto">loss = nn.CrossEntropyLoss()
input = torch.randn(3, 3, 5, requires_grad=True)
target = torch.empty(3, 3, dtype=torch.long).random_(5)
output = loss(input, target)
output.backward()

 Expected target size [3, 5], got [3, 3]
</code></pre>
<p>in this example, my batch size is 3 and each contains 3 categories each category contains a probability distribution of 5. so the shape is (3,3,5)</p>
<p>My target would be (batch size, categories index). In this example is (3,3)<br>
Unless I create a one-hot vector with each category’s index, converted it into (3,3,5). Otherwise, it won’t work but this is not a good approach.</p>

InnovArul: <aside class="quote no-group" data-username="Kelvin_Ting" data-post="1" data-topic="148788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kelvin_ting/48/45504_2.png" class="avatar"> Kelvin_Ting:</div>
<blockquote>
<pre><code class="lang-auto">loss = nn.CrossEntropyLoss()
input = torch.randn(3, 3, 5, requires_grad=True)
target = torch.empty(3, 3, dtype=torch.long).random_(5)
output = loss(input, target)
output.backward()
</code></pre>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="onebox" target="_blank" rel="noopener">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html</a><br>
From the documentation, you can see that the second dimension belongs to the number of classes (C).<br>
The below code works:</p>
<pre><code class="lang-auto">loss = nn.CrossEntropyLoss()
input = torch.randn(3, 5, 3, requires_grad=True)
target = torch.empty(3, 3, dtype=torch.long).random_(5)
output = loss(input, target)
output.backward()
</code></pre>
Pytorch tensor from loss function won't change data type after casting, which causes a bug

benkolber: <p>I am using the MSE loss function from pytorch nn module to calculate a loss during training as so:</p>
<pre><code class="lang-auto">        loss = criterion(prediction, truth)
        loss = loss.to(torch.double)

        optimizer.zero_grad()               
        loss.backward()                      
        optimizer.step()  
</code></pre>
<p>but for some reason I keep getting the error:</p>
<pre><code class="lang-auto">---&gt; 54             loss.backward()                      
     55             optimizer.step()                    
     56 
    RuntimeError: expected dtype Double but got dtype Long (validate_dtype at /Users/distiller/project/conda/conda-bld/pytorch_1587428061935/work/aten/src/ATen/native/TensorIterator.cpp:143)
</code></pre>
<p>Even though I cast my loss to a double, it still won’t work.</p>
<p>For further investigation purposes, when running it in a notebook the following occurs, which may contribute to the problem</p>
<pre><code class="lang-auto">print(loss)
--&gt; tensor(841645.2747, dtype=torch.float64, grad_fn=&lt;MseLossBackward&gt;)

l1 = loss.to(torch.double)
print(l1)
--&gt; tensor(841645.2747, dtype=torch.float64, grad_fn=&lt;MseLossBackward&gt;)

l2 = loss.double()
print(l2)
--&gt; tensor(841645.2747, dtype=torch.float64, grad_fn=&lt;MseLossBackward&gt;)
</code></pre>
<p>Any ideas?</p>

InnovArul: <aside class="quote no-group" data-username="benkolber" data-post="1" data-topic="148784">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/benkolber/48/32373_2.png" class="avatar"> benkolber:</div>
<blockquote>
<pre><code class="lang-auto">        loss = criterion(prediction, truth)
        loss = loss.to(torch.double)
</code></pre>
</blockquote>
</aside>
<p>Are <code>prediction, truth</code> variables holding <code>double</code> tensors?</p>

benkolber: <p>they weren’t. thank you so much!</p>
Is there any way to speed up the jit loading process of quantized model?

mkcho: <p>Hi there!</p>
<p>I am writing to ask if there is an efficient way to speed up the loading time of quantized model (using post training static int8 quantization).</p>
<p>As of now, when I broke down the loading process of full-precision(i.e, fp32) and int8 model, loading time on int8 model requires much longer time than that on fp32 (see below). I think this is due to quantized::conv2d_prepack() process, but I don’t know how to optimize or speed up.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2aa2a1eb8a967e8276a18d5f4f793d6f786dcce.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d2aa2a1eb8a967e8276a18d5f4f793d6f786dcce" title="qmodel"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2aa2a1eb8a967e8276a18d5f4f793d6f786dcce_2_690x384.jpeg" alt="qmodel" data-base62-sha1="u3CQmIRMY4YY1LoY9uhRxRVJgu2" width="690" height="384" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2aa2a1eb8a967e8276a18d5f4f793d6f786dcce_2_690x384.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2aa2a1eb8a967e8276a18d5f4f793d6f786dcce_2_1035x576.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2aa2a1eb8a967e8276a18d5f4f793d6f786dcce_2_1380x768.jpeg 2x" data-dominant-color="7F7D7C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">qmodel</span><span class="informations">2378×1326 334 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for reading my problem!<br>
MK</p>
Reinterpret casting funny business with BDD100k

5had3z: <p>BDD100k has its panoptic annotations in an RGBA packed format described <a href="https://doc.bdd100k.com/format.html#segmentation-formats" rel="noopener nofollow ugc">here</a>. I’m trying to cast this to a int32 so that I can use this to find unique labels for mask generation. However, when trying to bitwise extract the annotation data, I am getting bad category_ids (over the max category). If I check the raw “R” channel without loading, all the class ids seem to be valid, but when trying to extract from the upper byte of the int32, I get invalid classes. I would presume ideally I would cast to uint32, but there is no torch.uint32.</p>
<pre data-code-wrap="python3"><code class="lang-nohighlight">self.image_data = (
    torch.from_numpy(
        np.array(Image.open(data_path).convert("RGBA"), dtype=np.uint8)
    )
    .view(dtype=torch.int32)
    .permute(2, 0, 1)
)
...
for mask in self.image_data.unique():
    category_id = seg &gt;&gt; 24
    truncated = seg &amp; 2 ** 19
    occluded = seg &amp; 2 ** 18
    iscrowd = seg &amp; 2 ** 17
    ignore = seg &amp; 2 ** 16
    instance_id = seg &amp; 0xFFFF
</code></pre>

5had3z: <p>Good ol’ little endian results in the reversal of the byte ordering when unpacking from the bitcasting. This seems to be good for it now.</p>
<pre><code class="lang-auto">instance_id = (seg &gt;&gt; 16 &amp; 0xFF00) + seg &gt;&gt; 24
truncated = seg &amp; 2 ** 11
occluded = seg &amp; 2 ** 10
iscrowd = seg &amp; 2 ** 9
ignore = seg &amp; 2 ** 8
category_id = seg &amp; 0xFF
</code></pre>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (128x32 and 1056x120)

Jishnu_Saurav_Mittap: <p>Someone please help me out!</p>
<p>My model :<br>
class logCNN(nn.Module):</p>
<pre><code>def __init__(self, num_classes):
    super(logCNN, self).__init__()
    self.num_classes = num_classes
    self.features = nn.Sequential(
        
        nn.Conv2d(1, 16, kernel_size=2),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2),
        nn.Conv2d(16, 32, kernel_size=2),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2)
    )

    self.classifier = nn.Sequential(

        nn.Linear(1056, 120),
        nn.ReLU(),
        nn.Linear(120, 84),
        nn.ReLU(),
        nn.Linear(84, num_classes),
    )


def forward(self, x):
    x = self.features(x)
    x = torch.flatten(x, 1)
    logits = self.classifier(x)
    probas = F.softmax(logits, dim=1)
    return logits, probas
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86d81ce2ccec3e9f1054c45cfe44139f35feab90.png" data-download-href="https://discuss.pytorch.org/uploads/default/86d81ce2ccec3e9f1054c45cfe44139f35feab90" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86d81ce2ccec3e9f1054c45cfe44139f35feab90_2_690x195.png" alt="image" data-base62-sha1="jeT02wa1gBGpEmth3jbkOKLJUic" width="690" height="195" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86d81ce2ccec3e9f1054c45cfe44139f35feab90_2_690x195.png, https://discuss.pytorch.org/uploads/default/original/3X/8/6/86d81ce2ccec3e9f1054c45cfe44139f35feab90.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/6/86d81ce2ccec3e9f1054c45cfe44139f35feab90.png 2x" data-dominant-color="242625"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">847×240 37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on the error message the <code>in_features</code> of the first linear layer do not match the feature dimension of the flattened activation.<br>
Using:</p>
<pre><code class="lang-python">    self.classifier = nn.Sequential(
        nn.Linear(32, 120),
        ...
</code></pre>
<p>should work.</p>
How to negate the Bool type Tensor?

ElleryL: <pre><code class="lang-auto">x = torch.tensor([False], dtype=torch.bool)
~x # error
</code></pre>
<p>any simple way of doing negation ?</p>

InnovArul: <p>I’m not near a PC now to test. Does <code>not x</code> work?</p>

John_Deterious: <p>I have just tried this:</p>
<pre><code class="lang-auto">~
</code></pre>
<p>Works like a dream</p>
<p>However, my dtype is not bool, it is torch.uint8<br>
Which is the default output when you do logical operation on a tensor like x&gt;0</p>

John_Deterious: <p>He should change dtype to uint8 and tilde operator will work.</p>

drevicko: <p>Not sure which pytorch version you have, but in 1.4, ~x works fine on torch.bool tensors.</p>

aknirala: <p>~x does not work, however (x == False) works fine.</p>
Does nn.Flatten() count as a hidden layer?

Nurkhan_Laiyk: <p>Hi,<br>
I wanted to count the hidden layer of the model<br>
model = nn.Sequential(<br>
nn.Flatten(),<br>
nn.Linear(784,128),<br>
nn.ELU(),<br>
nn.Linear(128,128),<br>
nn.ELU(),<br>
nn.Linear(128,10))<br>
in this case the number of hidden layers 1 or 2?<br>
Does nn.Flatten() count as a layer?</p>

Matias_Vasquez: <p>Flatten will return a view of the tensor, which shares the actual data with the original tensor. Only how the data is viewed is changed.</p>
<p>So you are only “reshaping” it. I would <strong>not</strong> count it as a hidden layer since there are no weights to be trained.</p>

Nurkhan_Laiyk: <p>Thanks for your comment. In that case number of hidden layers should be 1?</p>

Matias_Vasquez: <p>Yes</p>
<p>1 change of view<br>
1 input<br>
1 hidden<br>
2 activations<br>
1 output</p>

InnovArul: <p>Typically the layers with learnable parameters are counted as layers.</p>
Regarding KL divergence in pytorch (vs Tensorflow)

AmoghRaut: <p>I was converting the following tensorflow code to pytorch,</p>
<pre><code class="lang-auto">import tensorflow.compat.v1 as tf

logit_true = tf.distributions.Categorical(probs=logit_true)
logit_aug = tf.distributions.Categorical(probs=logit_aug)

distillation_loss = tf.distributions.kl_divergence(logit_true,logit_aug,allow_nan_stats= False)
</code></pre>
<p>My pytorch implementation.</p>
<pre><code class="lang-auto">logit_true = torch.distributions.categorical.Categorical(probs=logit_true)
logit_aug = torch.distributions.categorical.Categorical(probs=logit_aug)

distillation_loss = torch.distributions.kl.kl_divergence(logit_true,logit_aug)
</code></pre>
<p>However, the model with tensorflow runs fine, but the one with my pytorch implementation somehow messes up . I wanted to know if there was any difference in both of the kl divergence implementations.</p>
<p>A little context: It’s a part of distillation loss</p>
<p>Pytorch vs Tensorflow:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c8a3d460170e79cac89c0e6ec5c21693a29d211.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6c8a3d460170e79cac89c0e6ec5c21693a29d211" title="joint"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c8a3d460170e79cac89c0e6ec5c21693a29d211_2_517x186.jpeg" alt="joint" data-base62-sha1="fubK52ZcffST41rPPlVRnzE5dvz" width="517" height="186" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c8a3d460170e79cac89c0e6ec5c21693a29d211_2_517x186.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c8a3d460170e79cac89c0e6ec5c21693a29d211_2_775x279.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c8a3d460170e79cac89c0e6ec5c21693a29d211_2_1034x372.jpeg 2x" data-dominant-color="393736"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">joint</span><span class="informations">1277×461 93.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Apply filters on gradients of feature maps

kbtorcher: <p>My gradients of feature map is of the size: <code>torch.Size([1, 64, 128, 128])</code>. My filters for the corresponding convolutional layer is: <code>torch.Size([64, 3, 5, 5])</code>. How do I multiply these two tensors to get the input: <code>torch.Size([1, 3, 128, 128])</code>?</p>

InnovArul: <p>An example implementation of 2D conv layer using <code>F.unfold()</code> is here: <a href="https://github.com/InnovArul/codesmart/blob/master/computer_vision/pytorch/discuss_pytorch/custom_conv.py" class="inline-onebox">codesmart/custom_conv.py at master · InnovArul/codesmart · GitHub</a></p>
<p>To answer your question, to get <code>grad_input</code>, try following these lines:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/InnovArul/codesmart/blob/8c1b61b7cec62d966a251eab36d5e61c3fcfba9c/computer_vision/pytorch/discuss_pytorch/custom_conv.py#L53-L58">
  <header class="source">

      <a href="https://github.com/InnovArul/codesmart/blob/8c1b61b7cec62d966a251eab36d5e61c3fcfba9c/computer_vision/pytorch/discuss_pytorch/custom_conv.py#L53-L58" target="_blank" rel="noopener">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/InnovArul/codesmart/blob/8c1b61b7cec62d966a251eab36d5e61c3fcfba9c/computer_vision/pytorch/discuss_pytorch/custom_conv.py#L53-L58" target="_blank" rel="noopener">InnovArul/codesmart/blob/8c1b61b7cec62d966a251eab36d5e61c3fcfba9c/computer_vision/pytorch/discuss_pytorch/custom_conv.py#L53-L58</a></h4>



    <pre class="onebox">      <code class="lang-py">
        <ol class="start lines" start="53" style="counter-reset: li-counter 52 ;">
            <li># grad_input depends on weights and grad_output</li>
            <li># unfold grad output for grad input calculation</li>
            <li>unfold_gradout = F.unfold(grad_output, kernel_size=kernel_size, dilation=1, padding=kernel_size//2, stride=1)</li>
            <li>unfold_gradout = rearrange(unfold_gradout, 'b (out_channels kh kw) (h w) -&gt; b out_channels kh kw h w', out_channels=out_channels, kh=kernel_size, kw=kernel_size, h=h, w=w)</li>
            <li>weights_for_gradinput = weights.flip(dims=[2,3])</li>
            <li>grad_input = torch.einsum('oipq,bopqhw-&gt;bihw', weights_for_gradinput, unfold_gradout) # p, q = kernel sizes | h,w = input &amp; output height, width</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Non Differentiable Function

kk1153: <p>Hi all,</p>
<p>Can someone please tell me how pytorch deals with non differentiable functions like torch.abs and L1 loss function?</p>

gaosh: <p>For most non-differential function including L1, I think it uses sub-gradients of the function. Also, You can always use proximal gradient  to calculate non-differential function, if you want to use some custom optimization method on the non-differential function</p>

OzymandiaSt: <p>Will the sub-gradient be applied implicitly?</p>

Adhithyan_Sakthivelu: <p>When we try to do autodiff of torch.min(x,abs(x)) to calculate the subgradient, it calculates as 0.5 which is not true. Why is this the case?</p>

AlphaBetaGamma96: <p>Can you share a minimal reproducible example of this incorrect behavior?</p>
Remove transform layer in Faster RCNN (torchvision, onnx)

bulatnv: <p>Hello to everyone,</p>
<p>This problem occurred after converting model to onnx for deploy.</p>
<p>Onnxruntime (for C#) can’t properly work with transform layer (There is a very big question why torchvision exports image/batch transformation as a layer?) Moreover because of this transformation, I have couldn’t convert model further to tensorRT.</p>
<p>As a solution, I would like to remove transform layer from Faster Rcnn torchvision model. Is there any solution to do that without constructing model from scratch?</p>
<p>Kind regards,</p>
<p>Bulat.</p>

miguelcsilva: <p>Did you find an answer for this? Currently facing the same issue</p>
Cifar10 dataset divide

vojkan7: <p>Hi everyone</p>
<pre><code class="lang-auto">
classes = ('plane', 'car', 'bird', 'cat',

           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,

                                        download=True, transform=transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=128,

                                          shuffle=True, num_workers=2)

</code></pre>
<p>Here in the trainloader are all 10 classes. And for example, I have 5 clients and I want each client to become only two classes (client 1 becomes classes 1 and 2, client 2 becomes classes 3 and 4, etc.)</p>
<p>Any ideas how to do that?</p>
<p>Or is there possibility to select classes and data from trainset to make trainloader with only two specific classes??</p>

Matias_Vasquez: <p>You can use the indices where the target meets your criteria to create a <code>Subset</code> for each client.</p>
<pre><code class="lang-python">classes = ('plane', 'car', 'bird', 'cat',

           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,

                                        download=True, transform=transform)

client_1_idx = np.where((np.array(trainset.targets) == 0) | (np.array(trainset.targets) == 1))[0]
client_2_idx = np.where((np.array(trainset.targets) == 2) | (np.array(trainset.targets) == 3))[0]
client_3_idx = np.where((np.array(trainset.targets) == 4) | (np.array(trainset.targets) == 5))[0]
client_4_idx = np.where((np.array(trainset.targets) == 6) | (np.array(trainset.targets) == 7))[0]
client_5_idx = np.where((np.array(trainset.targets) == 8) | (np.array(trainset.targets) == 9))[0]


client_1_ds = torch.utils.data.Subset(trainset, client_1_idx)
client_2_ds = torch.utils.data.Subset(trainset, client_2_idx)
client_3_ds = torch.utils.data.Subset(trainset, client_3_idx)
client_4_ds = torch.utils.data.Subset(trainset, client_4_idx)
client_5_ds = torch.utils.data.Subset(trainset, client_5_idx)

client_1_dl = torch.utils.data.DataLoader(client_1_ds, batch_size=128, shuffle=True, num_workers=2)
client_2_dl = torch.utils.data.DataLoader(client_2_ds, batch_size=128, shuffle=True, num_workers=2)
client_3_dl = torch.utils.data.DataLoader(client_3_ds, batch_size=128, shuffle=True, num_workers=2)
client_4_dl = torch.utils.data.DataLoader(client_4_ds, batch_size=128, shuffle=True, num_workers=2)
client_5_dl = torch.utils.data.DataLoader(client_5_ds, batch_size=128, shuffle=True, num_workers=2)
</code></pre>
Full libtorch build for Android

SamL: <p>Hi,</p>
<p>I would like to train (or finetune) a model on Android using the c++ api. Has anyone succeeded in building libtorch for Android ?</p>
<p>Thanks in advance !</p>

dbsx: <p>I got similar request.  In order to use the c++ api directly on android, do we have to build it manually?<br>
Here, I did refer to some tutorial like <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwju_4qDkon3AhXD_WEKHeVTCbgQFnoECBIQAQ&amp;url=https%3A%2F%2Fpytorch.org%2Fmobile&amp;usg=AOvVaw1KB33vZioaiNgE2eqnNHH4" rel="noopener nofollow ugc">this</a>, but it seems to have to use python version.</p>
<p>Update:<br>
I found <a href="https://github.com/ljk53/pytorch-android-cpp-demo" rel="noopener nofollow ugc">one</a> which may be helpful.</p>
Pytorch error loading .dll file

Kevinzhu: <p>Hi:</p>
<p>I have a problem when running pytorch cuda version. I got an error message as below:</p>
<pre><code class="lang-auto">OSError: [WinError 127] The specified procedure could not be found. Error loading
"C:\Users\adm\anaconda3\envs\mytorch\lib\site-packages\torch\lib\shm.dll" or one of its
dependencies.
</code></pre>
<p>Can anyone understand this error and let me know how to solve it?</p>
<p>Kevin</p>
Working with tensors larger than GPU memory

Art-MC: <p>Hello, I am developing a program that will use auto-differentiation to iteratively reconstruct a multi-dimensional input tensor. The issue I am facing is that the forward model has a step that requires a very large tensor when fully vectorized. Giving a simplified example: the input tensor T1 has dimensions (dz, dy, dx), and it is reshaped to dimensions (dz<em>dy</em>dx, 1, 1). A second tensor T2 has shape (1, dy, dx), and I need to perform the following operation:</p>
<blockquote>
<p>out = torch.exp( T1 * T2 * -1j).sum(dim=0)</p>
</blockquote>
<p>So my output is only of dimensions (dy, dx), but because a function must be performed on every input voxel the maximum tensor size in the middle is (dz<em>dy</em>dx, dy, dx). This very quickly becomes larger than my available GPU memory, and I therefore have two questions.</p>
<p>First, is there a clever way of avoiding the large tensor that I’m not seeing? Due to the exponential after multiplying but before summing it’s not as easy as a matmul, but the ideal solution would be to find a vectorized way of performing this calculation that doesn’t scale terribly.</p>
<p>Second, assuming I am stuck with the large tensor, what is the fastest way of performing this calculation? At one extreme I could effectively loop over each of the input voxels, summing the output tensor as I go. This is of course very slow but requires no additional memory. I’m currently slicing over the reshaped input tensor and performing the vectorized calculation one chunk at a time, but again this is slow and it feels like there should be a better way.</p>
<p>Any help is appreciated, thanks!</p>
Vectorizing layer usage in pytorch

TheTorchPea: <p>I have a batch of sentences of the shape: (batch_size, seq_length, emb_size), (batch_size, seq_length, domain_id). My model has separate linear layers for every domain_id. I wish to choose different linear layers according to the domain_id.</p>
<p>For now, my code uses a for loop to do so:</p>
<p>output = [ ]<br>
for id, sentence in zip(domain_ids, batch):<br>
output.append(self.domain_layers[ id ](sentence))</p>
<p>output = torch.stack(output,dim=0)</p>
<p>But this seems to slow down the training a lot. Is there a way to vectorize this part? I tried putting all domain_layers into a nn.ModuleList but nn.ModuleList accepts a single integer for indexing.</p>
<p>Any help is greatly appreciated. Thanks!</p>
How to use BatchSampler to choose from different indexes in different batches

Nimrod_Daniel: <p>Hi,</p>
<p>I have a custom dataset, that is pretty straightforward (path, image, label). For each label, I know it  relevant indexes. For the training phase, I need each batch to sample from one label only (my architecture supports that). For example - dusring the first 3 batches sample only from ‘label1’, during the 4th to 6th batch sample from ‘label2’ only.<br>
It looks like BatchSampler, that wraps another sampler to yield mini-batches, is the way to go, but I’m not sure how it could support that in order to create one dataloder.</p>
<p>Assuming I have a custom dataset - “train_dataset”, and I know the indexes of all the targets, such as:<br>
indx_dict = {‘label1’: [20,21,22,23,24,25],<br>
‘label2’ : [25,26,27,28,29, 30]}</p>
<p>How can I create one dataloader that samples twice for example from “label1” indexes (bs=3), then samples twice from “label2” indexes (bs=3)?<br>
Preferably do that with replacement.<br>
The solution should support more than only 2 labels, that’s just a short example</p>
Unhandle exception when using "forward"

artzers: <p>My environment:<br>
cuda 11.3, pytorch1.11, libtorch-win-shared-with-deps-Debug-1.11.0+cu113, win10, vs 2017, qt 5.12.</p>
<p>My network: An 3D image processing network.</p>
<p>The PT file is export by Python code:<br>
model.load_state_dict(torch.load(’./save_models/mynet.pth’,map_location=‘cpu’))<br>
model.eval()<br>
example = torch.rand(1, 1,16,32,32)<br>
traced_script_module = torch.jit.trace(model, example)<br>
traced_script_module.save("./model_CPP.pt")</p>
<p>My C++ code:<br>
device_type = at::kCPU;<br>
try {<br>
module=<br>
torch::jit::load(“model_CPP.pt”, device_type);<br>
module.eval();<br>
}<br>
catch (const c10::Error&amp; e) {<br>
QMessageBox::warning(this, “warning”, “error loading the model.”);<br>
std::cerr &lt;&lt; “error loading the model\n”;<br>
return false;<br>
}<br>
module.to(device_type);<br>
torch::Tensor img_tensor1 = torch::rand({ 1,16,32,32 });<br>
img_tensor1 = img_tensor1.to(torch::kFloat32).to(device_type);</p>
<pre><code>std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(img_tensor1);

torch::NoGradGuard no_grad;
at::Tensor output;
try {
	auto res = module.forward(inputs);
	output = res.toTensor();
}
catch (const c10::Error&amp; e) {
	std::cout &lt;&lt; e.what() &lt;&lt; std::endl;
}
</code></pre>
<p>The error information when executing module.forward():<br>
Unhandled exception at 0x00007FFBAA574F69 in Test.exe: Microsoft C++ exception: std::runtime_error at memory location 0x000000B6DFCF29D8.</p>
<p>Could you help me solve this problem?</p>
CrossEntropyLoss applied on a batch

tom.urkin: <p>Hello all,<br>
I am trying to understand how the nn.CrossEntropyLoss() applied on a batch behaves.<br>
Assuming I am performing a binary classification operation and the batch size is B - so the output of my CNN is of dimensions BX2.</p>
<p>I calculate the loss by the following:<br>
loss=criterion(y,st) where y is the model’s output and st is the correct labels (0 or 1) and y is of dimensions BX2.</p>
<p>From my understanding for each entry in the batch it computes softmax and the calculates the loss.<br>
Then it sums all of these loss values and divides the result by the batch size.</p>
<p>So the dimensions of ‘loss’ in this case is of a 1X1 tensor.</p>
<p>Did I get this right?</p>

ksmdanl: <p><a class="mention" href="/u/tom.urkin">@tom.urkin</a> that sounds about right to me. If you don’t specify the parameter “reduction”, you will get averaged loss over batch size.</p>

tom.urkin: <p><a class="mention" href="/u/ksmdanl">@ksmdanl</a> thank you very much for your response.</p>
<p>Is it the same for all other PyTorch loss functions? Meaning that if the loss of a batch is calculated the result is by default the mean loss at that given iteration (batch)?</p>
<p>Thanks in advance!</p>

ksmdanl: <p><a class="mention" href="/u/tom.urkin">@tom.urkin</a> yes as far as I know that should be the case:)</p>

mMagmer: <aside class="quote no-group" data-username="tom.urkin" data-post="1" data-topic="148262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/aeb1de/48.png" class="avatar"> tom.urkin:</div>
<blockquote>
<p>So the dimensions of ‘loss’ in this case is of a 1X1 tensor.</p>
</blockquote>
</aside>
<p>just for clarification,<br>
it retruns scalar tensor not 1x1 tensor if you use reduction.</p>
Changing Model Architecture after DDP

Chung-Hao_Ku: <p>Hi, I would like to know if there is a way to modify the model architecture during training, say for example, the last fc layer of a network, after the model was wrapped by DDP. The reason for this is because my training loop works as starting off with some number of warm up iterations and then after these iterations I’d have to change the last fc layer with a different output size. The difficulty lies in the way of trying to reference the last fc layer, since after DDP it became a DDP object. I wonder if there is a tweak for this? Thanks.</p>

Chung-Hao_Ku: <p>found my solution here <a href="https://discuss.pytorch.org/t/access-to-attributes-of-model-wrapped-in-ddp/130572" class="inline-onebox">Access to attributes of model wrapped in DDP</a></p>

pritamdamania87: <p><a class="mention" href="/u/chung-hao_ku">@Chung-Hao_Ku</a> There might be unintended consequences of modifying the model under DDP. The reason being that DDP builds buckets and state based on the model passed in. Now if the model changes (ex: parameters added/removed), this might break DDP where it might no longer synchronize gradients for the new parameters.</p>
<p>My suggestion would be to create a new DDP instance with the new model and then continue training.</p>

Chung-Hao_Ku: <p>Yes. I realised later on it’s not common practice to do this and instead found an alternative. Thanks for your detailed explanation.</p>
What's the difference between batched matrix multiplication and multiplying each matrix in batch respectively?

JohnsonLC: <p>As the tile says, I want to know what the difference between batched matrix multiplication and multiplying each matrix in batch respectively.</p>
<p>In the intermediate step of my network, I get a tensor <strong>x</strong> with shape <strong>[B, N, C]</strong> and a tensor <strong>y</strong> with shape <strong>[B, C, N]</strong>.</p>
<p>The first way:</p>
<pre><code class="lang-auto">masked_x = x[mask].view(B, -1, C)
f = torch.matmul(masked_x, y)
</code></pre>
<p>The second way:</p>
<pre><code class="lang-auto">masked_x = [x[i][mask[i]] for i in range(B)]
f = [torch.matmul(masked_x[i], y[i]) for i in range(B)]
</code></pre>
<p><strong>mask</strong> has a shape [B, N] with bool values. Each <strong>mask[i]</strong> has the same number of True values.</p>
<p>Then, when I trained the network, the first way took up more GPU memorys than the second way during loss.backward(). What’s the reason?</p>

AlphaBetaGamma96: <p>The first method is doing all matmuls at once whereas the second method is doing 1 at a time via list comprehension. The second method will be slower than the 1st method but it’ll require less memory.</p>
<p>Make sure your mask has <code>requires_grad</code> set to <code>False</code> that’ll free up some additional memory if you’re trying to minimize the memory footprint.</p>

JohnsonLC: <p>Thanks for reply! So the second method is just using time for space, and two methods will get exactly the same behavior?</p>

AlphaBetaGamma96: <p>They’ll give the same behaviour but you should double check and compare the two Tenors via <code>torch.allclose</code></p>

JohnsonLC: <p>I’ll check that. Thanks a lot for your answers!</p>
Where is the object file of PyTroch located when building from the source?

mkcho: <p>Hi there!</p>
<p>I downloaded the source of PyTorch and built the code on my own, but I can’t find the location of resulting object file. I really wanted to know the location of object file of PyTorch!</p>
<p>Thanks!</p>

ptrblck: <p>The location depends on the actual build command (as you can install it “locally” or into an environment etc.) and you can find it via <code>print(torch.__path__)</code>.</p>
RNN time complexity issue

Nathan_Goedeke: <p>Hello,</p>
<p>I’m trying to train my RNN on a dataset of 540 MB of text with a validation set of 26 MB of text. I run into a complexity issue by how long this is going to take to fully train. For each individual epoch, the number of forward passes needed to traverse the entire dataset would be:</p>
<p>26 MB x (540 MB / 10) + 540 MB = 1.4e15</p>
<p>When I try using a batch size of 128 and sequence length of 100, then each forward pass seems to last about one second. So the total training time would be:</p>
<p>1.4e15 / (128 x 100) / (3600 x 24 x 365) = 3,478 years</p>
<p>So it seems that training this way is not very practical. I can make the forward pass faster by decreasing the batch size, but that makes each batch cover less ground, so the net result seems to be worse instead of better.</p>
<p>Is there some way for PyTorch to speed up training with multithreading? I have an i7 CPU but I’ve never had to make use of it before.</p>
<p>My architecture is two layers of LSTM, size 512 each.</p>
<p>Thanks,<br>
Nathan</p>
nn.Sequential giving error while nn.Linear is not

mba: <p>I am kind of newbie to pytorch. I am trying to change the head of transformer which was:</p>
<p>lm_head = nn.Linear(768, 32128)</p>
<p>For this part, the model runs without error, However when I tried to add another fully connected layer using the Sequential function like:</p>
<p>lm_head = nn.Sequential(<br>
nn.Linear(768, 32128),<br>
nn.ReLU,<br>
nn.Linear(768, 32128)<br>
)</p>
<p>I am using a batch size of 32, and I am getting the following error:</p>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x32128 and 768x32128)</p>
<p>shouldnt both of these format give me the same output? I am confused!</p>

ptrblck: <p>As the error message explains, the <code>out_features</code> of the first linear layer in the <code>nn.Sequential</code> block do not match the <code>in_features</code> of the second linear layer.<br>
A fix would be:</p>
<pre><code class="lang-python">lm_head = nn.Sequential(
    nn.Linear(768, 32128),
    nn.ReLU,
    nn.Linear(32128, 32128)
)
</code></pre>
Finding synchronization points

tjoseph: <p>Hi,<br>
is there a way (e.g. a ContextManager) to find cpu-gpu synchronization points in my code?#</p>
<p>Best,<br>
Tim</p>

ptrblck: <p>You could use <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_sync_debug_mode.html#torch-cuda-set-sync-debug-mode"><code>torch.cuda.set_sync_debug_mode()</code></a>.</p>
How to do scaled L2 distance calculation in pytorch?

MUnique: <p>Hello,</p>
<p>I wanted to do traceable scaled L2 distance calculation in Pytorch: Something like this:<br>
A_ik = G_k (X_i - Y_k)^2, where X is in format of (B * N * D), Y in format of (B * K * D), and G in format of (B * K) so the output A is in format of (B * N * K). Does anyone know that how I can do that? Thanks in advance</p>

mxahan: <p>Did you check pytorch <a href="https://pytorch.org/docs/stable/generated/torch.einsum.html" rel="noopener nofollow ugc">einsum</a>?</p>
Decompose NxN kernel into 1xN & Nx1 kernels

Muhammad_Maaz: <p>Hi,</p>
<p>I am trying to decompose the NxN kernel in my custom ConvNet into two 1xN &amp; Nx1 kernels. The inference speed of the model is slightly reduced as compared to the model with the NxN kernel, however, the training speed decreased a lot (i.e. reduced almost by 2 - 2.5x). Is there any way that I can train the model with 1xN &amp; Nx1 Conv kernels faster in PyTorch? Currently, I am using 1.10.0+cu113 version of PyTorch.</p>
<p>Any guidance/help would be really appreciated. Thank you</p>

KFrank: <p>Hi Muhammad!</p>
<aside class="quote no-group" data-username="Muhammad_Maaz" data-post="1" data-topic="148615" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muhammad_maaz/48/23987_2.png" class="avatar"> Muhammad_Maaz:</div>
<blockquote>
<p>I am trying to decompose the NxN kernel in my custom ConvNet into two 1xN &amp; Nx1 kernels.</p>
</blockquote>
</aside>
<p>Could you show us explicitly – with as little code as possible – how you<br>
intend to structure this decomposition?</p>
<p>Concretely, let’s say that <code>N = 7</code> and we have:</p>
<pre data-code-wrap="python"><code class="lang-python">conv = torch.nn.Conv2d (1, 1, 7, bias = False)
</code></pre>
<p>(with <code>bias</code> turned off for simplicity).</p>
<p><code>conv</code> is basically a <code>7 x 7</code> kernel.  What, concretely, does your scheme<br>
replace it with?</p>
<p>Best.</p>
<p>K. Frank</p>

Andrei_Cristea: <p>I wonder if OP is trying to reduce the Convolution to its first principal component by breaking it up into its nearest representation that factors into a product of N-by-1 and 1-by-N matrices. If that is the case, I have no idea if one should expect to get any speed-up from such a “simplification”.</p>

Muhammad_Maaz: <p>Hi, as pointed out by <a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a>,</p>
<p>I am trying to decompose the 7x7 convolution into 7x1 &amp; 1x7 convolutions.</p>
Dataloader with num_workers = 4 slower than default?

jayz: <p>Hello,</p>
<p>I am using a dataloader with a custom dataset and batch sampler. I just tried setting num_workers=4 to see whether it speeds up, but it dramatically slows down. My system is Windows and I am running from a jupyter notebook.</p>
<p>In my dataloader, I use a collate_function that creates tensors like:<br>
<code>torch.tensor(..., device='cuda')</code>. Could this be the reason? That there is excessive copying between cuda and cpu or sth like that?</p>
<p>Any ideas?</p>
<p>Best, JZ</p>
Embedding weight renormalization is inconsistent on CUDA?

Sam-gege: <p>Hi, when I run the following code in a jupyter notebook cell multiple times, the result may be different each time. This happens only when it runs on CUDA.<br>
I’ve recorded a short video of it: <a href="https://www.youtube.com/watch?v=K8wvAFPXmmA" rel="noopener nofollow ugc">here</a></p>
<pre><code class="lang-python">import torch
import numpy as np

np.random.seed(0)

x_np = np.random.randint(0, 5, (30, 2))
weight_np = np.random.randn(5, 8).astype(np.float32)

m = torch.nn.Embedding(5, 8, max_norm=1)
m.weight.data = torch.tensor(weight_np)
x = torch.tensor(x_np)

m = m.cuda()
x = x.cuda()

y = m(x)

print(m.weight[0, 0])
</code></pre>
<p>Do you think I did something wrong? or is this a bug? Thanks for any help!</p>
How to custom the fasterrcnn_resnet50_fpn with the pre-trained model?

AllenChen: <pre><code class="lang-auto">def fasterrcnn_resnet50_fpn(pretrained=False, progress=True,
                            num_classes=91, pretrained_backbone=True, trainable_backbone_layers=3, **kwargs):
    assert trainable_backbone_layers &lt;= 5 and trainable_backbone_layers &gt;= 0
    # dont freeze any layers if pretrained model or backbone is not used
    if not (pretrained or pretrained_backbone):
        trainable_backbone_layers = 5
    if pretrained:
        # no need to download the backbone if pretrained is set
        pretrained_backbone = False
    backbone = resnet_fpn_backbone('resnet50', pretrained_backbone, trainable_layers=trainable_backbone_layers)
    model = FasterRCNN(backbone, num_classes, **kwargs)
    if pretrained:
        ### update faster rcnn pretrained model 
        pretrained_dict = load_state_dict_from_url(model_urls['fasterrcnn_resnet50_fpn_coco'],
                                              progress=progress)
        model_dict = model.state_dict() 
        pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}

        # update &amp; load
        model_dict.update(pretrained_dict) 
        model.load_state_dict(model_dict)
    return model
</code></pre>
<pre><code class="lang-auto">model = torchvision.models.detection.fasterrcnn_resnet50_fpn(
        pretrained=False, image_mean=image_mean, image_std=image_std)
model.backbone.body.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2,  padding=3, bias=False)
</code></pre>
<p>I’ve custom the resetnet to 4 channels and would like to load the pre-trained dict into my custom model.  But the load_dict method seems not working. Is there any advice for this? thanks</p>
[FSDP] Is there Any way to unwrap FSDP when I want to save module's state?

111114: <p>Hello, I’m trying to train my asr model with FullyShardedDataParallel.<br>
I directly saved FSDP wrapped model like below.</p>
<pre><code class="lang-auto">from torch.distributed.fsdp import FullyShardedDataParallel as FSDP

torch.distributed.barrier()
model_state = model.state_dict() # model = FSDP(mymodel())
if rank==0:
  torch.save({'model':model_state},  model_path)
</code></pre>
<p>But when I load state my model,  It return’s flattened parameters.</p>
<pre><code class="lang-auto">model = mymodel()
checkpoint = torch.load(model_path, map_location=torch.device('cpu'))
model.load_state_dict(checkpoint['model'])

Unexpected key(s) in state_dict: "_fsdp_wrapped_module.flat_param"
</code></pre>
<p>And when I wrap model by FSDP, and try loading state, It seems only rank-0’s params have been saved.</p>
<pre><code class="lang-auto">Error(s) in loading state_dict for FullyShardedDataParallel:
	size mismatch for _fsdp_wrapped_module.flat_param: copying a param with shape torch.Size([2430345]) from checkpoint, the shape in current model is torch.Size([14582066]).
</code></pre>
<p>Is there any way to save unwrapped model state?</p>
<p>Thank you.</p>

kumpera: <p>FSDP keeps the model sharded outside of forward/backwards so that’s why you’re seeing that.</p>
<p>In order to both see the unwrapped and unsharded model state you can use the <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/fsdp/fully_sharded_data_parallel.py#L1435" rel="noopener nofollow ugc"><code>summon_full_params</code> context manager</a>. Unfortunately this method is not available on PyTorch 11.1 and you’ll need to use nightly builds.</p>
<p><a class="mention" href="/u/rohan_varma">@Rohan_Varma</a> you have more visibility on FSDP checkpointing, could you help on that point?</p>

111114: <p>Thank you for your reply.</p>
<p>I installed Nightly build and try to test like below.</p>
<pre><code class="lang-auto">if rank == 0:
    model.clip_grad_norm_(1)
    model_state = model.summon_full_params()
    
    torch.save(
            {
                'model' : model_state,
            },
            model_path
        )
</code></pre>

kumpera: <p><code>summon_full_params</code> is a context manager and should be used in a <code>with</code> statement.<br>
Beyond that, you have to pass the unwrapped module to torch.save to skip unserializable objects like ProcessGroup.</p>
<pre><code class="lang-auto">if rank == 0:
  with model.summon_full_params():
    torch.save( 
        {
            'model' : model.module,
         },
         model_path
     )
</code></pre>
<p>As you’re using nightly you can now use new <code>state_dict</code> APIs that were designed for loading/saving: <a href="https://pytorch.org/docs/master/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel.state_dict" class="inline-onebox" rel="noopener nofollow ugc">FullyShardedDataParallel — PyTorch 1.12 documentation</a></p>