akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
torch.nn.Linear BUG

chandler-bing: <p>Hi, everyone, I encountered a very simple but weird BUG!</p>
<pre><code class="lang-auto">import torch

m = torch.load(f'./gate_linear.pt',map_location='cpu')
print(m)
#Linear(in_features=1024, out_features=2816, bias=False)

x1 = torch.load(f'./x1.pt',map_location='cpu')
x2 = torch.load(f'./x2.pt',map_location='cpu')
print(x1.size(),x2.size())
#torch.Size([1, 128, 1024]) torch.Size([1, 129, 1024])

print(torch.equal(x1,x2[:,:-1,:]))
#True

o1 = m(x1)
o2 = m(x2)

print(torch.equal(
    o1,
    o2[:,:-1,:]
    ))

#False

</code></pre>
<p>the code is simple enough, as the<code> True</code> result of<code> torch.equal</code> suggest, input <code>x1, x2[:,:-1,:]</code> should be same.<br>
why it become different after a simple Linear model?<br>
I upload my pt files (about 4M size) to reproduce the result, <a href="https://drive.google.com/file/d/1E1ShH3j6QrgS8wRRqopz8pLQST-0qZ_T/view?usp=sharing" rel="noopener nofollow ugc">google drive</a><br>
any thoughts? thank you guys</p>

chandler-bing: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  could u help me with this? thank u <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

chandler-bing: <p>minimal reproducible code …</p>
<pre><code class="lang-auto">import torch
torch.set_printoptions(precision=8)
x = torch.tensor([[[ 0.0451, -0.8093],
                   [-0.3275, -0.5304]]]
                 )
x1 = x[:,:-1,:]
x2 = x
w = torch.tensor(   [[ 0.29,  0.05],
                     [-0.11, -0.29],
                     [ 0.05,  0.29],
                     [-0.03, -0.05]], requires_grad=True)
w = w.transpose(0,1)

print(x.size(),x.dtype)
print(w.size(),w.dtype)
#torch.Size([1, 2, 2]) torch.float32
#torch.Size([2, 4]) torch.float32

o1 = torch.matmul(x1,w)
o2 = torch.matmul(x2,w)

print(o1)
print(o2)
# tensor([[[-0.02738600,  0.22973600, -0.23244201,  0.03911200],
#          [-0.12149499,  0.18984099, -0.17019099,  0.03634500]]],
#        grad_fn=&lt;UnsafeViewBackward0&gt;)
print(torch.equal(o1,o2[:,:-1,:]))
#False
</code></pre>
<p>seems like a loss of precision?</p>

ptrblck: <p>Different algorithms can be used for different input and weight shapes and bitwise-identical outputs are not guaranteed. Neither of these results is more accurate and should show a similar error to a wider dtype.</p>

chandler-bing: <p>got it, thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
QAT specific layers of a model

Swaraj_Badhei: <p>Hello,<br>
I am working in NAS and want to quantize (QAT) specific layers of models inside the search space.<br>
The way I am thinking is to hook layers with a torch quantizer with the layers as soon as they are created, before being passed to a sequential module. I am not sure how to do it. One post suggested passing qconfig by directly accessing the qconfig field of layer but that does not seem to exist. Kindly let me know how this can be done. Also if anyone thinks there are other methods to deal with this, please tell me.</p>
<p>Thanks in advance</p>

jerryzh168: <aside class="quote no-group" data-username="Swaraj_Badhei" data-post="1" data-topic="199687">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/swaraj_badhei/48/65881_2.png" class="avatar"> Swaraj_Badhei:</div>
<blockquote>
<p>The way I am thinking is to hook layers with a torch quantizer with the layers as soon as they are created, before being passed to a sequential module. I am not sure how to do it. One post suggested passing qconfig by directly accessing the qconfig field of layer but that does not seem to exist. Kindly let me know how this can be done. Also if anyone thinks there are other methods to deal with this, please tell me.</p>
</blockquote>
</aside>
<p>current recommended flow is this: <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_qat.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Quantization-Aware Training (QAT) — PyTorch Tutorials 2.2.1+cu121 documentation</a>, here is an example to set configuration by module name: <a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/pt2e/test_xnnpack_quantizer.py#L330" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/quantization/pt2e/test_xnnpack_quantizer.py at main · pytorch/pytorch · GitHub</a></p>
Quantizing model I'm hitting createStatus == pytorch_qnnp_status_success INTERNAL ASSERT FAILED

BmanClark: <p>I’m quantizing and converting to tflite this model: <a href="https://github.com/Picsart-AI-Research/MI-GAN" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Picsart-AI-Research/MI-GAN: [ICCV 2023] MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices</a><br>
To do that I’m using this tool: <a href="https://github.com/alibaba/TinyNeuralNetwork" class="inline-onebox" rel="noopener nofollow ugc">GitHub - alibaba/TinyNeuralNetwork: TinyNeuralNetwork is an efficient and easy-to-use deep learning model compression framework.</a></p>
<p>I’m in the last step of conversion and I’m hitting:<br>
<code>RuntimeError: createStatus == pytorch_qnnp_status_success INTERNAL ASSERT FAILED at "../aten/src/ATen/native/quantized/cpu/BinaryOps.cpp":203, please report a bug to PyTorch. failed to create QNNPACK Add operator</code></p>
<p>In the output I notice an error:<br>
<code>Error in QNNPACK: failed to create add operator with 8.319039e-06 A-to-output scale ratio: scale ratio must be in [2**-14, 2**8) range</code><br>
I can see that error in <a href="https://github.com/pytorch/QNNPACK/blob/7d2a4e9931a82adc3814275b6219a03e24e36b4c/src/add.c" class="inline-onebox" rel="noopener nofollow ugc">QNNPACK/src/add.c at 7d2a4e9931a82adc3814275b6219a03e24e36b4c · pytorch/QNNPACK · GitHub</a><br>
Is this because I’m adding too things that are on very different scales? Could it be a warning, or does it make the quantization impossible?<br>
It’s a bit hard to work out what’s going on and how I can work around it.</p>

jerryzh168: <p>maybe you can try to set a min value for scale, we have this in pytorch: <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/observer.py#L345" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/ao/quantization/observer.py at main · pytorch/pytorch · GitHub</a></p>
<p>I’m not sure if this is relevant but I’ve never seen this error before</p>
Dequantize tensors from int8 to fp16

raunaks: <p>I have loaded an LLM in huggingface with load_in_8bit=True.<br>
I noticed the objects in the state_dict are structured something like</p>
<ol>
<li>model.layers.18.self_attn.k_proj.weight</li>
<li>model.layers.18.self_attn.k_proj.SCB</li>
<li>model.layers.18.self_attn.k_proj.weight_format</li>
</ol>
<p>The SCB and weight_format are present only in the quantized model. I think SCB refers to scale and bias that can help us in recreating the original tensor? Weight format is just a string that says “row”</p>
<p>I am not sure about the exact method to dequantize, but I tried the following:<br>
<code>(weight_SCB.unsqueeze(1) * weight)/127</code><br>
This is giving a tensor that is close to the original model (loading without load_in_8bit=True)<br>
However it is not the same.<br>
I think I am doing something wrong in the dequantization process. Would be great if someone could point me to some code or documentation on how I can recreate the exact original tensor (alternatives to huggingface work as well) from the weights.</p>
<p>As a follow up question, I know that for some models there are outlier values that are not quantized even though other values in the tensor are quantized. However I could not find this information in the state_dict. How can we find and handle these values during the dequantization process?</p>

jerryzh168: <p>Hi is this a huggingface quantized model? can you show all the code you used to quantize or load the model?</p>

raunaks: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> yes. This is the code I used:</p>
<pre><code class="lang-auto">model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", cache_dir="/path/to/dir", device_map="cuda:0", load_in_8bit=True)
state_dict = model.state_dict()
</code></pre>

jerryzh168: <p>I see, I’m not sure how they are quantizing the model, maybe you can open an issue in their github? <a href="https://github.com/huggingface/transformers" class="inline-onebox" rel="noopener nofollow ugc">GitHub - huggingface/transformers: 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.</a></p>
Running PyTorch MPS acceleration on Apple M1, get "Placeholder storage has not been allocated on MPS device!" error, but all seems to be on device

Travis_C_Cuvelier: <p>I’ve noticed that a few folks have had this problem. I’m running the nightly build of PyTorch 2.4.0.dev20240326 and am trying to use my Mac’s GPU to speed up training. I’m encountering the dreaded “RuntimeError: Placeholder storage has not been allocated on MPS device!”. I think I understand that this happens when “the things needed for the computation aren’t properly loaded onto the GPU”.</p>
<p>If I call:</p>
<pre><code class="lang-auto">torch.backends.mps.is_available()
</code></pre>
<p>I get true.</p>
<p>I’m generally confused about how and when I’m supposed to “move” things to the device to accelerate training, but per <a href="https://stackoverflow.com/a/75730534/11648574" rel="noopener nofollow ugc">this answer</a> I tried the following from a breakpoint:</p>
<pre><code class="lang-auto">state
tensor([0., 3., 0.])
ipdb&gt;  state.to(torch.device("mps"))
tensor([0., 3., 0.], device='mps:0')
ipdb&gt;  self.net.to(torch.device("mps"))
Policy_Network(
  (shared_net): Sequential(
    (0): Linear(in_features=3, out_features=4, bias=True)
    (1): Tanh()
    (2): Linear(in_features=4, out_features=4, bias=True)
    (3): Tanh()
  )
  (output_net): Sequential(
    (0): Linear(in_features=4, out_features=2, bias=True)
    (1): Softmax(dim=-1)
  )
)
ipdb&gt;  self.net.state_dict()
OrderedDict({'shared_net.0.weight': tensor([[ 0.2975, -0.2548, -0.1119],
        [ 0.2710, -0.5435,  0.3462],
        [-0.1188,  0.2937,  0.0803],
        [-0.0707,  0.1601,  0.0285]], device='mps:0'), 'shared_net.0.bias': tensor([ 0.2109, -0.2250, -0.0421, -0.0520], device='mps:0'), 'shared_net.2.weight': tensor([[ 0.0725, -0.0020,  0.4371,  0.1556],
        [-0.1862, -0.3020, -0.0838, -0.2157],
        [-0.1602,  0.0239,  0.2981,  0.2718],
        [-0.4888,  0.3100,  0.1397,  0.4743]], device='mps:0'), 'shared_net.2.bias': tensor([ 0.3300, -0.4556, -0.4754, -0.2412], device='mps:0'), 'output_net.0.weight': tensor([[ 0.4391, -0.0833,  0.2140, -0.2324],
        [ 0.4906, -0.2115,  0.3750,  0.0059]], device='mps:0'), 'output_net.0.bias': tensor([-0.2634,  0.2570], device='mps:0')})
ipdb&gt;  action_probs = self.net(state)
*** RuntimeError: Placeholder storage has not been allocated on MPS device!```

(the error is still there). This doesn't make sense to me as it seems like the model and the data (state) are on the MPS device. The network (as you can probably see) is a simple torch.nn.module. Does this make sense/does anyone have any advice as to why this is happening? I feel like [this](https://stackoverflow.com/questions/78168447/placeholder-storage-has-not-been-allocated-on-mps-device) might be relevant, but I haven't identified any other parameters...</code></pre>
Restricting output range in last layer of deep architecture [regression task]

Saimon: <p>HI, I am working with 3D point cloud data, and I want to predict point wise value. Or point-wise regression task.</p>
<ol>
<li>I normalized the output of all the data to be within range [0,1]. Now how can I restrict my last layer to only output values within this range. In general, I heard it is not a good idea to use any activation function (e.g., Sigmoid) in the last layer during regression task. I am wondering is there any better idea?</li>
<li>Is it a good idea to unnormalize within the model in my case? Currently I am unnormalizing after prediction using model.</li>
</ol>
<p>Thanks in advance!</p>

KFrank: <p>Hi Amirul!</p>
<aside class="quote no-group" data-username="Saimon" data-post="1" data-topic="199712" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/7cd45c/48.png" class="avatar"> Saimon:</div>
<blockquote>
<p>Now how can I restrict my last layer to only output values within this range.</p>
</blockquote>
</aside>
<p>I would not recommend restricting the output of your model – just let your<br>
model learn that predictions outside of the range <code>[0, 1]</code> are poor predictions.</p>
<p>Let’s say your (normalized) ground-truth target is <code>0.98</code> and your model<br>
predicts <code>1.02</code>.  Your regression loss (typically something like <code>MSELoss</code>)<br>
will penalize your prediction by a little bit.  But that’s appropriate, because<br>
the prediction is only a little be off.  If the prediction were <code>2.34</code>, it would be<br>
significantly off and would be penalized appropriately.</p>
<p>Would anything <em>break</em> if your model made a prediction greater than one?<br>
What would be worse: a prediction of <code>0.85</code> when the ground truth is <code>0.71</code><br>
or a prediction of <code>1.02</code> when the ground truth is <code>0.98</code>?</p>
<aside class="quote no-group">
<blockquote>
<p>Is it a good idea to unnormalize within the model in my case?</p>
</blockquote>
</aside>
<p>I wouldn’t.  I would expect that you could do so in a logically-consistent<br>
way, but I don’t see what benefit you would get.</p>
<aside class="quote no-group">
<blockquote>
<p>Currently I am unnormalizing after prediction using model.</p>
</blockquote>
</aside>
<p>There is often some benefit to normalizing your data (especially your<br>
input data).  But if the range of your ground-truth target data isn’t<br>
outlandish, things might work fine leaving it (for simplicity’s sake)<br>
unnormalized.</p>
<p>Best.</p>
<p>K. Frank</p>
Validating UNet architecture

phisanti: <p>Hi, this is my first time building a model with Pytorch, so I am translating a unit from TensorFlow. Thus, I was wondering if someone could give me a sanity check that the model looks valid:</p>
<pre data-code-wrap="python"><code class="lang-python">
class UNet(nn.Module):
    def __init__(self, in_channels=1, out_channels=1, init_features=64, pooling_steps=2):
        super(UNet, self).__init__()

        features = init_features
        self.encoders = nn.ModuleList()
        self.decoders = nn.ModuleList()
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.upconv = nn.ModuleList()

        for i in range(pooling_steps):
            self.encoders.append(UNet._block(in_channels if i==0 else features, features * (2**i), name=f"enc{i+1}"))
            self.decoders.insert(0, UNet._block(features * (2**(i+1)), features * (2**i), name=f"dec{i+1}"))
            self.upconv.insert(0, nn.ConvTranspose2d(features * (2**(i+1)), features * (2**i), kernel_size=2, stride=2))

        self.bottleneck = UNet._block(features * (2**(pooling_steps-1)), features * (2**pooling_steps), name="bottleneck")
        self.conv = nn.Conv2d(in_channels=features, out_channels=out_channels, kernel_size=1)

    def forward(self, x):
        encs = []
        for i, encoder in enumerate(self.encoders):
            x = encoder(x)
            encs.append(x)
            x = self.pool(x)

        x = self.bottleneck(x)

        for i, decoder in enumerate(self.decoders):
            x = self.upconv[i](x)
            x = torch.cat((x, encs[-(i+1)]), dim=1)
            x = decoder(x)

        return torch.sigmoid(self.conv(x))

    @staticmethod
    def _block(in_channels, features, name):
        return nn.Sequential(
            nn.Conv2d(
                in_channels=in_channels,
                out_channels=features,
                kernel_size=3,
                padding=1,
                bias=False,
            ),
            nn.BatchNorm2d(num_features=features),
            nn.ReLU(inplace=True),
            nn.Conv2d(
                in_channels=features,
                out_channels=features,
                kernel_size=3,
                padding=1,
                bias=False,
            ),
            nn.BatchNorm2d(num_features=features),
            nn.ReLU(inplace=True),
        )
</code></pre>
<p>Also, I was wondering what strategy could I follow in order to speed up inference. I read that it is possible to use half-precision (16 bits), but I am not sure how to implement it. Does it go with the tensor, or is it an attribute of the model?</p>
<p>Any other trick will be welcome.</p>

ptrblck: <p>I don’t see anything obviously wrong in the model implementation.</p>
<p>For mixed-precision training check <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">this tutorial</a>.</p>

phisanti: <p>Thanks for the feedback!</p>
How to reduce memory usage for large matrix calculations

bowensuuu: <p>A_ = torch.sigmoid(torch.matmul(x, x.t()))<br>
x is the feature of tens of thousands of nodes, the shape is 700,000*8, 8 is the number of features extracted from each node.<br>
Calculation requires several t of memory. How to reduce memory overhead?</p>

sc0v0ne: <p>One option would be to use Python Generator, I will leave a tutorial below if it can be useful for your application.</p>
<p><a href="https://medium.com/the-research-nest/optimizing-memory-usage-in-python-e8a30e0dddd3" rel="noopener nofollow ugc"> Optimizing Memory Usage in Python</a></p>

ptrblck: <p>I don’t understand how a generator would help here.</p>
<p><a class="mention" href="/u/bowensuuu">@bowensuuu</a> your output matrix will have a shape of <code>[700,000 * 8, 700,000 * 8]</code> and will thus use ~114TB of memory in <code>float32</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(7*8, 8)
A_ = torch.sigmoid(torch.matmul(x, x.t()))
print(A_.shape)
# torch.Size([56, 56])

(700000*8 * 700000*8 * 4) / 1024**4
# 114 TB
</code></pre>
<p>(my example uses <code>7*8</code> just to illustrate the shapes)</p>
<p>You would need to reduce the features significantly.</p>
[BUG?] Torch.load loads different values for the same file in each time

Ameen_Ali: <p>Hello</p>
<p>I have saved a pytorch tensor which I have obtained from some optimization procedure, and I save it as a file.</p>
<p>I have noticed that i get different values for the same tensor loaded in each time I load it.</p>
<p>First try:</p>
<pre><code class="lang-auto">tensor([[ 0.0423,  0.0015,  0.0025,  ..., -0.0053, -0.0032,  0.0013],
        [ 0.0048,  0.0129, -0.0160,  ...,  0.0002, -0.0005, -0.0035],
        [ 0.0763, -0.0221,  0.0303,  ..., -0.0008, -0.0018, -0.0009],
        ...,
        [-0.0067, -0.0207, -0.0157,  ..., -0.0009,  0.0042, -0.0030],
        [ 0.0059, -0.0294, -0.0113,  ..., -0.0066,  0.0115,  0.0053],
        [-0.0429, -0.0239, -0.0362,  ...,  0.0038,  0.0119,  0.0004]])
</code></pre>
<p>Second try:</p>
<pre><code class="lang-auto">tensor([[ 4.7368e-24,  0.0000e+00,  6.4595e-24,  ..., -5.3488e-03,
         -3.1589e-03,  1.3454e-03],
        [ 4.8110e-03,  1.2933e-02, -1.5969e-02,  ...,  2.2385e-04,
         -4.8361e-04, -3.5191e-03],
        [ 7.6261e-02, -2.2051e-02,  3.0303e-02,  ..., -7.5350e-04,
         -1.7584e-03, -9.4366e-04],
        ...,
        [-6.6854e-03, -2.0729e-02, -1.5737e-02,  ..., -9.4584e-04,
          4.1868e-03, -2.9629e-03],
        [ 5.8720e-03, -2.9389e-02, -1.1321e-02,  ..., -6.6035e-03,
          1.1492e-02,  5.3333e-03],
        [-4.2901e-02, -2.3902e-02, -3.6203e-02,  ...,  3.7728e-03,
          1.1884e-02,  4.2430e-04]])
</code></pre>
<p>Third try:</p>
<pre><code class="lang-auto">tensor([[-1.7079e+38,  5.6052e-45,  9.2569e-32,  ..., -1.7414e+38,
          3.5733e-43,  2.1366e-34],
        [ 1.4507e-39,  9.5536e-38,  1.1210e-44,  ...,  1.4507e-39,
          3.1854e-23,  1.1210e-44],
        [ 1.2180e-31,  1.3689e-39,  2.7985e-39,  ...,  2.1367e-34,
          2.9186e-39,  7.0129e-21],
        ...,
        [ 1.4486e-39,  3.3822e-26,  9.0042e-41,  ...,  1.4586e-39,
          1.2915e-19,  7.0312e-41],
        [ 3.7297e-34,  1.4586e-39,  1.3919e-16,  ...,  1.4481e-36,
          1.4586e-39,  3.3473e-39],
        [ 5.3249e-44,  3.0248e-39,  1.2986e-39,  ...,  3.0829e-44,
          5.8775e-39,  7.3361e-39]])
</code></pre>
<p>Can this be a bug?</p>
<p>Pytorch version is : 1.13.0+cu116</p>

ptrblck: <p>Looks like a bug assuming the file didn’t change. Could you update PyTorch and check if you still observe this issue?</p>
Does `torch.profiler` support NVTX markers?

songyuc: <p>Hi, guys,<br>
We plan to use <code>nvidia.dali</code> to accelerate our training, which says:</p>
<blockquote>
<p>As for profiling, DALI doesn’t have any built-in profiling capabilities, still it utilizes NVTX ranges and has a dedicated domain (so it is easy to find in the profile) to show its operations. So you can capture the profile using <a href="https://developer.nvidia.com/nsight-systems" rel="noopener nofollow ugc">NVIDIA Nsight Systems</a> or any Deep Learning profile that also supports NVTX markers.</p>
</blockquote>
<p>(Since I have little knowledge about NVTX,)<br>
I am wondering whether <code>torch.profiler</code> supports NVTX markers?</p>
<p>Your answer and guide will be appreciated!</p>

galv: <p>Hi sonyuc, torch.profiler uses Kineto, which itself uses the CUPTI library to caputre NVTX events as described here: <a href="https://docs.nvidia.com/cupti/main/main.html?highlight=nvtx#nvidia-tools-extension-callbacks" class="inline-onebox" rel="noopener nofollow ugc">2. Usage — Cupti 12.4 documentation</a>. At least this is my understanding.</p>

jon-chuang: <p>Hi, this is false. Kineto does not support NVTX markers via cupti. See: <a href="https://github.com/pytorch/kineto/issues/318#issuecomment-2022538594" class="inline-onebox" rel="noopener nofollow ugc">support of torch.cuda.nvtx.range_push() &amp; torch.cuda.profiler.start() · Issue #318 · pytorch/kineto · GitHub</a></p>
Why torch1.13.x JIT compiler doesn't support sm90

Jonson: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccaeb46aecd54c362a2f868512040799d9610b51.png" data-download-href="https://discuss.pytorch.org/uploads/default/ccaeb46aecd54c362a2f868512040799d9610b51" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccaeb46aecd54c362a2f868512040799d9610b51.png" alt="image" data-base62-sha1="tcHHQqhxwxrdGoaZEPhy1uiSGQx" width="690" height="91" data-dominant-color="242525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">872×116 4.55 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>These snippets are in <code>torch.utils.cpp_extension.py::_get_cuda_arch_flags</code>. And the latest supported arch is sm_86.</p>

ptrblck: <p>PyTorch 1.13 released with CUDA 11.6 and 11.7, which did not support Hopper architectures. Update to any recent stable or nightly binary and it’ll work.</p>
Batch balancing with different length datasets

Juan_Carlos_Martinez: <p>Hi everyone!</p>
<p>I am trying to build some experiments with PyTorch and PyTorch Lightning. My problem is how to approach a Dataset class that takes as input for example: [dataset_1_train_split.dat, dataset_2_train_split.dat, dataset_3_train_split.dat] . However, d1, d2, d3 are 10, 100 and 1000 lengths each. If I have a batch_size = 4, how can I force every batch to be as balanced as possible?. I assume a “balanced batch” is a batch with samples from every dataset. The epoch should end when all the samples from the biggest dataset have been part of a batch. In the example the batch should be: [sample_from_d1, sample_from_d2, sample_from_d3, sample_from_&lt;any_of_datasets&gt;]. Of course this should be dynamic depending on batch_size and number of input training splits.</p>
<p>Thank you in advance, hope it is possible. If not, how would you approach this problem?<br>
Juan Carlos</p>

Juan_Carlos_Martinez: <p>In case someone needs to do something similar. I ended creating an interleaved list between all the datasets and use it as the only dataset for the experiment.</p>
Multi-layer, Bi-directional LSTM's

Harshit_Goyal: <p>As per the documentation <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">on PyTorch LSTMs</a>, the final hidden state <code>hn</code>,  when <code>bidirectional = True</code> and <code>num_layers  = 4</code>  and <code>batch_first = True</code> will have shape <code>( 2 ∗ 4 , batch_size  , H_out)</code>. How can I extract out the hidden state output by only the last layer (or the “topmost” layer , not the last time-step). How can I extract out only the hidden state from forward propagation? I couldn’t find the correct scheme to <code>unflatten()</code> <code>hn</code>  to do the above.<br>
​</p>
More efficient autograd for matrix w.r.t matrix gradient computation?

sherlock.h: <p>Hello ! I have two tensors K (N, N) and X (N, T, F), and I want to calculate the derivative of K[i, j] with respect to X[j, -1].<br>
At the moment I’m using <code>torch.autograd.grad(K[i, j], X)</code> and getting the [j, -1]th output, but I’m wondering if it’s possible to differentiate directly with respect to X[j, -1], or even better.</p>
<p>The final goal is to calculate this:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.stack([
    torch.stack([
        torch.autograd.grad(K[i, j], X, retain_graph=True)[0][j, -1]
        for j in range(X.shape[0])
    ])
    for i in range(X.shape[0])
]).mean(1)
</code></pre>
<p>If you can think of a way to optimise this, please let me know!</p>
How to understand the high embedding vector value after training the model?

songsong0425: <p>Dear PyTorch community,</p>
<p>Greetings, always thank you for your effort in the active discussion about the overall deep learning paradigm.<br>
I have a general question about the embedding vector from the trained model.</p>
<p>I checked that my model returned node embedding vector after model training which results always predict samples into positive.</p>
<pre data-code-wrap="python"><code class="lang-python">model = MyModel(in_channels=256, hidden_channels=128, out_channels=64).to(device) #if sum, in_channels=256
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-3)
criterion = torch.nn.BCEWithLogitsLoss()

model.train()
for data in tqdm(train_loader):
    
    data = data.to(device)
    data.edge_class = data.edge_class[data.input_id]
    data.edge_index_class = torch.zeros(len(data.edge_index[0])).to(device)    
            
    for i in range(len(data.edge_index_class)):
        if ((data.edge_index[1][i] in data.edge_label_index[0]) or (data.edge_index[1][i] in data.edge_label_index[1])):
            data.edge_index_class[i] = data.edge_class[(data.edge_index[1][i]==data.edge_label_index[0])
                                                         |(data.edge_index[1][i]==data.edge_label_index[1])]
        else:
            data.edge_index_class[i] = torch.max(data.edge_index_class[(data.edge_index[1][i]==data.edge_index[0])])
            
    optimizer.zero_grad()
    
    z, a1, a2 = model(data.x[0], data.x[1], data.n_id, data.edge_index_class, data.edge_index) 
    
    out = ((z[data.edge_label_index[0]] * z[data.edge_label_index[1]]).sum(dim=-1)).view(-1) # product of a pair of nodes on each edge
    out_sig = ((z[data.edge_label_index[0]] * z[data.edge_label_index[1]]).sum(dim=-1)).view(-1).sigmoid()

print(out)
#tensor([ 8.4500,  7.3036,  7.7035,  8.7444, 11.7715,  8.8596,  7.1483, 11.5223,
#        11.2428, 10.7440,  7.6953, 10.4081,  7.4365,  7.4336,  9.0127,  7.7171,
#        10.3317,  8.9579,  6.8032, 11.4876,  7.3854,  5.2074,  6.3047, 10.6670,
#         7.2817,  9.7236,  6.9220, 10.3020], device='cuda:0',
#       grad_fn=&lt;ViewBackward0&gt;)

print(out_sig)
#tensor([0.9998, 0.9993, 0.9995, 0.9998, 1.0000, 0.9999, 0.9992, 1.0000, 1.0000,
#        1.0000, 0.9995, 1.0000, 0.9994, 0.9994, 0.9999, 0.9996, 1.0000, 0.9999,
#        0.9989, 1.0000, 0.9994, 0.9946, 0.9982, 1.0000, 0.9993, 0.9999, 0.9990,
#        1.0000], device='cuda:0', grad_fn=&lt;SigmoidBackward0&gt;)

print(data.edge_label)
# tensor([1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0,
#        1, 1, 1, 1], device='cuda:0')
</code></pre>
<p>In this case, where should I check the model?<br>
The experimental setting is as follows:<br>
dropout rate: 0.2<br>
batch size: 32<br>
learning rate: 0.001<br>
weight decay: 0.001</p>
<p>Thank you for reading my question.</p>
Change parameters for Swin-transformer

kyrie_rao: <p>I’m trying to change the Swin transformer parameter DROP_PATH_RATE to fine-tune by doing:</p>
<pre><code class="lang-auto">model = torchvision.models.swin_b(weights=torchvision.models.Swin_B_Weights.IMAGENET1K_V1, stochastic_depth_prob=0.2)
</code></pre>
<p>but I’m finding that it seems to be fixed in swin_transfomer.py.</p>
<pre><code class="lang-auto">return _swin_transformer(
        patch_size=[4, 4],
        embed_dim=128,
        depths=[2, 2, 18, 2],
        num_heads=[4, 8, 16, 32],
        window_size=[7, 7],
        stochastic_depth_prob=0.5,
        weights=weights,
        progress=progress,
        **kwargs,
    )
</code></pre>
<p>Are there any ways to define the DROP_PATH_RATE parameter when initializing the Swin transformer?</p>

kyrie_rao: <p>I answered myself.<br>
For Swin Transformer base with pre-trained ImageNet-1K:</p>
<pre><code class="lang-auto">weights = torchvision.models.Swin_B_Weights.IMAGENET1K_V1
model = torchvision.models.SwinTransformer(
        patch_size=[4, 4],
        embed_dim=128,
        depths=[2, 2, 18, 2],
        num_heads=[4, 8, 16, 32],
        window_size=[7, 7],
        stochastic_depth_prob=0.2
)
model.load_state_dict(weights.get_state_dict(progress=True, check_hash=True))
</code></pre>
How to set up audio data for audio classification tasks for lstm model?

Teddy_Salas: <p>How to setup audio data for audio classification task using lstm?</p>
<p>for example this is the setup for image data for image classification task ,like this how do i setup my data for audio classification,please help.</p>
<p>import torch<br>
from torch import nn<br>
import torchvision<br>
from torchvision import transforms</p>
<p>train_transforms = transforms.Compose([<br>
transforms.Resize(size=(224,224)),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.RandomRotation(10),<br>
transforms.TrivialAugmentWide(num_magnitude_bins=31),<br>
transforms.ToTensor(),</p>
<p>])</p>
<p>test_transforms = transforms.Compose([<br>
transforms.Resize(size=(224,224)),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.RandomRotation(10),<br>
transforms.ToTensor(),</p>
<p>])from torchvision import datasets</p>
<p>train_data = datasets.ImageFolder(<br>
root=train_dir,<br>
transform=train_transforms<br>
)</p>
<p>test_data = datasets.ImageFolder(<br>
root=test_dir,<br>
transform=test_transforms<br>
)from torch.utils.data import DataLoader</p>
<p>train_dataloader = DataLoader(<br>
dataset = train_data,<br>
batch_size = 28,<br>
num_workers = os.cpu_count(),<br>
shuffle=True<br>
)</p>
<p>test_dataloader = DataLoader(<br>
dataset = test_data,<br>
batch_size=28,<br>
num_workers = os.cpu_count(),<br>
shuffle=False<br>
)</p>
Unable to access Wikitext2 and Wikitext 103 datasets!

vighneshch: <p>Hi facing with the below error we I try to load the wikitext datasets from torchtext</p>
<pre><code class="lang-auto">from torchtext.data.functional import to_map_style_dataset
from torchtext import datasets
data = datasets.WikiText103(root='/data',split='train')
data_set = to_map_style_dataset(data)

</code></pre>
<p>Facing with the below error on colab</p>
<pre><code class="lang-auto">TTPError: 403 Client Error: Forbidden for url: https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip
This exception is thrown by __iter__ of HTTPReaderIterDataPipe(skip_on_error=False, source_datapipe=OnDiskCacheHolderIterDataPipe, timeout=None)
</code></pre>
<p>My torchtext version is 0.17.1</p>

Jia_Huang: <p>I face same issue here, anyone can help?</p>

vighneshch: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi can you please help us?</p>

vdw: <p>That’s not really an PyTorch issue, I think. When you just click the link</p>
<pre><code class="lang-auto">https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip
</code></pre>
<p>I don’t know who manages this dataset, but most likely not the PyTorch devs etc.</p>

vighneshch: <p>Yes, the link needs to be changed to a working one.</p>
<p>Anyways I downloaded the same dataset from huggingface datasets.</p>
Is there any way to dump the whole model's backward pseudo python code?

nahso: <p>The overall progress of my model is:</p>
<pre><code class="lang-auto">data = model(input)
grad = torch.autograd.grad(data, input)
output grad
</code></pre>
<p>The gradient with respect to the input data is the output of the python code. I plan to rewrite the whole model with pure C, but I can not get the straightforward backward code.</p>
<p>Is there any way to dump the whole model’s backward pseudo python code?</p>

ptrblck: <p>I’m not aware of a way to export the backward Python code in PyTorch for a full model.</p>
Guidance on Placement of Architecture-Specific Kernel Optimizations in PyTorch Codebase

abhishek-fujitsu: <p>Hello PyTorch Community,</p>
<p>I’m currently working on enhancing the performance of specific operations in PyTorch, targeting performance optimizations . My focus is on modifying kernel-level code within the aten/src/ATen/native/cpu/TensorCompareKernel.cpp file, particularly the min_kernel_impl and max_kernel_impl functions.My approach involves integrating CPU-specific optimizations.</p>
<p>I would like to ask for guidance on the following points:</p>
<p>Q1. Is it suitable to place CPU-specific, conditional (if-else) optimizations within TensorCompareKernel.cpp for functions like min_kernel_impl and max_kernel_impl?</p>
<p>Q2. If TensorCompareKernel.cpp isn’t the right place for such architecture-specific enhancements, could you suggest an alternative file or module within the PyTorch codebase where these optimizations could be appropriately integrated?</p>
<p>Q3. Any general advice or guidelines the community follows when contributing CPU/architecture-specific code to maintain the design philosophy and maintainability of PyTorch.</p>
<p>Insights, advice, or links to any relevant documentation or discussions would be highly valuable and appreciated.</p>
<p>Thank you for your time and help!</p>

ptrblck: <p>I would recommend creating an issue on GitHub explaining your approach and allowing code owners to review your suggestion. Generally, I would assume architecture-specific code can land, but since it’s CPU-specific I’m not deeply familiar with the structure.</p>
GPU Utilization

Ido_Avrahami: <p>Hello everyone,</p>
<p>I’m training a CNN, that looks like</p>
<p>model = nn.Sequential(<br>
sl.FlattenTime(),<br>
nn.Conv2d(2, 8, kernel_size=3, padding=1, bias=False),<br>
# sl.IAFSqueeze(batch_size=batch_size, min_v_mem=-1),<br>
nn.MaxPool2d(2),<br>
# sl.SumPool2d(2),<br>
nn.Conv2d(8, 16, kernel_size=3, padding=1, bias=False),<br>
# sl.IAFSqueeze(batch_size=batch_size, min_v_mem=-1),<br>
nn.MaxPool2d(2),<br>
# sl.SumPool2d(2),<br>
nn.Conv2d(16, 32, kernel_size=3, padding=1, bias=False),<br>
# sl.IAFSqueeze(batch_size=batch_size, min_v_mem=-1),<br>
nn.MaxPool2d(2),<br>
# sl.SumPool2d(2),<br>
nn.Conv2d(32, 64, kernel_size=3, padding=1, bias=False),<br>
# sl.IAFSqueeze(batch_size=batch_size, min_v_mem=-1),<br>
nn.MaxPool2d(2),<br>
# sl.SumPool2d(2),<br>
nn.Conv2d(64, 10, kernel_size=2, padding=0, bias=False),<br>
# sl.IAFSqueeze(batch_size=batch_size, min_v_mem=-1),<br>
nn.Flatten(),<br>
sl.UnflattenTime(batch_size=batch_size),<br>
).to(device)</p>
<p>sl is a module named sinabs that is based on pytorch and can be used with it.<br>
It also utilizes the gpu the same as pytorch.<br>
When I’m training the model the cuda utilisation graph on task manager shows spikes every few seconds of 50% usage.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b9eb44006abcd567256ae1ac86e83752450f0a4.png" alt="Screenshot 2024-03-25 213919" data-base62-sha1="d4vtfwEM0W88ep5OsibmaZ9YiQQ" width="290" height="74"><br>
It’s worth mentioning that every spike of usage matches the iteration step in the training loop.</p>
<p>It also takes long to train.<br>
My GPU is 1060 6gb and torch.cuda.is_available() returns True.</p>
<p>Also when using nvidia-smi it says 0% utilization most of the time and again it shows spikes of 100% utilization every few seconds…</p>

ptrblck: <p>You could profile your code via e.g. Nsight Systems to narrow down the bottlenecks and to isolate which part of the code is responsible for the low utilization.</p>
Modifying the matrix multiplication kernel

Tiger37: <p>I essentially want to replace the product operation within matrix multiplication to another type of operation. I have very little knowledge when it comes to writing a custom pytorch kernel, and so, I would like to take advantage of everything behind torch.matmul, but would need to make a few changes to the underlying code to change the operation. I have been looking everywhere, but is there an easy way to do this before I hash together a terribly optimized cuda kernel?</p>
<p>The source code for pytorch matrix multiplication is pointed to in this <a href="https://discuss.pytorch.org/t/matrix-multiplication-source-code/71071">topic</a>, but not really sure how to utilize this, and again, if there is an easier way, that would be great.</p>
<p>The aim is to have an efficient operation that can be used on the gpu (cuda).</p>

eqy: <p>The Triton tutorial on matmuls might be useful <a href="https://triton-lang.org/main/getting-started/tutorials/03-matrix-multiplication.html" class="inline-onebox" rel="noopener nofollow ugc">Matrix Multiplication — Triton documentation</a> .</p>
<p>Most CUDA matmuls in PyTorch are dispatched to cuBLAS which is closed-source. The closest thing would probably be CUTLASS kernels in e.g., <a href="https://github.com/NVIDIA/cutlass" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/cutlass: CUDA Templates for Linear Algebra Subroutines</a></p>

Tiger37: <p>That explains the pain in finding the source code. The triton tutorial looks very promising, thank you. Overall, this rabbit hole I have fallen into seems to be a very rare use case and thus going to take a lot more work than it is worth now. I may focus on a CPU implementation for now instead.</p>

bdusell: <p>FWIW, I’ve been told that cutlass is supposed to be very close to the closed-source implementation.</p>
<p>I once wanted to do exactly the same thing, so I wrote the “torch_semiring_einsum” package for PyTorch: <a href="https://github.com/bdusell/semiring-einsum" class="inline-onebox" rel="noopener nofollow ugc">GitHub - bdusell/semiring-einsum: Generic PyTorch implementation of einsum that supports different semirings</a></p>
<p>You should also check out genbmm: <a href="https://github.com/harvardnlp/genbmm" class="inline-onebox" rel="noopener nofollow ugc">GitHub - harvardnlp/genbmm: CUDA kernels for generalized matrix-multiplication in PyTorch</a></p>

Tiger37: <p>This is basically what I needed! With a few minor modifications, I think I could get the equation I need. Thank you!</p>
How to Adapt DDP Pipeline Tutorial for Multi-Node Training

jasonkrone: <p>Anyone have advice on how to adapt the <a href="https://github.com/pytorch/tutorials/blob/main/advanced_source/ddp_pipeline.py" rel="noopener nofollow ugc">ddp_pipeline</a> tutorial for multi-node training?</p>
<p>I’ve been using <code>torchrun</code> for multi-node <code>DistributedDataParallel</code> training; however, the <a href="https://github.com/pytorch/tutorials/blob/main/advanced_source/ddp_pipeline.py" rel="noopener nofollow ugc">ddp_pipeline</a> example relies on <code>mp.spawn</code>.</p>
<p>I tried to modify the the <a href="https://github.com/pytorch/tutorials/blob/main/advanced_source/ddp_pipeline.py" rel="noopener nofollow ugc">ddp_pipeline</a> script to run with <code>torchrun</code>, but it caused the <code>DistributedDataParallel(model)</code> call to hang.</p>
<p>Would love any suggestions for how to either: (1) adapt <a href="https://github.com/pytorch/tutorials/blob/main/advanced_source/ddp_pipeline.py" rel="noopener nofollow ugc">ddp_pipeline</a> for use with <code>torchrun</code> or (2) how to set up multi-node training using <code>mp.spawn</code>.</p>

jasonkrone: <p>Solved this. See my answer <a href="https://discuss.pytorch.org/t/problem-on-combining-model-parallelization-and-ddp-on-multi-nodes/197472/4">here</a>!</p>
Numerical change from transpose operation?

kheyer: <p>I found a numerical quirk where the output of a computation is changed by using the transpose operation. The difference is small, but I’m curious if anyone knows the underlying reason for this.</p>
<p>Start with:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([0.0014, -0.0306,  0.0005,  0.0011,  0.0012,  0.0022,  0.0017,  0.0011,
                  0.0017,  0.0011,  0.0012,  0.0017,  0.0014,  0.0015,  0.0010,  0.0006,
                  0.0006,  0.0004,  0.0009,  0.0007,  0.0008,  0.0007,  0.0013,  0.0013,
                  0.0015,  0.0023,  0.0007], dtype=torch.float64)
</code></pre>
<p>Now compute:</p>
<pre data-code-wrap="python"><code class="lang-python">y1 = (x.unsqueeze(0) @ torch.ones(x.shape[0], 1, dtype=torch.float64)).item()
&gt; 3.2526065174565133e-19

y2 = (x.unsqueeze(0) @ (torch.ones(1, x.shape[0], dtype=torch.float64).T)).item()
&gt; 2.4936649967166602e-18
</code></pre>
<p>I would not expect the transpose operation to have this effect. Does anyone know the underlying reason?</p>

ptrblck: <p>Different algorithms can be used for different input shapes which do not produce bitwise-identical outputs.</p>
Loss becomes NaN when introducing regularization

Jacob_Eriksen: <p>I have a model that trains well without any regularization, however, when I implement L2 regularization (by the weight_decay in adam optimizer), the loss becomes nan after some iterations. I have tried different values of weight_decay  in [0.1, 0.01, 0.00001] and tried to lower the learning rate, but it doesn’t seem to help.</p>
<p>The loss function does seem to decrease nicely up until some point where the loss becomes nan, and when lowering the learning rate, the convergence will go slower but the nan loss will still occur just after more iterations. The nan loss does seem to come roughly around the same loss value.</p>
<p>Does anyone have any suggestions as to why this happens? or what I could try to prevent it?</p>
<p>Notice, the model is a highly overparameterized model if that makes any difference</p>
Faster repeat_interleave?

Lin_Yang: <p>I’m trying to get [0,0,0, 1,1,1, 2,2,2]…<br>
tried two method, both very slow, is that a better approach?</p>
<pre><code class="lang-auto">from triton.testing import do_bench
import torch

size = 1*1024 * 1024

def _repeat_interleave(size, repeats):
    return torch.arange(size).repeat_interleave(repeats, output_size=size*d)

def _tile(size, repeats):
    return torch.arange(size).view(-1, 1).tile(1, d).flatten()

for d in [1, 2, 4, 8, 16, 32]:
    assert(torch.allclose(_repeat_interleave(size, d), _tile(size, d)))
    print(d, do_bench(lambda : _repeat_interleave(size, d)))
    print(d, do_bench(lambda : _tile(size, d)))
    print('-----')

1 0.37788161635398865
1 0.3799819350242615
-----
2 3.7355096340179443
2 3.932922124862671
-----
4 30.093482971191406
4 19.520463943481445
-----
8 52.662208557128906
8 53.49440002441406
-----
16 71.41273498535156
16 70.170654296875
-----
32 143.93128967285156
32 138.313720703125
-----
</code></pre>
I can't import pipeline from transformers library it shows an error

Aniket_Pal: <p>False ===================================BUG REPORT=================================== ================================================================================ The following directories listed in your path were found to be non-existent: {WindowsPath(‘C’), WindowsPath(‘/Users/anike/anaconda3/envs/deeplearning/lib’)} The following directories listed in your path were found to be non-existent: {WindowsPath(‘/matplotlib_inline.backend_inline’), WindowsPath(‘module’)} The following directories listed in your path were found to be non-existent: {WindowsPath(‘vs/workbench/api/node/extensionHostProcess’)} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths… The following directories listed in your path were found to be non-existent: {WindowsPath(‘/usr/local/cuda/lib64’)} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6. CUDA SETUP: To manually override the PyTorch CUDA version please</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba7d963bfb523ce3440819ce212327261e6d6a5f.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba7d963bfb523ce3440819ce212327261e6d6a5f" title="error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba7d963bfb523ce3440819ce212327261e6d6a5f_2_437x500.png" alt="error" data-base62-sha1="qBLSrowWwyLp502Q7xWOMgLOfjF" width="437" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba7d963bfb523ce3440819ce212327261e6d6a5f_2_437x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba7d963bfb523ce3440819ce212327261e6d6a5f_2_655x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba7d963bfb523ce3440819ce212327261e6d6a5f_2_874x1000.png 2x" data-dominant-color="1B1D21"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1478×1689 363 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The stacktrace points to <code>bitsandnytes</code> so you might want to create an issue in their repository.</p>
Torch Batch Norm

likhilb9: <p>How to copy the weights from one model to another (same model architecture) only for batch norm layer.</p>

ptrblck: <p>You can access the corresponding batchnorm layers and copy their <code>state_dict</code> via:</p>
<pre data-code-wrap="python"><code class="lang-python">modelA.batch_norm_layer1.load_state_dict(modelB.batch_norm_layer1.state_dict())
</code></pre>
<p>If you need to copy multiple layers, you can of course also iterate the <code>model.children()</code> and use an <code>if</code> condition to filter for e.g. <code>isinstance(child, nn.BatchNorm2d)</code>.</p>

likhilb9: <p>But from the above code, we have to write code for each layer to load.<br>
Let say, we have switched the model now. We have again changed the code.<br>
Is there any way the code is dynamic?</p>

ptrblck: <p>In this case you can follow the second part of my post e.g. by iterating the modules and using conditions:</p>
<pre data-code-wrap="python"><code class="lang-python">modelA = models.resnet50()
modelB = models.resnet50()

for (nameA, childA), (nameB, childB) in zip(modelA.named_modules(), modelB.named_modules()):
    if isinstance(childA, nn.BatchNorm2d) and isinstance(childB, nn.BatchNorm2d):
        print("Found {} and {}".format(nameA, nameB))
        childA.load_state_dict(childB.state_dict())

# verify
for (nameA, childA), (nameB, childB) in zip(modelA.named_modules(), modelB.named_modules()):
    if isinstance(childA, nn.BatchNorm2d) and isinstance(childB, nn.BatchNorm2d):
        print((childA.weight == childB.weight).all())
</code></pre>
<p>In the end the right approach depends on your use case and if you want to use the module names, a lookup table, or just iterate the modules assuming they were created in the same order.</p>