akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Convert .pth to .pt

Javier_Alonso: <p>Hi everyone, I’m new in this community and relative new in IA.</p>
<p>I downloaded the llama-2-7b-chat from meta and I want to run this in Java.</p>
<p>First I tried to load the module with:</p>
<pre><code class="lang-auto">Module mod = Module.load("llama-2-7b-chat/consolidated.00.pth", extraFiles, Device.CPU);
</code></pre>
<p>but I have theese errors:</p>
<pre><code class="lang-auto">Exception in thread "main" com.facebook.jni.CppException: PytorchStreamReader failed locating file constants.pkl: file not found
Exception raised from valid at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\caffe2\serialize\inline_container.cc:184 (most recent call first):
</code></pre>
<p>I read that is a problem with the model’s format and I should convert it into a pt (TorchScript model).</p>
<p>I’ve tried with a lot of code, and in my last attemp I wrote this:</p>
<pre><code class="lang-auto">import torch
from transformers import LlamaPreTrainedModel

#print(torch.__version__)

# Carga tu modelo entrenado en PyTorch
#model = torch.jit.load('consolidated.00.pth', map_location=torch.device('cpu'))
model = LlamaPreTrainedModel.from_pretrained(pretrained_model_name_or_path='consolidated.00.pth',
            config='params.json',)

print(type(model))

# Convierte el modcelo a TorchScript
scripted_model = torch.jit.script(model)

print(type(scripted_model))

# Guarda el modelo en formato TorchScript
#torch.save(scripted_model, 'llama_chat.pt')
</code></pre>
<p>I can see only the lines of the error but not the exception, my terminal only displays:<br>
StopIteration</p>
<p>Somebody managed to convert this model to torchscript?</p>
<p>Thanks for read!</p>

GithinjiHans: <p>did you find a solution for this?</p>

Javier_Alonso: <p>No. Sorry <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>I’ve been very busy.</p>
Can anyone correct this code c++

AishaMohamed2022: <p>i implement Algorithm merge sort by c++<br>
but it not sorted Array input</p>
<p><span class="hashtag-raw">#include</span><br>
using namespace std;<br>
void MERGE(int* A, int p, int q, int r) {<br>
int n1 = q - p + 1;<br>
int n2 = r - q;<br>
int* L = new int[n1];<br>
int * R=new int[n2];<br>
for (int i = 0; i &lt; n1 + 1; i++) {<br>
L[i] = A[p + i - 1];<br>
}<br>
for (int j = 0; j &lt; n2 + 1; j++) {<br>
R[j] = A[p + j];<br>
}<br>
int l = 0, t = 0;<br>
for (int k = p; k &lt; r + 1; k++) {<br>
if (L[l] &lt;= R[t]) {<br>
A[k] = L[l];<br>
l++;<br>
}<br>
else if (A[k] == R[t])<br>
t++;<br>
}<br>
}<br>
void Merge_sort(int* A, int start, int end) {<br>
if (start &lt; end) {<br>
int q = (start + end) / 2;<br>
Merge_sort(A, start, q);<br>
Merge_sort(A, q + 1, end);<br>
MERGE(A, start, q, end);<br>
}<br>
}</p>
<p>int main() {<br>
int A<span class="chcklst-box fa fa-square-o fa-fw"></span> = { 9,6,3,7,8,1,4,2,8,7 };<br>
int a = 0, o = 5, e = 10;<br>
cout &lt;&lt; “Array sort” &lt;&lt; endl;</p>
<pre><code>Merge_sort(A,0,10);
</code></pre>
<p>for (int s = 0; s &lt; e; s++) {<br>
cout &lt;&lt; A[s] &lt;&lt; " ";<br>
}<br>
}</p>
<p>that is out put of code<br>
-858993460 -858993460 -858993460 -858993460 3 1 1 1 1 7</p>

ptrblck: <p>General C++ questions are better asked on StackOverflow. Use <code>torch.sort</code> to sort PyTorch tensors.</p>
How to build a multi modals models in PyTorch?

Teddy_Salas: <p>Hello everyone can anyone please tell me how to build a How to build a multi modal models in PyTorch from scratch?</p>

JamesTrick: <p>Hi Teddy</p>
<p>You may want to checkout <a href="https://pytorch.org/blog/introducing-torchmultimodal/" rel="noopener nofollow ugc">TorchMultiModal</a>.</p>
<p>If you’re wanting to combine an image and text for example, you can process these separately, and eventually join the two tensors (image and text) together using something like <code>torch.cat()</code>. We can’t help you too much further without more information.</p>

Teddy_Salas: <p>okay, thank you i will inform you later.</p>
How to run Pytorch code on Kaggle's TPU's

Schroter: <p>Hi All,</p>
<p>I have some code which I borrowed from <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">here</a>. As seen in the tutorial, it runs on the GPU. Would anyone know how to convert it to run on <a href="https://www.kaggle.com" rel="noopener nofollow ugc">Kaggle’s</a> TPU’s please.</p>
<p>Thanks &amp; Best Regards</p>
<p>Michael Schtoter</p>
TransformerEncoder with 1 TransformerEncoderLayer

dato: <p>Hi,</p>
<p>I just study a bit the code of the pytorch transformer module.</p>
<p>There, the Class TransformerEncoder is a stack of N TransformerEncoderLayers. So TransformerEncoder(encoder_layer, num_layers=1) with 1 Encoder Layer should give the same output as encoder_layer?</p>
<p>So, I would assume to get the same output for both ways:</p>
<pre><code class="lang-auto">

encoder_layer = nn.TransformerEncoderLayer(d_model=6, nhead=2, batch_first=True)
src = torch.rand(2, 5, 6)
out = encoder_layer(src)
print(out, out.shape)

transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=1)
out = transformer_encoder(src)
print(out, out.shape)
</code></pre>
<p>The output is always very different:</p>
<pre><code class="lang-auto">tensor([[[ 1.0687, -0.0628, -0.3718, -0.1207,  1.2548, -1.7682],
         [ 0.0136,  0.2716, -0.6997, -0.2500,  1.9374, -1.2729],
         [-0.6875, -0.3608,  0.7908, -0.0981,  1.7106, -1.3550],
         [-1.6746, -0.1693,  0.2136,  0.8932,  1.3815, -0.6443],
         [ 0.3343, -1.9192,  0.0965,  1.1378,  0.8232, -0.4726]],

        [[-1.0233,  0.4987,  0.9853,  0.2254,  0.9698, -1.6559],
         [-0.9034, -0.5267,  1.3985,  0.5957,  0.8218, -1.3858],
         [ 0.9843, -1.5308,  0.2510, -0.5477,  1.4139, -0.5707],
         [-0.1907, -0.0046, -1.3699,  1.2456,  1.2743, -0.9548],
         [-0.6701, -0.6789,  0.4082,  0.9170,  1.4406, -1.4168]]],
       grad_fn=&lt;NativeLayerNormBackward0&gt;) torch.Size([2, 5, 6])
tensor([[[-0.0860, -1.5762, -0.2599,  0.2899,  1.8222, -0.1900],
         [-0.1013, -0.0221, -0.6439, -0.0912,  2.0414, -1.1828],
         [-0.4802, -0.2750,  0.6492, -0.1569,  1.7459, -1.4830],
         [-1.6553, -0.2178,  0.4153,  0.7626,  1.4004, -0.7052],
         [ 0.2467, -2.0493,  0.3164,  0.9811,  0.7766, -0.2716]],

        [[-1.3599,  0.1510,  0.9750,  0.0135,  1.3658, -1.1453],
         [-1.0742, -0.2285,  1.3229,  0.5419,  0.8574, -1.4195],
         [ 0.7563, -1.5332,  0.4498, -0.0397,  1.3679, -1.0011],
         [-0.2690, -0.0986, -1.1021,  0.9688,  1.5993, -1.0985],
         [-0.3656, -0.8214,  0.2097,  0.7191,  1.6452, -1.3870]]],
       grad_fn=&lt;NativeLayerNormBackward0&gt;) torch.Size([2, 5, 6])
</code></pre>

J_Johnson: <p>Add 1 line:</p>
<pre><code class="lang-auto">encoder_layer = nn.TransformerEncoderLayer(d_model=6, nhead=3, batch_first=True)
encoder_layer.eval() #&lt;&lt;&lt; here
src = torch.rand(2, 5, 6)
out = encoder_layer(src)
print(out, out.size())

transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=1)
out = transformer_encoder(src)
print(out, out.size())
</code></pre>
<p>Dropout layers are randomly set every time data gets input.</p>
RuntimeError: Cannot access data pointer of Tensor that doesn't have storage

whatdhack: <p>Seeing  "RuntimeError: Cannot access data pointer of Tensor that doesn’t have storage "  during loss.backward  (inputs = [ a batchnorm param] , retain_graph=True) . Non-BN layers seem to have no such issue.   Would appreciate an explanation and a fix.</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing this issue?</p>
Python: torch.cuda.nccl.unique_id()

gakadam: <p>Hi,</p>
<p>I facing the same issue as in this post:</p>
<p><a href="https://github.com/pytorch/pytorch/issues/38019" rel="noopener nofollow ugc">torch.cuda.nccl.init_rank does not handle “uid” properly, causing runtime error · Issue #38019 · pytorch/pytorch (github.com)</a></p>
<p>I am running the same example, and see this assert triggering <a href="https://github.com/pytorch/pytorch/blob/9f47580ad772c0da7e37a799711563534a51cba5/torch/csrc/cuda/python_nccl.cpp#L114" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/9f47580ad772c0da7e37a799711563534a51cba5/torch/csrc/cuda/python_nccl.cpp#L114</a>.</p>
<p>Now id_len required is 128 Bytes.</p>
<p>I have torch v2.2.</p>
<p>Any solutions / work around for this?</p>
<p>Thank you.</p>
Torch not compiled with CUDA enabled, torch.cuda.is_available() returns True

134_Bhavana_Kalivemu: <p>I have installed pytorch enable with cuda versions (torch 2.1.1+cu121). But Iam facing “AssertionError: Torch not compiled with CUDA enabled”.</p>
<p>torch.cuda.is_available() = True.</p>
<p>I tried re-installing many times. But still failed. Can please someone help.<br>
Thankyou.</p>

ptrblck: <p>The binaries work for me:</p>
<pre><code class="lang-python">python -c "import torch; print(torch.__version__); print(torch.randn(1).cuda())"
2.1.1+cu121
tensor([0.0681], device='cuda:0')
</code></pre>
<p>so you might need to check if you have e.g. multiple PyTorch installations in your current environment and uninstall the wrong ones.</p>
Understanding gumbel_softmax implementation

curiousEngRes: <p>Hi,<br>
I am reviewing the gumbel_softmax implementation in PyTorch (<a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#gumbel_softmax" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional — PyTorch 2.1 documentation</a>). I am not able to understand, what this line of code is trying to accomplish:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc78bdeb701075fd0f22283da4c57f72351de272.png" data-download-href="https://discuss.pytorch.org/uploads/default/dc78bdeb701075fd0f22283da4c57f72351de272" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc78bdeb701075fd0f22283da4c57f72351de272_2_690x239.png" alt="image" data-base62-sha1="vsnJmVkPHyvmODSdWUi2IOjPvaO" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc78bdeb701075fd0f22283da4c57f72351de272_2_690x239.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc78bdeb701075fd0f22283da4c57f72351de272_2_1035x358.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc78bdeb701075fd0f22283da4c57f72351de272_2_1380x478.png 2x" data-dominant-color="EDECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1562×542 61 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can anyone please explain what functionality this line of code is providing.</p>

ptrblck: <p>The forward pass will return <code>y_hard</code> since <code>- y_soft.detach() + y_soft()</code> results in <code>0</code>:</p>
<pre><code class="lang-python"># forward
ret = y_hard - y_soft.detach() + y_soft
ret = y_hard + 0
ret = y_hard
</code></pre>
<p>while the backward pass will go through <code>y_soft</code> since <code>y_hard</code> is not attached to the computation graph and created via <code>torch.zeros_like().scatter_()</code>. The second term <code>y_soft.detach()</code> is explicitly detached and thus won’t pass any gradients to previous layers so only the third term <code>y_soft</code> will be used in the backward.</p>
Do inplace operations have implicit synchronize inside?

youkaichao1: <p>My simple test code:</p>
<pre><code class="lang-python">import time
import random

import torch
a = torch.ones((10000, 10000)).cuda()

stream_1 = torch.cuda.Stream()
stream_2 = torch.cuda.Stream()

with torch.cuda.stream(stream_1):
    for i in range(100):
        a.add_(1)

time.sleep(random.random() / 100)

with torch.cuda.stream(stream_2):
    a.zero_()
    # a = a + 1

torch.cuda.synchronize()
print(a.mean().item())
</code></pre>
<p>Using <code>a.zero_()</code>, the order of operations between screams is ensured, and the output is deterministically <code>0</code>;<br>
Using <code>a = a + 1</code>, the order of operations between screams is not deterministic (and is expected, because I’m using this code to understand cuda behavior).</p>
<p>The output result seems to indicate that inplace operations have  implicit synchronization inside.</p>

ptrblck: <p>If you remove the <code>sleep</code> call you would get <code>99</code> (or a random number which would depend on the kernel execution order) so I don’t fully understand your claim:</p>
<aside class="quote no-group" data-username="youkaichao1" data-post="1" data-topic="192009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/f0a364/48.png" class="avatar"> youkaichao1:</div>
<blockquote>
<p>Using <code>a.zero_()</code>, the order of operations between screams is ensured, and the output is deterministically <code>0</code>;</p>
</blockquote>
</aside>

youkaichao1: <p>What is your runtime environment? I always get <code>0</code> printed, on RTX 4090 GPU.</p>

ptrblck: <p>I’m using a 3090, but I doubt the actually used GPU is that important and would rather try to make sure your CPU is fast enough to schedule all kernels in different streams.</p>

youkaichao1: <p>I use two servers to test the code, and the output is always <code>0</code>. One with RTX 4090, and another with V100. Do you really get random output?</p>
Export all operands from each conv layer

makkkiiii: <p>Hello everyone,</p>
<p>I’m interested in exporting each operation from a convolutional layer. Is there a straightforward method to accomplish this? The goal is to create a table with columns for op1 and op2. These operands will be used to test the circuits I’ve constructed.</p>
<p>I’m aware of the approach involving adjustments to the model class and the development of routines to export everything. However, is there a simpler method, perhaps utilizing pre-existing PyTorch functionalities, such as monitoring and storing or something similar?</p>
<p>Thanks!</p>
Masked_select -- inverse operation?

mbp28: <p>Hello,</p>
<p>The <code>masked_select</code> function allows me to extract elements from a tensor. Given a tensor <code>A</code> and a boolean mask <code>mask</code>, <code>masked_select</code> will give me the indexed elements as a 1D-vector (call it <code>b</code>), such that</p>
<blockquote>
<p>b = torch.masked_select(A, mask)</p>
</blockquote>
<p>My question is if there is an efficient way would be to achieve the inverse operation? This is, given both <code>b</code> and <code>mask</code>, how can I recover <code>A</code> or rather a version of <code>A</code> where the indexed elements are in the right position and all-non indexed elements of <code>A</code> can be some constant, e.g. zero.</p>
<p>My use case is that <code>A</code> is about 15k x 15k dimensions and <code>b</code> holds about `200k elements. So I need something fast that should also work with auto-grad. When I am implementing a simple for loop, it takes ages to propagate back through the graph.</p>

ptrblck: <p>You could try it with this code sample:</p>
<pre><code class="lang-auto">A = torch.randn(10, 10)
mask = torch.ByteTensor(10, 10).random_(2)
b = torch.masked_select(A, mask)

C = torch.zeros(10, 10)
C[mask] = b
print(A, mask, C)
</code></pre>
<p>I’m not sure if it fits your needs regarding the functionality with auto-grad.<br>
Could you test it or give a small code sample showing what you would like to do?</p>

mbp28: <p>Hey, this works a treat! Thank you. It’s appropriate for my use case, the speed up is huge. It does work with autograd too. Thanks a lot!</p>

AlbertZeyer: <p>Note, the inverse operation is called masked_scatter.</p>
No dashboards are active error for Tensorboard

Xeonen: <p>I’m using PyTorch version 2.0.1+cu118 and Tensorboard version v2023.10.1002992421 with VS Code. My summary writer is initialized as</p>
<pre><code class="lang-auto">self.writer = SummaryWriter() # Trainer.py Line 25
</code></pre>
<p>Then writing is done as  following</p>
<pre><code class="lang-auto">self.writer.add_scalar(f"{step_type} Loss", loss.item(), epoch * len(loader) + pbar.n) # Trainer.py Line 78
</code></pre>
<p>Closing is done as following</p>
<pre><code class="lang-auto">self.writer.close() # Trainer.py Line 96
</code></pre>
<p>I can see files written as “events.out.tfevents.1700214365.DESKTOP-05JQ386.26868.1” within /runs/.</p>
<p>When I run the Tensorboard, I get " No dashboards are active for the current data set." I tried to setting every combination of directory. Checked forums, used code that should be working (check doodle.py) to no avail. I failed, failed, failed and decided to get a punching bag or therapy, maybe both.</p>
<p>The code I have prepared is uploaded to GitHub for easy access if needed.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/Xeonen/Cats-and-Dogs-PyTorch-Example">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/Xeonen/Cats-and-Dogs-PyTorch-Example" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/23809de9066e4da90134e7f89cc56a4a13b0f46b8f6c26b769568c5fcdae3909/Xeonen/Cats-and-Dogs-PyTorch-Example" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/Xeonen/Cats-and-Dogs-PyTorch-Example" target="_blank" rel="noopener nofollow ugc">GitHub - Xeonen/Cats-and-Dogs-PyTorch-Example: This is a working example on...</a></h3>

  <p>This is a working example on how to develop, train, and save a basic categorization CNN model. - GitHub - Xeonen/Cats-and-Dogs-PyTorch-Example: This is a working example on how to develop, train, a...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How do I solve erratic train loss and validation loss?

vivekjoshy: <p>I’m only in the first epoch and it’s already very erratic. It stays the same no matter how many epochs I try. There is a general trend downwards, but changing the learning rate seems to have no effect. I’ve tired learning rates from 0.001 to 0.0003.</p>
<p>My GNN has approximately 5M parameters and is training on around 8M samples of graph data (node features and edge indexes). Values are also normalized, so it can’t be that. I’m using the Adam optimizer.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/618045071dee583666308f90ec0cebdb1acf1d7d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/618045071dee583666308f90ec0cebdb1acf1d7d" title="train_batch_mae"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/618045071dee583666308f90ec0cebdb1acf1d7d_2_690x317.jpeg" alt="train_batch_mae" data-base62-sha1="dUx87PwMHrNXgexkZT1BLGLyVbf" width="690" height="317" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/618045071dee583666308f90ec0cebdb1acf1d7d_2_690x317.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/618045071dee583666308f90ec0cebdb1acf1d7d_2_1035x475.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/618045071dee583666308f90ec0cebdb1acf1d7d_2_1380x634.jpeg 2x" data-dominant-color="F5E8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">train_batch_mae</span><span class="informations">1920×883 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/9845a446f500548a8d4e4982147d1fe4c053c4f0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9845a446f500548a8d4e4982147d1fe4c053c4f0" title="validation_batch_mae"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/9845a446f500548a8d4e4982147d1fe4c053c4f0_2_690x317.jpeg" alt="validation_batch_mae" data-base62-sha1="lJ3MuH8n3R93n82NJEbmb3KZDB6" width="690" height="317" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/9845a446f500548a8d4e4982147d1fe4c053c4f0_2_690x317.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/9845a446f500548a8d4e4982147d1fe4c053c4f0_2_1035x475.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/9845a446f500548a8d4e4982147d1fe4c053c4f0_2_1380x634.jpeg 2x" data-dominant-color="E7F2FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">validation_batch_mae</span><span class="informations">1920×883 154 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
TorchStudio DeepSpeech model with Common Voice dataset

solmorth: <p>After loading data from the Mozilla Common Voice Corpus 15.0 dataset, I encounter the error shown below. What parameters should I choose to build the model? The only parameters that changed the output were n_features=199296 and n_hidden=1 (still an exception), which is likely an incorrect approach.</p>
<p>Im using MacBook Pro (intel version) MacOs 14</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99ccb272d917119c0337548aff99347ae49478c2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/99ccb272d917119c0337548aff99347ae49478c2" title="Zrzut ekranu 2023-11-16 o 21.34.25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99ccb272d917119c0337548aff99347ae49478c2_2_307x500.jpeg" alt="Zrzut ekranu 2023-11-16 o 21.34.25" data-base62-sha1="lWzC5OWEDqLMEZ2cHK5ARpLl6hA" width="307" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99ccb272d917119c0337548aff99347ae49478c2_2_307x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99ccb272d917119c0337548aff99347ae49478c2_2_460x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99ccb272d917119c0337548aff99347ae49478c2_2_614x1000.jpeg 2x" data-dominant-color="2B2929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Zrzut ekranu 2023-11-16 o 21.34.25</span><span class="informations">998×1624 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
nn.Module with 2 outputs in nn.Modulelist

palace_painter: <p>I wonder if it’s possible to add an nn.Module with 2 outputs(means it has 2 return value in the ‘‘forward’’ function) into an nn.Sequential and specify that I want to treat the 2nd output of Module1 as the input of Module2???</p>
<p>Code:<br>
nn.Sequential(<br>
Module1,<br>
Module2</p>
<h1><a name="module1-has-output1-and-output2-i-want-to-use-output2-only-to-continue-the-process-of-forward-propagation-1" class="anchor" href="#module1-has-output1-and-output2-i-want-to-use-output2-only-to-continue-the-process-of-forward-propagation-1"></a>Module1 has output1 and output2 I want to use output2 only to continue the process of forward propagation</h1>
<p>)</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-can-i-pass-multiple-inputs-to-nn-sequential-layers/50868">This topic</a> discusses workarounds, which might be applicable for your use case.</p>

palace_painter: <p>thanks for your reply! I will check the topic.</p>
Using older version of pytorch and torchtext on colab

Josh_C: <p>Hi,<br>
I was trying use an older version of pytorch and torchtext,<br>
but after installing them, when I try to import them, it shows the version is still the latest version not the older version that I just downloaded.</p>
<p>Does anyone have an idea for this?</p>
<p>Thanks in advance.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f0c38fbd0da8838c2745e4c04f355a99dfb23205.png" data-download-href="https://discuss.pytorch.org/uploads/default/f0c38fbd0da8838c2745e4c04f355a99dfb23205" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f0c38fbd0da8838c2745e4c04f355a99dfb23205.png" alt="image" data-base62-sha1="ylTA349TowrwnMY0Z0IZesWemB7" width="690" height="329" data-dominant-color="393A3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1416×677 13.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Run the <code>uninstall</code> command a few times until <code>pip</code> and <code>conda</code> both complain that no package can be found.</p>
Loading data from HDF5 files takes longer when preceded by memory-intensive operations

5hadytru: <p>Why might loading data (w/ random access) from a large HDF5 file be slower (and get slower and slower as training progresses) if, right before I call <strong>getitem</strong> on my custom Dataset, I temporarily fill GPU memory (almost completely) with intermediate torch.nn.functional.cosine_similarity computations? I am also doing several dist.gather and dist.broadcast operations, if that could be the culprit for whatever reason.</p>
Tensor shape problem

Mohamed_Abokahf: <p>Hallo I got this message while training lstm model :</p>
<p>UserWarning: Using a target size (torch.Size([28, 7, 1, 1])) that is different to the input size (torch.Size([28, 7])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size. I need to know how to solve this …</p>
<p>This is the training loop:</p>
<p>def train_model():<br>
# Train the model<br>
num_epochs = 10<br>
for epoch in range(num_epochs):<br>
for tra_batch in train_loader:<br>
X_train, y_train = tra_batch<br>
optimizer.zero_grad()<br>
output = model(X_train)<br>
loss = criterion(output, y_train.squeeze(-1))<br>
loss.backward()<br>
optimizer.step()</p>
<pre><code>    # Validate the model
    model.eval()
    with torch.no_grad():
        val_losses = []
        for val_batch in valid_loader:
            X_val, y_val = val_batch
            val_output = model(X_val)
            val_loss = criterion(val_output, y_val.squeeze(-1))
            val_losses.append(val_loss.item())

    # Log training and validation loss with W&amp;B (outside the validation loop)
    wandb.log({"Training Loss": loss.item(), "Validation Loss": sum(val_losses) / len(val_losses)}) 
    print(f'Epoch [{epoch}/{num_epochs}], Training Loss: {loss.item():.4f}, Validation Loss: {(sum(val_losses) / len(val_losses)):.4f}')
</code></pre>
Pytorch Simaese model using Lstm

silkey: <p>I am building a siamese model using Lstm, I have trained and tested the model but I condn’t inference it on sigle sample</p>
<p>Here’s the model</p>
<pre><code class="lang-auto">class SiameseLstm(nn.Module):
    def __init__(self):
        super(SiameseLstm, self).__init__()
        self.embedding_model = nn.Embedding(model_vocab_size, embedding_dim)
        self.embedding_model.weight.data.copy_(embedding_matrix_model)
        self.embedding_student = nn.Embedding(student_vocab_size, embedding_dim)
        self.embedding_student.weight.data.copy_(embedding_matrix_student)
        self.lstm = nn.LSTM(input_size=embedding_dim, hidden_size=128, num_layers=2, batch_first=True, bidirectional=True)
        self.flat = nn.Flatten()
        self.dropout = nn.Dropout(0.1)
        self.dense = nn.Linear(544, 1)
        self.activation = nn.Sigmoid()
        
    def forward_model(self, input):
        embedded = self.embedding_model(input)
        out,(h, c) = self.lstm(embedded)
        #hidden = torch.cat((h[-2,:,:], h[-1,:,:]), dim = 1)
        output = out[:, -1, :]
        output = self.flat(output)
        return output
    
    def forward_student(self, input):
        embedded = self.embedding_student(input)
        out,(h, c) = self.lstm(embedded)
        #hidden = torch.cat((h[-2,:,:], h[-1,:,:]), dim = 1)
        output = out[:, -1, :]
        output = self.flat(output)
        return output



    def forward(self, inp1, inp2):
        out1 = self.forward_model(inp1)
        out2 = self.forward_student(inp2)
        x3 = torch.subtract(out1, out2)
        x3 = torch.multiply(x3, x3)
        x1_ = torch.multiply(out1, out1)
        x2_ = torch.multiply(out2, out2)
        x4 = torch.subtract(x1_, x2_)
        x5 = torch.cdist(out1, out2)
        merged = torch.concatenate((x5, x4, x3), dim=-1)
        merged = self.dropout(merged)
        merged = self.dense(merged)
        merged = self.activation(merged)
        return merged
        
</code></pre>
<p>train method:</p>
<pre><code class="lang-auto">Model.train()
    for i, (inp1, inp2, label) in enumerate(train_dataloader):  
        inp1 = inp1.cuda()
        inp2 = inp2.cuda()
        label = label.cuda()
        optimizer.zero_grad()
        output = Model(inp1, inp2)
        #print('output', output)
        #print('label', label)
        loss = criterion(output, label)
        acc = calculate_acc(output, label)
        TrainLoss += loss.item()
        TrainAcc += acc.item()
        loss.backward()
        optimizer.step()
</code></pre>
<p>And this is the error in inference:</p>
<pre><code class="lang-auto">runtimeerror: mat1 and mat2 shapes cannot be multiplied (1x513 and 544x1)```</code></pre>
JIT function changed the default encoding of the built-in `open`

Devymex_Wang: <p>This is a piece of reproduced code.</p>
<pre><code class="lang-auto">import sys, torch
@torch.jit.script
def magic_func(x, y):
    return torch.tanh(x + y)
x, y = torch.rand(8, device="cuda"), torch.rand(8, device="cuda")
before_encoding = open(sys.argv[0]).encoding
for _ in range(int(sys.argv[1])):
    output = magic_func(x, y)
after_encoding = open(sys.argv[0]).encoding
print(f'{before_encoding} vs. {after_encoding}')
</code></pre>
<p>Test results:</p>
<pre><code class="lang-bash">$ python utf8.py 3
UTF-8 vs. ANSI_X3.4-1968
$ python utf8.py 2
UTF-8 vs. ANSI_X3.4-1968
$ python utf8.py 1
UTF-8 vs. UTF-8
</code></pre>
<p>What’s going on here?</p>

ptrblck: <p>This seems to be an <a href="https://stackoverflow.com/questions/74044994/nvrtc-nvrtccompileprogram-is-changing-the-output-of-locale-getpreferredencoding">nvrtc issue</a> and you might need to set the encoding explicitly to restore it again as a workaround.</p>
Torch.compile and Type Hinting

abcamiletto: <p>Hi there,</p>
<p>I’m wondering whether using type hinting in the code helps / speeds up the compiling operation or it does not matter at all.<br>
I think in the old jit days, that was making a difference. I wonder how that works now with torch.compile.</p>
<p>Any input is appreciated!</p>

bdhirsh: <p>I think the answer here is no.</p>
<p>One of the core differences between <code>torch.jit.script</code> and PT2 is that:</p>
<ul>
<li>
<p><code>torch.jit.script</code> does static analysis of your python code to try to optimize it. Here, having the user give the static analysis infra more info via type hints can potential help with optimizing the code</p>
</li>
<li>
<p><code>torch.compile</code> doesn’t do static analysis - it traces your code with example inputs, and specializes the optimized code that it generates off of all of the concrete metadata that it found when tracing the program with your particular inputs. One downside is that we might specialize and need to recompile later when you provide different inputs, but in general the idea/hope is that recompiles will happen rarely (as long as the user’s code is written in a reasonable way). The upside of tracing + specialization is that we don’t need type hints to get any extra data about the program, since we have the concrete values from your example inputs.</p>
</li>
</ul>
<p>There are a few cases though where the user can add extra info to their program to help torch.compile. One example is if you’re using <code>torch.compile(m, mode="reduce-overhead")</code> and performing training: PT2 will try to guess when the start of your training loop is so it knows where to start the cudagraph, but you can explicit tell it with the <code>torch._inductor.cudagraph_mark_step_begin()</code> API (more info here: <a href="https://dev-discuss.pytorch.org/t/cudagraphs-in-pytorch-2-0/1428" class="inline-onebox" rel="noopener nofollow ugc">CUDAGraphs in Pytorch 2.0 - compiler - PyTorch Dev Discussions</a>)</p>
Why forward_hook has mistake?

divided-by-7: <p>I want to visual the BatchNorm2d performance in resnet18, and I use register_forward_hook to do.</p>
<pre><code class="lang-auto">bn2d_inputs = {}
bn2d_outputs = {}
relu_inputs = {}
relu_outputs = {}
def get_layer_details(idx):
    def hook(layer, input, output):
        if isinstance(layer,torch.nn.BatchNorm2d):
            print(f"layer.__class__.__name__: {layer.__class__.__name__}")
            print(f"Input shape: {input[0].shape}")
            print(f"Output shape: {output.shape}")
            bn2d_inputs[str(idx) + "_" + layer.__class__.__name__] = input
            bn2d_outputs[str(idx) + "_" + layer.__class__.__name__] = output
            print("weight:",layer.weight)
            print("bias:",layer.bias)
        if isinstance(layer,torch.nn.ReLU):
            relu_inputs[str(idx) + "_" + layer.__class__.__name__] = input
            relu_outputs[str(idx) + "_" + layer.__class__.__name__] = output
    return hook

for idx,(name, layer) in enumerate(resnet18.named_modules()):
    layer.register_forward_hook(get_layer_details(idx))
</code></pre>
<p>I got the bn2d outputs all positive(&gt;0),it’s seem like the bn2d+relu output, but my code wants to get the bn2d output.<br>
What’s my mistake?</p>

ptrblck: <p><code>torchvision.models.resnet18</code> uses inplace <code>nn.ReLU</code> modules. Replace them with their out-of-place version and you should see the batchnorm output.</p>
How to manage sparseconvtensor randomness?

jisoo0-0: <p>Hi, … … I’m having some trouble to control the randomness of the model called Virconv(<a href="https://github.com/hailanyi/VirConv/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hailanyi/VirConv: Virtual Sparse Convolution for Multimodal 3D Object Detection</a>)</p>
<p>I checked and found that randomness started on the sparseconvtensor of spconv module.</p>
<p>Is there a reason why randomness is out of control when my environment is as follows?</p>
<p>ccimport           0.3.7<br>
certifi            2023.7.22<br>
charset-normalizer 3.3.2<br>
click              8.1.7<br>
cloudpickle        2.2.1<br>
cumm-cu111         0.2.9<br>
cytoolz            0.12.0<br>
dask               2023.6.0<br>
easydict           1.11<br>
fire               0.5.0<br>
fsspec             2023.9.2<br>
idna               3.4<br>
imagecodecs        2023.1.23<br>
imageio            2.31.4<br>
importlib-metadata 6.0.0<br>
lark               1.1.8<br>
llvmlite           0.36.0<br>
locket             1.0.0<br>
networkx           3.1<br>
ninja              1.11.1.1<br>
numba              0.53.1<br>
numpy              1.23.1<br>
packaging          23.1<br>
partd              1.4.1<br>
pccm               0.3.4<br>
pcdet              0.3.0+ca5ad85 /home/workspace/jisoocv/KITTI/VirConv<br>
Pillow             10.0.1<br>
pip                23.3<br>
portalocker        2.8.2<br>
prefetch-generator 1.0.3<br>
protobuf           4.25.0<br>
pybind11           2.11.1<br>
PyWavelets         1.4.1<br>
PyYAML             6.0.1<br>
requests           2.31.0<br>
scikit-image       0.19.3<br>
scipy              1.11.3<br>
setuptools         68.0.0<br>
six                1.16.0<br>
spconv-cu111       2.1.20<br>
tensorboardX       2.6.2.2<br>
termcolor          2.3.0<br>
tifffile           2023.4.12<br>
toolz              0.12.0<br>
torch              1.8.1+cu111<br>
torch-scatter      2.0.8<br>
torchaudio         0.8.1<br>
torchvision        0.9.1+cu111<br>
tqdm               4.66.1<br>
typing_extensions  4.8.0<br>
urllib3            2.1.0<br>
wheel              0.41.2<br>
zipp               3.11.0</p>
<p>pytorch 1.8.1<br>
cudnn8<br>
Ubuntu 18.04<br>
NVIDIA CUDA 11.1</p>
<p>with two A6000 gpus(I’ve tried to train on multi-GPU mode with distributed learning).</p>

ptrblck: <p>Are you seeing the same effect using pure PyTorch?<br>
If not, you might want to follow up with the authors of this custom tensor class.</p>

jisoo0-0: <p>The pure pytorch seems ok in this environment,…<br>
I have followed the custom tensor class for spconv(<a href="https://github.com/traveller59/spconv" class="inline-onebox" rel="noopener nofollow ugc">GitHub - traveller59/spconv: Spatial Sparse Convolution Library</a>), but it seems no problem with how they use torch. I don’t know where the randomness came from…</p>

ptrblck: <p>This repository is not only using PyTorch but also implements custom backend operations as seen <a href="https://github.com/traveller59/spconv/blob/125a194d895b1bc3ad6ff907bc72641548397b32/spconv/csrc/sparse/convops.py#L188">here</a>, which is why I suggested checking these.</p>
Differences cuda_ctc_decoder and ctc_decoder

Macsim2: <p>Hi everyone<br>
I’m just curious about cuda_ctc_decoder and ctc_decoder in src/torchaudio/model/decoder/*.py (git hash ccd78ffba3208b4a9f5face4bd76e942ddc2d4a3)</p>
<ol>
<li>
<p>actually What I want to know is that why I couldn’t get <code>timesteps</code> like a output of ctc_decoder when I use cuda_ctc_decoder?<br>
It seem a differences of beamsearch score method(Lexicon vs Prefix), but Isn’t it possible getting a timesteps with cuda_ctc_decoder?</p>
</li>
<li>
<p>Another question is that How can I use ctc_decoder without n-gram LM…?<br>
like a cuda_ctc_decoder, only with token_list, Can I use ctc_decoder.py…?</p>
</li>
</ol>
<p>thank you</p>
Recommend wrrting more in NLP tutorials(for scratch up session)

applepieiris: <p>I am reading the NLP tutorial of official documents in <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html</a> and I think this tutorial is very good to understand RNN. However, I think it is not enough to learn about the pytorch with text processing. In this tutorial, it don’t use the recommend way to build the dataset such as <code>torch.utils.data.Dataset</code>, also it don’t show how to use dataloader to load the batch data to train the network.<br>
I recommend add this part into the tutorial and <a href="https://vl8r.eu/posts/2019/11/23/a-pytorch-rnn-with-variable-sequence-lengths/" rel="noopener nofollow ugc">this Blog</a> has introduced some details of implementing it. Also, the tutorial can introduce more about <code>pack_padded_sequence, opad_packed_sequence</code> because when it makes me really confused when I want to create the dataloader with variable length tensors. Although your forums has many questions on this topic, maybe this kind of common problems can be added into the professional tutorials other than letting user to identify the solutions from dozons of reference documents.<br>
Thanks!</p>
Single multi card training, suddenly the process is closed, want to know why

yangh0597: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d2b6580a91e07abe41a1086d5e31812d1efb444.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3d2b6580a91e07abe41a1086d5e31812d1efb444" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d2b6580a91e07abe41a1086d5e31812d1efb444_2_690x324.jpeg" alt="image" data-base62-sha1="8J86l4VXHNohlycMxfEVIgA4n9q" width="690" height="324" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d2b6580a91e07abe41a1086d5e31812d1efb444_2_690x324.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d2b6580a91e07abe41a1086d5e31812d1efb444_2_1035x486.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d2b6580a91e07abe41a1086d5e31812d1efb444_2_1380x648.jpeg 2x" data-dominant-color="2F4A5C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1568×738 153 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Do you have any feedback for PyTorch Distributed?

irisz: <p>Hey folks, the PyTorch Distributed team is planning for work on H2 and we would love to hear your feedback on areas we should focus on and better collaborate. Please msg me directly, or If you’re interested in chatting, I can setup a call.</p>

yang623: <p>Hi，I found a mistake in the pytorch.doc2.1, but I can’t find the feedback, so I leave a message here.</p>
<p><span class="mention">@torch.inference_mode</span>  will return a instance of inference_mode  but not a decorate method inference to func, so a parenthesis is missing here.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a332ed8f30970b099479eef9c058e8d0f9f6d156.png" data-download-href="https://discuss.pytorch.org/uploads/default/a332ed8f30970b099479eef9c058e8d0f9f6d156" title="Screenshot_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a332ed8f30970b099479eef9c058e8d0f9f6d156_2_429x500.png" alt="Screenshot_1" data-base62-sha1="nhIWiH7YaQ69Zm1ZtbvTLrPIBFA" width="429" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a332ed8f30970b099479eef9c058e8d0f9f6d156_2_429x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a332ed8f30970b099479eef9c058e8d0f9f6d156_2_643x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a332ed8f30970b099479eef9c058e8d0f9f6d156_2_858x1000.png 2x" data-dominant-color="F1F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_1</span><span class="informations">929×1082 64.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

irisz: <p>Thanks for your feedback. I will forward this over to our document engineers. Thanks again!</p>
Installing C++ distributions of PyTorch on Windows

mattpats: <p>I have followed the instructions provided by Pytorch to installing C++ distributions of PyTorch:</p>
<p><a href="https://pytorch.org/cppdocs/installing.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/installing.html</a></p>
<p>However, this command produces an error:</p>
<p><code>cmake --build . --config Release</code></p>
<p>I have tried adjusting many things, and I always get an error. I am starting to wonder if it is even possible to do this with VS Code, or do I need to use Visual Studio (which I don’t have and don’t want to use) on my Windows instead. I say this because on Pytorch’s GitHub:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/pytorch">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/pytorch" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/ae6d78350035c8f6832927cd0693a31ebca994f6331b0ba4e92e67f0c4fa7fa2/pytorch/pytorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorch/pytorch" target="_blank" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with...</a></h3>

  <p>Tensors and Dynamic neural networks in Python with strong GPU acceleration - GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>They mention that “the build tools do not come with Visual Studio Code by default.”</p>
<p>Based on the generated CMakeCache.txt file, the build system I am using is ninja and the compiler is G++. I am currently trying to install the CPU-version but am open to the CUDA-version if that would be easier for some reason.</p>
How to customize the module `state_dict` & `load_state_dict`

huahuanZ: <p>In my model, there is a fixed and pretrained sub-module which I don’t want to store it in <code>state_dict()</code> for smaller checkpoint size.</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self, ref_model):
        super().__init__()
        self.fc = nn.Linear(3,4)  # trainable parameters
        self.ref_m = ref_model    # fixed parameters
        for m in self.ref_m.parameters():
            m.requires_grad=False

ref_model=...
model = MyModel(ref_model)
# how to exclude ref_model from state_dict()?
checkpoint = model.state_dict()
</code></pre>

ptrblck: <p>You could manually delete the unwanted <code>ref_model</code> parameters and buffers directly as seen here:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(3,4)  # trainable parameters
        self.ref_m = nn.Linear(10, 10)    # fixed parameters
        for m in self.ref_m.parameters():
            m.requires_grad=False

model = MyModel()
checkpoint = model.state_dict()

keys_to_delete = []
for key in checkpoint:
    if 'ref_m' in key:
        keys_to_delete.append(key)
for key in keys_to_delete:
    del checkpoint[key]
print(checkpoint)
</code></pre>
<p>Note however, that you won’t be able to use <code>model.load_state_dict(checkpoint)</code> with the default setting of <code>strict=True</code> anymore since obviously keys are missing. In this case you would have to use <code>strict=False</code> and make sure that only the expected <code>ref_m</code> keys are missing.</p>
<p>I assume <code>ref_model</code> is an entire model and not just buffers, as you could use <code>self.register_buffer</code> with <code>persistent=False</code> in the latter case and could avoid manipulating the <code>state_dict</code>.</p>

huahuanZ: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, my current workaround is just as you suggested. I wonder if it’s possible to implement it in a more “elegant” way, i.e. we deal the case inside <code>MyModel</code>, such that the interface to upper invoking could be like as usual.</p>
<p>I also tried overwriting the <code>state_dict</code> and <code>load_state_dict</code> of the model. But it seems these functions is never used if we wrapped the model with DDP or DP.</p>

ptrblck: <p>I don’t know if there is a better approach. While internally state_dict hooks are used, they can change at any time and could thus easily break your code if you depend on them.</p>

Vijetha_Gattupalli: <p>I tried something like this from inside my <code>Model</code> which is a child of <code>torch.nn.Module</code>. It seems to work both when wrapped in DDP and without.</p>
<p>Could you let me know if you see any issues in this approach?</p>
<pre><code class="lang-auto">def state_dict(self, destination=None, prefix='', keep_vars=False):
    my_state_dict = super().state_dict(destination, prefix, keep_vars)
    ret_state_dict = {k: my_state_dict[k] for k in my_state_dict if 'lora_' in k} # your logic here
    return ret_state_dict
</code></pre>