akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to store data to torch.multiprocessing.Manager().list or dict in multiprocessing

Joeyoung: <p>Hi,</p>
<p>The following code shows the variable ‘data’ doesn’t store any updated data.<br>
How can I make it updated?</p>
<p>Thanks!</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
from torch.distributed import init_process_group, destroy_process_group

def func(local_rank, data):
     torch.cuda.set_device(local_rank)
     init_process_group( backend="nccl", rank=local_rank, world_size=2, init_method="tcp://localhost:12355" )
     data[local_rank]["a"] = [1,2,3,4,5]
     print("local_rank=", local_rank, ", data=", data[local_rank]) # prints {}
     destroy_process_group()

if __name__ == "__main__":
     with mp.Manager() as mgr:
          data = mgr.list([{},{}])
          mp.spawn(func, nprocs=2, args=(data,))
          print(data) # prints [{},{}]
</code></pre>
Strange Unet artifact

Vathsan: <p>I am using a Unet model (a simple encoder with average pooling and a decoder using <a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html" rel="noopener nofollow ugc">ConvTranspose2d</a> function) for image upsampling (super-resolution). A pair of images (input on the left and target on the middle) is used for training as shown below. Every other column from the target image is removed and zero-padded to come up with the input image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d26d7003a9613ace945f1e3c8ec9fb6e7637bbef.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d26d7003a9613ace945f1e3c8ec9fb6e7637bbef" title="Screenshot from 2024-10-11 23-34-14"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d26d7003a9613ace945f1e3c8ec9fb6e7637bbef_2_690x345.jpeg" alt="Screenshot from 2024-10-11 23-34-14" data-base62-sha1="u1wJJCWjXeWHfC9EKJSRwy2ZHMH" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d26d7003a9613ace945f1e3c8ec9fb6e7637bbef_2_690x345.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d26d7003a9613ace945f1e3c8ec9fb6e7637bbef_2_1035x517.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d26d7003a9613ace945f1e3c8ec9fb6e7637bbef_2_1380x690.jpeg 2x" data-dominant-color="868686"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-10-11 23-34-14</span><span class="informations">1920×962 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>During training, I could see the vertical line artifacts (image on the right) on the reconstructed image (model output) upon zooming in.</p>
<p>What can be done to rectify the artifacts? I am using L1 and L2 norms as loss functions. The training set contains 6000, 672 x 1024 grayscale images.</p>

J_Johnson: <p>Your output and target look close to the same, just flipped vertically. You sure you aren’t flipping one of them before viewing?</p>

Soumya_Kundu: <p>Pretty big images, are you patching them (if so experiment a bit more here?) or feeding them whole?</p>

Vathsan: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a> Yes, the output looks close to the target. I have uploaded the full-size validation target and output images at <a href="https://ibb.co/album/5GB5rY" class="inline-onebox" rel="noopener nofollow ugc">Unet — ImgBB</a>. You can see the artifacts when you zoom in on the output.</p>
<p><a class="mention" href="/u/soumya_kundu">@Soumya_Kundu</a>, I am feeding the whole image. I will give reducing/patching the images a try and see if that works.</p>
<p>Thanks for the response guys!</p>
Guarantee traversal order for optimiser states

Jackmin801: <p>Is there an easy way to consistently traverse the optimiser states?<br>
What I want to do is copy the optim states on gpu into a big flat tensor on cpu.<br>
In order to do this, I need to have an iterator that traverses the optimiser states in the same order everytime.</p>
<p>My current solution is to use <code>[optimizer.state[i] for i in optimizer.param_groups[0]['params']</code></p>
<pre data-code-wrap="python"><code class="lang-python">numels = sum(i.numel() for i in optimizer.param_groups[0]['params'])
flat_tensor = torch.empty((2 * numels,))

offset = 0
for i in optimizer.param_groups[0]['params']:
    for state_key in ['exp_avg', 'exp_avg_sq']:
        _target = optimizer.state[i][state_key]
        _temp_view = flat_tensor.as_strided(_target.shape, _target.stride(), offset)
        _temp_view.copy_(_target, non_blocking=True)
        offset += _target.numel()
</code></pre>
Why does pytorch always use static cuda runtime library now?

nooodles2023: <p>In old version, only use static before cuda-5.5</p><aside class="onebox githubcommit" data-onebox-src="https://github.com/pytorch/pytorch/commit/675b579bf995227cc327a4d72214ef6d17bfa8ce#diff-c6c4630a9acd3ec0a660c55c12b6e24d17da4ab5f79e0de2b1d91cd7d0e463a1L856">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/commit/675b579bf995227cc327a4d72214ef6d17bfa8ce#diff-c6c4630a9acd3ec0a660c55c12b6e24d17da4ab5f79e0de2b1d91cd7d0e463a1L856" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Commit">
    <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M10.86 7c-.45-1.72-2-3-3.86-3-1.86 0-3.41 1.28-3.86 3H0v2h3.14c.45 1.72 2 3 3.86 3 1.86 0 3.41-1.28 3.86-3H14V7h-3.14zM7 10.2c-1.22 0-2.2-.98-2.2-2.2 0-1.22.98-2.2 2.2-2.2 1.22 0 2.2.98 2.2 2.2 0 1.22-.98 2.2-2.2 2.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/commit/675b579bf995227cc327a4d72214ef6d17bfa8ce" target="_blank" rel="noopener nofollow ugc">cmake wrapper (#8797)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        committed <span class="discourse-local-date" data-format="ll" data-date="2018-06-22" data-time="19:29:25" data-timezone="UTC">07:29PM - 22 Jun 18 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/ssnl" target="_blank" rel="noopener nofollow ugc">
          <img alt="ssnl" src="https://avatars.githubusercontent.com/u/5674597?v=4" class="onebox-avatar-inline" width="20" height="20">
          ssnl
        </a>
      </div>

      <div class="lines" title="changed 17 files with 2064 additions and 2049 deletions">
        <a href="https://github.com/pytorch/pytorch/commit/675b579bf995227cc327a4d72214ef6d17bfa8ce" target="_blank" rel="noopener nofollow ugc">
          <span class="added">+2064</span>
          <span class="removed">-2049</span>
        </a>
      </div>
    </div>
  </div>
</div>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">if(NOT CUDA_VERSION VERSION_LESS "5.5")
  cuda_find_library_local_first(CUDA_cudart_static_LIBRARY cudart_static "static CUDA runtime library")
  mark_as_advanced(CUDA_cudart_static_LIBRARY)
endif()
</code></pre>
Interpolate + Conv1D to Match Res Connections Shape

Inigo_Parra: <p>Hi guys,<br>
I was wondering if this forward pass is correct to align the dims of the residual connections:</p>
<pre><code class="lang-auto">    def forward(self, x):
        # print(f"Decoder input: {x.shape}")
        x, self_attn = self.seq_attention(x)
        # print(f"After seq_attn: {x.shape}")
        x = self.activation(self.norm1(self.deconv1(x)))
        # print(f"After deconv1: {x.shape}")
        x = self.activation(self.norm2(self.deconv2(x)))
        # print(f"After deconv2: {x.shape}")
        residual_1 = x
        x = self.activation(self.norm3(self.deconv3(x)))
        # print(f"After deconv3: {x.shape}")
        x = self.activation(self.norm4(self.deconv4(x)))
        # print(f"After deconv4: {x.shape}")
        x = x + F.interpolate(residual_1, size=x.shape[2:], mode='nearest')
        # print(f"After residual interpolation 1: {x.shape}")

        x = self.final_layer(x)
        x = F.interpolate(x, size=self.final_shape, mode='linear', align_corners=False)
        x = self.tanh(x)
        # print(f"After final transform, interpolate, and tanh: {x.shape}")
        return x, self_attn
</code></pre>
<p>I would greatly appreciate any comments and potential pros and cons.<br>
Thank you!<img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
Transposed Convolutions

Inigo_Parra: <p>Hi!</p>
<p>I am having a hard time understanding the ConvTranspose1D layer. I would really appreciate if someone could help me understand the transposed convolutions better (in detail if possible)</p>
<p>Thank you!</p>

ptrblck: <p>The <a href="https://github.com/vdumoulin/conv_arithmetic/tree/master?tab=readme-ov-file#transposed-convolution-animations"><code>conv_arithmetic</code> repository</a> could help as it’s visualizing how the layers perform their computation. The linked paper also describes it in more detail.</p>

Inigo_Parra: <p>This is great! Thank you!</p>
Colab T4，always running

shiqi_ning: <p>I followd <code>https://captum.ai/tutorials/Llama2_LLM_Attribution</code>.<br>
But In Colab T4 gpu，It cost 20 mins,but still running.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/916a1d039af61c65d820c1efb194c3e54187d28d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/916a1d039af61c65d820c1efb194c3e54187d28d" title="c0442089bda380f061cd3799bff330f"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/916a1d039af61c65d820c1efb194c3e54187d28d_2_690x305.jpeg" alt="c0442089bda380f061cd3799bff330f" data-base62-sha1="kKozXtKn7hgISrYs3sqJO5Fuwt7" width="690" height="305" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/916a1d039af61c65d820c1efb194c3e54187d28d_2_690x305.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/916a1d039af61c65d820c1efb194c3e54187d28d_2_1035x457.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/916a1d039af61c65d820c1efb194c3e54187d28d_2_1380x610.jpeg 2x" data-dominant-color="F3F6EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">c0442089bda380f061cd3799bff330f</span><span class="informations">2702×1195 224 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Issues with PPO Tutorial and Custom Dictionary Observation Space

rgbarishian: <p>I am learning how to use TorchRL and am trying to implement my own environment while following the <a href="https://pytorch.org/rl/stable/tutorials/coding_ppo.html" rel="noopener nofollow ugc">PPO Tutorial</a>. I even used TorchRL’s own <a href="https://github.com/pytorch/rl/blob/main/examples/envs/gym_conversion_examples.py" rel="noopener nofollow ugc">Gymnasium Conversion Examples</a></p>
<p>My new environment is defined as the following</p>
<pre><code class="lang-auto">class MyEnv(gym.Env):
    def __init__(self):
        self.observation_space = spaces.Dict(
            obs0=spaces.Box(-1, 1, (2,)),
            obs1=spaces.Box(-1, 1, (3,))
        )
        self.action_space = spaces.Box(-1, 1, (1,))

    def step(self, action):
        return self.observation_space.sample(), 1, False, False, {}

    def reset(self, **kwargs):
        return self.observation_space.sample(), {}
</code></pre>
<p>I then follow the PPO tutorial and change this line from</p>
<pre><code class="lang-auto">base_env = GymEnv("InvertedDoublePendulum-v4", device=device)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">base_env = GymWrapper(MyEnv())
</code></pre>
<p>It seems to run fine until this line</p>
<pre><code class="lang-auto">print("Running policy:", policy_module(env.reset()))
</code></pre>
<p>but get this error:<br>
"KeyError: “Some tensors that are necessary for the module call may not have been found in the input tensordict: the following inputs are None: {“observation”}.”</p>
<p>This makes sense because after calling env.reset(), observation is not a key. I then updated the observation space to this</p>
<pre><code class="lang-auto"> self.observation_space = spaces.Dict(
    observation(
            obs0=spaces.Box(-1, 1, (2,)),
            obs1=spaces.Box(-1, 1, (3,))
        ))
</code></pre>
<p>but it returns this error</p>
<pre><code class="lang-auto">...
File /home/myuser/.../site-packages/torch/nn/modules/linear.py", line 259, in itialize_parameters
    self.in_features = input.shape[-1]
IndexError: tuple index out of range
</code></pre>
<p>For some reason the input is still in TensorDict format. All other tutorials using gym environments show the input data as a tensor. While a tensor has a shape, the TensorDict’s shape is torch.Size(<span class="chcklst-box fa fa-square-o fa-fw"></span>).</p>

rgbarishian: <p>I have solved this using the <a href="https://pytorch.org/rl/stable/reference/generated/torchrl.envs.transforms.CatTensors.html#torchrl.envs.transforms.CatTensors" rel="noopener nofollow ugc">CatTensors</a> class.</p>
<p>Following the PPO tutorial, I added the CatTensors transform to the Compose object.</p>
<pre><code class="lang-auto">env = TransformedEnv(
    base_env,
    Compose(
        # normalize observations
        CatTensors(in_keys=["obs0", "obs1", out_key="observation])
        ObservationNorm(in_keys=["observation"]),
        DoubleToFloat(),
        StepCounter(),
    ),
)
</code></pre>
Variable length time series data

Sai1: <p>Hello all, I got two questions,</p>
<ol>
<li>I have time series data and each entry has variable time steps. I understand that padding is one way to deal with this. But I wanted to ask if there are better ways to handle this ?</li>
<li>In text data, assume the embedding size is 300. If I’m using LSTMs, Usually, the hidden dimension is smaller than the embedding dimension. However, in my time series data, I have 11 channels (they correspond to embeddings dim in text data). Now is it a good idea to use an RNN with a hidden dim of say 128. I feel like this is not the way to go.<br>
Any suggestions are welcome.</li>
</ol>
<p>Thanks in advance.</p>

vdw: <p>You could organizes your sequences in such a way that all sequences in a batch have the same length using a <a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">sampler</a>.</p>
How to optimize a single index in a PyTorch tensor?

MartensCedric: <p>I have code like this optimizing an (N,3) tensor.</p>
<pre data-code-wrap="python"><code class="lang-python">deform_verts = torch.full(verts_shape, 0.0, device=device, requires_grad=True)
optimizer = torch.optim.SGD([deform_verts], lr=5e-2, momentum=0.9)
</code></pre>
<p>However, what I really want is to optimize a single index.</p>
<pre data-code-wrap="python"><code class="lang-python">deform_verts = torch.full(verts_shape, 0.0, device=device, requires_grad=True)
optimizer = torch.optim.SGD([deform_verts[275][1]], lr=5e-2, momentum=0.9)
</code></pre>
<blockquote>
<p>ValueError: can’t optimize a non-leaf Tensor</p>
</blockquote>
<p>How can I optimize this single index?</p>
How can I run backward() on individual image pixels without causing an error of trying to backward through the graph a second time?

MartensCedric: <p>I have this code</p>
<pre data-code-wrap="python"><code class="lang-python">...
vertex_id = 275

deform_verts.retain_grad() # input
predicted_silhouette.retain_grad() # output

impact_img = torch.zeros_like(predicted_silhouette, requires_grad=False)
for i in range(image_size):
  for j in range(image_size):
      pixel = predicted_silhouette[i][j]
      pixel.retain_grad()
      pixel.backward()
      impact = deform_verts.grad[vertex_id] 
      impact_img[i][j] += impact.sum()


plt.imshow(impact_img.detach().cpu().numpy())
</code></pre>
<p>I’m trying to create an image based off how a single entry of deform_verts affects the entire image. For this purpose, I go through every pixel of the output image and call <code>.backward()</code> and insert its gradient into a new image for visualization purposes. However, when I call <code>backward()</code> on a first pixel, I cannot call it on a second one. I tried to use <code>retain_graph</code> but I don’t think this does what I want since the image looks far from what I expected.</p>

ptrblck: <p>Use <code>retain_graph=True</code> in the <code>backward</code> call to keep the computation graph alive.</p>

MartensCedric: <p>Unfortunately, that didn’t work as I think it accumulates the gradients. What I did that worked was to rerun everything at every iteration</p>
<pre data-code-wrap="python"><code class="lang-python">vertex_id = 275


impact_img = torch.zeros_like(predicted_silhouette, requires_grad=False)
for i in range(image_size):
  for j in range(image_size):
    src_mesh = mesh_good_start
    verts_shape = src_mesh.verts_packed().shape
    deform_verts = torch.full(verts_shape, 0.0, device=device, requires_grad=True)

    new_src_mesh = src_mesh.offset_verts(deform_verts)
    predicted_silhouette = meshes_to_oc(new_src_mesh, camera) 
    pixel = predicted_silhouette[i][j]
    pixel.retain_grad()
    pixel.backward()
    impact = deform_verts.grad[vertex_id] 
    impact_img[i][j] += impact[1]


plt.imshow(impact_img.detach().cpu().numpy())
plt.colorbar()
</code></pre>

MartensCedric: <p>I finally got it to work. I needed <code>zero_()</code></p>
<pre data-code-wrap="python"><code class="lang-python">src_mesh = mesh_good_start
verts_shape = src_mesh.verts_packed().shape
deform_verts = torch.full(verts_shape, 0.0, device=device, requires_grad=True)
new_src_mesh = src_mesh.offset_verts(deform_verts)
predicted_silhouette = meshes_to_oc(new_src_mesh, camera)

impact_img = torch.zeros_like(predicted_silhouette, requires_grad=False)
for i in range(image_size):
  for j in range(image_size):
    pixel = predicted_silhouette[i][j]
    pixel.retain_grad()
    pixel.backward(retain_graph=True)
    impact = deform_verts.grad[vertex_id] 
    impact_img[i][j] += impact[1]
    deform_verts.grad.zero_()


</code></pre>
FSDP MixedPrecision vs AMP autocast?

anonymous_user_83259: <p>Given an arbitrary fp32 nn.Module that fits on a single GPU, is there a full enumeration of the differences between</p>
<ul>
<li>MixedPrecision(torch.bfloat16, torch.float32)</li>
<li>torch.autocast(“cuda”, dtype=torch.bfloat16)</li>
</ul>
<p>in <strong>computation</strong>?</p>
<p>I noticed that certain modules/methods do not execute with correct precision using FSDP MixedPrecision, so there exists a difference.</p>
Exporting multiple graph specializations

ajl95: <p>Hello,</p>
<p>I have a use case for <code>torch.compile()</code> for a model with varying input shapes, where trying to compile using dynamic shapes fails due to some of the operations in my model. To work around this, I am relying upon compiling multiple specializations of the graph, one for each of a small set of fixed input shapes, which I then pad my inputs to. This solution works well and speeds things up a lot, but it makes exporting the model tricky, forcing me to recompile all of the specializations ahead of time every time I want to do inference. I have two questions:</p>
<ol>
<li>Do multiple specializations of the compiled graph each store their own copy of the model weights on the GPU, or are they able to share the weights between them? Empirically I don’t seem to see the GPU memory usage increasing but it’s a little hard to tell since the memory usage varies depending on the input shape, so I wanted to check exactly what is going on here.</li>
<li>If it is indeed the case that the multiple specializations are able to share the weights, is there any way to replicate this behaviour using <code>torch.export</code>? Presumably if I exported the model for each input shape in turn and loaded them all back into a process, I would end up with n copies of the weights on the GPU?</li>
</ol>
<p>Many thanks,<br>
Angus</p>

ajl95: <p>Just bumping this in case anybody has any insight – some ability to export multiple graph specialisations at once would be really handy.</p>
AttributeError: module 'utils' has no attribute 'collate_fn'

Armin_Dashti: <p>I exactly copied this code from this <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#testing-forward-method-optional" rel="noopener nofollow ugc">tutorial</a> but I got error.<br>
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)<br>
dataset = PennFudanDataset(‘PennFudanPed’, get_transform(train=True))<br>
data_loader = torch.utils.data.DataLoader(<br>
dataset, batch_size=2, shuffle=True, num_workers=4,<br>
collate_fn=utils.collate_fn)</p>
<p>The error:<br>
AttributeError: module ‘utils’ has no attribute ‘collate_fn’</p>

Armin_Dashti: <p>I solved it.<br>
The problem was a conflict between the <a href="https://pypi.org/project/python-utils/" rel="noopener nofollow ugc">utils package</a> (Not related to PyTorch) and utils in PyTorch.<br>
You can see utils.py on <a href="https://github.com/pytorch/vision/tree/main/references/detection" rel="noopener nofollow ugc">Github</a>.<br>
I wonder why PyTorch didn’t mention this issue in its tutorial.</p>

Mohar_Kundu: <p>How did you solve it? can you provide the code here? please</p>

Armin_Dashti: <p><strong>utils</strong> is from the detection file.</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/vision/tree/main/references/detection">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/vision/tree/main/references/detection" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/vision/tree/main/references/detection" target="_blank" rel="noopener nofollow ugc">vision/references/detection at main · pytorch/vision</a></h3>

  <p><a href="https://github.com/pytorch/vision/tree/main/references/detection" target="_blank" rel="noopener nofollow ugc">main/references/detection</a></p>

  <p><span class="label1">Datasets, Transforms and Models specific to Computer Vision</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I just copied the file and then called it. like this:<br>
<code>from detection import utils</code><br>
Then you can call <strong>collate_fn</strong> like this:<br>
<code>utils.collate_fn</code><br>
You can download the file from github directly if you can’t find it.</p>

Mohar_Kundu: <p>it is showing error as “no module named detection”. I have copied the utils file from detection and have placed it in the working directory.</p>

Mohar_Kundu: <p>Resolved. Had to copy Detection folder in the working directory. Thanks.</p>

Armin_Dashti: <p>You’re welcome.<br>
It’s a better idea if you copy detection folder. Maybe you will need files in detection like engine.py</p>

wh0978449773: <p>Hi, is your main function available on GitHub?"</p>

Hoang-Hiep_Le: <p>Many thanks.<br>
I had the same problem and just solved it by following your guide.</p>
Passing torch.tensor to a CPP function

Milin_Bhade: <p>Is there a way to send torch.tensor to CPP file. And then use the tensor values as normal lists in CPP code</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">Custom C++ and CUDA Extensions Tutorial</a> might be a good start.</p>
IndexError: list index out of range using datasets.UCF101

AndyB: <p>Traceback (most recent call last):</p>
<p>File “/home/Andy/code/UCF101 with Pytorch.py”, line 33, in </p>
<p>train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True,</p>
<pre><code>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
</code></pre>
<p>File “/home/Andy /miniconda3/lib/python3.12/site-packages/torch/utils/data/dataloader.py”, line 351, in <strong>init</strong></p>
<p>sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]</p>
<pre><code>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
</code></pre>
<p>File “/home/Andy/miniconda3/lib/python3.12/site-packages/torch/utils/data/sampler.py”, line 143, in <strong>init</strong></p>
<p>if not isinstance(self.num_samples, int) or self.num_samples &lt;= 0:</p>
<pre><code>                 ^^^^^^^^^^^^^^^^
</code></pre>
<p>File “/home/Andy/miniconda3/lib/python3.12/site-packages/torch/utils/data/sampler.py”, line 150, in num_samples</p>
<p>return len(self.data_source)</p>
<pre><code>      ^^^^^^^^^^^^^^^^^^^^^
</code></pre>
<p>File “/home/Andy/miniconda3/lib/python3.12/site-packages/torchvision/datasets/ucf101.py”, line 122, in <strong>len</strong></p>
<p>return self.video_clips.num_clips()</p>
<pre><code>      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
</code></pre>
<p>File “/home/Andy/miniconda3/lib/python3.12/site-packages/torchvision/datasets/video_utils.py”, line 263, in num_clips</p>
<p>return self.cumulative_sizes[-1]</p>
<pre><code>      ~~~~~~~~~~~~~~~~~~~~~^^^^
</code></pre>
<p>IndexError: list index out of range</p>

AndyB: <p>OS:Ubuntu22.04LTS<br>
torch version:2.4.1</p>
How is pytorch tested before its release?

hlh: <p>Hello, I would like to ask:<br>
（1）How to conduct comprehensive testing before PyTorch release? Are there test cases and reports for unit testing, integration testing, and end-to-end testing?<br>
（2）For example, when testing the training function of PyTorch, do you compare the training loss and iteration time with the previous version?<br>
Looking forward to your answer, thank you very much</p>
Trusting the training of `RNN` with `PackedSequence`

Paulduf: <p>Hello,</p>
<p>I am starting with recurrent NNs, and I want to use <code>torch.nn.RNN</code>module and friends. I learned that they all accept <code>PackedSequence</code> and that this is the best way to wrap sequences of variable length for training.</p>
<p>My code basically looks like :</p>
<pre data-code-wrap="python"><code class="lang-python">optimizer = optim.Adam(
  ohemodel.parameters(), # basically a RNN + linear output layer
  lr=0.01
)

for epoch in range(1000):
    output = ohemodel.forward(X_packed.float()) # is a PackedSequence
    loss = criterion(output, y_up)
    print(f"loss {float(loss):.4f}")

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

</code></pre>
<p>My question is : can I be confident that the hidden state is reset to 0 / random for each new sequence in the batch. Should be the case as the <code>PackedSequence</code> wraps such information as a <code>batch_sizes</code> attribute, but how can I be sure everything is going as I guess ?</p>
<p>Bonus question: What if I want to change the initialization of the <code>h_0</code> for each sequence in the batch ? Is it a good practice anyway ? Where can I find good guidelines to start training RNNs (GRU, LSTM …)</p>
<p>Many thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
Usage of OpenXLA backend with NVIDIA GPUs

Maicon_Melo_Alves: <p>Hi folks!</p>
<p>I’m trying to execute some tests with the OpenXLA backend. I would like to use OpenXLA to generate code for NVIDIA GPUs; I’m not interested on generate code for TPU.</p>
<p>To install the torch_xla package, I’m following the instructions available at <a href="https://github.com/pytorch/xla/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/xla: Enabling PyTorch on XLA Devices (e.g. Google TPU)</a></p>
<p>Unfortunatelly, I’m getting the error “<strong>torch_xla-2.4.0-cp39-cp39-linux_x86_64.whl is not a supported wheel on this platform.</strong>” when trying to install the wheel <a href="https://storage.googleapis.com/pytorch-xla-releases/wheels/cuda/12.1/torch_xla-2.4.0-cp39-cp39-linux_x86_64.whl" rel="noopener nofollow ugc">https://storage.googleapis.com/pytorch-xla-releases/wheels/cuda/12.1/torch_xla-2.4.0-cp39-cp39-linux_x86_64.whl</a> on my machine.</p>
<p>I run PyTorch on a NVIDIA GPU on this same machine without any issues.</p>
<p>Has someone a clue about this problem?</p>
How to utilize shared GPU

nim9405: <p>Hi,<br>
I would like to ask how to fully use shared GPU. Currently, I only fully used dedicated GPU.<br>
Suggest some key words to search would be great help!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e32c4c68cda4138ef5b8c361a10cde836b45b366.png" data-download-href="https://discuss.pytorch.org/uploads/default/e32c4c68cda4138ef5b8c361a10cde836b45b366" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e32c4c68cda4138ef5b8c361a10cde836b45b366.png" alt="image" data-base62-sha1="wpFihlsXyrFUZgFiVsrrstMWFMi" width="640" height="500" data-dominant-color="F7F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">782×610 23.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
NN Output is different during training and when evaluate the model on the same data set

RFoldes: <p>Hi everybody,<br>
This is my first post, so I apologize for any mistakes.</p>
<p>I am trying to train and evaluate a simple NN model with the aim of reconstructing a 3D flow.<br>
The input is given by the coordinates (x,y,z,t) and the output is the 3 velocity components (u,v,w) as well as the pressure field p.</p>
<p>If I make a comparison of the reconstructed flow with the original one during the training phase, I can see the network learning how to create the flow. Indeed this is the situation at the last epoch of the training phase:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/463ab1255d469611c1d0a556711f72eb0bb2d2e7.png" data-download-href="https://discuss.pytorch.org/uploads/default/463ab1255d469611c1d0a556711f72eb0bb2d2e7" title="Vz_2Dplot_29000_edit"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/463ab1255d469611c1d0a556711f72eb0bb2d2e7.png" alt="Vz_2Dplot_29000_edit" data-base62-sha1="a1hax8wxtq46YiHVfOvYv5m89qT" width="500" height="500" data-dominant-color="D6F7CA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Vz_2Dplot_29000_edit</span><span class="informations">1200×1200 39.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This correctly reflect the low loss value at the end of the training phase: ~10^-4.</p>
<p>However, If I use the command:</p>
<pre><code class="lang-auto">model.eval(True)
y_eval = model(x_train)
</code></pre>
<p>The output makes no sense with respect to what previously observed.<br>
(I cannot attach more than one media as a novel user.)</p>
<p>Thanks in advance!</p>

RFoldes: <p>Here is the comparison between original and reconstructed field after the training phase.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f4e590e8693846fa297b51509226b494e6e82da.png" alt="u_test_edit" data-base62-sha1="mJhMc9BXahdCvmLcxEY7d9EF5AS" width="543" height="269"></p>

RFoldes: <p>Solved it.<br>
I mean, it was not a pytorch issue of course, it was me that I was messing with the order of inputs.<br>
During training it was (x,y,z,t) while for the evaluation I was using (t,x,y,z).</p>
nn.Parameter in DDP

Joeyoung: <p>Hi,</p>
<p>I am optimizing(training) nn.Module and one nn.Parameter together.<br>
DDP allows only nn.Module as its argument.<br>
If I give nn.Module to DDP, the optimization works well in multigpu, but the nn.Parameter value is not updated after optimization is done. Does it require any postprocessing to make it updated?<br>
How can I make the nn.Parameter value to be updated?<br>
Do I need to get the parameter value per each gpu, and average them?</p>
<p>Thank you!</p>

Joeyoung: <p>I create a nn.Module class to store nn.Parameter into the class. And wrap it to another DDP, which solves the problem.</p>

Joeyoung: <p>class MyModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
self._params = nn.ParamList()<br>
def append(self, p):<br>
self._params.append(p)<br>
def forward(self, x): # dummy forward<br>
return x</p>
<p>def train( local_rank, model, par ):<br>
…<br>
model = model.to(local_rank)<br>
optimizer = Adam( list(model.parameters()) … )<br>
DDP(model, device_ids=[local_rank])<br>
…<br>
for epoch in range(10):<br>
…<br>
loss = (par-2.0)**2<br>
loss.backward()<br>
optimizer.step()</p>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
model = MyModel()<br>
p = nn.Parameter( torch.tensor(1.0), requires_grad=True )<br>
model.append(p)<br>
mp.spawn(train, args=(world_size, model, p), nprocs=world_size)</p>
<p>In the above code, p is nn.Parameter instance, which is created before starting multiprocessing. It is used in loss calculation. In the loss calculation, is p’s gradient different between subprocesses? If so, how can it be possible? p is a single instance which is created in if <strong>name</strong> == “<strong>main</strong>” block.</p>
<p>Thank you!</p>

Joeyoung: <p>I have confirmed that its gradient is different for each subprocess.</p>
Can I make zero-division in pytorch return torch.inf instead of NaNs?

ilykuleshov: <p>The title says it all. Zero-division in pytorch returns NaNs, while mathematically it should return infinity. In certain cases, <code>torch.inf</code> would convert to normal values further on (e.g. <code>exp(-torch.inf)</code>, <code>1/torch.inf</code>, etc.), while NaNs on the other hand propagate endlessly, messing up model gradients. Pytorch functions already yield correct results for infinite values, so all that’s left is to return infinities after division. Is there any way to achieve this without resorting to masks or <code>torch.where</code>?</p>

Soumya_Kundu: <pre><code class="lang-auto">torch.nan_to_num(torch.div(numerator, denominator), nan=0.0, posinf=float('inf'), neginf=float('-inf'))
</code></pre>
<p>Not sure how efficient this is though. But seems like the ideal solution according to the <a href="https://pytorch.org/docs/stable/generated/torch.nan_to_num.html" rel="noopener nofollow ugc">docs</a></p>

ptrblck: <p>Additionally to <a class="mention" href="/u/soumya_kundu">@Soumya_Kundu</a>’s recommendation I also see that <code>Inf</code> is returned:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor(1.)
y = torch.tensor(0.)

print(x / y)
# tensor(inf)
</code></pre>

Alexander_Shekhovts: <p>In my experience if there is infinity in the forward computation, there will be a NaN in the backward. Like what do you expect from the gradient of exp(-1/x) at x=0+ in your example?</p>
<p>If you have y=1/x, is better to make sure that x is positive and bounded away from zero such that subseqent operations like y**2 do not overflow and gradient is somehow bounded. E.g.<br>
x = softplus(t)<br>
y= 1/(x + eps)</p>

wildfly: <p><a class="mention" href="/u/ilykuleshov">@ilykuleshov</a> Mathematically speaking I think this is an ambiguous case because:</p>
<p>with any x &gt; 0<br>
As you mentioned<br>
1/0+x &gt; 0<br>
and in limit 0 ← x (coming from positive numbers) it goes to infinity</p>
<p>But with any x &lt; 0<br>
1/0+x &lt; 0<br>
and in the limit x → 0 (coming from negative numbers) it goes to -infinity</p>
<p>So what is the value for x=0 negative or positive infinity?</p>
<p>I would say Nan is is the right behavior if ,the case 1/0, is not explicitly treated by the definition of the function you use.</p>
<p>But if positive infinity is your intended behaviour and not the default, which <a class="mention" href="/u/ptrblck">@ptrblck</a> 's example suggests, you can write your own function that checks if the denominator is 0 and returns torch.inf and otherwise call the normal torch implementation of that function or if equivalent what <a class="mention" href="/u/soumya_kundu">@Soumya_Kundu</a> proposed.</p>

ilykuleshov: <p>Thank you very much! Indeed, seems that pytorch handles this correctly, my bad. Turns out I had a  0/0 situation, in which case NaNs make total sense. Solved it with torch.where, similar to what <a class="mention" href="/u/wildfly">@wildfly</a> proposed.</p>
How long are nightly whls available

ashari4: <p>how long are nightly whls available? i noticed that the whl version <code>2.1.0.dev20230602+cpu </code> was taken down today:</p>
<p><a href="https://download.pytorch.org/whl/nightly/torch/" rel="noopener nofollow ugc">download.pytorch.org/whl/nightly/torch/</a></p>

ptrblck: <p>CC <a class="mention" href="/u/malfet">@malfet</a> who might know the life time of the nightly binaries.</p>

malfet: <p>There aren’t really taken down, just removed from the index after 60 days, see <a href="https://github.com/pytorch/builder/blob/2a03668317eae474844f3ae1f31578facf7a1688/s3_management/manage.py#L102-L103" rel="noopener nofollow ugc">https://github.com/pytorch/builder/blob/2a03668317eae474844f3ae1f31578facf7a1688/s3_management/manage.py#L102-L103</a></p>
<p>But one can still download the wheel, i.e.<br>
<code>curl -OL https://download.pytorch.org/whl/nightly/cpu/torch-2.1.0.dev20230602%2Bcpu-cp310-cp310-linux_x86_64.whl</code> will get you 2.1.0.dev20230602 wheel for Python-3.10</p>

Kaschi: <p>unfortunately your solution is not working for me. I installed the nightly version as follows:<br>
pip install --pre torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/nightly/cu124" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu124</a></p>
<p>However my dependencies require:<br>
PyTorch 2.5.0.dev20240705+cu124 with CUDA 1204 (you have 2.6.0.dev20241008+cu124)</p>
<p>Now I tried to install the 2.5.0 dev version like this:<br>
pip install <a href="https://download.pytorch.org/whl/nightly/cu124/torch-2.5.0.dev20240705%2Bcu124-cp310-cp310-linux_x86_64.whl" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu124/torch-2.5.0.dev20240705%2Bcu124-cp310-cp310-linux_x86_64.whl</a></p>
<p>I get this error:<br>
ERROR: Could not find a version that satisfies the requirement pytorch-triton==3.0.0+dedb7bdf33; platform_system == “Linux” and platform_machine == “x86_64” and python_version &lt; “3.13” (from torch) (from versions: 0.0.1)<br>
25.24 ERROR: No matching distribution found for pytorch-triton==3.0.0+dedb7bdf33; platform_system == “Linux” and platform_machine == “x86_64” and python_version &lt; “3.13”</p>
Confused by RuntimeError when checking for parameter in list

sammlapp: <p>Let’s say I want to check if a parameter is in a list of parameters, for instance to add it to an optimizer param group:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch, torchvision
m = torchvision.models.resnet18()
p0 = next(iter(m.fc.parameters()))
p_list = list(m.parameters())
p0 in p_list
</code></pre>
<p>Raises RuntimeError: The size of tensor a (7) must match the size of tensor b (512) at non-singleton dimension 3</p>
<p>But I should just be comparing object identities, not values/shapes of tensors. What’s going on here? As a workaround, I could compare object ids but this doesn’t seem like it should be necessary.<br>
Thanks!</p>

Eduardo_Lawson: <p>When you use <code>p0 in p_list</code>, python calls the <code>__contains__</code> method of the list. It iterates through its elements and uses the <code>==</code> operator to check for equality.<br>
Pytorch has overloaded the <code>==</code> operator for tensors to perform element-wise comparison rather than identity comparison. When comparing tensors of different shapes, PyTorch raises the <code>RuntimeError</code> you observed.</p>
<p>As a workaround for comparing objects identities, I would suggest doing it with <code>is</code> operator:</p>
<pre><code class="lang-auto">any(p0 is p for p in p_list)
</code></pre>
<p>or with id:</p>
<pre><code class="lang-auto">any(id(p0) == id(p) for p in p_list)
</code></pre>
How to best speed up for-loop for Kalman Filter

SaremS: <p>Hi,</p>
<p>I am looking to speed up some custom Kalman Filter models. It looks pretty obvious to me that the for-loop is the main performance bottleneck. I have tried to jit-compile the loop-part which gives some improvement but is still far from what I expect to be possible.</p>
<p>Is there any best practice approach that I might still be missing?</p>
<p>My next step would be to write a custom C++ module for the loop logic, but I’m still hopeful that there is a better way to make things faster.</p>
<p>Thanks for any help!</p>

XiaoliangWang2001: <p>Hi, I have recently run into a similar situation. Have you found a neat solution to this problem?</p>
Isn't Bias normally int Quantized in INT8 PTSQ model?

minjun_jo: <p>I am a student who studies AI accelerators.<br>
We are doing related research by performing Pytorch’s Eager mode Quantization.<br>
As a result of the output, it was confirmed that quantization was made for the weight and activation of Qint8, but the bias is not.</p>
<p>I have a few questions, please.</p>
<ol>
<li>Doesn’t Bias usually Quantize to INT8 in Quantization model?</li>
</ol>
<p>2.Isn’t it recommended to run Eager mode to experiment with the Quantization model for INT8? Usually, it seems to be customized, but I’m not very good at coding.</p>
<ol start="3">
<li>
<p>If you want to experiment with INT8 quantization, it is said that QINT8 is decoded with FP32 and then calculated. If you want to study only with INT8, what method should I use? (Eager mode, FX, ONNX or Tensorflow…</p>
</li>
<li>
<p>Also, most backend(fbgem or quantization backend) doesn’t support INT8 operation well, is there a way to support it well?</p>
</li>
</ol>
<p>Please answer what you know.</p>

Khumbaba: <p>For your first question, no, bias is not usually quantized to int8. It is quantized to int32 (or even int64). From my understanding, the goal is to speed up the multiplications, which will be done in int8, with the result stored in int32. Which in turn makes sense why the bias would want to be in int32.</p>
<p>An extract from <a href="https://arxiv.org/pdf/1712.05877" rel="noopener nofollow ugc">Google’s quantization paper</a> which I recommend reading if you want to get to know quantization better:</p>
<blockquote>
<p>Note that the biases are not quantized because they are<br>
represented as 32-bit integers in the inference process, with<br>
a much higher range and precision compared to the 8 bit<br>
weights and activations. Furthermore, quantization param-<br>
eters used for biases are inferred from the quantization pa-<br>
rameters of the weights and activations</p>
</blockquote>
Why does torch not support onnx export with opset>17? dynamo_export not working properly?

Bodo: <p>So it would really be nice if the latest opset version of onnx would work because they are necessary to go from experimental level to production. Not having the option of opset &gt;17 somehow negates the possibility to use torch for training models, since the new nvidia inference server with tensor rt rely on it.</p>
<p>I also tried dynamo_export for opset=18 but when tryining to use dynamic_shapes it does not work, which makes the model not batchable. It just makes dynamic shapes for the last two dimensions but not for batching.</p>
<p>My code for the dynamo_export is:</p>
<pre><code class="lang-auto">input = torch.randn([1, 3, 112, 112])
opts = torch.onnx.ExportOptions()
opts.dynamic_shapes = True
ox_p = torch.onnx.dynamo_export(model, input, export_options=opts)
</code></pre>
<p>Maybe you have an idea how to make the first dimension dynamic in dynamo_export?</p>
<p>Bets regards</p>
Does ExecuTorch have an interface to load the model chunk by chunk?

dbort: <p>(reposting this Meta-internal question and response)</p>
<blockquote>
<p>We have a decompression engine on DSP and would like to know if it can be used on ExecuTorch models. So far we have been using <code>BufferDataLoader</code> by specifying buffer address and buffer size.</p>
<p>Is there any other interface to load the model chunk by chunk so that we can decompress the model on the fly?</p>
</blockquote>
<p>You could write your own DataLoader subclass to load the data chunk by chunk, though it does require random access into the decompressed data to satisfy the <a href="https://github.com/pytorch/executorch/blob/b6e6d062e30455c96ba8b58c52a6200788721296/runtime/core/data_loader.h#L91" rel="noopener nofollow ugc">load()</a> behavior. This is usually a pain when loading from compressed data, because naive compression schemes will require re-decompressing from the beginning of the stream every time.</p>
<p>Block compression can avoid some of that by having sync points inside the compressed data, so that you can start decompressing from further in the stream. It typically requires extra metadata and compression logic, though.</p>
<blockquote>
<p>Another question. After the model is loaded, can we reclaim the buffer memory space?</p>
</blockquote>
<p>Depends on the model. The ET runtime uses the FreeableBuffer objects returned by DataLoader to manage program data lifetime. When they’re freed, the data behind that FreeableBuffer can be reclaimed. But the standard BufferDataLoader can’t free anything, because a) it doesn’t know how its memory was allocated, and b) even if it knew that the memory was allocated with malloc(), there’s no way to free a hole inside a malloc buffer (apart from OS tricks like madvise()). That was the whole motivation behind DataLoader and FreeableBuffer.</p>
<p>As for when those buffers are freed: today, then only time that this happens is for backend delegate data blobs, during model load. If, after initialization, the backend decides that it doesn’t need its data blob, it can free it. <a href="https://github.com/pytorch/executorch/blob/b6e6d062e30455c96ba8b58c52a6200788721296/backends/xnnpack/runtime/XNNPACKBackend.cpp#L90-L91" rel="noopener nofollow ugc">XNNPACK</a> does this, though some backends do not.</p>
<p>Apart from those backend data segments, all other loaded data is required for model execution, and needs to remain resident for the lifetime of the Method. Although we could add some kind of API for freeing the Method data, it would be equivalent to destroying the Method itself, which is already possible today. So if you need to reclaim data between executions, you could destroy the Method – but there’s the obvious tradeoff with the latency of re-loading the Method the next time you want to execute it.</p>
No initializer for Module "TransformerEncoder"

palace_painter: <p>There is no initializer(the “reset_parameters” function) for Module “nn.TransformerEncoder” as other Modules used by users(e.g. “nn.Linear”).</p>
<p>Is “TransformerEncoder” not supposed to be used by users in their code?<br>
If so, which module should I use instead if I want to build a Encoder-only transformer in pytorch?</p>
<p>Thank you!</p>

ptrblck: <p><code>nn.TransformerEncoder</code> needs an <code>encoder_layer</code> input, which can be e.g. <code>nn.TransformerEncoderLayer</code> which is internally initializing modules and thus calling their corresponding <code>reset_parameters</code> function.</p>

palace_painter: <p>Thanks for your generous help!<br>
By the way, I’m not familiar with how people initialze the built-in modules (e.g. TransformerEncoder) in pytorch.<br>
Do users rely on the initializers of submodules’(e.g. nn.Linear) to initialize the whole module instead of initializing the model with a customized “reset_parameters()” function?</p>
<p>Thank you!</p>