akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Wrapping with DDP increases GPU memory

username: yiftach
cooked: <p>Hi,<br>
This is probably trivial, but I saw in <a href="https://pytorch.org/tutorials/recipes/zero_redundancy_optimizer.html" rel="noopener nofollow ugc">this example for ZeRO optimizer</a> that after wrapping a model with DDP, the peak GPU memory almost doubles.<br>
What is the reason? I thought DDP’s only job was synchronizing gradients wisely during the backward, and sure what job it needs to do during initialization. I also think it has some function during forward() that I’m not aware of, is that the case?</p>
<p>Thanks,<br>
Yiftach</p>

username: smth
cooked: <p>as mentioned on that page, each worker has a separate optimizer state, so that overhead adds to the extra memory usage.</p>
<blockquote>
<p>The idea of <a href="https://pytorch.org/docs/master/distributed.optim.html">ZeroRedundancyOptimizer</a> comes from <a href="https://github.com/microsoft/DeepSpeed">DeepSpeed/ZeRO project</a> and <a href="https://github.com/marian-nmt/marian-dev">Marian</a> that shard optimizer states across distributed data-parallel processes to reduce per-process memory footprint.</p>
</blockquote>

username: yiftach
cooked: <p>Thanks <a class="mention" href="/u/smth">@smth</a>, but as far as I can tell, the print that shows more memory is being allocated happens before any optimizer is instantiated:</p>
<pre><code class="lang-auto">...
    ddp_model = DDP(model, device_ids=[rank])
    print_peak_memory("Max memory allocated after creating DDP", rank)

    # define loss function and optimizer
    loss_fn = nn.MSELoss()
    if use_zero:
        optimizer = ZeroRedundancyOptimizer(
            ddp_model.parameters(),
            optimizer_class=torch.optim.Adam,
            lr=0.01
        )
    else:
        optimizer = torch.optim.Adam(ddp_model.parameters(), lr=0.01)
...
</code></pre>
Understand attn_mask param in nn.MultiheadAttention module

username: richardatpytorch
cooked: <p>I have a point cloud with n points and each point has d features. I want to use the <code>attn_mask</code> in <code>nn.MultiheadAttention</code> module to do local attention to only points in neighbors. Should I pass the adjacency matrix to achieve that target?</p>

username: HELLORPG
cooked: <p>Well, I think this would work (but you should reverse the adjacency matrix, because <code>True</code> in <code>attn_mask</code> means masking this token), but NOT a good choice.</p>
<p>If you have N points, and every point has M neighbor, which M &lt;&lt; N. The cost of this attention layer which is only within neighbors should be O(NxM). But if using the <code>attn_mask</code> directly to the attention layer, the cost will be O(NxN). That’s because the <code>attn_mask</code> just uses <code>-inf</code> to neutralize the softmax for masked tokens but still compute with them. If you have so many points (like 5000) but M is small (like 10), this will lead to unnecessary consumption.</p>

username: richardatpytorch
cooked: <p>I understand what you are concerned about. However, this is the only way I know to utilize the parallelization of GPU. Do you have any suggestions?</p>

username: HELLORPG
cooked: <p>In 2D image processing, <a href="https://github.com/SHI-Labs/Neighborhood-Attention-Transformer" rel="noopener nofollow ugc">Neightborhood Attention</a> has similar needs.</p>
<p>There are also some other methods in my memory. For example, grouping the N tokens and their M neighbors, then getting (N, M) tokens. Processing <code>attention</code> on these tokens by treating N as the batch size. This will require consideration of the trade-off between computational and memory effort.</p>
Error when training LorA

username: ledggerald
cooked: <p>I want to train my LorA IN kohya, I have Pytorch and Xformers installed and it shows TRUE in the address bar, but when training I get these errors:</p>
<p>ValueError: torch.cuda.is_available() should be True but is False. xformers’ memory efficient attention is only<br>
available for GPU</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/5701eca1165e7e0cc72fcd9ba1c27f0ead2ec776.png" data-download-href="https://discuss.pytorch.org/uploads/default/5701eca1165e7e0cc72fcd9ba1c27f0ead2ec776" title="2023-09-18_02-30-35"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5701eca1165e7e0cc72fcd9ba1c27f0ead2ec776_2_690x388.png" alt="2023-09-18_02-30-35" data-base62-sha1="cpHF5Mfu6OlFUvIHbGOSXYJNtD8" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5701eca1165e7e0cc72fcd9ba1c27f0ead2ec776_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5701eca1165e7e0cc72fcd9ba1c27f0ead2ec776_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5701eca1165e7e0cc72fcd9ba1c27f0ead2ec776_2_1380x776.png 2x" data-dominant-color="1A1919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2023-09-18_02-30-35</span><span class="informations">1920×1080 126 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What have i do?(in Dreambooth i haven`t this problem but there are other problems)</p>
<p>(Windows 10)</p>

username: J_Johnson
cooked: <p>Whoever developed the code you’re using stated that it’s only available for GPU. Do you have a GPU and do you have the appropriate Nvidia drivers installed?</p>

username: ledggerald
cooked: <p>Reinstal kohya and it works…)</p>
Using Dropout in reinforcement learning gym environment

username: andyjh1222
cooked: <p>Hi</p>
<p>I’m using Ray (rllib) and dropout with a gym environment and a fully connected forward network.</p>
<p>Since I’m using evolution strategies which injects a different set of parameter noise into the model for each rollout worker and corresponding mini-batch, it would be helpful to know does nn.dropout disable a different set of neurons for each environment step? Or is it the same mask for each network initialization during worker initialization and mini-batch?</p>
<p>Rllib just abstracts the torch model, it doesn’t change any of the underlying torch code.</p>

username: J_Johnson
cooked: <p>When your model is run in <code>model.train()</code> mode, dropout will zero out a random distribution of layer parameters based on the p value set. Every time data goes through the model, a new set of parameter values are zeroed out.</p>
<p>On a side note, dropout is generally beneficial when you have a fixed training set in which the model may overfit on. It helps compensate for “holes” or “excesses” in the dataset. However, in reinforcement learning, the training data is produced on the fly and there is no risk of overfitting. The data is a random distribution of possible environments and more uniform with no repetition. So generally you won’t see dropout layers applied in reinforcement learning models.</p>

username: andyjh1222
cooked: <p>thank you for the response, definitely very helpful!</p>
Training with gloo gets slow for multiple nodes

username: Cpuguy
cooked: <p>I’m training with DDP on a slurm server using CPUs and gloo. I set_num_threds to the CPUs per process and Slurm_n_tasks equals the number of nodes.</p>
<p>When I train on more than one node it gets much slower and takes up a lot of extra memory, requiring a decrease in batch size per process. The result is I get a sizable performance loss, even though scaling withing a single node is pretty good.</p>
<p>I’m wondering if anyone has any ideas why this happens and how I might improve it.</p>

username: HELLORPG
cooked: <p>Well, using DDP will slow the speed compared with training on a single GPU without DDP. But I’m not sure whether it needs more GPU memory.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="HELLORPG" data-post="2" data-topic="188499">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hellorpg/48/53892_2.png" class="avatar"> HELLORPG:</div>
<blockquote>
<p>Well, using DDP will slow the speed compared with training on a single GPU without DDP.</p>
</blockquote>
</aside>
<p>It won’t, since you are able to process much more data and thus reduce the epoch time significantly.</p>

username: HELLORPG
cooked: <p>Yeah, it will reduce the epoch time but increase the iteration time.</p>
TransformerEncoder output size doesn't match input size

username: joshadel
cooked: <p>I’m looking for suggestions on how to debug the following issue, where the output size of a <code>nn.TransformerEncoder</code> does not match the input. As part of a larger network, I have a <code>nn.TransformerEncoder</code> layer containing a single <code>nn.TransformerEncoderLayer</code>. It’s created as:</p>
<pre><code class="lang-python">enc_layers = nn.TransformerEncoderLayer(16, 2, 
    dim_feedforward=32, dropout=0.0, activation='gelu',
    batch_first=True,
)
self.encoder = nn.TransformerEncoder(enc_layers, 1)
</code></pre>
<p>In the <code>forward</code> method, the snippet where this is called is:</p>
<pre><code class="lang-python">xs = xs.view(-1, 20, 16)
print(xs.size())
xsa = self.encoder(xs, src_key_padding_mask=mask)
print(xsa.size())
</code></pre>
<p>I’m getting:</p>
<pre><code class="lang-auto">torch.Size([1024, 20, 16])
torch.Size([1024, 14, 16])
</code></pre>
<p>The strange thing is that when I run a single batch in isolation in a notebook, the sizes match and everything looks fine, but when run in the training script, I see the size mismatch. I’ve printed the sizes of all of the model parameter tensors in both, and they match.  The mask size is `torch.Size([1024, 20]).</p>
<p>I’ve also used <code>python -m pdb</code> to drop in and inspect things when the training script fails at the next step because of the unexpected tensor size. When I re-pass <code>xs</code> to <code>self.encoder</code> I’m getting the expected size, so this is really mysterious to me.</p>
<p>I’m using torch 2.0.0 on a cuda gpu, but the same thing happens when running on the CPU as well.</p>
<p>Is there a situation where a <code>nn.TransformerEncoder</code>’s output could have a different size than the input? Thoughts/suggestions for how to further debug this?</p>

username: AleTL
cooked: <p>Hello,</p>
<p>I am experimenting exactly same problem. Changing the batch size of my model for some reason triggers this behaviour.</p>
<p><code>transformer_output = self.encoder(embeddings, is_causal=self.autoregressive, src_key_padding_mask=attention_mask)</code></p>
<p>where <code>embeddings</code> is shape <code>[9, 2048, 256]</code> and <code>attention_mask</code> is size <code>[9, 2048]</code>, but the output is size <code>[9, 2041, 256]</code>.<br>
Is there any explanation? I don’t know what is going on and it’s difficult to debug.</p>
<p>Thanks!</p>
How to parallelly process a tensor according to a list of segments?

username: ppp
cooked: <p>I am attempting to process a tensor based on a list of segments. Each segment in the tensor will undergo a different operation. I have tried using the vmap function to solve this problem, but it is resulting in an error: “RuntimeError: vmap: It looks like you’re calling .item() on a Tensor.”. Here is a simplified version of my code:</p>
<pre><code class="lang-auto">import torch

segment_size = torch.randint(1, 6, size=(6,))
idx_end = torch.cumsum(segment_size, dim=0)
idx_start = torch.roll(idx_end, 1)
idx_start[0] = 0
idx_start_end = torch.cat((idx_start.unsqueeze(0), idx_end.unsqueeze(0)), dim=0)
x = torch.randn(idx_end[-1])

def calculate(start_end, x):
    start = start_end[0]
    end = start_end[1]
    selected_x = x[start:end]
    selected_x = 0
    return selected_x

torch.vmap(calculate, in_dims=(1,None))(idx_start_end,x)
</code></pre>
<p>I would like to know how to resolve this issue or if there is a more efficient way to implement it.</p>
I want to know how to find my tensor data when I debug the pytorch

username: YCB-1027
cooked: <pre><code>  const auto gates = params.linear_hh(hx).add_(
    pre_compute_input ? input : params.linear_ih(input));
  auto chunked_gates = gates.unsafe_chunk(4, 1);
  auto ingate = chunked_gates[0].sigmoid();
  auto forgetgate = chunked_gates[1].sigmoid();
  auto cellgate = chunked_gates[2].tanh();
  auto outgate = chunked_gates[3].sigmoid();
  auto cy = (forgetgate * cx).add_(ingate * cellgate);
  auto hy = outgate * cy.tanh();
  hy = params.matmul_hr(hy);
  return std::make_tuple(std::move(hy), std::move(cy));
</code></pre>
Breaking down data frame

username: NolimitDougie
cooked: <pre data-code-wrap="import"><code class="lang-plaintext">import pandas as pd
import os
import pickle
from glob import glob
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
from torch.utils.data import Dataset, DataLoader, TensorDataset
import torch.nn as nn
import torch
import torchvision
from torchvision.models import ResNet50_Weights
import seaborn as sns
from tqdm import tqdm
from PIL import Image
from itertools import chain
import torch.nn.functional as F
import torch.optim as optim
from sklearn.utils import resample
import torch.optim.lr_scheduler as lr_scheduler
from sklearn.metrics import f1_score, accuracy_score, precision_score, recall_score, roc_auc_score, \
    multilabel_confusion_matrix, roc_curve, auc, classification_report

# Device configuration GPU support for MAC
if torch.backends.mps.is_available():
    mps_device = torch.device("mps")
else:
    print("MPS device not found.")

# Paths to Images and DataEntry file
all_xray_df = pd.read_csv('NihXrayData/Data_Entry_2017_v2020.csv')
allImagesGlob = glob('NihXrayData/images*/images/*.png')
# eof

all_image_paths = {os.path.basename(x): x for x in
                   allImagesGlob}
# print('Scans found:', len(all_image_paths), ', Total Headers', all_xray_df.shape[0])
all_xray_df['path'] = all_xray_df['Image Index'].map(all_image_paths.get)
all_xray_df.sample(3)

# # Data Pre Processing ####
# # Simplifying to 15 primary classes (adding No Finding as the 15th class)
condition_labels = ['Atelectasis', 'Consolidation', 'Infiltration', 'Pneumothorax', 'Edema', 'Emphysema', 'Fibrosis',
                    'Effusion', 'Pneumonia', 'Pleural_Thickening',
                    'Cardiomegaly', 'Nodule', 'Mass', 'Hernia', 'No Finding']
for label in condition_labels:
    all_xray_df[label] = all_xray_df['Finding Labels'].map(lambda result: 1.0 if label in result else 0)
all_xray_df.head(20)

all_xray_df['disease_vec'] = all_xray_df.apply(lambda target: [target[condition_labels].values], 1).map(
    lambda target: target[0])

all_xray_df.head()


train_df, test_df = train_test_split(all_xray_df, test_size=0.30, random_state=2020)
</code></pre>
<p>Hello I’m trying to set up 3 batches with 5 classes to train the model to do multi-task learning. I’m having trouble breaking the data frame based on top 5 classes in one batch second 5 in one third 5 in the third. Also any suggestions on how to set up the model for accomplish this will be appreciated</p>
Mismatch when converting TransformerEncoder to onnx

username: Mohamed_Hassan
cooked: <p>I am convering a model containing TransformerEncoder. However I noticed that the outputs I get in PyTorch and onnxruntimes are different.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        trans_enc_layer = nn.TransformerEncoderLayer(d_model=32,
                                                     nhead=4,
                                                     dim_feedforward=64)

        self.seqTransEncoder = nn.TransformerEncoder(trans_enc_layer,
                                                     num_layers=1)

    def forward(self, x):
        return self.seqTransEncoder(x)


model = Model()
x = torch.randn(7, 1, 32)
y = model(x)

output_file_name = 'model.onnx'
torch.onnx.export(
    model,
    x,
    output_file_name,
    training=torch.onnx.TrainingMode.EVAL,
)
ort_session = ort.InferenceSession(output_file_name)
ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(x)}
ort_outs = ort_session.run(None, ort_inputs)
np.testing.assert_allclose(to_numpy(y), ort_outs[0], rtol=1e-03, atol=1e-05)
</code></pre>
<p>When I run the code above, I get the following</p>
<pre><code class="lang-auto">Not equal to tolerance rtol=0.001, atol=1e-05

Mismatched elements: 224 / 224 (100%)
Max absolute difference: 0.96960294
Max relative difference: 14.14618
</code></pre>

username: ptrblck
cooked: <p>You are not calling <code>model.eval()</code> on the PyTorch model before creating the reference output and are thus using the default dropout layer. Check if this would reduce the numerical mismatches.</p>

username: Mohamed_Hassan
cooked: <p>Thanks a lot. That was the issue indeed.</p>
Zero_grad() zeros intermediate steps as well?

username: gandolfin69420
cooked: <p>Well, tried to get the answer out of chatgpt, but it was too easily influenced by my assumptions.</p>
<p>Let’s say I have a loss that is calculated after some intermediate steps. Also, the optimizer is setup using the model’s parameters, so we have.</p>
<p>input =&gt; model’s parameters =&gt; intermediate steps =&gt; loss</p>
<p>and when we loss.backward()</p>
<p>input (accumulated gradient) =&gt; model’s parameters (accumulated gradient) =&gt; intermediate steps (accumulated gradient) =&gt; loss</p>
<p>then we optimizer.step() and optimizer.zero_grad(), so does this zero_grad() zero all accumulated gradients, or only the model’s parameters gradients since the optimizer was only initialized with the model’s parameters? that is, compare these</p>
<p>input =&gt; model’s parameters =&gt; intermediate steps =&gt; loss<br>
input (still has accumulated gradient, albeit not relevant for future calculations) =&gt; model’s parameters (zeroed) =&gt; intermediate steps (still has accumulated gradient, interferes with future backprops?) =&gt; loss</p>
<p>oh wait, I am stupid, I think. requires_grad==False by default, but you can still backward() a grad through a tensor that has requires_grad==False, so that’s why it’s fine, ye I am dumb, maybe just someone confirm this if true just to make sure</p>

username: ptrblck
cooked: <p>Intermediate steps don’t have accumulated gradients as they are stored only temporarily and recreated in each forward step.<br>
The gradients will be accumulated to the <code>.grad</code> attribute of the used leaf variables, i.e. parameters and the optimizer will zero these attributes from all passed parameters.</p>

username: gandolfin69420
cooked: <p>if i understand correctly, non-leaf nodes in the computational graph do not store any gradients, so they can be backpropped through multiple times without any “accumulated gradient reset”-process required. zeroing gradients is only required for leaf nodes, which store the gradients as a numerical value?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="gandolfin69420" data-post="3" data-topic="188404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gandolfin69420/48/73311_2.png" class="avatar"> gandolfin69420:</div>
<blockquote>
<p>so they can be backpropped through multiple times without any “accumulated gradient reset”-process required</p>
</blockquote>
</aside>
<p>Yes, intermediate forward activations are used to compute the gradients and do not receive gradients themselves. After a backward pass they will be deleted to save memory and wou won’t be able to reuse them multiple times unless you specify <code>.backward(retain_graph=True)</code>.</p>
Doubt regarding Implementation of Hierarchical Attention Network

username: Srinjoy_Mukherjee
cooked: <p>Hey all,<br>
I was reading <a href="https://www.cs.cmu.edu/~./hovy/papers/16HLT-hierarchical-attention-networks.pdf" rel="noopener nofollow ugc">this paper</a> and came across a problem, during implementation.</p>
<p>During dataset creation, I created batches as 4 dimensional tensor as such( batch_size x document_size x sentence_size x embedding_size).</p>
<p>Now when I pass though the GRU, it says nn.GRU only accept 3D tensors. So my question is what changes should I make to the model or dataset creation?</p>
<p>Thanks in advance.</p>
How to profile memory footprint and breakdowns

username: oracal
cooked: <p>I was making inference using Pytorch and I wanna get the memory footprint in the process of inference and breakdown it into seperate parts to analyse for further study. How can I do that and thanks</p>

username: cqynvd
cooked: <p>This page may help you. <a href="https://pytorch.org/tutorials/beginner/profiler.html" rel="noopener nofollow ugc">Profiling your PyTorch Module — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>
Recreating Issue for self-referential list in torch._dynamo

username: abewtra
cooked: <aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/109504">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/109504" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/109504" target="_blank" rel="noopener nofollow ugc">[dynamo] torch._dynamo.exc.Unsupported: comparison SymNodeVariable() &lt;built-in function is_&gt; ListVariable()</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2023-09-18" data-time="15:25:02" data-timezone="UTC">03:25PM - 18 Sep 23 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/ezyang" target="_blank" rel="noopener nofollow ugc">
          <img alt="ezyang" src="https://avatars.githubusercontent.com/u/13564?v=4" class="onebox-avatar-inline" width="20" height="20">
          ezyang
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          good first issue
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: pt2
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: dynamo
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

Self explanatory.

Occurred when implementing self-ref<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">erentiality check in torch.tensor ref

```
            if cur_item is obj:
                raise TypeError("new(): self-referential lists are incompatible")
```

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I’m not exactly sure how to recreate this issue. Does anyone have any tips?</p>

username: marksaroufim
cooked: <p>Might just be best to discuss this on the github issue</p>
Updating a model with the loss of another

username: timmy1691
cooked: <p>I am wondering if I have a two neural networks, netA, and netB. and I have some output from<br>
y_A = netA(X)<br>
y_B = netB(X)<br>
and I want to use the loss calculated from model A, such as<br>
loss = Loss(y_A, y)</p>
<p>but then I want to use this loss to update my model netB.</p>
<p>Because I am coming to this from a pure computational perspective, such as linear regression we can differentiate the mean squared error in terms of the parameters to get some values. and the backpropagation with respect to linear algebra is a dot product between the input of the layer and the output of the layer.</p>
A small snippet for lambda modules

username: Roulbac
cooked: <p>Hi all,<br>
I’ve been trying to build a U-Net lately and I wanted to stick with the nn.Module way of thinking, to make skip connection modules.<br>
After some thinking, the following snippet helped me with my troubles and I felt like sharing it.<br>
It’s a general-purpose Module that wraps python lambdas</p>
<pre><code class="lang-auto">class LambdaModule(nn.Module):
    def __init__(self, lambd):
        super().__init__()
        import types
        assert type(lambd) is types.LambdaType
        self.lambd = lambd

    def forward(self, x):
        return self.lambd(x)
</code></pre>
<p>The way I used it was the following:</p>
<pre><code class="lang-auto">            middle_lambda = lambda x: torch.cat([x, self.lower(x)], 1)
            middle = LambdaModule(middle_lambda)
            block = [left, middle, right]
            self.block = nn.Sequential(*block)
</code></pre>
<p>where left and right are the modules left and right of the skip connection module</p>

username: d0uwe
cooked: <p>Very useful, thanks!</p>
Detect CUDA graph capture in a TorchScript model

username: RaulPPelaez
cooked: <p>Hi, I need to check for graph capturing on a module that is compiled with torch.jit.script:</p>
<pre><code class="lang-python">class Module(torch.nn.Module):
  def forward(self, x):
        if torch.cuda.graphs.is_current_stream_capturing():
              print("Capturing underway")

module = torch.jit.script(Module())
</code></pre>
<p>However, this function is not supported by TorchScript and the following error is printed:</p>
<pre data-code-wrap="shell"><code class="lang-plaintext">Python builtin &lt;built-in function _cuda_isCurrentStreamCapturing&gt; is currently not supported in Torchscript:
  File "/shared/raul/mambaforge/envs/openmmtorch-test/lib/python3.10/site-packages/torch/cuda/graphs.py", line 25
    If a CUDA context does not exist on the current device, returns False without initializing the context.
    """
    return _cuda_isCurrentStreamCapturing()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>Would you help me come up with a workaround?<br>
Thanks</p>
Runtime Error: NNPACK

username: lint_weaver
cooked: <p>RuntimeError: NNPACK SpatialConvolution_updateOutput failed</p>
<p>I’m getting the above error when I run code on my local machine (Mac), but when I run the same code on google collab, it does not display the error. My version of PyTorch is 2.0.1 on my Mac, which is the same version on collab. What gives?</p>
How to exclude submodule's parameters for a outer module?

username: batx
cooked: <p>I’m implementing a module based on the paper “Emergence of Invariance and Disentanglement in Deep Representations”, and sample parameters of a base module from a given distribution. So I have to convert all parameters of the base module into Intermediate variables instead of module parameters.</p>
<p>But if I create a base module as a submodule in the outer module, its paramerters will be included into the result of parameters() method automatically. How to exclude submodule’s parameters?</p>

username: interfish
cooked: <p>Same question here. Check this <a href="https://discuss.pytorch.org/t/how-to-exclude-parameters-from-model/6151">how-to-exclude-parameters-from-model</a>. Wrap it in a list can solve but also will lose effects when using something like <code>to("cuda").</code></p>

username: batx
cooked: <p>I knew it, and was trying to find a simpler way, but I finally used it.</p>
How can I extract features?

username: samm
cooked: <p>I’m using a model for object detection for images . After training data set I got files with .pth extention for 15 epochs … what should I do for extracting the features for the images</p>

username: Kapil_Rana
cooked: <p>So I assume that you have saved the model in pth file after training for 15 epochs.<br>
I am not sure from which layer you want to extract features.</p>
<p>model(input) will provide the features or output from the last layer.<br>
For getting features from intermediate layer you can explicitly return in forward function of the model.</p>
<p>or you can get features using the layer variable names.<br>
model = new_model(output_layer = ‘layer4’)</p>
<p>or</p>
<p>new_classifier = nn.Sequential(*list(model.children())[:-1])</p>

username: samm
cooked: <p>thanks a lot for replying . I found the method called forward … do you mean this method ?</p>
<p>def forward(self, x):<br>
x = self.forward_features(x)<br>
return x</p>

username: Kapil_Rana
cooked: <p>Yes, in this function you can save output from any layer and return it.</p>
<p>suppose<br>
x1 = self.cnn1(x)<br>
x2 = self.cnn2(x1)</p>
<p>return x1,x2</p>

username: samm
cooked: <p>excuse me so this function save the features as it return x  ?</p>

username: Kapil_Rana
cooked: <p>this function take an input and generate a output ragardign that input.<br>
Suppose in image classification you will pass and input and in the end you will get a output from last (this is usual case in all the code forums.)</p>
<p>But what I am trying to say you can return any variable in forward function. Forward function is  just how each layer is being called and passed to next. so you can return output from any layer.</p>

username: samm
cooked: <p>does the result will be like this</p>
<p>[{391895: [array([[3.43202576e+02, 2.43713760e+01, 4.96343414e+02, 3.32774170e+02,<br>
8.81948233e-01],</p>
<p>or</p>
<p>{‘391895’: array([[1.15911709e-03, 3.19864594e-05, 1.06177040e-05, 6.39094651e-07,<br>
1.44435398e-06, 1.01086480e-05,</p>
<p>i mean the structure of format of the features will start with symbol { or will be a list that start with symbol [ ?</p>

username: Kapil_Rana
cooked: <p>Can you show me the model</p>

username: samm
cooked: <p>which part do you need ? is there any specific function ?</p>

username: Kapil_Rana
cooked: <p>I need model code and forword function mainly and which features you need</p>

username: kirtac83
cooked: <p>There is a wrapper for <a href="https://pytorch.org/vision/stable/feature_extraction.html" rel="noopener nofollow ugc">feature extraction</a> now.</p>
Python code using DataLoader not running from C++

username: gerstla
cooked: <p>Hi</p>
<p>When I run the <a href="https://drive.google.com/file/d/1OksnEnKlJGSIYxVdSB3vWH0P9bZXb0Zs/view?usp=drive_link" rel="noopener nofollow ugc">transferTrainingNoImages.py</a> code from Python it runs with no problems.</p>
<p>But when I try running it from C++ like below<br>
then it goes into an infinite loop. I tried debugging the python code and it seems there is some problem in line 63 in the python file.</p>
<blockquote>
<p><span class="hashtag-raw">#define</span> PY_SSIZE_T_CLEAN<br>
<span class="hashtag-raw">#include</span> &lt;python.h&gt;</p>
<p>using namespace std;</p>
<p>int main()<br>
{<br>
char filename[  ] = “transferTrainingNoImages.py”;</p>
<pre><code>FILE* fp;

Py_Initialize();

auto err = fopen_s(&amp;fp, filename, "r");

PyRun_SimpleFile(fp, filename);

Py_Finalize();
</code></pre>
<p>}</p>
</blockquote>
<p>Can anyone point me to the problem.</p>
<p>Thanks</p>

username: Ella_Mendelson
cooked: <p>I have the problem. can’t run python codes with DataLoader from c++. any solution?</p>
Enable NUMA binding with torch.distributed.launch

username: hubertlu-tw
cooked: <p>For the distributed workloads without torch.distributed.launch API, we are able to manually spawn python processes and leverage CPU/GPU affinity by “numactl” to get better performance. For example, <a href="https://github.com/mlperf/training_results_v0.7/blob/master/NVIDIA/benchmarks/ssd/implementations/pytorch/run_and_time.sh#L49" rel="noopener nofollow ugc">NVIDIA MLPerf SSD run script</a> with <a href="https://github.com/mlperf/training_results_v0.7/blob/master/NVIDIA/benchmarks/ssd/implementations/pytorch/bind_launch.py#L117" rel="noopener nofollow ugc">bind_launch.py</a> and the PyTorch tuning guide: <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#utilize-non-uniform-memory-access-numa-controls" rel="noopener nofollow ugc">CPU specific optimizations - Utilize Non-Uniform Memory Access (NUMA) Controls</a>. However, how can we enable NUMA binding for the workloads with torch.distributed.launch API?</p>

username: kwen2501
cooked: <p><a class="mention" href="/u/d4l3k">@d4l3k</a> Wondering if you have any idea? Thanks!</p>

username: GuWei007
cooked: <p>Encountered the same problem when training GPT 18.4B, is there any standard solution?<br>
<a class="mention" href="/u/hubertlu-tw">@hubertlu-tw</a> <a class="mention" href="/u/kwen2501">@kwen2501</a></p>
What is the underlying implementation of torch.tensor.to?

username: zy2uestc
cooked: <p>I’m a freshman of PyTorch and C++, I can easily use ‘torch.tensor.to’ function to transfer model between CPU and GPU. But I want to understand what is the underlying implementation of torch.tensor.to, especially how to malloc memory and copy memory.</p>
<p>I can find def to() in source code ‘torch/nn/modules/module.py’, but I don’t know how to interact with C++ and CUDA source code underlying. Where is this part source code?</p>
<p>Can anyone help to explain this question? Thanks</p>

username: ptrblck
cooked: <p>In case you are looking for the CUDA memory management take a look at <a href="https://pytorch.org/docs/stable/notes/cuda.html#memory-management">these docs</a> and if you are interested about the internal implementation of the <code>CUDACachingAllocator</code> check the code <a href="https://github.com/pytorch/pytorch/blob/7f7f6267e99f8dd5990e43dd2301ac37bb66e299/c10/cuda/CUDACachingAllocator.cpp#L50">here</a>.</p>

username: zy2uestc
cooked: <p>Thanks. I’ll follow your suggestion to learn.</p>
How to use FSDP and ema together?

username: walle_autoscale
cooked: <p>Hi, currently in pytorch2.0, FSDP model do not support deepcopy, how can I copy a model param as ema and update it?</p>
<p>example, my fsdp model:</p>
<pre><code class="lang-auto">        sharding_strategy=torch.distributed.fsdp.ShardingStrategy.SHARD_GRAD_OP
        model = FSDP(model, sharding_strategy=sharding_strategy,
                     ignored_parameters = not_trainable, )
</code></pre>
<p>I can’t call <code>deepcopy(model)</code> directly, how should I achive ema?</p>

username: yiftach
cooked: <p>In general (even without FSDP) it is advised to only store/copy the model’s <code>state_dict()</code>, not the model itself. Have you tried that?</p>

username: walle_autoscale
cooked: <p>Thanks. I will try only store/copy the model’s <code>state_dict()</code>.</p>

username: walle_autoscale
cooked: <p>I tried to use <code>summon_full_params</code> to get full param and update ema.<br>
However, this seem to cause too much memory fragments (reserved mem &gt;&gt; allocated mem).</p>
<p>So I abandoned FSDP and use deepspeed instead.</p>