akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torchsample has been revamped and renamed!

ncullen93: <p>Hi pytorch community,</p>
<p>A long time ago I created the torchsample package to make it easier to train pytorch models on medical imaging datasets. I have recently completely reworked the package to provide even better high-level tools for augmentation, training, and visualizing medical imaging AI models. It is now available at <a href="https://github.com/ncullen93/nitrain" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ncullen93/nitrain: Augmentation, training, and visualization tools for medical imaging AI models</a></p>
<p>Feel free to check it out and I would love for any feedback on how we can keep improving the tools available for the medical imaging AI community.</p>
Any different between model(input) and model.forward(input)

ThaiThien: <pre><code>class MyModel(nn.Module):
    def __init__(self, cuda, word_dim, tag_dim, mem_dim, criterion):
        super(MyModel, self).__init__()
   
    def forward(input):
         .. # do something
         return output
</code></pre>
<p><code>model = MyModel()</code></p>
<p>Is there any different if I called</p>
<p><code>model.forward(input)</code></p>
<p>rather than</p>
<pre><code>model(input)
</code></pre>
<p>Because only when I call model.forward(input), IDE (in this case, Pycharm) suggest me argument for forward function.</p>

fmassa: <p>You should avoid calling <code>Module.forward</code>.<br>
The difference is that all the hooks are dispatched in the <code>__call__</code> function, so if you call <code>.forward</code> and have hooks in your model, the hooks won’t have any effect</p>

Mithun_Paul: <p>But who/what exactly calls the .forward() function? is it the <strong>init</strong>()? In my case I am being told in error that the .forward is not getting all the arguments it needs, but I am not able to figure out where in the control flow its getting lost.</p>

ptrblck: <p>As <a class="mention" href="/u/fmassa">@fmassa</a> said, <code>forward</code> is called in the <code>.__call__</code> function. Have a look at the <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L485">line of code</a>.</p>

Oleksandra_Sopova: <p>What so you mean by the “hooks”?</p>

dngros: <p>Just avoiding using forward directly doesn’t really solve the issue that it breaks IDE support. One potential solution could be this <a href="https://discuss.pytorch.org/t/solution-to-different-forward-for-train-and-inference-ide-support-for-forward-args/33756" class="inline-onebox">Solution to different forward for train and inference + IDE support for forward args</a> . Here I just wrapped nn.Module to allow a forward method with both hooks and IDE support is retained.</p>

ThaiThien: <p>I haven’t use “hooks” yet, but it is like event trigger.</p>

Gillian: <p>Hello from 6 years later, here is a link targetting the commit instead of the main branch, showing the right line (as i suppose) <a href="https://github.com/pytorch/pytorch/blob/d564ecb4a515e34184c976617f57b2eb01665660/torch/nn/modules/module.py#L485" rel="noopener nofollow ugc">pytorch/torch/nn/modules/module.py at d564ecb4a515e34184c976617f57b2eb01665660 · pytorch/pytorch · GitHub</a></p>
Runtime error: Detected that your are using FX to symbolically trace a dynamo optimized function. This is not supported at the moment

Irfanuet191: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc8292d0f22d9f457cc70836835c772a4deb332e.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc8292d0f22d9f457cc70836835c772a4deb332e" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc8292d0f22d9f457cc70836835c772a4deb332e.png" alt="Capture" data-base62-sha1="tbb9He7eYn1VFSFDE0B0m76RzGu" width="690" height="266" data-dominant-color="0E0D0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1633×631 33.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
“Runtime error: Detected that your are using FX to symbolically trace a dynamo optimized function. This is not supported at the moment.”<br>
When load increase on the server I get this error. When there are small number of requests it work pretty great but as soon as the load increase I started to get this error. To tackle it I have to start the service again.</p>

gug: <p>I am facing the same error.</p>
<p>When im using multithreading with compiled model, i get this error. but if i dont compile the model. the program runs smoothly.</p>
Per Sample Gradients

likhilb9: <p>Is there any way to calculate the per sample gradients with batch norm.<br>
I guess Opacus convert the batch norm to Group norm and then calculate the per sample gradients is there any efficient way to calculate?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/likhilb9">@likhilb9</a>,</p>
<p>Can you share a minimal reproducible example?</p>

likhilb9: <p>from nn.utils._per_sample_grad import call_for_per_sample_grads<br>
res = call_for_per_sample_grads(model)(input)<br>
loss(res, labels).backward()<br>
for param in model.parameters():<br>
param.grad_sample # instead of the grad, per sample gradients are in the grad_sample field<br>
The above code does not work for batch norm layers</p>
PyTorch version of scipy.special.hyperu

AlphaBetaGamma96: <p>Hi All,</p>
<p>Is there an equivalent of <code>scipy.special.hyperu</code> within PyTorch? I’ve read the docs within the <code>torch.special</code> namespace and the <code>hyperu</code> function isn’t there.</p>
<p>Is there an equivalent for the confluent hypergeometric function?</p>
Use gradcam gradients during training

Giovanni_Cena: <p>Hi,</p>
<p>I’m trying to implement a custom training procedure with gradcam maps involved. In particular, I want to use them inside the loss function.<br>
To do so, the generated maps need to track the activation function, to perform the backpropagation.</p>
<p>The question I’m asking is: Is it possible to ask Pytorch to keep gradients while tensors are manipulated by a library?</p>
<p>Is there another way around for keeping gradients in the heatmap tensors produced by a gradcam library without reimplementing it from scratch? Or do I have to do the re-implementation keeping always the gradients (so, practically speaking, checking if the feature maps taken with hooks, which are tensors, are always being manipulated with grad_fn with them?)</p>
<p>Thanks in advance!</p>
Relation between `torchvision.datasets.folder.ImageFolder`, `torch.utils.data.ConcatDataset` and `torch.utils.data.DataLoader`

palguna_gopireddy: <p>I have seen the <a href="https://github.com/davin11/CNN-NLM" rel="noopener nofollow ugc">following code</a>. I don’t understand does it produce different patches of an image images multiple times, (or) same patch of an image multiple times.</p>
<pre><code class="lang-auto">    trainsetiters = 640

    transform_train = Compose([
        sar_dataset.RandomCropPil(patchsize),                     # to crop randomly
        sar_dataset.Random8OrientationPil(),                      # to rotate randomly
        sar_dataset.PilToGrayTensor(bayes=1.0,scale=scale_img),   # To convert to grey scale
    ])

    trainset = PlainImageFolder(dirs=train_data_folder, transform=transform_train, cache=True)
    trainset = torch.utils.data.ConcatDataset([trainset] * trainsetiters)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=batchsize, shuffle=True, num_workers=20)
</code></pre>
<p>The <a href="https://github.com/davin11/CNN-NLM/blob/master/dataset/sar_dataset.py" rel="noopener nofollow ugc">PlainimageFolder</a> function used above is a custom function inspired from <code>torchvision.datasets.folder.ImageFolder</code></p>
<p>My doubt is does this <code>torch.utils.data.DataLoader</code> produces:<br>
multiple (640 times) different patches from the same image in an epoch<br>
(or)<br>
the same patch of an image multiple times (640 times) is repeated in an epoch?</p>
Asynchronous Training of Deep Learning Models

Sar2580P: <p>I am thinking of how would it be if I can create asynchronous <strong>forward</strong> function in sub-class of nn.Module . When I came across architecture in attached image, I felt that it would be faster if we could train pass through each branch parallelly.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df2666c201824770dd42be16f1acdf8dea6792f8.png" alt="pic" data-base62-sha1="vQ4Kx1euD0ye5DfadsBQQfWc8mA" width="491" height="303"></p>
<p>Another scenario I have come across is when input is of shape (B, C ,V ,T) , where :<br>
<code>B: batch_sz , C: node_features , V : num_nodes , T : time_stamps</code><br>
So, there is a graph for each time-stamp corresponding to each instance in the batch. Now there is a module which calculates Adj_matrix of a graph based on its node_features… It would be compute intensive if we club (B,C,V,T)-&gt;(BxT, C,V) , and then treat (B*T) as new batch , where T~=250 , B=32…</p>
<p>Is there some work-around that we can  evaluate adj_mat efficiently… This problem is not only limited to calculation of adj_mat through my custom module, but rather to use any graph layer from PyG, we need to club B,T</p>
<p><strong>I’m not talking about training on multiple GPU, but making calls asynchronously rather than sequentially.</strong></p>
<p>Please answer both the question , thanks in advance</p>
Partial derivatives using autograd for function with multiple parameters

THarIKH_M: <p>Hi<br>
I have a function <code>f(x,pi(x;theta))</code> where pi is a neural network with parameter theta. x is input with n dimension. I also have a function <code>w(x,u)</code> where <code>u = pi(x;theta)</code> and u is of m-dimension. I need to find <code>f_x, dw/dx, matmul(f_u, u_theta), matmul(w_u, pi_theta)</code> where f_x means partial derivative w.r.t x, similarly for f_u (f with respect to u), u_theta (u with respect to theta), w_u (w with respect to u), pi_theta (pi with respect to theta). I will use these derivatives to find an approximations for finding <code>dL/dtheta</code> to update my neural network parameter. I want an estimator of above derivatives in an efficient and in batched manner.<br>
So can anyone help me in finding these derivatives.<br>
Thanks in advance<br>
regards<br>
Tharikh</p>
Pytorch not working with Nvidia 4070Ti

shanze: <p>NVIDIA GeForce RTX 4070Ti with CUDA capability sm_89 is not compatible with the current PyTorch installation.</p>
<p>The error is as follows,<br>
NVIDIA GeForce RTX 4070 Ti with CUDA capability sm_89 is not compatible with the current PyTorch installation.\nThe current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70</p>
<p>The torch now is 2.2.1+cu118, cuda is 11.8. However, I check the torch.cuda.get_arch_list(), and get [‘sm_50’, ‘sm_60’, ‘sm_70’, ‘sm_75’, ‘sm_80’, ‘sm_86’, ‘sm_37’, ‘sm_90’].<br>
There is no result for sm_89.</p>
<p>My OS is Ubuntu 18.04.01<br>
Please help me solve this issue…<br>
I can provide more information, if you tell me what Info I should provide</p>

ptrblck: <p>All current stable and nightly binaries support your 4070Ti, as sm_89 is binary compatible to sm_86 and sm_80.</p>
Creating array mask based on matching subintervals

u7122029: <p>Suppose I have a list of encoded tokens and an encoded string of words</p>
<pre><code class="lang-auto">tokens = [[5], [243,290], [0,2,0,4], ...]
prompt = [243, 5, 290, 5, 0, 2, 243, 290, 0, 2, 0, 4, ...]
</code></pre>
<p>I want to create a 0-1 mask where the 1-s correspond to where the tokens match exactly with the prompt. In the example the mask would look something like</p>
<pre><code class="lang-auto">mask = [0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, ...]
</code></pre>
<p>Notice that the 243 and 290 at the beginning of the prompt are 0 because the two numbers are not next to each other, and the first <code>0, 2</code> are not marked as 1 because they are only part of the <code>0,2,0,4</code> token.</p>
<p>I am thinking that the way to implement this would be to loop over each of the tokens, and perform a sliding window technique for each one, but I think this can be better optimised, and I am not sure how. Could I please have some suggestions for this?</p>
<p>Thanks in advance!</p>
Text Multiclass CNN loss Problem

7lotus7: <h2><a name="hello-im-getting-this-error-while-trainig-my-cnn-model-on-multicalss-text-dataset-1" class="anchor" href="#hello-im-getting-this-error-while-trainig-my-cnn-model-on-multicalss-text-dataset-1"></a>Hello , I’m getting this error while trainig my CNN model on multicalss text  dataset</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
Cell In[136], line 1<br>
----&gt; 1 train(model,data,10,0.001,16)</p>
<p>Cell In[133], line 31, in train(model, data, epochs, learning_rate, batch_size)<br>
28 print(y_pred)<br>
30 # Calculate the cross-entropy loss<br>
—&gt; 31 loss = criterion(y_pred, y_batch)<br>
33 # Clean gradients<br>
34 optimizer.zero_grad()</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1518, in Module._wrapped_call_impl(self, *args, **kwargs)<br>
1516     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]<br>
1517 else:<br>
 → 1518     return self._call_impl(*args, **kwargs)</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1527, in Module._call_impl(self, *args, **kwargs)<br>
1522 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1523 # this function, and just call forward.<br>
1524 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks<br>
1525         or _global_backward_pre_hooks or _global_backward_hooks<br>
1526         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1527     return forward_call(*args, **kwargs)<br>
1529 try:<br>
1530     result = None</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/loss.py:1179, in CrossEntropyLoss.forward(self, input, target)<br>
1178 def forward(self, input: Tensor, target: Tensor) → Tensor:<br>
 → 1179     return F.cross_entropy(input, target, weight=self.weight,<br>
1180                            ignore_index=self.ignore_index, reduction=self.reduction,<br>
1181                            label_smoothing=self.label_smoothing)</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:3053, in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)<br>
3051 if size_average is not None or reduce is not None:<br>
3052     reduction = _Reduction.legacy_get_string(size_average, reduce)<br>
 → 3053 return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)</p>
<p>RuntimeError: Expected floating point type for target with class probabilities, got Long<br>
when i printed the y batch and y_pred , im getting this :tensor([3, 1, 3, 0, 2, 2, 2, 1, 2, 1, 3, 2, 1, 2, 0, 3])<br>
tensor([-0.0000, -0.0000, -0.0000, -0.0000, -0.2270,  0.1651, -0.2672, -0.0000,<br>
-0.3112, -0.0181, -0.0000, -0.2580, -0.0000, -0.1150, -0.0000, -0.0351],<br>
grad_fn=)<br>
Can you please help me to find out the problem</p>
Accessing attention weights for ViT models

rezaarezvan: <p>Hey!</p>
<p>How can I get the attention weights of the available pre-trained ViT (In my case im using the <code>vit_b_16</code>)</p>
<p>I ultimately want to visualize an attention map over the original input image.</p>
<p>I’ve read the source code for the Vision transformer class and from what I understand we explicitly set the <code>need_weights</code> flag to <code>False</code> at the Encoder block.</p>
Why is loss not converging?

CMJr: <p>I’m trying to make a deep Q network for the Lunar Lander v2 environment.  I got the model to run, but it isn’t converging at all; the loss and environment rewards aren’t improving.  What’s wrong with my code?<br>
Here’s a link to my code on Colab: <a href="https://colab.research.google.com/drive/1w8VuD3ETgvt3-m7fLHqhaWvMhlJPUqWN?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a><br>
Some of the more important parts:</p>
<pre><code class="lang-auto">def __init__(self):
        super(ActorCritic, self).__init__()

        self.action_layer = nn.Linear(128, 4)
        self.value_layer = nn.Linear(128, 1)

        self.state_size = 8
        self.action_size = 4

        self.state_dense = nn.Linear(self.state_size, 256)
        self.action_dense = nn.Linear(self.action_size, 256)

        self.hl1 = nn.Linear(256, 512)
        self.dp1 = nn.Dropout(0.25)
        self.hl2 = nn.Linear(512, 512)
        self.dp2 = nn.Dropout(0.25)

        self.outputs = nn.Linear(512, 1)

        self.state_values = []
        self.rewards = []

    def forward(self, state, action):

        state = torch.from_numpy(state).float()
        state = F.relu(self.state_dense(state))

        action = np.array(action)
        action = torch.from_numpy(action).float()
        action = F.relu(self.action_dense(action))

        combined = torch.mul(state, action)

        value = F.relu(self.hl1(combined))
        value = self.dp1(value)
        value = F.relu(self.hl2(value))
        value = self.dp2(value)
        value = self.outputs(value)

        self.state_values.append(value)

        return value

    def calculateLoss(self, gamma=0.99):

        # calculating discounted rewards:
        rewards = []
        dis_reward = 0
        for reward in self.rewards[::-1]:
            dis_reward = reward + gamma * dis_reward
            rewards.insert(0, dis_reward)

        # normalizing the rewards:
        rewards = torch.tensor(rewards)
        rewards = (rewards - rewards.mean()) / (rewards.std())

        loss = 0
        for value, reward in zip(self.state_values, rewards):
            value = value.to(torch.float64)
            value_loss = F.mse_loss(value, reward)
            loss += value_loss
        return loss
</code></pre>
<p>How I update the network:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss = policy.calculateLoss(gamma)
loss.backward()
optimizer.step()
policy.clearMemory()
</code></pre>
PyTorch standard Coco dataset (datasets.CocoDetection) not compatible with Faster R-CNN object detection model

ranjaniocl: <p>Hi,<br>
I am trying to train and evaluate pre-trained Faster R-CNN model with standard coco dataset. I am getting the following error</p>
<p>TypeError: RandomIoUCrop() requires input sample to contain tensor or PIL images and bounding boxes. Sample can also contain masks.</p>
<p>Here are the high level steps</p>
<ol>
<li>Downloaded the COCO 2017 dataset</li>
<li>Prepared PyTorch dataset using standard steps from <a href="https://pytorch.org/vision/main/auto_examples/transforms/plot_transforms_e2e.html#sphx-glr-auto-examples-transforms-plot-transforms-e2e-py" class="inline-onebox" rel="noopener nofollow ugc">Transforms v2: End-to-end object detection/segmentation example — Torchvision main documentation</a></li>
<li>Training and evaluating Faster R-CNN model using steps from <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.2.1+cu121 documentation</a></li>
</ol>
<p>Any help will be appreciated. Thanks.</p>

ranjaniocl: <p>Here is colab notebook to view the steps and error I am getting.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1Tbu2Thf-thn0lLG12dM3bq_BMZDihX2Y?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/a01fcfa49b15ed5edf590b766bd247d1/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1Tbu2Thf-thn0lLG12dM3bq_BMZDihX2Y?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1Tbu2Thf-thn0lLG12dM3bq_BMZDihX2Y?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to train a vision transformer model without pretrained parameters?

Knight_None: <p>I’m only a freshman for transformer.  I trained my vit model on flower102 dataset in 400 epochs, but however I adjust my hyper-parameters, the test accuracy reachs only 30%!<br>
Then I tried pytorch official vit-b model without any pretrained parameters, the result is same!  No matter what data augmentation options I select, the vit model’s test accuracy only stops at about 30%.</p>
<p>Is it indispensable for me to use pretrained parameters of Google’s to fine-tune my vit model?<br>
Sometime, I need to change the structure of MSA or SW-MSA to design a new model for my work, so I can’t use pretrained parameters.</p>
Make torch linear layer probabilistic

Yauheni_Mardan: <p>I’d like to make a native torch nn.Linear <code>layer</code> probabilistic by</p>
<ol>
<li>Adding to the <code>layer</code> weight and bias distribution objects using <code>register_module</code> method. For example:</li>
</ol>
<pre><code class="lang-auto">layer.register_module('weight_dist', some_weight_dist_obj)
layer.register_module('bias_dist', some_bias_dist_obj)
</code></pre>
<p>Where some_weight_dist_obj and some_bias_dist_obj are objects of the class <code>GaussianVariable </code></p>
<pre><code class="lang-auto">class GaussianVariable(AbstractVariable):
    ...
    def sample(self) -&gt; Tensor:
        epsilon = torch.randn_like(self.sigma)
        return self.mu + self.sigma * epsilon. # self.mu and self.sigma are trainable nn.Parameter objects
</code></pre>
<ol start="2">
<li>Replacing the old <code>forward</code> with a new one:</li>
</ol>
<pre><code class="lang-auto">def new_forward(self, input: torch.Tensor) -&gt; torch.Tensor:
    sampled_weight = self.weight_dist.sample()
    sampled_bias = self.bias_dist.sample()
    return F.linear(input, a, b)
...
layer.forward = new_forward.__get__(layer, nn.Module)
</code></pre>
<p><strong>In this setting I don’t face any problem</strong> during network training i.e. self.mu and self.sigma are being updated.</p>
<p>However, I don’t want to reimplement a forward function as I showed in <span class="hashtag-raw">#2</span> (For <code>nn.Linear</code> it is ok, but next I want to use also other, more complex layers). Rather I would like to use <code>nn.Linear.forward</code> again. For that, I tried many options, but none of them worked:</p>
<ol>
<li>I did not replace <code>forward</code> with <code>new_forward</code>, but used hooks:</li>
</ol>
<pre><code class="lang-auto">def forward_hook(module, inputs):
    nn.utils.vector_to_parameters(module.weight_dist.sample().flatten(), module.weight)
    nn.utils.vector_to_parameters(module.bias_dist.sample().flatten(), module.bias)
    return inputs
...
layer.register_forward_pre_hook(forward_hook)
</code></pre>
<ol start="2">
<li>Used the following <code>new_forward</code>:</li>
</ol>
<pre><code class="lang-auto">def new_forward(self, input: torch.Tensor) -&gt; torch.Tensor:
    nn.utils.vector_to_parameters(self.weight_dist.sample().flatten(), self.weight)
    nn.utils.vector_to_parameters(self.bias_dist.sample().flatten(), self.bias)
    return nn.Linear.forward(self, input)
</code></pre>
<ol start="3">
<li>Used different assignments:</li>
</ol>
<pre><code class="lang-auto">nn.utils.vector_to_parameters(self.weight_dist.sample().flatten(), self.weight)
or
self.weight.data = self.weight_dist.sample()
or
self.weight = nn.Parameter(self.weight_dist.sample())
</code></pre>
<p>Could you help me with that? How to properly reuse <code>forward</code> method in this setting?</p>

Yauheni_Mardan: <p>P.S. I don’t want to inherit from nn.Linear and create own ProbLinear or something</p>
Image segmentation debugging using the diffusion model and detectron2

raushankm: <p>I am training a diffusion model built on top of detectron2 on the Publaynet dataset for instance segmentation. But the output I am getting after multiple iterations is it is segmenting the whole document as the figure. It doesn’t segment individual elements like tables and texts in the document. The loss function is decreasing very well at a learning rate of 0.00005. Total loss decreases to 1.6. Loss_bbox is around 0.200 and loss:giou is 0.3573.</p>
<p>I did the sanity check the input in the model seems to be correct. I visualized the input and bounding boxes and also checked the targets. Everything seems to be correct. I don’t know if I should train the model further or change any other hyperparameters. I tried with multiple learning rate this learning rate seems to be good. NUM_PROPSALS is 500. Should I raise to 1000. WHat hyperparameters should I be specifically care about. The code is not from scratch. I am taking the repo from here <a href="https://github.com/chenhaoxing/DiffusionInst" class="inline-onebox" rel="noopener nofollow ugc">GitHub - chenhaoxing/DiffusionInst: This repo is the code of paper "DiffusionInst: Diffusion Model for Instance Segmentation" (ICASSP'24).</a>. So I am not building the model from scratch. Let me know if anyone has any idea.</p>
<p>Below is the hyperparameters list from config file.</p>
<pre><code class="lang-auto">MODEL:
  META_ARCHITECTURE: "DiffusionInst"
  WEIGHTS: "detectron2://ImageNetPretrained/torchvision/R-50.pkl"
  PIXEL_MEAN: [123.675, 116.280, 103.530]
  PIXEL_STD: [58.395, 57.120, 57.375]
  BACKBONE:
    NAME: "build_resnet_fpn_backbone"
  RESNETS:
    OUT_FEATURES: ["res2", "res3", "res4", "res5"]
  FPN:
    IN_FEATURES: ["res2", "res3", "res4", "res5"]
  ROI_HEADS:
    IN_FEATURES: ["p2", "p3", "p4", "p5"]
  ROI_BOX_HEAD:
    POOLER_TYPE: "ROIAlignV2"
    POOLER_RESOLUTION: 7
    POOLER_SAMPLING_RATIO: 2
SOLVER:
  IMS_PER_BATCH: 2
  BASE_LR: 0.0000125
  STEPS: (210000, 250000)
  MAX_ITER: 270000
  WARMUP_FACTOR: 0.01
  WARMUP_ITERS: 1000
  WEIGHT_DECAY: 0.0001
  OPTIMIZER: "ADAMW"
  BACKBONE_MULTIPLIER: 1.0  # keep same with BASE_LR.
  CLIP_GRADIENTS:
    ENABLED: True
    CLIP_TYPE: "full_model"
    CLIP_VALUE: 1.0
    NORM_TYPE: 2.0
SEED: 40244023
INPUT:
  MIN_SIZE_TRAIN: (480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800)
  CROP:
    ENABLED: False
    TYPE: "absolute_range"
    SIZE: (384, 600)
  FORMAT: "RGB"
TEST:
  EVAL_PERIOD: 20000
DATALOADER:
  FILTER_EMPTY_ANNOTATIONS: False
  NUM_WORKERS: 2
VERSION: 2
</code></pre>
Instantiate, loading and saving different model with HYBRID_SHARD FSDP

Qinsheng_Zhang1: <p>Assume I am using HYBRID_SHARD FSDP and model are sharded in one node, is it possible to create different model and loading different model checkpoint for different node, and save different checkpoint per node.</p>
<p>This is under the scenario where the FSDP model does not engage in the model’s forward and backward passing.</p>
<pre data-code-wrap="py"><code class="lang-py">coef = node_id
model = ModelClass(coef)
# how to construct FSDP
# how to load checkpoint based on node_id
# how to save FSDP checkpoint based on node_id 
```</code></pre>
Equivalent to PyArray_SetBaseObject for tensors, to free memory

tomgrek: <p>I’m creating a tensor in C++, turning it into a PyObject* with THPVariable_Wrap, packing that into a tuple, then running some Python code (user code, run on the interpreter embedded in C++) which uses that tuple. It leaks the memory.</p>
<p>With NumPy arrays you do <code>PyArray_SetBaseObject(array, tuple)</code> so when the tuple’s ref counter goes to zero, the array data is freed.</p>
<p>Is there any equivalent to that in libtorch - or other way to free the tensor’s data some time later?</p>

tomgrek: <p>Playing around, this solves 99% of the leaked tensor data:</p>
<p>…sometime later, in another thread far far away…</p>
<pre><code class="lang-auto">auto tensor = THPVariable_Unpack(PyObject*)
auto ptr = tensor.unsafeReleaseIntrusivePtr();
ptr.get()-&gt;release_resources();
</code></pre>
"print(torch.cuda.nccl.version())" throws "AttributeError: module 'torch._C' has no attribute '_nccl_version'"

Verner_Mednis: <p>That’s what it does.<br>
ChatGPT suggested to try running:<br>
"import torch</p>
<p>print(torch._<em>version</em>_)<br>
print(torch.cuda.is_available())<br>
print(torch.version.cuda)<br>
print(torch.cuda.nccl.version())" before trying the llama chatbot.</p>
<p>The only one that didn’t work was “print(torch.cuda.nccl.version())” with the error:<br>
“Traceback (most recent call last):<br>
File “&lt;stdin&gt;”, line 1, in &lt;module&gt;<br>
File “C:\Python310\lib\site-packages\torch\cuda\nccl.py”, line 35, in version<br>
ver = torch._C._nccl_version()<br>
AttributeError: module ‘torch._C’ has no attribute ‘_nccl_version’”.</p>
<p>P. S.<br>
If this helps then:<br>
“nvcc --version” returns “Build cuda_12.4.r12.4/compiler.33961263_0”,<br>
“torch.version.cuda” returns “12.1”</p>

ptrblck: <p>Your binary does not ship with NCCL as it’s not supported on Windows.<br>
I don’t know what you are trying to do exactly, but you won’t be able to use NCCL for a multi-GPU run and would need to use e.g. Gloo.</p>
Use SeamlessM4Tv2Model, I want to slow down the rate of speech of audio output

lam_vu_Nguyen: <pre><code class="lang-auto">text_inputs = processor(text="I have a daughter 2 years old, I wanted her name to be Hương Ly", src_lang="eng", return_tensors="pt").to(device)
audio_array = model.generate(**text_inputs, tgt_lang=language)[0].cpu().numpy().squeeze()
file_path = 'audio_from_text.wav'
sf.write(file_path, audio_array, 16000)
</code></pre>
<p><a href="https://huggingface.co/docs/transformers/main/en/model_doc/seamless_m4t_v2" rel="noopener nofollow ugc">doc</a><br>
[<a href="https://huggingface.co/spaces/Imadsarvm/Sarvm_Audio_Translation/resolve/b5b7fa364f8c567c4eb330583f673cd4c600976b/app.py?download=true" rel="noopener nofollow ugc">ex</a>]</p>
<p>it has returned a 3 seconds audio</p>
<p>I try adding <code>speech_temperature=0.2</code> or <code>speech_do_sample=True</code> to <code>generate()</code> but there is no change, it still has returned a 3 seconds audio, for example, I want to change the rate of speech so it will be 5 seconds audio<br>
any ideal ?</p>
Almost same file size when saving model's state_dict vs full model in Google Colab

asad: <p>As far as i could see from documentation and forums, the recommended method to save the model for inference later is to save with</p>
<p><code>torch.save(model_name.state_dict(), "path_of_file.pt")</code></p>
<p>I tried just that in a Jupyter notebook on Google colab after training the model, but it seems to have saved the complete model as the file size is almost same as when I saved the full model (18MB). Is it possible for the two file sizes to be same? And can I verify without loading either of them if the state_dict was successfully saved?</p>
<p>My model is a custom inherited class of torch.nn.Module(), and torch version is 2.2.1+cu121</p>
<p>I have verified the size again, and the state_dict file is only around 5KB smaller than the full model. What could be the cause of this?<br>
My model had around 4 Million parameters and around 15 mixed layers of Conv2d, Relu, BatchNorm2d, MaxPool2d, dropout and Linear. It was set to evaluation mode twice after training, to validate and test, and then I tried to save it.</p>

ptrblck: <p>I would expect the majority of the size of a checkpoint would be the <code>model.state_dict()</code>, not the actual model code, so I think your observation is reasonable.</p>
PyTorch project no more working after upgrade to Ubuntu 22.04

Pavils_Jurjans: <p>I think I’m missing some general universal understanding how to approach these kind of things.</p>
<p>I have a Python project (<a href="https://github.com/AUTOMATIC1111/stable-diffusion-webui" class="inline-onebox" rel="noopener nofollow ugc">GitHub - AUTOMATIC1111/stable-diffusion-webui: Stable Diffusion web UI</a>) that has it’s own virtualenv with all the relevant modules.</p>
<p>It used to run on Ubuntu 20.04, before I upgraded to Ubuntu 22.04. Now it does not pass the <code>torch.cuda.is_available()</code> test.</p>
<p>Some version info:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.__version__)
2.0.1+cu118

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

From NVIDIA Server app:
Operating System: Linux-x86_64
NVIDIA Driver Version: 525.116.04
Graphics Processor: NVIDIA GeForce RTX 2070 Super
CUDA Cores: 2560
Total Memory: 8192 MB
</code></pre>
<p>I guess there’s some driver/version incompatibility issue going on, but have no idea from where to start. How can I debug this situation and move towards solution?</p>

ptrblck: <aside class="quote no-group" data-username="Pavils_Jurjans" data-post="1" data-topic="180794">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pavils_jurjans/48/56234_2.png" class="avatar"> Pavils_Jurjans:</div>
<blockquote>
<p>I guess there’s some driver/version incompatibility issue going on, but have no idea from where to start.</p>
</blockquote>
</aside>
<p>I agree as the failure points towards a broken driver installation.<br>
I don’t know if you’ve reinstalled the NVIDIA drivers after updating to Ubuntu 22.04, but I would assume the update might have broken your setup.</p>

Pavils_Jurjans: <p>Well, after purging the NVIDIA drivers, rebooting in safe mode, and reinstalling them back, PyTorch now says that cuda_is_available. But not without several completely dead boots in between, where it stopped with almost black screen, just some system messages, and then nothing. At one point almost decided I have to reinstall the whole desktop. Quite stressful experience, do not recommend.</p>

Crunchy: <p>My stack repeatedly breaks after Ubuntu updates also, I think the kernel updates are doing it. It puts the kernel modules out of sync with the install cuda libs. I think it’s because the kernel update needs a reboot but the libs don’t. It’s a paint though as it happens frequently.</p>
Pre-compute minibatch

stevethesteve: <p>I’d like to do data augmentation and feature extraction on the fly. However, this is computationally expensive. Is there a way to pre-compute a minibatch (data augmentation + feature extraction) on the CPU while the <em>previous</em> minibatch is being used to train the model on the GPU?</p>

ptrblck: <p>Yes, by setting <code>num_workers&gt;=1</code> in the <code>DataLoader</code> multiple processes will be spawned which will preload the batches in the background.</p>
How to add Attribute and use after building the model?

Hrridoyv2: <blockquote>
<p>I’ve built a neural network model and I’d like to incorporate custom functions, <code>encodeImage</code> and <code>encodeText</code> , for pre-processing data. Ideally, I want these functions to be callable both during model definition and after training (post-build). However, including them directly within the model definition restricts their use to before Just-In-Time (JIT) compilation. Calls made after model building result in the functions being undefined</p>
</blockquote>
<pre><code class="lang-auto"># The Custom Attributes I wan to add in the Model
    def encode_image(self, image):
      return self.visual(image.type(self.dtype))

    def encode_text(self, text):
      x = self.token_embedding(text).type(self.dtype)  # [batch_size, n_ctx, d_model]

      x = x + self.positional_embedding.type(self.dtype)
      x = x.permute(1, 0, 2)  # NLD -&gt; LND
      x = self.transformer(x)
      x = x.permute(1, 0, 2)  # LND -&gt; NLD
      x = self.ln_final(x).type(self.dtype)

      # x.shape = [batch_size, n_ctx, transformer.width]
      # take features from the eot embedding (eot_token is the highest number in each sequence)
      x = x[torch.arange(x.shape[0]), text.argmax(dim=-1)] @ self.text_projection

      return x
  # Image Classifier Neural Network
  class ImageClassifier(nn.Module):
      def __init__(self, n_qubits, n_layers, encode_image):
          super().__init__()
          self.model = nn.Sequential(
              qlayer,
              ClassicalLayer(2)
          )
  
      def forward(self, x):
          result = self.model(x)
          return result
</code></pre>
<pre><code class="lang-auto">  batch_size = 28
      channels = 1
      height = 28
      width = 28
      example_input = torch.randn(height, width)
  
      traced_model = torch.jit.trace(clf, example_input)
  
      # Save JIT archive
      traced_model.save('qunn.pt')
      with open('model_state.pt', 'wb') as f:
          save(clf.state_dict(), f)
</code></pre>
How do I determine the longest sequence length that fits into memory?

Goldname: <p>I am trying to fine tune a transformer embedding model to embed long documents of around 8000-10000 tokens. However, I keep running into CUDA out of memory. The embedding model is only about a few hundred megabytes and can fit inside my 40gb gpu memory. I tried using deepspeed and fsdp distributed training in order to split the model across 8 gpus, but i still run out of memory.</p>
<p>I am wondering how can I determine the longest sequence length I can train with my model, either through calculations or code?<br>
Additionally, shouldn’t fsdp/deepspeed be supplying enough memory? My intuition says that 8*40Gb should be plenty enough for 10000 sequence length and 200 mb model, but I don’t know how to check this statement.</p>
How to set up audio data for audio classification tasks using PyTorch and torchaudio?

Teddy_Salas: <p>How to setup audio data for audio classification task using lstm?</p>
<p>for  example this is the setup for image data for image classification task ,like this how do i setup my data for audio classification,please help.</p>
<p>import torch<br>
from torch import nn<br>
import torchvision<br>
from torchvision import transforms</p>
<p>train_transforms = transforms.Compose([<br>
transforms.Resize(size=(224,224)),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.RandomRotation(10),<br>
transforms.TrivialAugmentWide(num_magnitude_bins=31),<br>
transforms.ToTensor(),</p>
<p>])</p>
<p>test_transforms = transforms.Compose([<br>
transforms.Resize(size=(224,224)),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.RandomRotation(10),<br>
transforms.ToTensor(),</p>
<p>])from torchvision import datasets</p>
<p>train_data = datasets.ImageFolder(<br>
root=train_dir,<br>
transform=train_transforms<br>
)</p>
<p>test_data = datasets.ImageFolder(<br>
root=test_dir,<br>
transform=test_transforms<br>
)from torch.utils.data import DataLoader</p>
<p>train_dataloader = DataLoader(<br>
dataset = train_data,<br>
batch_size = 28,<br>
num_workers = os.cpu_count(),<br>
shuffle=True<br>
)</p>
<p>test_dataloader = DataLoader(<br>
dataset = test_data,<br>
batch_size=28,<br>
num_workers = os.cpu_count(),<br>
shuffle=False<br>
)</p>
How to deploy a trained PyTorch model in android playstore?urgent help

Teddy_Salas: <p>i have a trained  PyTorch model with me, i wanted to deploy it on android play store, how can i do that?, urgent help.</p>

marksaroufim: <p>You can take a look at the recent executorch examples for android <a href="https://github.com/pytorch/executorch/tree/main/examples/demo-apps/android" class="inline-onebox">executorch/examples/demo-apps/android at main · pytorch/executorch · GitHub</a></p>

Teddy_Salas: <p><a class="mention" href="/u/marksaroufim">@marksaroufim</a> okay thank you</p>
Exceptions in Torch Export

fabian_schutze: <p>Exception are currently not supported in torch.export.export. See the example below:</p>
<pre><code class="lang-auto">import os
import torch

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        raise RuntimeError("Exception")

with torch.no_grad():
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model = Model().to(device=device)
    example_inputs=(torch.randn(8, 10, device=device),)
    torch.export.export(model, example_inputs)
</code></pre>
<p>which results in the following (abbreviated) error:</p>
<pre><code class="lang-auto">Unsupported: call_function BuiltinVariable(RuntimeError) [ConstantVariable(str)] {}

from user code:
   File "/tmp/main.py", line 9, in forward
    raise RuntimeError("Exception")

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
</code></pre>
<p>However, exception are an idiomatic way to report errors and bail-out early. For example, I count 278 uses of exceptions in <a href="https://github.com/huggingface/diffusers/tree/main/src/diffusers/models" rel="noopener nofollow ugc">diffusers models</a> folder.</p>
<p>Is there a way to handle exceptions in exported code or what would be an idiomatic way to bail out early in programs? I can think of using <code>torch.cond</code> and separate functions in two paths and return error codes if needed, but that doesn’t seem to be very ergonomic.</p>
Train loss unexpectedly increase after epochs 2

adfad1: <p>Hi, I want to fine-tune vit-tiny/16 model on imagenet-1k with fp16 datatype, however the train loss unexpectedly increase after epochs 2, what may happen?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ce1244d80f9413046cdc549ac6305dffb968ba6.png" alt="trainloss" data-base62-sha1="47tJsGWWys767SGtBVqimqIsjdk" width="336" height="258"></p>