akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pdist vs. cdist performance

cinna: <h2><a name="i-have-seen-this-question-asked-in-the-past-but-i-am-getting-a-strange-result-where-pdist-is-almost-20-times-slower-than-cdist-i-believe-that-in-the-previous-versions-of-pytorch-i-did-not-see-this-difference-i-am-using-211-here-are-the-steps-to-reproduce-my-timings-1" class="anchor" href="#i-have-seen-this-question-asked-in-the-past-but-i-am-getting-a-strange-result-where-pdist-is-almost-20-times-slower-than-cdist-i-believe-that-in-the-previous-versions-of-pytorch-i-did-not-see-this-difference-i-am-using-211-here-are-the-steps-to-reproduce-my-timings-1"></a>I have seen this question asked in the past but I am getting a strange result where pdist is almost 20 times slower than cdist. I believe that in the previous versions of pytorch I did not see this difference (I am using 2.1.1)<br>
Here are the steps to reproduce my timings:</h2>
<p>import torch<br>
import scipy.spatial.distance # to convert pdist vector to matrix</p>
<p>x = torch.randn(3000,200,device=‘cuda’,dtype=torch.float32)<br>
x_unsqueeze = x.unsqueeze(0) # need batch dimension for cdist</p>
<p>pdist = torch.pdist(x).cpu().numpy()<br>
pdist = scipy.spatial.distance.squareform(pdist)</p>
<p>cdist = torch.cdist(x_unsqueeze,x_unsqueeze).squeeze().cpu().numpy()<br>
np.fill_diagonal(cdist,0) # set diagonal to 0 for comparison with pdist result</p>
<p>np.allclose(cdist,pdist) # verify that both computations give the same result</p>
<p>timeit torch.cdist(x_unsqueeze,x_unsqueeze).squeeze().cpu().numpy()<br>
<strong>13.1 ms</strong> ± 8.17 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)</p>
<p>timeit torch.pdist(x).cpu().numpy()<br>
<strong>215 ms</strong> ± 1.67 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</p>
NER model wrong output on Android

Mujahid_Khan: <p>Hi guys, I am very new to the ML world. Need an urgent help, I am using a pre-built MobileBERT NER model from huggingface and converted to PyTorch Lite at .PTL file and it’s working great with python.</p>
<p>On Android I was able to create same tokenizer, prediction decoder but it’s very wrong on Android, same model performing well on python environment.</p>
<p>I tried reconverting the model with optimised and also not. Nothing is working, can anyone give me any hints please</p>
<p>Hint: The raw output of the model on Android are bigger numbers than than it’s showing on python environment</p>
Using cropped background images as background class

Dot_Pixis: <p>Hello.</p>
<p>I’m currently working on a binary image classification problem using high resolution (up to 6000x4000 pixels) images with complex backgrounds, and CNN transfer learning.</p>
<p>In order to reduce Images size and complexity, I applied an offline crop to all images in a way that keeps the Region Of Interest (ROI) distinguishable. This resulted with three types of cropped images:</p>
<ol>
<li>images with background only,</li>
<li>images with ROI only, and</li>
<li>images with both ROI and background.</li>
</ol>
<p>Only images of type 1 and 2 were used to train validate and test the CNN model.</p>
<p>I want to use the background-only images (type 1) as a background class in order to make my model fucus only on ROI and ignore backgrounds. How can I deal with the background class (number of images, loss function…) ?</p>
<p>Thanks.</p>

shalom_p: <p>Hi for reducing complexity you can just resize the image to a lower resolution this way you don’t loose valuable information.<br>
An ideal scenario would be that you have the entire frame with your focus object in frame and also the entire frame without your focus object in frame.<br>
If you can please elaborate on what you are trying to classify I think I can help you better.</p>
The time consumption of torch.where() is a bit strange

Mart1n: <p>Recently, my face detection code (which should be a model modified based on retina-face) needs to use torch.where to filter prediction boxes. I found that the main time-consuming post-processing is to calculate the torch.where line of code, specifically:<br>
inds = torch.where(scores &gt; args.confidence_threshold)[0],<br>
where scores are confidence and are placed on the GPU. I found this line of code to be very time consuming and difficult to optimize. Moreover, the time it takes to execute torch.where() multiple times is the same as the time it takes to execute it once, about 100ms.<br>
Any possible suggestions would be greatly appreciated!</p>

ptrblck: <p>Your profiling is most likely invalid as you are not properly synchronizing the code.<br>
<code>torch.where</code> adds data-dependent control flow and thus synchronizes the code as also seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.cuda.set_sync_debug_mode("warn")

x = torch.randn(10, device="cuda")
torch.where(x &gt; 0.)
#  UserWarning: called a synchronizing CUDA operation (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:147.)
#  torch.where(x &gt; 0.)
</code></pre>
<p>If CUDA kernels are still executed on the device, this call will accumulate their execution time.<br>
Check if this is the case in your use case.</p>
Is it safe to update nvidia driver with old pytorch version

Yubin_Qin: <p>Hello, we are using a shared V100 GPU server with driver 410 and pytorch 1.7.1 cuda 9.2. And I want to run a project based on torch2.0 with cuda 12. Is it safe to update the nvidia driver to a new version say 535, while keeping my teammates’ projects built on a docker environment with torch 1.7.1 cu92 able to run?<br>
Thanks!</p>

ptrblck: <p>Yes, this should be the case as newer drivers support old CUDA runtimes. However, CUDA 9.2 is quite old that I don’t know what its support is.</p>
How does dropout in LSTM/GRU work?

mhenning: <p>In the docs it is stated that dropout is applied to the output of intermediate layers. My question is, what kind of dropout? Is is the normal Dropout layer, which drops completely random? Or is it something like Dropout2D, which drops along the feature axis (still not optimal for time series data)? Unfortunately, the source code does not reveal an answer, as the dropout parameter is given into some C function, which already outputs the results.</p>
<p>Bonus question, does the dropout parameter work correctly with Packed Sequences as input?</p>
<p>Double bonus question, if I want to have spatial dropout (drop along the time axis) and implement it myself while working with packed sequences, would it be correct to do the following:<br>
x = pack_padded_sequence(x, …)<br>
x, _ = lstm1(x)  # only one layer<br>
x = pad_packed_sequence(x, …)<br>
x = x.permute(0,2,1)<br>
x = Dropout2D()(x)<br>
x = x.permute(0,2,1)<br>
x = pack_padded_sequence(x, …)<br>
x, _ = lstm2(x)  # only one layer<br>
x = pad_packed_sequence(x, …)</p>

tom: <p>Back in the day, the dropout was just randomly on each element without any structure.<br>
This was discussed a bit including on this forum in the early days when Gal and Ghahramani (<a href="https://arxiv.org/abs/1512.05287" class="inline-onebox" rel="noopener nofollow ugc">[1512.05287] A Theoretically Grounded Application of Dropout in Recurrent Neural Networks</a>) proposed a dropout scheme that is kept fixed across the “time” dimension. You can have more control by using LSTMCell and doing the time loop yourself, at the expense of some speed which you may (partially?) recover through pre-compilation.</p>
<p>Best regards</p>
<p>Thomas</p>
Potential suggestion to speed up the following snippet?

vanila_siri: <p>Hi. I am new to this forum. The following code is extremely slow and I am seeking ways to speed it up. The distance tensor is with shape (batch_size, D, D). Literally, I want to obtain the average of every three points in the distance tensor. Is there any ways / native ops that can be used to speed this op up?</p>
<pre><code class="lang-auto">@torch.jit.script
def cal_3p_distance(distance):
    E = torch.zeros(distance.shape[0], distance.shape[1], distance.shape[1], distance.shape[1]).to(distance.device)
    for i in range(distance.shape[1]):
        for j in range(distance.shape[1]):
            for k in range(distance.shape[1]):
                E[:, i, j, k] = (distance[:, i, j] + distance[:, i, k] + distance[:, j, k])
    E = E / 3
    return E
</code></pre>

tom: <p>You want to use broadcasting</p>
<p><code>E = distance[:, :, None] + distance[:, :, None, :] + distance[:, None, :, :] / 3</code> should do the trick.</p>
<p>Best regards</p>
<p>Thomas</p>
SVT backward error for Low-rank structure

xianggkl: <p>I’m suffering from the backward of the SVD thresholding method, which always brings out the NAN errors due to it removes the small singular values to keep a Low-rank structure. Pytorch warns that if a Low-rank matrix or tensor, whose singular value list <code>S</code> is thresholded by some certain <code>tau</code> thresholding operator, the gradients would get NANs due to similar singular values or zeros (see warnings in torch.linalg.svd document).<br>
From this blog <a href="https://discuss.pytorch.org/t/modify-gradient-computation-of-torch-linalg-svd/138290">Modify gradient computation of <code>torch.linalg.svd</code></a>, I learn that I should add a <code>torch.eps</code> function to the bottom of the <code>1/(sigma_i**2 - sigma_j**2)</code>.<br>
But I have some questions about this:</p>
<ol>
<li>Do this ‘adding method’ cause the imprecise Low-rank results?</li>
<li>And what do you think is the reason Pytorch doesn’t include this mechanism (Because as I understand it, as long as backpropagation about SVD is used, there will be such a problem.)?</li>
<li>How can I figure this Low-rank problem out in a more elegant way?</li>
</ol>
<p>I would really appreciate it if you can help me out.</p>

souryadey: <p>Hi, I had a similar issue as you have linked. I found <a href="https://github.com/google/jax/issues/2311#issuecomment-984131512" rel="noopener nofollow ugc">this Github thread</a>, which led me to <a href="https://github.com/wangleiphy/tensorgrad/blob/master/tensornets/adlib/svd.py" rel="noopener nofollow ugc">this SVD implementation</a>, which solved my issue.</p>

xianggkl: <p>Thanks for your reply! And it does help me figure out the SVD backward problem.</p>
Pytorch model calibration

klark: <p>Hi,<br>
I tried to calibrate my pytorch model with sklearn.<br>
I used the following code</p>
<pre><code class="lang-auto">valid_X, valid_y = get_input_and_target_from_loader(valid_loader)
# calibrate model on validation data
calibrator = CalibratedClassifierCV(model, cv='prefit', method='isotonic')
calibrator.fit(valid_X, valid_y)
</code></pre>
<p>I get an error:<br>
<code>sklearn.utils._param_validation.InvalidParameterError: The 'estimator' parameter of CalibratedClassifierCV must be an object implementing 'fit' and 'predict_proba', an object implementing 'fit' and 'decision_function' or None.</code></p>
<p>Is there any equivalent function for Pytorch models ?</p>

soulitzer: <p>I’m not familiar with sklearn + calibration, so I have a dumb question: what is the difference between calibrating a model and training a model (e.g. forward, backward, optimizer step update parameters), or do you mean something closer to optimizing hyper parameters?</p>

klark: <p>I didn’t mean to optimize hyper-parameters. Here is a link explaining calibration.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://wttech.blog/blog/2021/a-guide-to-model-calibration/">
  <header class="source">
      <img src="https://wttech.blog/img/favicon-32x32.png" class="site-icon" width="32" height="32">

      <a href="https://wttech.blog/blog/2021/a-guide-to-model-calibration/" target="_blank" rel="noopener nofollow ugc">Wunderman Thompson Technology</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/360;"><img src="https://wttech.blog/static/df1f49d87396559b9ff2a62efc392b3a/7187e/featured-image.jpg" class="thumbnail" width="690" height="360"></div>

<h3><a href="https://wttech.blog/blog/2021/a-guide-to-model-calibration/" target="_blank" rel="noopener nofollow ugc">A guide to model calibration</a></h3>

  <p>Calibration is important, albeit often overlooked, aspect of training machine learning classifiers. It gives insight into model uncertainty, which can be later communicated to end-users or used in further processing of the model outputs. In this...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


dsesto: <p>If I’m not mistaken, in order to use sklearn’s <code>CalibratedClassifierCV</code>, you need to pass it a sklearn Model in the <code>model</code> argument.<br>
This won’t work out of the box with PyTorch models.</p>
<p>However, the link you shared in your last post is very illustrative and shares some examples to implement your own calibration using sklearn’s <a href="https://scikit-learn.org/stable/modules/isotonic.html#isotonic" rel="noopener nofollow ugc">IsotonicRegression</a> model.</p>
Update only sub-elements of weights

jan.olle: <p>Hi everyone.</p>
<p>So I’m trying to implement a deep neural network composed of a few linear layers. Every layer is composed of a 2x2 weight matrix and no biases. The peculiarity of my network is that I would only like to train one of the elements of each weight and leave the others untouched.</p>
<p>To make things more concrete, every weight looks like:</p>
<p><code>W = torch.tensor([[number1, number2],[number3, part I want to train]])</code></p>
<p>Things I’ve thought is proceed as usual (i.e. calculate the loss function and then use some optimizer) and then somehow zero the gradients for the parts of the weights I don’t want to touch but I haven’t managed to succeed. Any ideas? I can share parts of the code if that helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=6" title=":smile:" class="emoji" alt=":smile:"><br>
Thanks!</p>

ptrblck: <p>You could use a hook to zero out all other gradients.<br>
Here is a small example for a simple model:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Linear(2, 2),
    nn.Sigmoid(),
    nn.Linear(2, 2)
)
# Create Gradient mask
gradient_mask = torch.zeros(2, 2)
gradient_mask[0, 0] = 1.0
model[0].weight.register_hook(lambda grad: grad.mul_(gradient_mask))

optimizer = optim.SGD(model.parameters(), lr=1.0)
criterion = nn.CrossEntropyLoss()

batch_size = 10
x = torch.randn(batch_size, 2)
target = torch.randint(0, 2, (batch_size,))

# Get weight before training
w0 = model[0].weight.detach().clone()

# Single training iteration
optimizer.zero_grad()
output = model(x)
loss = criterion(output, target)
loss.backward()
print('Gradient: ', model[0].weight.grad)
optimizer.step()

# Compare weight update
w1 = model[0].weight.detach().clone()
print('Weights updated ', w0!=w1)
</code></pre>

Mandy: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thanks for you solution, it’s a good way to freeze some weights. However, if I set <code>weight_decay</code> or <code>momentum</code> in <code>optim.SGD</code>, you method will not work appropriately. Like</p>
<pre><code class="lang-auto">optimizer = optim.SGD(model.parameters(), lr=1.0,momentem=0.9, weighe_decay=0.001)
</code></pre>
<p>I have the same goal as Jan. I was wondering if you could tell me how to solve this problem.</p>

ptrblck: <p>In that case you shouldn’t pass these parameters to the optimizer (and could additionally set their <code>requires_grad</code> attribute to <code>False</code>).</p>

Lakshya1: <p>but if we don’t pass it to optimizer then even the parts we want to update will not be updated, right?</p>
Is it possible to split all large models and perform training in parallel over multiple GPUs?

thefreeman: <p>Is it possible to split all large models and perform training in parallel over multiple GPUs?</p>
<p>If so, what is the point of creating/using GPUs with larger VRAM?</p>
<p>If it’s because a GPU with larger VRAM is faster, how much faster is a GPU with 80GB VRAM than a dual GPU setup of the same GPU with 40GB VRAM?</p>
<p>I am asking this question because I have read many times that <em><strong>“If you want to train larger models, you need a GPU with larger VRAM.”</strong></em> However, if we can split the model and train it through multiple GPUs in parallel, or even a single GPU with smaller VRAM, it means what they are saying isn’t entirely accurate?</p>

eqy: <p>Your question has many moving parts that must be fixed in place before it can be answered, such as:<br>
the ratio of FLOPs to memory bandwidth of each GPU,<br>
the interconnect speed between each GPU (e.g., is this PCI-E or NVLink, and is there P2P support, …),<br>
the model sharding strategy (TP, PP, DDP, FSDP, or some combination of them?),<br>
the number of GPUs being compared.</p>
<p>Intuitively, whenever you are using more GPUs, there will be some overhead introduced (bubbles in the pipeline if pipeline parallelism is used, activation communication if model parallel is used, gradient allreduce communication of data parallel is used, weight communication if fully-sharded data-parallel is used) as some examples which would either be reduced or nonexistent in the single GPU case. Whether the additional computational FLOPs of adding GPUs makes up for this overhead depends on the communication required by your use case and the communication speed of the GPU relative to its computational throughput. This is a recent paper that shows how complex the task of choosing the best parallelization strategy for a given number of GPUs can be: <a href="https://arxiv.org/pdf/2304.11277.pdf" rel="noopener nofollow ugc">2304.11277.pdf (arxiv.org)</a></p>

thefreeman: <p>I really liked your detailed answer!<br>
I asked this question pairing a couple of 4090s in my mind. So the communication method between GPUs would be PCIe and no p2p support based on the other topic I read about multiple 4090s build.<br>
TFLOPS to memory bandwidth is 2.6 I think.<br>
I don’t know about the sharding strategy.</p>
<p>I just wanted to ask a general question to see is a dual 4090s build somehow a waste of money(compared to a dual 3090s or a single 4090 setup) or it’s worth it?</p>
Keep off-diagonal elements only from square matrix

calincru: <p>I have a (square) pairwise distances matrix, say <code>n x n</code>, and I would like to get rid of the diagonal and hence work with a <code>n x (n - 1)</code> matrix.  What’s a fast way of doing that?</p>
<p>One way is <code>a.masked_select(~torch.eye(n, dtype=bool)).view(n, n - 1)</code> but I was curious if there’s a faster approach.</p>

andreaskoepf: <p>I do not know if it is really faster, but just for the fun of it you could try as an alternative:</p>
<pre><code class="lang-auto">a.flatten()[1:].view(n-1, n+1)[:,:-1].reshape(n, n-1)
</code></pre>

zzhang99: <p>This is indeed much faster!</p>

Yuval_S: <p>Even faster is</p>
<pre><code>a * (1 - torch.eye(n))
</code></pre>

tanyjnaaman: <p>For those who want a visualization of how this works:</p>
<pre><code class="lang-auto">a = [
    [1,2,3]
    [4,5,6]
    [7,8,9]
]
# flatten
a.flatten() #  [1,2,3,4,5,6,7,8,9]
a.flatten()[1:]  # [2,3,4,5,6,7,8,9] # removes the first diagonal

# view 
a.flatten()[1:].view(n-1, n+1) # takes a snaking line starting from after a diagonal, ending in a diagonal
# [
#    [2,3,4,5]
#    [6,7,8,9]
# ]

# cut and reshape
a.flatten()[1:].view(n-1, n+1)[:, :-1] # removes last element from every row, which is a diagonal
# [
#    [2,3,4]
#    [6,7,8]
# ]


</code></pre>
Handling module alias in FSDP

jiach: <p>Hi,</p>
<p>I’m trying to train a model that has module alias with FSDP. I got the following error when loading the checkpoint file.</p>
<pre><code class="lang-auto">  ......
  File "***/checkpoints.py", line 67, in load_model_checkpoint
    state_dict = torch.load(checkpoint_path, map_location="cpu")
  File "***/lib/python3.8/site-packages/torch/serialization.py", line 1014, in load
    return _load(opened_zipfile,
  File "***/lib/python3.8/site-packages/torch/serialization.py", line 1422, in _load
    result = unpickler.load()
  File "***/lib/python3.8/site-packages/torch/_utils.py", line 202, in _rebuild_tensor_v2
    tensor = _rebuild_tensor(storage, storage_offset, size, stride)
  File "***/lib/python3.8/site-packages/torch/_utils.py", line 181, in _rebuild_tensor
    return t.set_(storage._untyped_storage, storage_offset, size, stride)
RuntimeError: Trying to resize storage that is not resizable
</code></pre>
<p>Here’s a minimum example to reproduce the error.</p>
<pre data-code-wrap="python"><code class="lang-python">import os
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP

DIM = 16


class Model(torch.nn.Module):
    def __init__(self, dim=512) -&gt; None:
        super().__init__()
        self.linear1 = torch.nn.Linear(dim, dim)
        self.linear2 = torch.nn.Linear(dim, dim)
        self.linear3 = torch.nn.Linear(dim, dim)

        self.linear_alias = self.linear3


def setup(rank, world_size):
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "14355"
    torch.cuda.set_device(rank)
    # initialize the process group
    dist.init_process_group("nccl", rank=rank, world_size=world_size)


def cleanup():
    dist.destroy_process_group()


def demo_basic(rank, world_size):
    setup(rank, world_size)

    model = Model(dim=DIM).to(rank)
    model = FSDP(model, device_id=rank)

    state_dict = model.state_dict()
    if rank == 0:
        torch.save(state_dict, "model.pt")

    cleanup()


if __name__ == "__main__":
    world_size = torch.cuda.device_count()
    mp.spawn(demo_basic, args=(world_size,), nprocs=world_size, join=True)
    chkpt = torch.load("model.pt", map_location="cpu")

</code></pre>
<p>Should we use module alias at all? If I must use alias, how should I handle them in FSDP?</p>
<p>I didn’t find much information about using alias modules with FSDP online. Would anyone please help me with some suggestions? Thank you!</p>
Loading .tfrecord file as pytorch dataset

Filip_Mirkovic: <p>Hello dear Torch firends!</p>
<p>My problem is the following, I have a fairly large dataset that is stored in .tfrecord format. Since I am way to deep into the project to switch to tensorflow I would like to train my model with this additional data using Pytorch. The issue is that  am not sure how to parse the binary stream stored in .tfrecord as a pytorch dataset, also the dataset is to large to be completely stored in memory.</p>

ShuokaiPan: <p>I was thinking this problem as well not long ago. I didn’t find a solution, but use a simple workaround which is to parse tfrecords using TF and save them into a different format such as hdf5, which can then be used in Pytorch.<br>
I guess it would be quite difficult to parse tfrecords in Pytorch, but would be happy to be proved wrong if other people know how to do it.</p>

Filip_Mirkovic: <p>Thanks for the reply!<br>
This is a solution I implemented for now, however this is far less efficient in terms of memory and loading time compared to tfrecord. It seems pytorch needs its own dataformat… torchrecord?</p>
How should be used register_forward_hook?

Mahdi_Amrollahi: <p>I need to use <code>register_forward_hook</code> method to receive some layers output but I will not receive the results when I call <code>.forward()</code> for any module however, when I change it to just calling that module like <code>linear1()</code>, I get the results on hooks. Is that a bug?</p>
<p>This does not work:</p>
<pre data-code-wrap="python"><code class="lang-python">from torch import nn

class cls_model(nn.Module):
    def __init__(self):
        super(cls_model,self).__init__()
        self.l1 = nn.Linear(3,2)
        self.l2 = nn.Linear(2,4)
        self.l3 = nn.Linear(4,5)
    def forward(self, x):
        y1 = self.l1.forward(x)
        y2 = self.l2.forward(y1)
        y3 = self.l3.forward(y2)
        return y3
    
def forward_hook(module, input, output):
    print(f"Inside forward hook for {module.__class__.__name__}")
    print(f"Input shape: {input[0].shape}")
    print(f"Output shape: {output.shape}")
    print("--------")
model = cls_model()
hook_handle = model.l2.register_forward_hook(forward_hook)
x = torch.randn((6,3))
model.forward(x)
</code></pre>
<p>This works</p>
<pre data-code-wrap="python"><code class="lang-python">from torch import nn

class cls_model(nn.Module):
    def __init__(self):
        super(cls_model,self).__init__()
        self.l1 = nn.Linear(3,2)
        self.l2 = nn.Linear(2,4)
        self.l3 = nn.Linear(4,5)
    def forward(self, x):
        y1 = self.l1(x)
        y2 = self.l2(y1)
        y3 = self.l3(y2)
        return y3
    
def forward_hook(module, input, output):
    print(f"Inside forward hook for {module.__class__.__name__}")
    print(f"Input shape: {input[0].shape}")
    print(f"Output shape: {output.shape}")
    print("--------")
model = cls_model()
hook_handle = model.l2.register_forward_hook(forward_hook)
x = torch.randn((6,3))
model.forward(x)
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Mahdi_Amrollahi" data-post="1" data-topic="192754">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_amrollahi/48/62481_2.png" class="avatar"> Mahdi_Amrollahi:</div>
<blockquote>
<p>however, when I change it to just calling that module like <code>linear1()</code>, I get the results on hooks. Is that a bug?</p>
</blockquote>
</aside>
<p>No, this is the exact reason why you should call the module directly instead of its <code>.forward</code> pass, as e.g. hooks will be skipped otherwise.</p>
How to properly handle inference cancelation in a TorchServe handler?

miroslav_lalev: <p>Hi all,</p>
<p>I’ve been wondering about the proper way of handling and stopping inference if a client cancels their request. My use case is that I have a custom TorchServe handler, which implements the <code>initialize</code>, <code>preprocess</code>, <code>inference</code>, and <code>postprocess</code> methods. The process takes a while - maybe around 50-60 seconds, so I thought it would be a good idea to handle any cancelations by interrupting the inference progress so that further requests can continue faster.</p>
<p>Is there any method, which I can implement to give me details about the underlying connection, such as when it was forcefully closed by the client?</p>
<p>Thanks,<br>
Miro</p>
Vanishing gradient observed in the SE block in the lower layer of the efficeintnetv2 model

Foram: <p>Hi all,<br>
I am working on implementing a binary classification model for breast cancer histopathology data using an EfficientNet V2-S model. However, I observed that after 5-10 epochs the gradients in the SE blocks are going to 0 in the lower layers. This is observed only is SE blocks. Should this be an expected behavior of the SE layer?</p>
<p>The dataset used comprises of 7909 images. with 80% data used for training and 10% for testing and validation. Schedular used is reduceOnPlateau, with initial learning rate of 0.0009 and factor = 0.5. The optimzer used is and Adams optimizer with weight decay of 0.005.</p>
<p>I have tried decreasing and increasing the learning rate but i get the same result.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24b049490353c87592172efdb732708989781006.png" alt="image" data-base62-sha1="5eySAKAAMeMVroxx4p4m33oXZ0a" width="507" height="304"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d130d392f6a5d17107adaa65f295299dc054a48.png" alt="image" data-base62-sha1="mpxMpyaHL4oEjqIoCtrQJSZ4oYM" width="508" height="304"><br>
These images are of the weights and biases observed for fc layer in SE block in layer 2, for 20 epochs.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40b2a73c2259013b4024d306c1cad7f231f79eca.png" alt="image" data-base62-sha1="9eljTKLoo6xYzgnV7hU3QEIN3RM" width="499" height="296"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f262a39fc7899b388b2fd8e39a0ad02fda2d6487.png" alt="image" data-base62-sha1="yAeSDAHJLFqRd2d3jo4hlua5Eeb" width="509" height="302"><br>
These images are of the weights and biases observed for fc layer in SE block in layer 3, for 20 epochs.</p>
<p>It would great if I could get some insights on why this is occurring and how can I solve this issue.</p>
<p>Would replacing the linear layers with 1x1 convolution make any difference?</p>
<p>Thanks!</p>
Inference Memory consumption is higher than expected

rprabhu: <p>I am trying to run inference for a gpt style model [i.e. a model the transformer block repeating itself n number of times].<br>
My model has 48 layers and a hidden size of 4096. so, the model parameters occupy 20 GB space[model is set to fp16]. I’ve set the <strong>model to eval</strong>, I’m running this on an <strong>A100 80GB</strong> GPU. I’ve used <strong>torch.no_grad</strong> decorator over the forward pass of the main model block that has all the layers. <strong>There is no KV cache in this implementation</strong></p>
<p>Theoretically, I should be able to fit &gt;1000 batch size. but i’m only able to 100. Why is this happening?<br>
I’ve tried using torch.cuda.empty_cache() and deleting variables that aren’t necessary. Doesn’t eleviate the problem.</p>
Old cuda version for lastest pytorch

tianlhcs: <p>I have a docker machine which limits the cuda version at 11.7, if I still want to install pytorch 2.1 which requires cuda11.8, can I use conda to install cudatoolkit-11.8 for a walkaround? I tried but it failed, I  don’t  know if I should set some environtment variable or something</p>

ptrblck: <p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or custom CUDA extensions, since the PyTorch binaries ship with their own runtime dependencies.</p>
OverflowError: (34, 'Numerical result out of range')

wasiahmad: <p>I am getting the following error (see the stacktrace) when I ran my code in a different GPU (Tesla K-20, cuda 7.5 installed, 6GB memory). Code works fine if I run in GeForce 1080 or Titan X GPU.</p>
<p><strong>Stacktrace</strong>:</p>
<pre><code>File "code/source/main.py", line 68, in &lt;module&gt;
    train.train_epochs(train_batches, dev_batches, args.epochs)
  File "/gpfs/home/g/e/geniiexe/BigRed2/code/source/train.py", line 34, in train_epochs
    losses = self.train(train_batches, dev_batches, (epoch + 1))
  File "/gpfs/home/g/e/geniiexe/BigRed2/code/source/train.py", line 76, in train
    self.optimizer.step()
  File "/gpfs/home/g/e/geniiexe/BigRed2/anaconda3/lib/python3.5/site-packages/torch/optim/adam.py", line 70, in step
    bias_correction1 = 1 - beta1 ** state['step']
OverflowError: (34, 'Numerical result out of range')</code></pre>

smth: <p>do you have a question?</p>

wasiahmad: <p>Yes, what can be the reason to get such error in a different GPU (Tesla K-20) while it works fine in GeForce or Titan X GPU? Moreover what the error means? Is it related to memory overflow which I don’t think so.</p>

jaiabhayk: <p>The exactly same issue/error i got (with K-40 and cuda 7.5), any suggestions ?</p>

jaiabhayk: <p>Workaround for this:-</p>
<p>Replace the following lines in <a href="http://adam.py" rel="nofollow noopener">adam.py</a>:-</p>
<p>bias_correction1 = 1 - beta1 ** state[‘step’]<br>
bias_correction2 = 1 - beta2 ** state[‘step’]</p>
<p>WITH</p>
<p>bias_correction1 = 1 - beta1 ** min(state[‘step’],1022)<br>
bias_correction2 = 1 - beta2 ** min(state[‘step’],1022)</p>

wasiahmad: <p>Why this works? What is the reason of taking <code>min</code> between <code>state</code> and 1022?</p>

jaiabhayk: <p>It is just for numerical reasons, practically beta1 or beta2 are 0.9 (between 0 and 1).<br>
And 0.9^1000 = 1.7478713e-46, which is almost zero for all practical purposes. (You can take 1000 instead of 1022 and nothing will change).</p>

suzannejin: <p>So here the state[‘step’] means the number of training step at the time.<br>
The total training step can be calculated as number of epochs * number of batches.<br>
At some point, when step is very big, the calculation of beta1 ** step raises overflow.</p>
<p>For me, changing the Python version solved the issue.</p>
<p>So, with Python 3.7.4, up to a point I get:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; 0.9 ** 7073
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
OverflowError: (34, 'Numerical result out of range')
</code></pre>
<p>Whereas with Python 3.8 or Python 3.10, I get:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; 0.9 ** 7073
0.0
</code></pre>
Torch.istft (center=False) Error reported when using window parameter

lxpnever: <pre data-code-wrap="python"><code class="lang-python">import torch
import torchaudio
import matplotlib.pyplot as plt


sample_rate = 44100
duration = 5.0
t = torch.arange(0, duration, 1.0 / sample_rate)
waveform = 0.5 * torch.sin(2.0 * 3.14159 * 220.0 * t) + 0.5 * torch.sin(2.0 * 3.14159 * 440.0 * t)

NFFT = 512
hop_size = 256

print("waveform", waveform.shape)

S = torch.stft(waveform, n_fft=NFFT, hop_length=hop_size, win_length=NFFT,
               center=False,
               window=torch.hann_window(window_length=NFFT), return_complex=True)

expected_length = (S.shape[-1] - 1) * hop_size + NFFT+NFFT
centerFalse_wav = torch.istft(S, n_fft=NFFT, hop_length=hop_size, win_length=NFFT,
                              center=False,
                              window=torch.hann_window(window_length=NFFT),
                              length=expected_length)
print(f"reconstructed_waveform shape: {centerFalse_wav.shape}")

</code></pre>
<p>line 31, in <br>
centerFalse_wav = torch.istft(S, n_fft=NFFT, hop_length=hop_size, win_length=NFFT,<br>
RuntimeError: istft(CPUComplexFloatType[257, 860], n_fft=512, hop_length=256, win_length=512, window=torch.FloatTensor{[512]}, center=0, normalized=0, onesided=None, length=220928, return_complex=0) window overlap add min: 1<br>
[ CPUBoolType{} ]<br>
If you don’t use window=torch. han_ Window (window_length=NFFT) will not report an error</p>
LSTMs producing same output for different batches of data

Linson_Andrew: <p>I am trying to design a Auto-Encoder model to convert my discrete my one-hot Encoded data into continuous data (more like some soft-max values . Imagine my training data is in dimension [64,10,98] where 64 is the batch size, 10 is the sequence length and 98 is the feature size . The code for Auto-Encoder i have used :<br>
import torch<br>
import torch.nn as nn</p>
<p>class EncoderDecoderModel(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size):<br>
super(EncoderDecoderModel, self).<strong>init</strong>()</p>
<pre><code>    self.encoder_lstm = nn.LSTM(input_size, hidden_size,num_layers = 10,batch_first=True)

  
    self.decoder_lstm = nn.LSTM(hidden_size, input_size, num_layers = 10,batch_first=True)

def forward(self, input_tensor):
  
    oru, (encoder_hidden, cell) = self.encoder_lstm(input_tensor)

 
    decoder_output, _ = self.decoder_lstm(oru)

    return decoder_output
</code></pre>
<p>input_size = 98<br>
hidden_size = 512<br>
batch_size = 64<br>
sequence_length = 10</p>
<p>model = EncoderDecoderModel(input_size, hidden_size)</p>
<p>input_tensor = data_one_hot.float()</p>
<p>output_tensor = model(input_tensor)<br>
print(output_tensor.shape)</p>
<p>The problem is that it is producing the same output for different batches of data in the training loop .</p>
<p>Training Loop :<br>
from torch.autograd import Variable<br>
import torch.nn.functional as F<br>
iters = 200000<br>
model.train()<br>
for iter in range(iters):<br>
data = next(pass_next)<br>
data = data.astype(np.float32)<br>
data_tensor = Variable(torch.tensor(data,requires_grad=True).long())<br>
data_one_hot = Fun.one_hot(data_tensor,num_classes=len(charmap))<br>
output = model(data_one_hot.float())<br>
ae_optim.zero_grad()<br>
loss  = F.cross_entropy(data_one_hot.float(),output)<br>
loss.backward()<br>
torch.nn.utils.clip_grad_norm_(model.parameters(),1.0)<br>
ae_optim.step()</p>
<p>if iter%10==0 and iter&gt;0:</p>
<pre><code>print("********************************")
samples = torch.argmax(output,dim=2)
# samples = decode_passwords(samples,inv_charmap)
# output = decode_passwords(samples,inv_charmap)
print(" samples : ",samples)
print("iter :",iter)
print("loss :",loss)
print("********************************")
</code></pre>
<p>Please Help !</p>
DDP mode ，GPU memory always increases during the validation phase, and sometimes it gets stuck

xusir.m02: <p>When using DDP mode, at the start of the first epoch during validation, GPU0 suddenly increases its memory usage by 2GB and keeps it that way. GPU1’s utilization sometimes reaches its maximum, causing the validation time to increase. Which part of the code should I pay attention to？yolo</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68a6a19ec5e8cf3a31f25eda0a5f2952b9d73abf.png" data-download-href="https://discuss.pytorch.org/uploads/default/68a6a19ec5e8cf3a31f25eda0a5f2952b9d73abf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68a6a19ec5e8cf3a31f25eda0a5f2952b9d73abf.png" alt="image" data-base62-sha1="eVMF6cdlCdCzw293cXZoZSwVl2T" width="690" height="60" data-dominant-color="5A3D23"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1211×107 5.02 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Bert classifier and integrated gradient using Captum

hossein: <p>Hi,<br>
I’m going to add interpretability to my model using the <code>integrated gradient</code> method of the Captum library.<br>
In the following, I will represent my codes:</p>
<pre><code class="lang-python">from transformers import BertModel

class Bert(nn.Module):
  def __init__(self):
    super(Bert,self).__init__()
    self.bert = BertModel.from_pretrained('bert-base-uncased')
    self.drop = nn.Dropout(0.5)
    self.fc = nn.Linear(768,5) 
  def forward(self,ids,masks):
    _ , cls = self.bert(input_ids = ids, attention_mask = masks, return_dict = False) 
    y = self.drop(cls)
    y = self.fc(y)
    return y

model = Bert()
</code></pre>
<p>Now, using tokenizer, I extract ids and masks of a given text:</p>
<pre><code class="lang-python">from transformers import BertTokenizer 
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
A = ['this is an example','this is not an example']
tokens = tokenizer(A, max_length = 512, padding = 'max_length', truncation = True, return_tensors = 'pt')
dataset = TensorDataset(tokens.input_ids, tokens.attention_mask)
data_loader = DataLoader(dataset, batch_size = 1)
</code></pre>
<p>Well, I created a data loader to load data one by one.<br>
Now, it is time to set Captum! This is how I am trying to run that (in which <code>i</code> and <code>j</code> is read through data loader):</p>
<pre><code class="lang-python">def predict(x,y):
  y = model(ids = x.long(), masks = y.long())
  return F.softmax(y, dim=-1)

integrated_gradients = IntegratedGradients(predict)
attributions_ig = integrated_gradients.attribute((i,j), target=0)
</code></pre>
<p>Now, My questions are:</p>
<ol>
<li>Am I setting the Captum correctly? Is everything right?!</li>
<li>When I implement these codes, it gives rise to this error:<br>
<code>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</code><br>
How should I fix that?</li>
</ol>
<p>Many thanks in advance.</p>

IneedMrmeeseeks: <p>Have you solve this problem?<br>
I also want to apply Integrated Gradients on custom Bert classification model.</p>
<p>Any updates? Thx!</p>

IneedMrmeeseeks: <p>For the guys who also have a same problem, read <a href="https://captum.ai/tutorials/Bert_SQUAD_Interpret" rel="noopener nofollow ugc">this captum tutorial</a>.</p>
<p>It’s not classification problem, but the underlying mechanism is same I think.</p>
<p>Also I read <a href="https://github.com/pytorch/captum/issues/150" rel="noopener nofollow ugc">this post</a> and <a href="https://github.com/pytorch/captum/issues/311" rel="noopener nofollow ugc">this one</a> for some insights.</p>
<p>Good Luck to your research. <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20"></p>
My ViT-b/16 fine-tuning accuracy has plateaued after 5 epochs

External_happy: <p>I am fine-tuning a ViT-b/16 model and have observed that the accuracy initially increases from 45% to 51% after 5 epochs. However, after this initial gain, the accuracy plateaus and never increases again. I have experimented with different batch sizes and learning rates using the Adam optimizer with a layer-wise learning rate scheduler. However, in all cases, the accuracy still drops after 5 epochs and never recovers.<br>
Could you please help me identify the potential causes of this early plateauing and suggest strategies to improve the model’s performance?</p>

shalom_p: <p>few main causes for this could be.<br>
1)model is not complex enough.<br>
2)data is not enough(either in size or variation)<br>
3)it is unable to come out from a local minimum.</p>
<p>for issue number 1 and 2 you can handle it.<br>
for issue number 3 try learning rate restart… use a cosine annealing restart scheduler it should fix the problem if issue number is the main cause of this.</p>
Pytorch geometric collate issue

Navya: <p>File “/home/ubuntu/Desktop/new_origami/Origami/origami/data/utils.py”, line 91, in <br>
collated_data_dict = { ligand_id : collate(data_list = list(data_dict[ligand_id]),cls=PairData,follow_batch=[‘atom_type_1’, ‘atom_type_2’])<br>
File “/home/ubuntu/anaconda3/envs/origami/lib/python3.9/site-packages/torch_geometric/data/collate.py”, line 32, in collate<br>
out = cls(_base_cls=data_list[0].<strong>class</strong>)  # Dynamic inheritance.<br>
TypeError: <strong>init</strong>() got an unexpected keyword argument ‘_base_cls’</p>
<p>I am trying to collate data based on the an id. the data is a bipartite graph, where ids can be repetative.<br>
Pytorch geometric version is 2.0.4.</p>

Navya: <p>Could it be because of versions or something else? Any help is appreciated.</p>

eqy: <p>I’m not sure <code>_base_cls</code> was ever a keyword argument of <code>collate</code>: see an example usage of <code>collate</code> here: <a href="https://pytorch-geometric.readthedocs.io/en/2.4.0/_modules/torch_geometric/data/batch.html" rel="noopener nofollow ugc">torch_geometric.data.batch — pytorch_geometric documentation (pytorch-geometric.readthedocs.io)</a></p>