akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.no_grad() vs detach()

username: artichoke
cooked: <p>I understand that when using the no_grad() environment, the Autograd does not keep track of the computation graph and it’s similar to temporarily setting requires_grad to False whereas the detach() function returns a tensor which is detached from the computation graph.</p>
<p>My question is, is there any place where using detach() is necessary? It seems to me that we can always do everything using no_grad() instead.</p>
<p>Thanks</p>

username: tom
cooked: <p>There is quite a bit of fine print to this rough “they have the same effect”. For example:</p>
<ul>
<li>even in <code>no_grad</code>-mode views will be tracked (and have requires_grad set if they are views of a tensor that has).</li>
<li>detach is a more versatile operation in that you can control what you want to not have gradients (e.g. if you want to train only the last (few) layer(s) for fine-tuning).</li>
</ul>

username: ooohh
cooked: <p><a class="mention" href="/u/tom">@tom</a><br>
I met a similar problem, I have a code like follows, only last frame will do bp and calculate loss. But all frames use the same parameters.<br>
It works with find_unused_parameters=True, but the weights cannot be updated. I need the weights to be updated by last frame’s bp. In this case, what I can do is using detach()?</p>
<pre><code class="lang-auto">class DummyDataset(torch.utils.data.Dataset):

    def __init__(self, seq_len=5):
        self.seq_len = seq_len
        

    def __len__(self):
        return 200

    def __getitem__(self, idx):
        return np.random.rand(self.seq_len, 10).astype(np.float32), np.random.rand(1).astype(np.float32)
        
class DummyNet(nn.Module):

    def __init__(self):
        super().__init__()

        self.mlp_fea = nn.Linear(10, 10)
        self.mlp_out = nn.Linear(10, 1)
    
    def forward(self, inputs, labels, training=False):
        inputs = inputs.cuda()
        labels = labels.cuda()

        B, SEQ, C = inputs.shape
        print(self.mlp_fea.weight)
        for i in range(SEQ):
            # print('seq: ', i)
            x = inputs[:, i]
            if i == 0:
                fea_prev = torch.zeros_like(x)

            if i &lt; SEQ - 1:
                self.eval()
                with torch.no_grad():
                    fea_prev = self.mlp_fea(x + fea_prev)
                self.train()
            else:
                fea_prev = self.mlp_fea(x + fea_prev)
                out = self.mlp_out(fea_prev)
            if i == SEQ - 1:
                loss = nn.L1Loss(reduction='mean')(out, labels)
                return loss
</code></pre>
Multivariate uniform distribution in pytorch?

username: Minsoo
cooked: <p>Hi, all.</p>
<p>I’m tying to implement multivariate uniform distribution.</p>
<pre><code class="lang-auto">x = torch.FloatTensor([[3,6]]).to(dtype=torch.float32)
lower_bound = torch.FloatTensor([[0,1]]).to(dtype=torch.float32)
upper_bound = torch.FloatTensor([[3,3]]).to(dtype=torch.float32)
x_init = x.new_empty((1,2)).uniform_(lower_bound, upper_bound)
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-25-d3abb0ac3983&gt; in &lt;module&gt;
      2 lower_bound = torch.FloatTensor([[0,1]]).to(dtype=torch.float32)
      3 upper_bound = torch.FloatTensor([[3,3]]).to(dtype=torch.float32)
----&gt; 4 x_init = x.new_empty((1,2)).uniform_(lower_bound, upper_bound)

TypeError: uniform_(): argument 'from' (position 1) must be float, not Tensor
</code></pre>
<p>I checked source code for torch.distributions.uniform,(<a href="https://pytorch.org/docs/stable/_modules/torch/distributions/uniform.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/distributions/uniform.html</a> )<br>
but I have no idea how to implement multidimensional lower and upper bound.<br>
Any ideas or examples are welcome.<br>
Thanks for reading.</p>

username: some_user1
cooked: <p>I guess you could do something like:</p>
<pre><code class="lang-auto">x = torch.FloatTensor([[3,6]]).to(dtype=torch.float32) # contains numbers from U(0,1)
res = (upper_bound - lower_bound) * x + lower_bound
</code></pre>
<p>This way, <code>res</code> will contain numbers from the desired distribution (provided elements DO NOT interact between them), i.e. <code>res</code>_i \sim U(<code>lower_bound</code>_i, <code>upper_bound</code>_i).</p>
<p>Is this what you are looking for ?</p>
Manually gathering tensors to avoid CUDA out of memory error

username: jef
cooked: <p>Hi, I am implementing a retrieval model with DDP 8 GPUs.<br>
I have to encode all Wikipedia articles (5.9M) in the model and save the encoded results (Transformer output corresponds to CLS).</p>
<p>By using <code>DistributedSampler</code>, each GPU can encode the articles on DDP condition. But when I called <code>torch.distributed.all_gather</code>, CUDA out of memory occurred.</p>
<p>So I gave up using <code>all_gather</code>. I started manually gathering the results from each GPU by saving and loading the results.</p>
<pre><code class="lang-python">torch.save(cand_vecs, os.path.join(exp_root, f"cand_vecs_{self.trainer.local_rank}.pt"))
dist.barrier()  # wait all cand_vecs are saved
all_rank_cand_vecs = []
for rank in range(dist.get_world_size()):
    all_rank_cand_vecs.append(torch.stack(torch.load(os.path.join(exp_root, f"cand_vecs_{rank}.pt")), dim=0))
all_rank_cand_vecs = torch.concat(all_rank_cand_vecs, dim=0)  # (num_candidates, H=768)

# I also saved the data indices to recover the original dataset order.
all_rank_example_indices = [None for _ in range(dist.get_world_size())]
dist.all_gather_object(all_rank_example_indices, example_indices)	

# recover the original dataset order
...
</code></pre>
<p>Is my code correct? I am a bit afraid because the order of my manual gathering is different from the naitive <code>all_gather</code>. If so, I could not recover the original dataset order.<br>
I believe that the order of returned list from <code>all_gather</code> corresnponds to <code>rank0, 1, 2, 3, 4....</code>.</p>

username: rvarm1
cooked: <p>Would you be able to move the results to CPU and call all_gather on the CPU results, or is that too slow for your use case? That would be a workaround to avoid CUDA OOM.</p>

username: Xu_Hui
cooked: <p>seems like all_gather will move these results first to GPU, and then gather these results on GPU, no matter where the gathered results originates from</p>
Crazy deadlock bug in child process

username: yousiyu
cooked: <pre><code class="lang-auto">import torch as tc
import torch.multiprocessing as mp

def do_something():
    b = tc.randn(1000, 1000)
    print(f'A tensor is created in process {mp.current_process().name}')
    print(b.sum())  # This line hangs forever

def bug():
    # a = tc.randn(1000, 1000)
    a = tc.ones(1000, 1000)  # comment out this line and revive the line above will make the bug go away
    p = mp.Process(target=do_something, args=())
    p.start()
    p.join()

if __name__ == '__main__':
    bug()
</code></pre>

username: yousiyu
cooked: <p>My torch version is 1.10.2<br>
It is crazy that creating a tensor in parent process causes a deadlock in the child process, and the tensor is not even passed to the child process.<br>
It is even crazier that if I replace the tensor in parent process with anther one, the deadlock goes away!<br>
Is it generally not recommended to use forked subprocess in pytorch?</p>
<p>thanks</p>

username: Fei_Liu
cooked: <p>I ran your code and did not see any bugs. The code just completed smoothly. My torch is 1.10.0</p>

username: Senlody
cooked: <p>I’m having a similar problem with python3.10 and torch 2.0/2.2 (didn’t try 2.1), here is my test code</p>
<pre><code class="lang-auto">import torch
import multiprocessing
import numpy as np

def test(n):
    x=np.zeros((n,n))
    print('tensor')
    x=torch.tensor(x) # the subprocess stucks here
    print('done')
    return x

def main():
    tensor = torch.zeros(200,1000)
    # tensor = torch.zeros(200,100) # this works, seems the bug goes away with small tensor
    
    # neither torch.multiprocessing nor built-in multiprocessing works
    # pool=torch.multiprocessing.Pool(1)
    pool=multiprocessing.Pool(1)
    pool.apply(test,(1000,))
    return

main()

</code></pre>
<p>Did you figure out a solution, or have any clues?</p>
<p>thanks</p>
<p>edit: The bug seems happens on linux, but goes away on win</p>
<p>edit2: found the cause <a href="https://github.com/pytorch/pytorch/issues/21956#issuecomment-636284398" rel="noopener nofollow ugc">here</a></p>
Installation problems libtorch

username: andrewshvecov7
cooked: <p>I have done all the requirements,but it doesn’t work. My os is windows ,I work on visual studio.I installed cuda toolkit 12.1, cmake the latest version, and libtorch 12.1 release version. I followed the instructions from the page <a href="https://pytorch.org/cppdocs/installing.html,I" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/installing.html,I</a> had to download nsight nvtx 11.8,but further wasn’t any problems.</p>
<p>after the last command nothing changed , I still can’t use pytorch c++, cmake prints next:</p>
<pre><code class="lang-auto">Severity	Code	Description	Project	File	Line	Suppression State	Details
Error		CMake Error at C:\Users\Acer\C++ projects\fraimworks\example-app\CMakeLists.txt:4 (find_package):
  By not providing "FindTorch.cmake" in CMAKE_MODULE_PATH this project has
  aske</code></pre>

username: ptrblck
cooked: <p>Which path did you use for <code>-DCMAKE_PREFIX_PATH</code>?<br>
The absolute path or the one from the Python package?</p>

username: andrewshvecov7
cooked: <p>I wrote the path to the libtorch cmake -DCMAKE_PREFIX_PATH=C:\Packages\libtorch …</p>
Simulating multiple masters in federated learning setting

username: Srinija_Ramichetty
cooked: <p>Hi everyone,</p>
<p>I am looking for a way to create a federated learning scenario with three masters and client network, and this functions similar to centralized federated learning, does pytorch support this , are there any libraries,policies, algorithms in pytorch that would let me achieve this?</p>
<p>Any insights would be appreciated<br>
Thank you in advance</p>
Torch.cuda.is_available() returns false even after accessing cuda toolkit 11.8

username: nplusonedims
cooked: <p>Problem is what the title says.<br>
I installed torch with the following command from the PyTorch website: pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a></p>
<p>Following are the relevant packages returned by Conda list:<br>
nvidia-cublas-cu11        11.11.3.6                pypi_0    pypi<br>
nvidia-cuda-cupti-cu11    11.8.87                  pypi_0    pypi<br>
nvidia-cuda-nvrtc-cu11    11.8.89                  pypi_0    pypi<br>
nvidia-cuda-runtime-cu11  11.8.89                  pypi_0    pypi<br>
nvidia-cudnn-cu11         8.7.0.84                 pypi_0    pypi<br>
nvidia-cufft-cu11         10.9.0.58                pypi_0    pypi<br>
nvidia-curand-cu11        10.3.0.86                pypi_0    pypi<br>
nvidia-cusolver-cu11      11.4.1.48                pypi_0    pypi<br>
nvidia-cusparse-cu11      11.7.5.86                pypi_0    pypi<br>
nvidia-nccl-cu11          2.19.3                   pypi_0    pypi<br>
nvidia-nvtx-cu11          11.8.86                  pypi_0    pypi<br>
torch                     2.2.0+cu118              pypi_0    pypi<br>
torchaudio                2.2.0+cu118              pypi_0    pypi<br>
torchvision               0.17.0+cu118             pypi_0    pypi</p>
<p>And still I get the following output.<br>
python3 -c ‘import torch; print(torch.backends.cudnn.enabled)’<br>
True<br>
python3 -c ‘import torch; print(torch.cuda.is_available())’<br>
False<br>
python3 -c ‘import torch; print(torch.version.cuda)’<br>
11.8</p>
<p>Output of Nvidia-smi is NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0</p>
<p>nvcc -v shows version Cuda compilation tools, release 11.8, V11.8.89</p>
<p>I don’t understand why am I still getting False on torch.cuda.is_available()</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue using the latest <code>torch==2.2.0+cu118</code> wheels:</p>
<pre data-code-wrap="python"><code class="lang-python">pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
...
Successfully installed nvidia-cublas-cu11-11.11.3.6 nvidia-cuda-cupti-cu11-11.8.87 nvidia-cuda-nvrtc-cu11-11.8.89 nvidia-cuda-runtime-cu11-11.8.89 nvidia-cudnn-cu11-8.7.0.84 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.3.0.86 nvidia-cusolver-cu11-11.4.1.48 nvidia-cusparse-cu11-11.7.5.86 nvidia-nccl-cu11-2.19.3 nvidia-nvtx-cu11-11.8.86 torch-2.2.0+cu118 torchaudio-2.2.0+cu118 torchvision-0.17.0+cu118
...
/workspace# python3 -c 'import torch; print(torch.backends.cudnn.enabled)'
True
/workspace# python3 -c 'import torch; print(torch.cuda.is_available())'
True
/workspace# python3 -c 'import torch; print(torch.version.cuda)'
11.8
</code></pre>

username: nplusonedims
cooked: <p>In the Successfully installed block, I have a few extra names; I’m not sure where are these coming from and if these could be the reason - mpmath-1.3.0 networkx-3.2.1 sympy-1.12 triton-2.2.0</p>
<p>I am looking for some direction to debug this.</p>
Variable length in each batch

username: Abdulrahman_Ahmed
cooked: <p>i have a dummy data with 2 batches. After padding using pad_sequence, the first batch size was (5,10) however, the second batch size was (5,12) according to the max sequence length in padding.<br>
but, when I use the Linear layer the model expects to find the input_size. In my case, I have a variable input size so what should I do?</p>

username: vdw
cooked: <p>Since your first layer seems to be a <code>nn.Linear</code>, I assume you’re implementing a basic MLP model. MLPs – in contrast to RNNs – are not designed to handle sequential data.</p>
<p>You can still use a MLP, though. The common approach would be to push your batch through an <code>nn.Embedding</code> layer, and the average the embedding vectors across each sentence in your batch. Of course, approach treats your input not as a sequence but as a bag of words.</p>
<p>But maybe I’m making the wrong assumption. What exactly are you trying to do?</p>
Computing loss and gradients for batches of parameters and data using torch.func

username: armal
cooked: <p>Hi all,</p>
<p>Is there a way to efficiently compute the loss and graident for a batch of parameters and a corresponding batch of data in torch.func?</p>
<p>For example, is there a way to do the following psuedocode using vmap anf functional_call or other methods?:</p>
<pre><code class="lang-auto">losses = torch.empty(N)
grads = torch.empty(N)

for i in range(N):
    # params is will be inserted into the model/net
    params = all_params[N]

    # get the batch of data to be processed for the loss and gradient 
    batch = all_data[N]

    # replace the parameters of the neural network model with params

    # compute the loss wrt the params and the batch of data 
    loss = compute_loss(params, batch)

    # compute the data wrt the model parameters we just inserted and the loss
    losses[i] = loss

    # save both the gradients and the loss so we have an N size tensors 
    gradients = compute_grad(loss)
    grads[i] = gradients

return loss, gradients</code></pre>
Manual Gradient Computation and weight update in Pytorch

username: rajoy99
cooked: <p>I do not want to use torch’s default loss.backward function for gradient computation. Instead I am calculating the gradients manually from the loss function (via torch.autograd.grad). But my gradients become zero after a few steps. The same code works if I use the loss.backward function. Is there any hidden transformation that torch applies on the gradients under the hood ? ( such as clipping the gradients in range [-1,1], detaching gradient etc. )</p>
<p>I am taking the gradients like this :</p>
<pre><code class="lang-auto">first_gradient=torch.autograd.grad(HSNR,NN.parameters(),retain_graph=True)
</code></pre>
<p>Here is how I am updating the weights :</p>
<pre><code class="lang-auto">    with torch.no_grad():
        for param, newgrad in zip(NN.parameters(),final_gradient):
            param.grad = newgrad
    
    optimizer.step() 

</code></pre>

username: albanD
cooked: <p>Hey!</p>
<p>There is no special transformation that happens no.<br>
I’m surprised that you set “retain_graph=True”, that should not be needed unless you’re explicitly re-using the same model for multiple calls to backward.</p>
DataLoader with workers in Jupyter doesn't work

username: CodingInProgress
cooked: <p>I am using Jupyter in VSCode on Windows and this doesn’t seem to work for me. Here is a simple reproduction for it.</p>
<pre><code class="lang-auto">import torch

class SomeIterableDataset(torch.utils.data.IterableDataset):
  def __init__(self):
    super(SomeIterableDataset).__init__()
  
  def generate(self):
    while True:
      result = torch.rand((16, 8, 3))
      yield result

  def __iter__(self):
    return iter(self.generate())


def main():
  #num_workers = 0 # Works in Jupyter
  num_workers = 1 # Doesn't work in Jupyter (works when executed as .py)

  dataset = SomeIterableDataset()
  dataset_loader = torch.utils.data.DataLoader(dataset, batch_size=4, num_workers=num_workers)

  for i, x in enumerate(dataset_loader):
    print(f'{i}: {x.size()}')
    if i &gt; 42:
      break

if __name__ == '__main__':
  main()
</code></pre>
<p>This leads to:</p>
<blockquote>
<p>RuntimeError: DataLoader worker (pid(s) 9156) exited unexpectedly</p>
</blockquote>
<p>Does someone know if there is a workaround for this issue?</p>

username: CodingInProgress
cooked: <p>Would it make sense to submit a bug report for this?</p>

username: ptrblck
cooked: <p>This might be related to <a href="https://github.com/pytorch/pytorch/issues/62943">this issue</a>.</p>
<blockquote>
<p>I think this issue is beyond the scope of Pytorch.<br>
It should be an issue of <a href="https://github.com/ipython/ipython" class="inline-onebox">GitHub - ipython/ipython: Official repository for IPython itself. Other repos in the IPython organization contain things like the website, documentation builds, etc.</a><br>
Python multiprocessing has different behaviors on spawn and fork, interactive and script mode.<br>
So far, any python multiple processing program are not supported in interactive mode with spawn.</p>
</blockquote>

username: CodingInProgress
cooked: <p>Thanks for the answer. I missed that closed issue.<br>
That was my assumption. Though, I was still surprised that I never ran into this issue in TensorFlow. They likely just implemented it differently.</p>
Multi-modal fusion Transformer

username: TeDataPro
cooked: <p>Hello,</p>
<p>I would like to implement the “Multimodal Fusion” block from the following approach :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b5e1ee89004a3d41cd88b2d2b8221e44af1a73d.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b5e1ee89004a3d41cd88b2d2b8221e44af1a73d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b5e1ee89004a3d41cd88b2d2b8221e44af1a73d_2_668x500.png" alt="image" data-base62-sha1="hBmnEUUwFpqhI7v4UuZLfMBPIDz" width="668" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b5e1ee89004a3d41cd88b2d2b8221e44af1a73d_2_668x500.png, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b5e1ee89004a3d41cd88b2d2b8221e44af1a73d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b5e1ee89004a3d41cd88b2d2b8221e44af1a73d.png 2x" data-dominant-color="E7EBE1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">852×637 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It is about image captioning for a 3D scene, and training is performed with next token prediction.<br>
As you can see, the Multimodal Fusion takes as input visual embeddings (blue) and textual embeddings (green). Textual embeddings are then updated with the Multimodal Fusion module taking into account visual embeddings when computing attentions.</p>
<p>I’m very new to multimodal attention, so my question is : how to implement this module ? Taking into account the fact that the number of input tokens may change ?</p>
<p>Does the code exist? If so, where can I find it?</p>
<p>Thank you so much in advance for your help !</p>

username: JuanFMontesinos
cooked: <p>Esentiallt it’s a pytorch transformer encoder module.<br>
As you will find there, it asks for a mask as kwargs.</p>
<p>Let’s say you pass 100 elements<br>
40 will be video, 60 text.<br>
You will need to pass forward(cat(video,text),mask)<br>
And this will create a sequence of 100 elements.<br>
You pick the last 60 ones (actually you can pick any 60 as the network will learn to positon them properly)<br>
if you signal lengths are different you need to add padding.</p>
<p>Sorry not to be more concise <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">  but lack of time. Hope it helps!</p>
Why can't I effectively parallelize my reinforcement learning programs using process based parallelism?

username: desert_ranger
cooked: <p>My objective is to run multiple reinforcement learning programs, using the Stable_Baselines3 library, at the same time. What I notice is that as I increase the number of programs, the iteration speed of the program gradually decreases, which is quite surprising since each program should be running on a different process (core).</p>
<p>Here is my program:</p>
<pre><code class="lang-auto">from joblib import Parallel, delayed

import gym
# from sbx import SAC
import torch

from stable_baselines3 import SAC
def train():


    env = gym.make("Humanoid-v4")

    model = SAC("MlpPolicy", env, verbose=1)
    model.learn(total_timesteps=7e5, progress_bar=True)

def train_model():

    train()



if __name__ == '__main__':
    num_of_programs = 1
    Parallel(n_jobs=10)(delayed(train)() for i in range(num_of_programs))
</code></pre>
<p><code>num_of_programs</code> is used to control the number of programs I am trying to run in parallel. Here are some statistics -</p>
<pre><code class="lang-auto">    Number of programs  Iteration speed
1   1   ~102 it/s
2   3   ~60 it/s
3   10  ~ 20 it/s
</code></pre>
<p>I made sure to request enough resources so that there isn’t a resource constraint. This is how I request my resources using slurm - <code>srun --time=10:00:00 --nodes=1 --cpus-per-task=16 --mem=32G  --partition=gpu --gres=gpu:a100-pcie:1 --pty /usr/bin/bash</code></p>
<p>Therefore I have 16 cpus, 32G memory and a 40 GB GPU.</p>
<p>I noticed the same issue when I moved from <code>stable_baselines3</code> to <code>sbx</code>. While <code>stable_baselines3</code> using <code>torch</code> as its deep learning library, the latter uses <code>JAX</code>.</p>
Need Help with Linear regression

username: thereayoo
cooked: <p>Hello everyone! I have a problem that my model cannot correctly predict the numbers I need. Tried various functions of activation, regularization, scaler, scheduler and others. In vandb, according to me, the calculation of the loss of MaE goes like this: 1 epoch of loss 23, 2 epoch 35, 3 epoch 1000, 4 epoch 50, 5 epoch 1000, 6 epoch 25 and so on without changing. But the accuracy steadily moves from 0 to 90%. And when transferring data to the model that it has not seen or seen, the result is bad. Also, I have already logged the gradients. I use a 3-layer linear model, the first layer is 10000 (and this size is mandatory), the second is 5000 (but I tried to experiment) and the last is 100.<br>
Maybe someone has an idea what my problem is?</p>
LRP for ResNet returns weird heatmap

username: razla
cooked: <p>Hey,</p>
<p>I’m trying to use your LRP implementation on a pretrained model resnet34/32 (ImagetNet and CIFAR10/100 respectively).</p>
<p>I needed to convert the ReLU layers to inplace=False in order to make it work and then when I execute this - it works but the output is weird and inconsistent…</p>
<p>these are the images for example:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/339800d7da5e625999391366b809e8f8f2d13a09.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/339800d7da5e625999391366b809e8f8f2d13a09" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/339800d7da5e625999391366b809e8f8f2d13a09.jpeg" alt="image" data-base62-sha1="7mq11BKYh8aL4wibMMgDMtAd15n" width="690" height="334" data-dominant-color="7A7A80"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">805×390 40.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and these are the explanations (heat maps)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/f/dfccbdf792447333b571e16e9e11d483ce3cef77.png" data-download-href="https://discuss.pytorch.org/uploads/default/dfccbdf792447333b571e16e9e11d483ce3cef77" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/dfccbdf792447333b571e16e9e11d483ce3cef77.png" alt="image" data-base62-sha1="vVP8i7WWjYO1NSv5lnb5XprmFrF" width="690" height="338" data-dominant-color="DCDAD9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">806×395 45.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>which doesnt make sense since the LRP is always readable and coherent with the images themselves…</p>
<p>thanks!</p>

username: JonasKlotz
cooked: <p>I have excactly the same problem, with the LRP ResNET. Also I’ve tried different rules. Did you found any solution?</p>
Torch.jit.trace consumes more than 40x gpu memory

username: Junxuan_Chen
cooked: <p>I’m using torch.jit.trace to export a vov99 model to torchscript Module on a A100 with 80GB gpu memory.</p>
<p>When I inference with the torch original torch.nn.Module model, it works well and only consumes around 2GB gpu memory. However when I trace the model with the same example input tensor, it runs out of 80GB gpu memory.</p>
<p>here is my code:</p>
<pre><code class="lang-auto">    backbone = Net(cfgs)
    checkpoint = load_checkpoint(
        backbone, args.weights, map_location='cuda', strict=False,
        logger=logging.Logger(__name__, logging.ERROR)
    )
    backbone.cuda()
    backbone.eval()

    # nn.Module inference tooks around 2GB gpu memory
    example_forward_input = torch.rand(1, 3, 640, 1600).cuda()
    original_feats = backbone(example_forward_input)
    for r in original_feats:
        print(r.shape)

    time.sleep(5)

    # Trace a specific method and construct `ScriptModule` with
    # here a OOM error raised: RuntimeError: CUDA out of memory. 
    # Tried to allocate 142.00 MiB (GPU 0; 79.33 GiB total capacity; 78.28 GiB already allocat
    # ed; 7.81 MiB free; 78.81 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; 
    # allocated memory try setting max_split_size_mb to avoid fragmentation.  
    # See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
    backbone_module = torch.jit.trace(backbone, example_forward_input)
    backbone_module.save('outputs/traced_backbone.pt')
</code></pre>

username: AppNet
cooked: <p>hi <a class="mention" href="/u/junxuan_chen">@Junxuan_Chen</a> ,<br>
did you get any solution of this?<br>
I am facing the same issue with my pytorch model which is around 100MB and I want to convert it into torch script, when I use a small dummy input [1,3, 240, 328] while converting it works fine but I want the input size (1, 3, 720, 1280) which may be the cause of running out of memory. I have tried to run on 8GB RAM GPU and also on CPU with 30GB RAM.</p>
<p>I have also tried different methods and versions for converting, but the issue still persists. however, I have noticed that there’s same issue with pytorch to onnx conversion.</p>
Compute Gramian of Jacobian using func's jvp and/or vjp

username: Pierre_Quinton
cooked: <p>Hi,</p>
<p>I am trying to compute the Gramian matrix of the jacobian matrix. That is if for instance my function is</p>
<pre><code class="lang-auto">def func(x):
  return A @ x
</code></pre>
<p>For some matrix <code>A</code>, then its jacobian is <code>A</code> and the Gramian of its Jacobian is <code>A @ A^T</code>.</p>
<p>Using <code>torch.func.vjp</code> and <code>torch.func.vmap</code>, we can compute the Jacobian as</p>
<pre><code class="lang-auto">import torch
from torch.func import vjp, vmap
matrix = torch.randn([4, 5], requires_grad=True)
input = torch.randn([5])
def func(x):
  return matrix @ x
output, vjp_fn = vjp(func, input)
jacobian = vmap(vjp_fn)(torch.eye(matrix.shape[0]))[0]
</code></pre>
<p>Now in principle I could then compute <code>jacobian @ jacobian.T</code> but this feels inefficient.</p>
<p>I was then thinking of computing the Gramian using the forward mode <code>func.jvp</code>. The idea would be to say that the Gramian of the jacobian is <code>J @ I @ J.T</code> instead of <code>I @ J @ J.T @ I</code> for the backward mode. So in principle, what I would like to do is to vectorise jvp, give it <code>torch.eye(matrix.shape[1])</code> and provide the result to jvp again.</p>
<p>My problem is that it seems that jvp (<a href="https://pytorch.org/docs/stable/generated/torch.func.jvp.html" class="inline-onebox" rel="noopener nofollow ugc">torch.func.jvp — PyTorch 2.2 documentation</a>) doesn’t seem to have the same prototype as vjp and it requires the tangeants as input, i.e. it does not return a function for computing the gradients. This means that it will be less efficient, for instance if I put it in vmap.</p>
<p>Here is my implementation using jvp :</p>
<pre><code class="lang-auto">import torch
from torch.func import jvp, vmap
matrix = torch.randn([4, 5], requires_grad=True)
input = torch.randn([5])
def func(x):
    return matrix @ x

output = func(input)
jvp_fn = lambda v: jvp(func, (input,), (v,))[1]
# Here we throw away the first output which is exactly output, this is a bit of a waste
jacobian_map = vmap(jvp_fn)
jacobian = jacobian_map(torch.eye(matrix.shape[1])).T
gramian = jacobian_map(jacobian)
</code></pre>
<p>Any idea is most welcome !</p>
How can I increase GPU usage?

username: theorem_on
cooked: <p>When the GPU usage is low, what is the common approach?</p>
<p>I know that simply increasing the batch size can affect the learning outcomes.</p>
<p>Are there other methods I should consider?</p>

username: ptrblck
cooked: <p>Increasing the batch size or using a larger model will quickly fill up your GPU memory. E.g. if your current model trains fine you could try to increase the number of layers and/or parameters to check if the training would benefit from it.</p>

username: theorem_on
cooked: <p>Thank you for your response.</p>
<p>Won’t increasing the batch size affect learning outcomes? Is increasing batch size until GPU memory is full the right choice?</p>
<p>My current average GPU utilization is 25%, so I was hoping to reduce the 3+ hours it takes to train my model. If my model and data are not heavy, is it impossible to reduce the training time?</p>

username: vdw
cooked: <aside class="quote no-group" data-username="theorem_on" data-post="3" data-topic="196225">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/theorem_on/48/62078_2.png" class="avatar"> theorem_on:</div>
<blockquote>
<p>Won’t increasing the batch size affect learning outcomes?</p>
</blockquote>
</aside>
<p>Is some sense probably, but it can be for the better. Larger batch sizes mean that your gradient for the update steps is an average over more training samples. This means typically, the larger the batch, the smaller the gradient. Hence, it’s very common that increasing the batch size allows you to increase the learning rate.</p>

username: theorem_on
cooked: <p>Thank you.<br>
I’ll create a larger batch size and train from it.</p>

username: theorem_on
cooked: <p>I’m sorry, but this may be a basic question.<br>
When increasing the batch size, would it be okay to set a very large size like 1024, 2048, 4096, 8192, etc. I know you’ve already answered this, but I’d like to know additionally if it’s valid for such large sizes.</p>
<p>If this is an issue, I would also like to know what size batch sizes you typically use to increase the batch size.</p>
What does grad_fn=<PowBackward0> mean exactly?

username: Ziyu_Huang
cooked: <p>Hi! I am interested in the exact meaning of grad_fn=. Thank you!!!</p>

username: ptrblck
cooked: <p>The simple explanation is: during the forward pass PyTorch will track the operations if one of the involved tensors requires gradients (i.e. its <code>.requires_grad</code> attribute it set to <code>True</code>) and will create a computation graph from these operations. To be able to backpropagate through this computation graph and to calculate the gradients for all involved parameters, PyTorch will additionally store the corresponding “gradient functions” (or “backward functions”) of the executed operations to the output tensor (stored as the <code>.grad_fn</code> attribute). Once the forward pass is done, you can then call the <code>.backward()</code> operation on the output (or loss) tensor, which will backpropagate through the computation graph using the functions stored in <code>.grad_fn</code>.<br>
In your case the output tensor was created by a <code>torch.pow</code> operation and will thus have the <code>PowBackward</code> function attached to its <code>.grad_fn</code> attribute:</p>
<pre><code class="lang-python">x = torch.randn(2, requires_grad=True)
out = torch.pow(x, 2)
print(out)
# tensor([0.4651, 1.1575], grad_fn=&lt;PowBackward0&gt;)

out.sum().backward()
print(x.grad)
# tensor([-1.3640, -2.1517])
</code></pre>

username: Ziyu_Huang
cooked: <p>Thank you very much for your detailed answer!!!</p>

username: unrenormalizable
cooked: <p>Can you point to the pytorch codebase where the grad_fns are implemented? functions like SumBackward0, PowBackward0 etc.</p>

username: ptrblck
cooked: <p>You can search the derivative of functions in the <a href="https://github.com/pytorch/pytorch/blob/e3cde685340d2c5c752428b37449ba75f59488af/tools/autograd/derivatives.yaml#L1330-L1332"><code>derivatives.yaml</code></a> file and will see for e.g. <code>pow</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">- name: pow.Tensor_Scalar(Tensor self, Scalar exponent) -&gt; Tensor
  self: pow_backward(grad, self, exponent)
  result: auto_element_wise
</code></pre>
<p>Searching for <code>pow_backward</code> points to <a href="https://github.com/pytorch/pytorch/blob/e3cde685340d2c5c752428b37449ba75f59488af/torch/csrc/autograd/FunctionsManual.cpp#L466">this function</a>.</p>
PyTorch distributed Peer to Peer training without initialising a specific group size

username: Nikolay_Blagoev
cooked: <p>My question is whether PyTorch supports Peer to Peer training without the need to initialise a group, as there may be a situation where the number of processes is not known apriori or can change throughout the training process. I saw the P2POp class, but it still seem to require that a group be initialised beforehand.</p>
<p>From what I understood torch elastic fails a sub group if one node fails in it. This is still undesired behaviour.</p>
<p>I am looking for a low level functionality which can establish a connection between two nodes, send/receive tensors through it, and allow for continued training even if some nodes disconnect from the group</p>
Sequence inference

username: jeduapf
cooked: <p>Hello guys, I’ve trained two YOLOv7 models for different purposes but I want to chain them in a better model (they can only be chained since the input of the second model is a cropped image of the input of the first one). So my question is, can I load both models in the same GPU and call one or the other depending on which image I have ( original input or cropped image after the first detection )?</p>

username: jeduapf
cooked: <aside class="quote no-group" data-username="rtwolfe94022" data-post="2" data-topic="196263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rtwolfe94022/48/67286_2.png" class="avatar"> rtwolfe94022:</div>
<blockquote>
<p>This approach provides a flexible way to utilize both models within the same pipeline, leveraging the GPU’s computational power for both initial and secondary detections. Adjust the memory management and parallel processing strategies according to your specific use case and hardware capabilities.</p>
</blockquote>
</aside>
<p>Thank you mister Wolf, I’ll try the approch today ! I’m using the yolov7 respository from WongKinYiu (<a href="https://github.com/WongKinYiu/yolov7" class="inline-onebox" rel="noopener nofollow ugc">GitHub - WongKinYiu/yolov7: Implementation of paper - YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors</a>) and in the code they trace the model, so after tracing both of them they will be well allocated in the same GPU right?</p>
<pre><code class="lang-auto">    # Initialize
    set_logging()
    device = select_device(opt.device)
    half = device.type != 'cpu'  # half precision only supported on CUDA

    # Load model
    model = attempt_load(weights, map_location=device)  # load FP32 model
    stride = int(model.stride.max())  # model stride
    imgsz = check_img_size(imgsz, s=stride)  # check img_size

    if trace:
        model = TracedModel(model, device, opt.img_size) # 

    if half:
        model.half()  # to FP16
</code></pre>
<p>So I just do it for both models and let it shine? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
Keeping optimizer states in FP32

username: MFajcik1
cooked: <p>Based on <a href="https://arxiv.org/pdf/2112.11446.pdf" rel="noopener nofollow ugc">Rae et al., 2021</a>, Figure A7, training LLMs in BF16 could hurt its performance, and they recommend keeping optimizer states in FP32 (model/gradients can stay in BF16).</p>
<p>I was wondering how to implement such a behavior in torch. Especially when combined with FSDP.  For instance for bf16, and adamW, can I simply cast <code>exp_avg</code>, and <code>exp_avg_sq</code> from optimizer.state_dict()?</p>
<p>Do you know any existing implementation of this?</p>
Understanding potential issues with transformers

username: kirk86
cooked: <p>Hi, I’m trying to understand the whether there’s a problem with my training procedure of transformer encoder.</p>
<p>So, basically I’m trying to train a transformer encoder for classification on synthetic dataset of size <code>[batch_size, vocab_size, dim]</code> generated on the fly where the vocab_size is of varying size.</p>
<p>During training I’ve observed that the loss and accuracy oscillates a lot. Meaning it improves but occasionally hits a hard example then it falls and takes a lot of time to pick up again.</p>
<p>When I plot train and loss values I get plots like the following, since I dunno much about nlp and transformer I was wondering if this is normal or is an indication of something wrong going on?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/5453f868ed4f89d047b88dc3de724a8229ad472f.png" alt="image" data-base62-sha1="c200OBCEsDGkYEEP7gDvWCJp9UP" width="409" height="297"></p>

username: J_Johnson
cooked: <p>Have you tried adjusting your learning rate? Also, what have you set your dropout to?</p>

username: kirk86
cooked: <p>Yes, I’ve tried changing the learning rate, also dropout is at 0.1.<br>
But, I’m using also weight decay but really small value 5e-4 and also gradient clipping at 0.5.</p>
Run Time Error: expected mat1 and mat2 to have same dtype while implementing GraphSage via Pytorch Geometric

username: Data_Science
cooked: <p>Hi<br>
While implementing GraphSage code via Pytorch Geometric, I am using the Neighbour_Loader function to do minibatching. I am working on graphdata wit size in 410 million.<br>
Graphsage takes 3 parameters edge index, x and y. When I am running the code, I am getting the run time error expected mat1 and mat2 to have same dtype, but got long int!=float. I have not type casted any of the pytorch geometric objects or the tensors. All of them are int64. Please someone solve this error.</p>
About stft and istft

username: Franz_Kaufmann
cooked: <p>class InitialReconstruction(BaseModule):<br>
def <strong>init</strong>(self, n_fft, hop_size):<br>
super(InitialReconstruction, self).<strong>init</strong>()<br>
self.n_fft = n_fft<br>
self.hop_size = hop_size<br>
window = torch.hann_window(n_fft).float()<br>
self.register_buffer(“window”, window)</p>
<pre><code>def forward(self, stftm):
    angle = torch.angle(stftm)
    magnitudes = torch.abs(stftm)
    stft = magnitudes * torch.exp(1j * angle)
    istft = torch.istft(stft, n_fft=self.n_fft, 
                    hop_length=self.hop_size, win_length=self.n_fft, 
                    window=self.window, center=True,return_complex=False,onesided=False)
</code></pre>
<p>return istft.unsqueeze(1)</p>
<p>class FastGL(BaseModule):<br>
def <strong>init</strong>(self, n_mels, sampling_rate, n_fft, hop_size, momentum=0.99):<br>
super(FastGL, self).<strong>init</strong>()<br>
self.n_mels = n_mels<br>
self.sampling_rate = sampling_rate<br>
self.n_fft = n_fft<br>
self.hop_size = hop_size<br>
self.momentum = momentum<br>
self.pi = PseudoInversion(n_mels, sampling_rate, n_fft)<br>
self.ir = InitialReconstruction(n_fft, hop_size)<br>
window = torch.hann_window(n_fft).float()<br>
self.register_buffer(“window”, window)</p>
<pre><code>@torch.no_grad()
def forward(self, s, n_iters=32):
    c = self.pi(s)
    x = self.ir(c)
    x = x.squeeze(1)
    c = c.unsqueeze(-1)
    prev_angles = torch.angle(s).clone().detach()
    for _ in range(n_iters):        
        s_complex = torch.stft(x, n_fft=self.n_fft, hop_length=self.hop_size, 
                       win_length=self.n_fft, window=self.window, 
                       center=True, return_complex=False, onesided=True)
       magnitudes = torch.abs(s_complex)
        angles = torch.angle(s_complex)
        prev_angles = angles
       
       s = c * (magnitudes*torch.exp(1j*(angles + self.momentum * (angles - prev_angles))))
        x = torch.istft(s, n_fft=self.n_fft, hop_length=self.hop_size, 
                                       win_length=self.n_fft, window=self.window, 
                                       center=True,return_complex=False, onesided=False)
</code></pre>
<p>return x.unsqueeze(1)</p>
<p>from these snippets, errors like “RuntimeError: Cannot have onesided output if window or input is complex” or “RuntimeError: istft(CUDAComplexFloatType[1, 513, 602], n_fft=1024, hop_length=256, win_length=1024, window=torch.cuda.FloatTensor{[1024]}, center=1, normalized=0, onesided=0, length=None, return_complex=1) : expected the frequency dimension (3rd to the last) of the input tensor to match n_fft when onesided=False, but got 513(when istft return complex = True)” still keeps occuring. what is the reason?</p>
How to convert at::ScalarType to c++ type?

username: shadowshadow
cooked: <p>I have a template class have two typename</p>
<pre data-code-wrap="c++"><code class="lang-c++">// key maybe uint32_t or uint64_t, elem can be any interger or float type
template &lt;typename KeyType, typename ElemType&gt;
class Cache{
  // ....
};
</code></pre>
<p>I want to use pybind to provide python API for this Cache class. And since pybind API need to specify the KeyType and ElemType. So I need to write a wrapper class like below:</p>
<pre data-code-wrap="c++"><code class="lang-c++">class CacheWrapper{
   CacheWrapper(at::Tensor t, CacheConfig cfg){
         dtype = t.scalar_type();
         if(cfg.capacity &gt; UINT32_MAX){
               lru_cache = reinterpret_cast&lt;void*&gt;(AT_DISPATCH_ALL_TYPES(t.type(), "uint32_cache",[&amp;]{NewUint32LRUCache&lt;scalar_t&gt;(cache, cfg);}));
            }else{
               lru_cache = reinterpret_cast&lt;void*&gt;(AT_DISPATCH_ALL_TYPES(t.type(), "uint64_cache",[&amp;]{NewUint64LRUCache&lt;scalar_t&gt;(cache, cfg);}));
            }
    }
private:
     void* cache;
     at::ScalarType dtype;
};
</code></pre>
<p>my question is can I directly use the dtype to instantiate the template class like <code>Cache&lt;uint32_t, dtype&gt;</code>, if not, how can I convert dtype to c++ type;</p>

username: tom
cooked: <p><code>ScalarTypeToCPPType&lt;scalar_type&gt;.type</code> might do the trick (in c10/core/ScalarType.h).</p>
<p>Best regards</p>
<p>Thomas</p>
BatchNorm Learnable Parameters

username: barista
cooked: <p>In the batch normalization’s pre-activation scaling, are the gamma and beta parameters learnable?</p>

username: ptrblck
cooked: <p>BatchNorm layers define trainable parameters by default, where the <code>weight</code> corresponds to the <code>gamma</code> parameter from the original paper and the <code>bias</code> corresponds to the <code>beta</code> parameter.</p>