akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Correct Implementation of Beta-VAE Reconstruction Loss with ViT Encoder-Decoder Architecture

Janina: <p>I am implementing a Vision Transformer (ViT) Encoder-Decoder architecture trained within a Beta VAE framework on noisy latent codes. My encoder returns the learned mean (<code>mu</code> ) and log variance (<code>logvar</code> ) of the latent space. Despite various attempts, I haven’t achieved the expected results for my decoded samples, yet. I suspect that the issue might also be in the definition of my objective, particularly the reconstruction loss. I have come across various ways of implementing the reconstruction loss in different works and am unsure if my current approach is correct.</p>
<p><strong>Current Implementation</strong> :<br>
I am using <code>F.mse_loss</code> with <code>reduction='none'</code> to get the pixel-wise error, then summing the errors over the height, width, and channel dimensions to get the loss per sample. I noticed an improvement when summing over the height, width, and channel dimensions in the variance of my decoded samples. Additionally, I have been experimenting with implementing the Negative Log-Likelihood (NLL) Loss, as I am working with noisy (normally distributed) latent codes of shape <code>(B, 4, 32, 32)</code>.</p>
<p><strong>Questions</strong>:</p>
<ol>
<li>Is the reconstruction loss correctly implemented for a ViT decoder?</li>
<li>Are there any potential issues or improvements that I could consider?</li>
</ol>
<p><strong>Code Sample</strong>:</p>
<pre><code class="lang-auto">    def reparametrize(self, mu, logvar):
        """Reparametrization trick to sample latent variable z ~ N(mu, sigma^2)."""
        if self.training:
            std = torch.exp(0.5 * logvar)
            eps = torch.randn_like(std)  # Sample from standard normal distribution
            return eps * std + mu
        else:
            return mu

    def reconstruction_loss(self, x, recon_x):
        """Reconstruction loss."""
        if self.recon_loss_type == 'L2':
            # Standard MSE Loss (L2, pixel-wise no uncertainty)
            mse_loss = F.mse_loss(recon_x, x, reduction='none')  # Pixel-wise error
            recon_loss = mse_loss.sum(dim=(1, 2, 3))  # Sum over all samples when (B, C, H, W)
        elif self.recon_loss_type == 'NLL':
            # Negative Log-Likelihood with learnable scale (models aleatoric uncertainty)
            var = torch.exp(self.scale)  # Convert to variance (log ensures positive values)
            dist_normal = torch.distributions.Normal(recon_x, torch.sqrt(var))  # Learned variance
            log_pxz = dist_normal.log_prob(x)
            recon_loss = -log_pxz.sum(dim=(1, 2, 3))  # Sum over all samples when (B, C, H, W)
        else:
            raise ValueError('Undefined reconstruction loss type.')

        return torch.mean(recon_loss, dim=0)  # Mean over the batch dimension

    def kl_divergence(self, mu, logvar):
        """KL divergence between approximate posterior and prior."""
        kld_loss = torch.mean(-0.5 * torch.sum(1 + logvar - mu ** 2 - logvar.exp(), dim=1), dim=0)
        return kld_loss
</code></pre>
The dtype of optimizer states in PyTorch AMP training

Chaohao_Yang: <p>Hello. I know that when using PyTorch AMP for training, the model weights are of type float32, while the gradients are 16 bits. I’m now wondering what the type of optimizer states is. For example, if I use Adam optimizer for training in the PyTorch AMP context, will the dtype of the states of Adam be 32bit or 16bit? Thanks.</p>

ptrblck: <p>The optimizer states should use the same dtype as the parameters but you can quickly check it by printing the <code>optimizer.state_dict()</code>.</p>
Is there a danger when doing DDP that different processes compiling kernels will overwrite each other?

divinho: <p>I’m in a situation where I have dynamic shapes, it’s a limited number, but there is more than one shape so I’m wondering what happens if I’m training on many GPUs. What if different processes are writing out kernels to the same location could that cause issues?</p>
<p>Would it be better for each process to use a different local disk location for its kernels (rather than setting <code>PYTORCH_KERNEL_CACHE_PATH</code> equally for all kernels)?</p>
Weight Decay for tied weights (embedding and linear layers)

dhruvbird: <p>So I have a model where I have an embedding layer (nn.Embedding) and a final nn.Linear projection layer that are sharing weights via weight tying.</p>
<p>It seems like the best practice is not to perform weight decay on embedding weights, but to perform decay on linear layer weights. What should I do in this situation?</p>
<p>Here are the pages I have checked unsuccessfully in search of an answer.</p>
<ol>
<li><a href="https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994" class="inline-onebox">Weight decay in the optimizers is a bad idea (especially with BatchNorm)</a></li>
<li><a href="https://github.com/karpathy/minGPT/pull/24#issuecomment-679316025" class="inline-onebox" rel="noopener nofollow ugc">Weight decay exclusions by michaellavelle · Pull Request #24 · karpathy/minGPT · GitHub</a></li>
<li><a href="https://github.com/karpathy/minGPT/blob/3ed14b2cec0dfdad3f4b2831f2b4a86d11aef150/mingpt/model.py#L136" rel="noopener nofollow ugc">https://github.com/karpathy/minGPT/blob/3ed14b2cec0dfdad3f4b2831f2b4a86d11aef150/mingpt/model.py#L136</a></li>
<li><a href="https://stats.stackexchange.com/questions/576463/why-not-perform-weight-decay-on-layernorm-embedding" class="inline-onebox" rel="noopener nofollow ugc">regularization - Why not perform weight decay on layernorm/embedding? - Cross Validated</a></li>
<li><a href="https://github.com/pytorch/examples/blob/main/word_language_model/model.py#L28" rel="noopener nofollow ugc">https://github.com/pytorch/examples/blob/main/word_language_model/model.py#L28</a></li>
<li><a href="https://stackoverflow.com/questions/49299609/tying-weights-in-neural-machine-translation" class="inline-onebox" rel="noopener nofollow ugc">python - Tying weights in neural machine translation - Stack Overflow</a></li>
<li><a href="https://discuss.pytorch.org/t/weight-decay-only-for-weights-of-nn-linear-and-nn-conv/114348" class="inline-onebox">Weight decay only for weights of nn.Linear and nn.Conv*</a></li>
</ol>

Avelina9X: <p>If you are weight tying you are effectively creating a linear layer that points to the embedding weight matrix rather than it’s own weights, so when you “search” for the weights of all linear layers the tied embeddings shouldn’t show up because those tied weights – although <em>used</em> by the linear layer – don’t <em>belong</em> to the linear layer.</p>
<p>In practice this depends on how you implement the weight tying. In the case of something like HuggingFace’s <code>PreTrainedModel</code> (with <code>tie_word_embeddings=True</code> in the config and a correctly set <code>_tied_weights_keys = [ 'lm_head.weight' ]</code> class attribute) the weights of <code>self.lm_head</code> will NOT be included when iterating through all child weights and modules, and will only be included in the <code>nn.Embedding</code> layer. So as long as your code knows not to enable weight decay for embedding modules the tied linear layer will not receive weight decay.</p>
Not able to download Pytorch

PierrunoYT: <p>ERROR: Could not find a version that satisfies the requirement torch (from versions: none)<br>
ERROR: No matching distribution found for torch</p>
<p>ERROR: Could not find a version that satisfies the requirement torchvision (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3)<br>
ERROR: No matching distribution found for torchvision</p>

ptrblck: <p>Could you post more information about your setup, in particular which Python version and OS you are using as well as the used install command, please?</p>

PierrunoYT: <p>I use Windows 10 and the latest python version.</p>
<p>I tried multiple commans like the ones from the website and other ones I found on the web.</p>

ptrblck: <p>The “latest” Python version might be too new, so maybe try to downgrade it to 3.10.</p>

BarnabyKincaid: <p>I am also seeing this issue:</p>
<pre><code class="lang-auto">C:\Users\name&gt;python --version
Python 3.7.3

C:\Users\name&gt;pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
Looking in indexes: https://download.pytorch.org/whl/cu118
Collecting torch
  Could not find a version that satisfies the requirement torch (from versions: )
No matching distribution found for torch
</code></pre>
<p>This is on Windows 10. I had just installed a fresh CUDA toolkit v 11.8</p>

ptrblck: <p>Python=3.7 is not supported anymore as seen in the <a href="https://pytorch.org/blog/pytorch-2.0-release/#highlightsperformance-improvements">release notes</a>:</p>
<blockquote>
<p><a href="https://pytorch.org/blog/deprecation-cuda-python-support/">Deprecation of Cuda 11.6 and Python 3.7 support</a> for PyTorch 2.0</p>
</blockquote>

BarnabyKincaid: <p>Ah, my mistake, thanks!</p>

mono: <p>i am having the same error:<br>
(environmentname) C:\Users\ahsan&gt;pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu124" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu124</a><br>
Looking in indexes: <a href="https://download.pytorch.org/whl/cu124" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu124</a><br>
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)<br>
ERROR: No matching distribution found for torch</p>
<p>windows11<br>
python version is 3.10.0<br>
just installed cuda:<br>
Cuda compilation tools, release 12.4, V12.4.99<br>
Build cuda_12.4.r12.4/compiler.33961263_0</p>
<p>i have tired python 3.11.0 as well as 3.12 but i get the same error</p>

mono: <p>it is solved: i had to download the pytorch manully from <a href="https://download.pytorch.org/whl/torch/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch/</a>. I had to make sure i choose the correct cuda and cp version.Then i had to manually install it using pip.</p>

Venom_Rock: <p>how did you install it manually using pip please</p>

Venom_Rock: <p>and how to make sure whats my correct cuda and cp version please</p>
Question about PyTorch and TensorFlow math

Aosmgpem_Ipsnagpisne: <p>Hello,</p>
<p>If y’all don’t mind, I have a weird question.</p>
<p>Let’s say I create the same model in PyTorch and in TensorFlow and then train it on the same dataset, everything the same</p>
<p>Then if I use the API for both to get the intermediate result of lets say layer 5 for the same input data, will the intermediate result matrix be identical in both the cases?</p>
<p>Or, another way to phrase this question would be, the math that would happen in both the models, will it be exactly the same?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>No, the math could slightly differ (I believe the handling of small <code>eps</code> values e.g. in normalization layers was different between frameworks). Even if the math is equal, the used algorithms might differ so you should at least expect small numerical errors caused by the limited floating point precision and a different order of operations.</p>
Dataloader yields copied batches

ilykuleshov: <p>Hello everyone! I have a small val dataset (~386 entries), so with a batch_size=256 the dataloader only does two iterations. With num_workers &lt;= 1 that is. As I found out, setting num_workers to a larger number yields copies of batches: for num_workers=2, the dataset is yielded twice; for num_workers=4 the dataset is yielded 4 times (totalling to 8 batches), and so on. The batches are not empty, so since I stumbled across this on a prediction step, I got a dataset several times bigger than the original one. This took a while for me to figure out, it’s not indicated in the docs, and it seems that such behavior is unexpected.</p>

eqy: <p>That would be interesting behavior indeed but I cannot reproduce it on my end:</p>
<pre><code class="lang-auto">$ cat dataloader.py
import torch

class ToyDataset(torch.utils.data.Dataset):
    def __getitem__(self, idx):
        data = torch.tensor(idx)
        return data

    def __len__(self):
        return 386

dataset = ToyDataset()
loader = torch.utils.data.DataLoader(dataset, batch_size=256, num_workers=2)
for batch in loader:
    print(batch.size(0))
</code></pre>
<pre><code class="lang-auto">$ python3 dataloader.py
256
130
</code></pre>

ilykuleshov: <p>Thank you very much for answering! I could not reproduce it as well; the problem seems to have fixed itself. And yet I distinctly remember changing the num_workers argument and seeing the len(dm.dataloader()) change accordingly. Sorry I couldn’t be of more help.</p>

ilykuleshov: <p>Finally was able to reproduce, sorry for bumping this old thread. Seems the issue is with using the <code>IterableDataset</code>:</p>
<pre><code class="lang-auto">import torch


class ToyDataset(torch.utils.data.IterableDataset):
    def __iter__(self):
        data = torch.arange(len(self))
        yield from data

    def __len__(self):
        return 386


dataset = ToyDataset()
loader = torch.utils.data.DataLoader(dataset, batch_size=256, num_workers=2)
print(len(loader), len(list(loader))) # 2 4
</code></pre>
Correct way of using foreach_worker and foreach_env

abidmeeraj: <p>I am quite new to Reinforcement Learning and can’t understand it. I am unable to update configurations for the batch data using PPO.</p>
<p>I am using my custom-defined GYM environment, and want to train it using PPO and my external data which I’m loading in the form of torch DataLoader.<br>
I am using Python 3.11 and Ray 2.40.0. Following is the relevant code:</p>
<pre><code class="lang-auto">import ray
from ray.rllib.algorithms.ppo import PPOConfig
from ray.tune.registry import register_env
from torch.utils.data import DataLoader

train_dataset = MultimodalDataset(
    csv_file=config.TRAIN_CSV_PATH, max_images=config.MAX_IMAGES_RL
)
train_loader = DataLoader(train_dataset, batch_size=config.BATCH_SIZE, shuffle=True)

# Define PPO configuration
ppo_config = (
    PPOConfig()
    .training(gamma=0.9, lr=0.01)
    .environment(env="MultimodalSummarizationEnv", env_config=default_env_config)
    .framework("torch")
    .resources(num_gpus=0, num_cpus_per_worker=1)
)

# Create PPO trainer
trainer = ppo_config.build()

# Function to update worker environments
def update_env_config_and_reset(worker, new_env_config):
    worker.foreach_env(lambda env: env.reset(env_config=new_env_config))

# Training loop
for batch_idx, batch in enumerate(train_loader):
    # Prepare batch-specific env_config
    new_env_config = {
# new data for the batch_idx
    }

    # Update and reset environments for all workers
    trainer.workers.foreach_worker(
        lambda worker: update_env_config_and_reset(worker, new_env_config)
    )

    # Train PPO
    result = trainer.train()
ray.shutdown()
</code></pre>
<p>However, when running the code I get the error on foreach_worker as follows:</p>
<blockquote>
<p>‘function’ object has no attribute ‘foreach_worker’</p>
</blockquote>
<p>Please help me identify where am I getting it wrong.</p>
Using snapdragon x elite npu for training

count_von_count: <p>Dear community,</p>
<p>i want to use the npu unit of my snapdragon x elite to train my models. is there any kind of pytorch integration or any other way for a medium skilled programmer to unleash its 45 tops?</p>
<p>Thanks in advance!</p>

XLin: <p>I’d love to know too, but there are really too few related questions.</p>
Uint64 tensors do not wraparound on overflow

kmaeng: <p>Dear experts,</p>
<p>I am seeing that uint64 datatype in torch does not have a wraparound behavior on overflow/underflow (which is the expected behavior in C/C++ from my understanding). Below is my code:</p>
<pre><code class="lang-auto">import torch

A = torch.tensor([2 ** 40], dtype=torch.uint64)
B = torch.tensor([2 ** 40], dtype=torch.uint64)
C = A * B
print(C)
A = torch.tensor([2 ** 40], dtype=torch.long)
B = torch.tensor([2 ** 40], dtype=torch.long)
C = A * B
print(C)
</code></pre>
<p>Both prints out 0. For torch.long, I see this makes sense as overflow/underflow is usually undefined behaviour. But I am surprised to see this happening for uint64.</p>
<p>Below are some of the questions:</p>
<ol>
<li>Is this expected (is the unsigned number overflow behavior deviating from C/C++ for torch)?</li>
<li>Is there a way for me to make it wraparound (e.g., by compiling the torch with a certain flag?)</li>
<li>If not, is there any way I can achieve my desired behavior?</li>
</ol>
<p>I appreciate any pointers in advance.</p>

KFrank: <p>Hi kmaeng!</p>
<aside class="quote no-group" data-username="kmaeng" data-post="1" data-topic="213926" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kmaeng/48/23707_2.png" class="avatar"> kmaeng:</div>
<blockquote>
<p>I am seeing that uint64 datatype in torch does not have a wraparound behavior on overflow/underflow</p>
</blockquote>
</aside>
<p>You are seeing the correct behavior (as do I on pytorch version 2.5.1).</p>
<p>I think you are being fooled by the fact that <code>2**40 * 2**40 = 2**80</code> and<br>
<code>2**80 = 0 mod 2**64</code>.  So <code>0</code> is the correct “wraparound” result.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.5.1'
&gt;&gt;&gt; A = torch.tensor([2 ** 40 - 1], dtype=torch.uint64)
&gt;&gt;&gt; B = torch.tensor([2 ** 40], dtype=torch.uint64)
&gt;&gt;&gt; A * B
tensor([18446742974197923840], dtype=torch.uint64)
&gt;&gt;&gt; 
&gt;&gt;&gt; A = torch.tensor([2 ** 40], dtype=torch.long)
&gt;&gt;&gt; B = torch.tensor([2 ** 40], dtype=torch.long)
&gt;&gt;&gt; A * B
tensor([0])
&gt;&gt;&gt; 
&gt;&gt;&gt; A = torch.tensor([2 ** 40 - 1], dtype=torch.long)
&gt;&gt;&gt; A * B
tensor([-1099511627776])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

kmaeng: <p>Oops. Yeah, this was a stupid mistake. I thought I had -1 but not sure where it went. Thanks for letting me know <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20"></p>
Write after Read dependency for async communication

Tianshu_Yu: <p>I’m trying to use torch.distributed.batch_isend_irecv(). The documentation said that this operation is async. However, I don’t know whether this mean that it means it could do Read after Write violation or Write after Read violation or both.</p>
<p>I’ve observed that if I do.<br>
p2p_op_lst = [<br>
torch.distributed.P2POp(torch.distributed.irecv, x, peer=src, group=self.comm_group)<br>
]<br>
reqs = torch.distributed.batch_isend_irecv(p2p_op_lst)<br>
x = a(x)<br>
The communication could overlap with the operation of kernel a, since communication happens on a different stream than the stream a is on.</p>
<p>But I have never observed the violation of the WAR.<br>
x = a(x)<br>
reqs = torch.distributed.batch_isend_irecv(p2p_op_lst)<br>
i.e. the recv never overlaps with a in this case.</p>
<p>Does torch.distributed’s async guarantee that overlapping in the second case would never happen? Does the user have to do explicit synchronization between communication stream and computation stream? How to do that? req.wait() is only for synchronization after communication but it’s not for synchronization before communication.</p>
Fold an overlapping 3D tensor?

bartley: <p><strong>Problem:</strong><br>
Fold/unfold overlapping patches from 3D tensors.</p>
<p><strong>Context:</strong><br>
I am working on a segmentation problem where I make a prediction for each patch, and then use mean pooling to combine predictions from overlapping patches.</p>
<p>Using code from <a href="https://discuss.pytorch.org/t/patch-making-does-pytorch-have-anything-to-offer/33850/11">here</a>, non-overlapping patches works great, but I have been unable to adapt the code to overlapping patches.</p>
<pre><code class="lang-auto">import torch

# Unfold data
x = torch.randn(1, 256, 256, 256)  # batch, c, h, w
kc, kh, kw = 64, 64, 64  # kernel size
dc, dh, dw = 64, 64, 64  # stride
patches = x.unfold(1, kc, dc).unfold(2, kh, dh).unfold(3, kw, dw)
unfold_shape = patches.size()
patches = patches.contiguous().view(patches.size(0), -1, kc, kh, kw)

# Reshape back
patches_orig = patches.view(unfold_shape)
output_c = unfold_shape[1] * unfold_shape[4]
output_h = unfold_shape[2] * unfold_shape[5]
output_w = unfold_shape[3] * unfold_shape[6]
patches_orig = patches_orig.permute(0, 1, 4, 2, 5, 3, 6).contiguous()
patches_orig = patches_orig.view(1, output_c, output_h, output_w)

# Check for equality
print((patches_orig == x).all())
</code></pre>
<p>Is it possible to use stride=16 and use mean pooling to combine overlapping patches?</p>
<p>Cheers,<br>
Brendan</p>

bartley: <p>The best option I found was to use Monai’s <a href="https://docs.monai.io/en/stable/inferers.html#sliding-window-inference-function" rel="noopener nofollow ugc">sliding window inference function</a>. This was exactly what I was looking for.</p>
Saving: `torchscript` or `state_dict`?

Aknw_Fen: <p>About <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">saving for inference</a>, the docs assert:</p>
<blockquote>
<p>Using the TorchScript format, you will be able to load the exported model and run inference without defining the model class</p>
</blockquote>
<p>And also “Recommend”:</p>
<blockquote>
<p>Save/Load <code>state_dict</code> (Recommended)</p>
</blockquote>
<p>I’m aware that Torchscript does not need model definition, and hence must store it within the pth file.</p>
<p>However, I’m not sure which one to use. Are there any other criteria that is important to be aware of?</p>
Use jit to speed up unrolled RNN

yousiyu: <p>I have the following customized GRU class, which unrolls the for loop, how can I use jit to speed it up?</p>
<pre><code class="lang-auto">class GRU(tc.nn.Module):
    """ GRU with zoneout support, input is batch first
    """
    def __init__(self, input_dim, hidden_size, zoneout_p=0):
        super().__init__()
        self.gru_cell = tc.nn.GRUCell(input_dim, hidden_size)
        self.hidden_size = hidden_size
        self.zoneout_p = zoneout_p

    def forward(self, x):
        assert len(x.shape) == 3, x.shape
        hidden_states = [self.gru_cell(x[:, 0, :])]
        for i in range(1, x.shape[1]):
            h_new = self.gru_cell(x[:, i, :], hidden_states[-1])  # BH
            if self.training:
                s = tc.rand_like(h_new) &lt; self.zoneout_p
                h_new = s * hidden_states[-1] + (~s) * h_new
            hidden_states.append(h_new)
        return tc.cat([hh[:, None, :] for hh in hidden_states], dim=1), hidden_states[-1][None]
</code></pre>
<p>I am running the code on GPU, my typical input size is (10000, 50, 7)<br>
I have tried subclassing the class from tc.jit.ScriptModule and decorating the forward method with tc.jit.script_method following <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/#lstm-layer-forward" class="inline-onebox" rel="noopener nofollow ugc">Optimizing CUDA Recurrent Neural Networks with TorchScript | PyTorch</a><br>
But it actually made the code even slower…</p>
Differentiable Optimizer Not Working For Simple Example

ekrws: <pre><code class="lang-auto">import torch
import torch.nn as nn

class Net(torch.nn.Module):

    def __init__(self):
        super().__init__()
        self.device = (
        "cpu"
        )
        self.Initialize()

    def Initialize(self):
        self.FC = nn.Sequential(
                nn.LayerNorm(10),
                nn.Linear(10, 5),).to(self.device)
    
        
    def Inference(self, s):
        ou = self.FC(s)
        return ou


    def forward(self, s):
        return self.Inference(s)


model = Net()
model.train()

optimizer = torch.optim.Adam(model.parameters(), lr=0.001, differentiable=True)

z = torch.randn((10,10))

a = model(z)

loss = (a - 1).mean()

optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>As above.</p>
<p>Error produced:<br>
File “…\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\torch\optim\adam.py”, line 413, in <em>single_tensor_adam<br>
param.addcdiv</em>_(exp_avg, denom)<br>
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</p>
<p>Pytorch 2.4</p>

janeyx99: <p>This is expected! The use case of a differentiable optimizer is to usually learn the best hyperparameters for the optimizer, e.g., the LR or weight decay, and so the key is to make the parameters non-Leaf as one does that. One way to make the parameters non-Leaf is to clone them all before passing them into the optimizer.</p>
<p>(As an aside, PyTorch support for Tensor LR for differentiable optimizers is admittedly sparse and is now on my mind as something to improve.)</p>
How to share Dataset among different processes in Pytorch？

forestbat: <p>My dataset for training Pytorch model needs plenty of I/O to generate, so I hope the program will use only 1 process to generate it, and this is my code:</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

# world_size is GPUs' num
mp.spawn(train_worker, args=(world_size, config_data), nprocs=world_size, join=True)

def train_worker(rank, world_size, cfgs):
    trainer = MyDistributedTask(rank, world_size, cfgs)
    trainer.run()

class MyDistributedTask():
    def make_dataset(self, is_tra_val_te: str):
        # Use process 0 to generate dataset
        if self.rank == 0:
            data_cfgs = self.cfgs["data_cfgs"]
            dataset = MyDataset(data_cfgs, is_tra_val_te)
            return dataset
</code></pre>
<p>However, <code>MyDataset</code> generated in process 0 is not sent to other processes, so how can I share <code>MyDataset</code> among processes? And I don’t want to use <code>mp.Lock</code> because it will make program slower and make <code>polars</code> appear deadlock mistakes.</p>
<p>Thank for your reply.</p>
Bug with isin assume_unique=True

IlanCosman: <p>I have the following incredibly strange bug with isin and assume_unique=True.</p>
<p>Without compiling, the code runs fine for any <code>n</code> and <code>assume_unique</code> value.</p>
<p>When compiling, the code runs fine for any <code>n</code> when <code>assume_unique=False</code>. When <code>assume_unique=True</code>, the code runs fine for <code>n &lt; 12</code>, and errors for <code>n &gt;= 12</code>.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn, Tensor

symbols = 2
bars = 3

all_false = torch.full((symbols, bars), False, device="cuda")  # (symbols, bars)

@torch.compile
class MyModel(nn.Module):
    def __init__(self, n: int, assume_unique: bool) -&gt; None:
        super().__init__()
        self.ignored_symbols = torch.arange(n, device="cuda")
        self.assume_unique = assume_unique

    def forward(
        self,
        symbol_ids: Tensor,  # (symbols, 1)
    ) -&gt; Tensor:
        mask_ignored_symbols = torch.isin(
            symbol_ids, self.ignored_symbols, assume_unique=self.assume_unique
        )  # (symbols, 1)

        return all_false | mask_ignored_symbols

# +50 to avoid the ignored symbols, not important
symbol_ids = torch.arange(symbols, device="cuda").unsqueeze(1) + 50
print(symbol_ids)

model = MyModel(n=12, assume_unique=True).cuda()

output: Tensor = model(symbol_ids)
</code></pre>
<p>The following error is what appears:</p>
<pre><code class="lang-auto">torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
RuntimeError: Attempting to broadcast a dimension of length 2 at -1! Mismatching argument at index 1 had torch.Size([2]); but expected shape should be broadcastable to [2, 3]

While executing %or_ : [num_users=1] = call_function[target=operator.or_](args = (%all_false, %mask_ignored_symbols), kwargs = {})
Original traceback:
  File "/home/ilan/monorepo/src/Ayin3/foo.py", line 28, in forward
    return all_false | mask_ignored_symbols
</code></pre>
<p>I guess there seems to be some weird squeezing going on when using <code>assume_unique=True</code> in compile.</p>

ptrblck: <p>Could you report this error on GitHub in case you can reproduce it in the latest nightly binary, please?</p>
Suggestion for ML approach to mimic FEM results?

LainuUrdin: <p>Hello everyone.</p>
<p>I’m new to Pytorch and machine learning methods, so I hope I’m not asking a question that is too obvious. As part of my research, I am trying to develop a surrogate model that accurately captures the results calculated by the finite element method, but in a much shorter time frame (evaluating my finite element model is computationally very expensive). In particular, I am trying to mimic the scalar field response of a 3D surface defined by a mesh (defined by nodes and connectivities), the inputs being a series of scalar floats.  As a first idea, I thought I could do this with a series of linear layers of neural networks with certain activation functions, but I realised that I would need a very large number of results to get a sufficiently high accuracy. My hypothesis is that the underlying field shape is somehow C0 and perhaps C1 continuous, so if I can somehow incorporate the information in the form of correlations between neighbouring nodes, I should be able to get the same result with a smaller number of outputs. I have seen cases in the literature where convolutional neural networks are used to replicate the results of finite element models, but in these cases they also use images as input data. In my particular case, the inputs are a few scalar values. In principle, it would be preferable for the method to work for a structured and unstructured model, but if it were only for a structured model, it could work for me. Do you have any thoughts on an ML based approach or method to incorporate this correlation between neighbouring nodes? Any suggestions or ideas would be very welcome.  Thanks in advance for your help.</p>
In GPU-memory compression

Theophile_Champion: <p>I need to compress grayscale images in the GPU memory.</p>
<p>I tried <code>encode_png((img * 255).type(torch.uint8).unsqueeze(dim=0))</code>, however this does not support GPU.</p>
<p>I tried <code>encode_jpeg((img * 255).type(torch.uint8).unsqueeze(dim=0))</code>, however this does not seem to support grayscale images, despite what the <a href="https://pytorch.org/vision/0.20/generated/torchvision.io.encode_jpeg.html#torchvision.io.encode_jpeg" rel="noopener nofollow ugc">documentation</a> says:</p>
<ul>
<li><strong>input</strong> (<em>Tensor</em> <em>[</em> <em>channels</em> <em>,</em> <em>image_height</em> <em>,</em> <em>image_width</em> <em>] or</em> <em>List</em> <em>[</em> <em>Tensor</em> <em>[</em> <em>channels</em> <em>,</em> <em>image_height</em> <em>,</em> <em>image_width</em> <em>]</em> <em>]</em>) – (list of) uint8 image tensor(s) of <code>c</code> channels, where <code>c</code> must be 1 or 3</li>
</ul>
<p>What would be the recommended way to do it?</p>
Dino training- input image

Barbo: <p>Hey there I am attempting to train dino to classify my images but I have some issues with the input image size. My images are actually 512x512 but most of it is just a black background and only a tiny part of the image is actually my object of interest. Do you have any recommendations on how to resize the image without affecting its quality?</p>

Eduardo_Lawson: <p>Use this <a href="https://pytorch.org/vision/main/generated/torchvision.transforms.Resize.html" rel="noopener nofollow ugc">Function</a>, you can just follow the example</p>

Barbo: <p>Hi, thanks a lot!! Yes I thought about this one however, the dino input should not be smaller than 224x224 so this means that I cannot resize the images to a smaller size (?). However, my objects are about 50x50 pixels so I worry that 224x224 would still leave a lot of empty space</p>

Eduardo_Lawson: <p>Well, maybe you can do a <a href="https://pytorch.org/vision/0.19/generated/torchvision.transforms.functional.crop.html" rel="noopener nofollow ugc">crop</a> and then Resize it. That will solve your issue</p>

Barbo: <p>Oh I see, sorry my bad I didn’t get the resize function right!! Thanks a lot!</p>
Out of memory when I use torch.cuda.empty_cache

Fandi_Wu: <p>Hi,<br>
I want to release GPU memory, so I use torch.cuda.empty_cache, but it throw a error that<br>
file ‘python3.6/site-packages/torch/cuda/<strong>init</strong>.py’, line 426, in empty_cache<br>
torch._C._cuda_emptyCache()<br>
RuntimeError: CUDA error: out of memory.</p>
<p>If I do not use torch.empty_cache(), everything work well. There is enough memory in GPU, but once I use torch.cuda.empty_cache, and it throw this error. Can someone tell me the reason?</p>

ptrblck: <p>This issue sound a bin weird.<br>
Are you able to create a dummy tensor on your GPU before <code>empty_cache</code> raises this OOM error?<br>
How much memory is being used before the call?</p>

Fandi_Wu: <p>I solve this problem,<br>
The reason is , torch.cuda.empty_cache() write data to gpu0 (by default) ,about 500M<br>
when I meet this problem, my gpu0 was fully occupied. so if I try</p>
<pre><code class="lang-auto">with torch.cuda.device('cuda:1'):
    torch.cuda.empty_cache()
</code></pre>
<p>no memory allocation occurs on gpu0.</p>

Richard_Wang: <p>Thank you, Wu,<br>
This saves my day !!!</p>

L_D: <p>Hello Fandi,<br>
Thank you so much for your solution. It really helps me.<br>
Best,</p>

xinming_wang: <p>Thank you. Really helpful. Can’t believe this method is not cleared in the document</p>

vindy: <aside class="quote no-group" data-username="Fandi_Wu" data-post="3" data-topic="57898">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/5f9b8f/48.png" class="avatar"> Fandi_Wu:</div>
<blockquote>
<p>I solve this problem,<br>
The reason is , torch.cuda.empty_cache() write data to gpu0 (by default) ,about 500M<br>
when I meet this problem, my gpu0 was fully occupied. so if I try</p>
<pre><code class="lang-auto">with torch.cuda.device('cuda:1'):
    torch.cuda.empty_cache()
</code></pre>
<p>no memory allocation occurs on gpu0.</p>
</blockquote>
</aside>
<p>I don’t understand this entirely… Did you mean <code>torch.cuda.empty_cache()</code> by default try to empty the cuda:0, therefore we have to specify which device before calling <code>torch.cuda.empty_cache()</code>?</p>
The strangest bug in model.forward() for multiple batches

Ella_Mendelson: <p>Hello,</p>
<p>I’m running a pytorch model with cuda.</p>
<p>My code gets as input about 100 images, divide them into groups with the size of BATCH_SIZE, and send to the model batch by batch.</p>
<p>In the first batch: the model.forward() function running time is very high (&gt;1000 ms)</p>
<p>In the second batch: I get the several occurrences of error message:</p>
<p><em>[W C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\jit\codegen\cuda\manager.cpp:336] Warning: FALLBACK path has been taken inside: torch::jit::fuser::cuda::runCudaFusionGroup. This is an indication that codegen Failed for some reason.</em><br>
<em>To debug try disable codegen fallback path via setting the env variable <code>export PYTORCH_NVFUSER_DISABLE=fallback</code></em></p>
<ul>
<li>(function runCudaFusionGroup)*</li>
</ul>
<p>but the detection is good.</p>
<p>up from the third batch the model.forward() runs as accepted: no errors, low running time (~ 10 ms) and good detection.</p>
<p>This strange behavior occurred regardless the batch size, the images sent to the model, or their number.</p>
<p>Any idea?<br>
Thank you in advanced!</p>

ptrblck: <p>Based on the warning you are using an older PyTorch release <code>&lt;=1.13</code> and are using the <code>TorchScript</code> workflow. Update to any recent PyTorch release and use <code>torch.compile</code> instead.</p>

Ella_Mendelson: <p>thank you for your answer!<br>
I upgraded the pytorch version, and now I don’t get the error messages. however, the running time issue is the same:</p>
<p>first batch running time: ~10,000 ms<br>
second batch running time: ~3,000 ms<br>
up from third batch: &lt;10 ms</p>
<p>as for model.compile - is is supported in the cpp?</p>
<p>thanks!</p>

ptrblck: <p>Good to hear the warning disappeared. Initial iterations could be slower due to the optimization of the computation graph.</p>
<p>No, <code>torch.compile</code> is not directly exposed in the C++ API.</p>
Implementing captum with pytorch-lightning

ik362: <p>Hi there,</p>
<p>I also posted this question here: <a href="https://github.com/pytorch/captum/issues/726#issue-950425016" class="inline-onebox" rel="noopener nofollow ugc">Implementing captum with pytorch-lightning · Issue #726 · pytorch/captum · GitHub</a></p>
<p>I am trying to use LayerGradCam in captum to interpret a particular layer in my model.</p>
<p>Part of the problem/complication seems to be that my model and forward method are defined in a pytorch-lightning module.</p>
<p>My pytorch-lightning module is:</p>
<pre><code class="lang-python">class model(pl.LightningModule):
    def __init__(self, learning_rate = float):
        super().__init__()
        self.learning_rate = learning_rate
        self.criterion = nn.BCEWithLogitsLoss()
        self.cam = LayerGradCam(self.forward, 'model.5')
        self.model = nn.Sequential(cnnBlock1(), cnnBlock2(), cnnBlock3(), linearBlock())

    def forward(self, x):
        return self.model(x)

    def train_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        train_loss = self.criterion(y_hat, y)
        self.log('train_loss', train_loss)
        return train_loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        val_loss = self.criterion(y_hat, y)
        self.log('val_loss', val_loss)
        return val_loss

    def test_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        attr = self.cam.attribute(x)
        return attr

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr = self.learning_rate
        return optimizer
</code></pre>
<p>However, when I run the test step I am getting the error:</p>
<pre><code class="lang-python">AttributeError: 'str' object has no attribute 'register_forward_hook'
</code></pre>
<p>I have two questions then:</p>
<ol>
<li>What does this error mean and how do I fix it?</li>
<li>How do I/what is best practice for implementing captum with pytorch-lightning?</li>
</ol>
<p>Thanks for your help!</p>

blackpearl006: <p>Did you solve it ?<br>
Facing similar problem !</p>
Cholesky rank-1 update in PyTorch

szkk22zc: <p>Hi there,</p>
<p>I am trying to write a function for rank-1 update with its corresponding backpropagation in PyTorch as it is not originally supported. However, I find that it is slower than <code>torch.linalg.cholesky</code>. The function I wrote is shown as follows.</p>
<pre><code class="lang-auto">@torch.jit.script
def cholesky_rank1_update_jit(L, v):
    L_new = L.detach().clone()
    n = L.shape[0]
    v = v.clone()  # Clone v to avoid modifying the original vector
    for k in range(n):
        Lkk = L_new[k, k]
        vk = v[k]
        r = torch.sqrt(Lkk**2 + vk**2)
        c = r / Lkk
        s = vk / Lkk
        L_new[k, k] = r
        if k + 1 &lt; n:
            L_ik = L_new[k+1:n, k]
            v_i = v[k+1:n]
            L_new[k+1:n, k] = (L_ik + s * v_i) / c
            v[k+1:n] = c * v_i - s * L_ik
    return L_new
</code></pre>
<p>Can I have any idea to improve the performance of this code?</p>
Is a Dataset copied as part of Dataloader with multiple workers?

escorciav: <p><img src="https://discuss.pytorch.org/images/emoji/apple/question.png?v=9" title=":question:" class="emoji" alt=":question:"> Does the Dataloader copy the Dataset on each worker?</p>
<ul>
<li>
<p>The documentation doesn’t use plain English. It does mention details of multiprocessing that I dunno <img src="https://discuss.pytorch.org/images/emoji/apple/sweat.png?v=9" title=":sweat:" class="emoji" alt=":sweat:"></p>
</li>
<li>
<p>I assume it copies as I have done hacks to avoid unpickable objects e.g. using HDF5</p>
</li>
</ul>
<p>I gotta implement a quick &amp; dirty hack to improve the speed of one dataset.</p>

ptrblck: <p>Yes, each worker would create a copy of the <code>Dataset</code>.</p>

escorciav: <p>Thanks!</p>
<p>Is there any (blog) post discussing how to finding bottlenecks and improving the speed of a Dataset?</p>
<ul>
<li>
<p>If so, please comment.</p>
</li>
<li>
<p>If you would be interested in reading one, <img src="https://discuss.pytorch.org/images/emoji/apple/raised_back_of_hand.png?v=9" title=":raised_back_of_hand:" class="emoji" alt=":raised_back_of_hand:"></p>
</li>
<li>
<p>If you are interested in writing one, react with <img src="https://discuss.pytorch.org/images/emoji/apple/smirk.png?v=9" title=":smirk:" class="emoji" alt=":smirk:"></p>
</li>
</ul>

ptrblck: <p>You could take a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a>, which explains data loading bottlenecks and some potential workarounds.</p>

escorciav: <p>Awesome. Added to my bookmarks!</p>

ado_sar: <p>Is this a deep or shallow copy?</p>
Forward and Backward hook

ShnekaM: <p>Hi,<br>
I am working on visualizing the attention layers of the <code>deit_tiny_patch16_224</code> model. To achieve this, I registered forward and backward hooks on the <code>attn_drop</code> layer using <code>register_forward_hook</code> and <code>register_full_backward_hook</code>. However, when I run the model, the hooks for the <code>attn_drop</code> layer are not being triggered.</p>
<p>Interestingly, if I register the same hooks on a different layer, such as <code>k_norm</code>, they work as expected.</p>
<p>I suspect this issue might be related to the dropout probability being set to 0.0 in the <code>attn_drop</code> layer. From my understanding, the dropout probability shouldn’t affect the execution of hooks, but I might be overlooking something. Could the disabled dropout (p=0.0) be causing the hooks to behave unexpectedly, or is there another reason the hooks aren’t working for this layer?</p>

ptrblck: <p>Disabling <code>dropout</code> layers via <code>p=0.0</code> won’t change the hook logic as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

class MyModel(nn.Module):
    def __init__(self, p):
        super().__init__()
        self.fc1 = nn.Linear(4, 4)
        self.drop = nn.Dropout(p)
        
    def forward(self, x):
        x = self.fc1(x)
        x = self.drop(x)
        return x

x = torch.randn(1, 4)

model = MyModel(p=0.5)
model.drop.register_forward_hook(lambda m, x, o: print("input {}, output {}".format(x, o)))
model.drop.register_full_backward_hook(lambda m, gi, go: print("ginput {}, goutput {}".format(gi, go)))

out = model(x)
# input (tensor([[-0.5069, -0.0430,  0.9238, -0.5910]],
#        grad_fn=&lt;BackwardHookFunctionBackward&gt;),), output tensor([[-1.0138, -0.0859,  1.8477, -1.1820]], grad_fn=&lt;MulBackward0&gt;)

out.mean().backward()
# ginput (tensor([[0.5000, 0.5000, 0.5000, 0.5000]]),), goutput (tensor([[0.2500, 0.2500, 0.2500, 0.2500]]),)


model = MyModel(p=0.0)
model.drop.register_forward_hook(lambda m, x, o: print("input {}, output {}".format(x, o)))
model.drop.register_full_backward_hook(lambda m, gi, go: print("ginput {}, goutput {}".format(gi, go)))

out = model(x)
# input (tensor([[ 0.1606, -0.6628, -0.7668, -0.4123]],
#        grad_fn=&lt;BackwardHookFunctionBackward&gt;),), output tensor([[ 0.1606, -0.6628, -0.7668, -0.4123]],
#        grad_fn=&lt;BackwardHookFunctionBackward&gt;)

out.mean().backward()
# ginput (tensor([[0.2500, 0.2500, 0.2500, 0.2500]]),), goutput (tensor([[0.2500, 0.2500, 0.2500, 0.2500]]),)
</code></pre>
<p>However, I don’t know if your implementation removes these layers to optimize the model.</p>
Summing tensor values with the same index from an index tensor

maartenvds: <p>I have a data tensor and an index tensor:</p>
<pre><code class="lang-auto">d = torch.Tensor([ 0.0300,  0.1000, -0.0500,  0.0600])
idxs = torch.LongTensor([0, 1, 1, 3])
</code></pre>
<p>I want to sum all values in the data tensor where the values in the index tensor are equal. For example 0.1 and -0.05 should be summed because the second and third value of idxs are equal. I know this can be done for a single value, for example index 1:</p>
<pre><code class="lang-auto">torch.sum(d[idxs == 1])
</code></pre>
<p>which will give:</p>
<pre><code class="lang-auto">tensor(0.0500)
</code></pre>
<p>How can I apply this for a range of indices without using a for loop?<br>
With a for loop it could look like this:</p>
<pre><code class="lang-auto">torch.Tensor([torch.sum(d[idxs == i]) for i in range(len(idxs))])
</code></pre>
<p>which will give (expected result):</p>
<pre><code class="lang-auto">tensor([0.0300, 0.0500, 0.0000, 0.0600])
</code></pre>

Gilf641: <p>Hi <a class="mention" href="/u/maartenvds">@maartenvds</a>!. Since you want to do it for a range of indices, I guess for loop is the best option. BTW, what’s your concern on using Loops?</p>

maartenvds: <p>Thanks for the response <a class="mention" href="/u/gilf641">@Gilf641</a> . ‘d’ is in fact a gradient tensor and its calculated inside a backward function of an autograd.Function. Since this function is used many times in my network, I wanted to make sure its as fast as possible (without writing custom C++/cuda code). I guess this is as efficient as it goes.</p>

ucl98: <p>Using <code>scatter_add_</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">src = torch.tensor([0.03, 0.1, -0.05, 0.6])
index = torch.tensor([0, 1, 1, 3])
torch.zeros(index.max() + 1).scatter_add_(0, index, src)
# result: tensor([0.0300, 0.0500, 0.0000, 0.6000])
</code></pre>
<p>Using <code>scatter_</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">src = torch.tensor([0.03, 0.1, -0.05, 0.6])
# 4 entries and highest index is 6 (5+1) -&gt; 6 x 4 tensor nedded
index = torch.tensor([0, 1, 1, 5])

a = torch.zeros(6, 4, dtype=src.dtype).scatter_(0, index[None], src[None])
torch.sum(a, dim=1)
# result: tensor([0.0300, 0.0500, 0.0000, 0.0000, 0.0000, 0.6000])
</code></pre>