akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Turning PYTORCH_JIT mode on/off dynamically

pd1: <p>Hi,<br>
We are planning to use PyTorch JIT in production and our top-level modules use JIT Script annotations. We require the JIT Scripting to turn off during training and we need to switch it back on for model export (with inference). However PYTORCH_JIT is an environment variable that is set statically.</p>
<ol>
<li>How can we achieve this without explicitly using PYTORCH_JIT=0 statically while launching the process?</li>
<li>While exporting the model, we perform a correctness check and need to turn it off/on dynamically for the 2 models (with and without JIT) to be loaded. How can we achieve this?</li>
</ol>
<p>Thanks!</p>

SimonW: <p><a class="mention" href="/u/apaszke">@apaszke</a> told me that PYTORCH_JIT=0 is only meant for debugging use. In your case, I would argue that you should do jit.script before exporting, where you can also compare both versions for correctness.</p>

reuben: <p>I assume you mean using the module normally during training and then call <code>torch.jit.trace</code> at export time to create the JIT version. While this works for fully traceable models, for my use case I have mixed tracing and TorchScript, and I don’t see any way to disable the TorchScript annotations, the JIT is used for script functions/methods even if torch.jit.trace was never called. Is there any other way to achieve this?</p>

Valiox: <p>I have these methods:</p>
<pre><code class="lang-auto">def set_jit_enabled(enabled: bool):
    """ Enables/disables JIT """
    if torch.__version__ &lt; "1.7":
        torch.jit._enabled = enabled
    else:
        if enabled:
            torch.jit._state.enable()
        else:
            torch.jit._state.disable()


def jit_enabled():
    """ Returns whether JIT is enabled """
    if torch.__version__ &lt; "1.7":
        return torch.jit._enabled
    else:
        return torch.jit._state._enabled.enabled
</code></pre>
<p>When training, I call <code>set_jit_enabled(false)</code> before instantiating modules that have JIT annotations. I haven’t found the proper way of doing this, but this hack works just fine.</p>

benzic: <p>I’ve created the decorator <code>jit_script</code> in <a href="https://github.com/BenZickel/torch_arima/blob/master/ARIMA/torch_utils.py" class="inline-onebox" rel="noopener nofollow ugc">torch_arima/ARIMA/torch_utils.py at master · BenZickel/torch_arima · GitHub</a>.</p>
<p>Functions decorated with this decorator will dynamically respect the _jit_enabled flag without needing to reload modules.</p>
What will happen if I set create_graph=False after computing second derivative?

lyy132: <p>I am trying to get first and second output derivatives w.r.t to network inputs. Specifically, the input  size is (Batch_size, 2) and output size is (Batch_size,70) and (Batch_size,200) for out1 and out2 respectively. I want a result that is the same size with network output, and in line <code>i</code> of the desired result each entry "result_<code>ij</code> "should be $\partial(output_<code>ij</code>\x_<code>i</code>)$.<br>
Here is my code:</p>
<pre><code class="lang-auto">    out1, out2 = model(torch.cat((col_x, col_t), dim=1))
    out1_x = torch.zeros_like(out1)
    out1_t = torch.zeros_like(out1)
    for i in range(out1.shape[1]):
        r_g = torch.zeros_like(out1)
        r_g[:, i] = 1
        res = torch.autograd.grad(out1,[col_x,col_t],grad_outputs = [r_g,r_g],retain_graph=True,create_graph=True,\
                                  allow_unused=True)
        out1_x[:,i] = res[0].squeeze()
        out1_t[:,i] = res[1].squeeze()

    # print(torch.cuda.memory_summary(device=None, abbreviated=False))
    out2_x = torch.zeros_like(out2)
    out2_t = torch.zeros_like(out2)
    for i in range(out2.shape[1]):
        r_g = torch.zeros_like(out2)
        r_g[:, i] = 1
        res = torch.autograd.grad(out2,[col_x,col_t],grad_outputs = [r_g,r_g],retain_graph=True,create_graph=True,\
                                  allow_unused=True)
        out2_x[:,i] = res[0].squeeze()
        out2_t[:,i] = res[1].squeeze()
    # print(torch.cuda.memory_summary(device=None, abbreviated=False))
    out1_xx = torch.zeros_like(out1)
    out2_xx = torch.zeros_like(out2)
    for i in range(out1.shape[1]):
        r_g = torch.zeros_like(out1)
        r_g[:, i] = 1
        partial_out1_xx = \
        torch.autograd.grad(out1_x, col_x, grad_outputs=r_g, retain_graph=True, create_graph=False, allow_unused=True)[0]
        out1_xx[:, i] = partial_out1_xx.squeeze()
    # print(torch.cuda.memory_summary(device=None, abbreviated=False))
    for i in range(out2.shape[1]):
        r_g = torch.zeros_like(out2)
        r_g[:, i] = 1
        partial_out2_xx = \
        torch.autograd.grad(out2_x, col_x, grad_outputs=r_g, retain_graph=True, create_graph=False, allow_unused=True)[0]
        out2_xx[:, i] = partial_out2_xx.squeeze()
</code></pre>
<p>In this code I set create_graph=False when I’m computing second derivative. Am I doing right? Or is there any quicker way to get results I want, without traversing columns?</p>

soulitzer: <p>Yes, there should be no need to create_graph=True when you already calling .grad( to compute the second derivative.</p>
How to continue BP using intermediate gradients

randomuserT: <p>Say I have an original model that I split into two chunks (at a so-called cut-layer); a client-side and server-side. When I perform backpropagation on the server-side, I need to continue the backward pass on the client-side as well to finalize a single backward pass over the entire model.</p>
<pre><code class="lang-auto">class ClientModel(nn.Module):

    def __init__(self):
        super(ClientModel, self).__init__()

        num_flat_features = 3 * 224 * 224

        self.layers = nn.Sequential(
            nn.Flatten(),
            nn.Linear(num_flat_features, 128),
            nn.ReLU()
        )

    def forward(self, x):
        x = self.layers(x)

        return x

class ServerModel(nn.Module):

    def __init__(self):
        super(ServerModel, self).__init__()

        self.cut_layer = nn.Linear(128, 64)
        self.relu = nn.ReLU()
        self.cls_layer = nn.Linear(64, 10)

    def forward(self, x):
        x = self.cut_layer(x)
        x = self.relu(x)
        x = self.cls_layer(x)

        return x
</code></pre>
<p>If after BP on the server-side I obtain the cut-layer gradients via <code>cut_layer.weight.grad</code>, <strong>how do I then continue the backpropagation on the client-side using these cut-layer gradients?</strong></p>
<p>Any help is much appreciated.</p>

soulitzer: <p>Doing two backward calls back to back and passing the intermediate gradient is equivalent to doing a single backward <code>torch.autograd.backward(intermediate_output, grad_tensors=(cut_layer.weight.grad,))</code></p>

randomuserT: <p>Thanks for your reply!</p>
<p>When using what you mentioned, I receive the following error;<br>
<code>RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([64, 128]) and output[0] has a shape of torch.Size([512, 128]).</code></p>
<p>My batch size is 512. So it looks like the gradients of the cut-layer are not of the same dimension as the activations when I do this. Which I suppose makes sense but then how would I go about this?</p>

soulitzer: <p>Ah, you should use retain_grad to ensure that .grad is recorded for the activation as well, and make sure you use the .grad wrt the activation.</p>

randomuserT: <p>I’m indeed already using activations.retain_grad()</p>
<p>How do I explicitly use the .grad wrt the activations? Should I extract those cut-layer gradients in a different manner than just <code>server_model.cut_layer.weight.grad</code>?</p>

soulitzer: <p>If you have a reference to the activation Tensor somewhere, you can access its .grad attribute. If not, there’s a little more work you would need to do, but still possible.</p>
Understanding DataLoader performance

b.buchhold: <p>I have a pretty basic DataSet and DataLoader and a relatively tiny toy model. Everything works as expected.</p>
<p>First of all, training seems to be dominated by data loading. Given the small model I use, I think that is somewhat plausible. Further, I noticed that having the dataset on the GPU (it fits without problems) is faster than having to on CPU and using pin_memory with multiple workers.</p>
<p>However, I noticed something I don’t quite understand:<br>
Increasing the size of X_train slows down the time per microbatch significantly, e.g., going from 1m items in train to 10m means I need about 3 times as long per batch (thus 30times as long per epoch where I would have expected only a factor of 10).</p>
<p>I would expect some overhead, because of shuffle taking longer and because more cache misses are to be expected. But such a big factor surprises me.</p>
<p>Is there an explanation? What would be a good way for me to approach and debug/profile this? Is this maybe indicative of something I am doing wrong?</p>

b.buchhold: <p>I figured this out myself. My targets were stored in a SparseTensor and the recommendations from here: <a href="https://discuss.pytorch.org/t/dataloader-loads-data-very-slow-on-sparse-tensor/117391/4" class="inline-onebox">Dataloader loads data very slow on sparse tensor - #4 by drj3122</a> helped me resolve the problem.</p>
<p>Essentially grabbing a batch from the SparseTensor scaled with the size of that Tensor and thus more training data mean slower micro batches.</p>
Inquiry Regarding PyTorch Data Mirroring and Proxy Services

gante_danca: <p>Dear PyTorch Team,</p>
<p>I trust this message finds you well.</p>
<p>We’d like to set a pytorch public mirror here in china aims to enhancing the accessibility and speed of technical resources for the developer community. We are considering offering a mirror and proxy service for the PyTorch datasets hosted at <a href="https://download.pytorch.org/whl/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/</a> to improve network performance for users.</p>
<p>We understand the importance of quick and reliable access to PyTorch’s extensive machine learning resources, and we believe that by providing a mirror and proxy service, we can significantly reduce latency and offer a more robust experience for users in our region.</p>
<p>Here are the key points we would like to discuss with you:</p>
<ol>
<li>How can I get the permission to mirror the content found at the aforementioned URL.</li>
<li>Any technical requirements or guidelines you may have for parties mirroring PyTorch data.Such as advice of how to sync the whole repository( for example rsync?).</li>
<li>How can we stay updated of PyTorch informations and updates.</li>
</ol>
<p>Our goal is to provide this service free of charge to the opensource users. We are more than willing to adhere to any terms of service or usage policies that you may have in place.</p>
<p>We are looking forward to the possibility of collaborating with the PyTorch team to ensure that our services are in line with your goals and expectations.</p>
<p>Please let us know if this is something that is feasible and what steps we need to take to move forward. Your guidance and support would be greatly appreciated.</p>
<p>Thank you for considering our request. We eagerly await your response.</p>
<p>Best regards,</p>
<p>motor</p>

ptrblck: <p>While this sounds like a great idea, I would generally be concerned about security guarantees and how the verification could work making sure packages were not manipulated.<br>
CC <a class="mention" href="/u/malfet">@malfet</a></p>

gante_danca: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> , the subsequent discussion will be moved to <a href="https://github.com/pytorch/pytorch/issues/128940" class="inline-onebox" rel="noopener nofollow ugc">Inquiry Regarding PyTorch Data Mirroring and Proxy Services · Issue #128940 · pytorch/pytorch · GitHub</a>. This topic is closed.</p>
CUDA driver initialization failed you might not have a CUDA gpu

belle213225: <p>I have Redhat (9) VM with A100 GPU.<br>
Nvidia-smi 550.54.15<br>
Cuda 12.4<br>
python 3.9</p>
<p>but when I tried check my pytorch detect gpu it’s shown<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04e80d6242d6d53b141beebfcd3ff873848d8bae.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/04e80d6242d6d53b141beebfcd3ff873848d8bae" title="S__13811725"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04e80d6242d6d53b141beebfcd3ff873848d8bae_2_690x44.jpeg" alt="S__13811725" data-base62-sha1="Hp4VYV4H0MEutt2yZGNWm2YiwC" width="690" height="44" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04e80d6242d6d53b141beebfcd3ff873848d8bae_2_690x44.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04e80d6242d6d53b141beebfcd3ff873848d8bae_2_1035x66.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04e80d6242d6d53b141beebfcd3ff873848d8bae_2_1380x88.jpeg 2x" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">S__13811725</span><span class="informations">1562×100 48.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
nvcc:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0cc77683cc5a555493bb2afdfebd508756c0733.jpeg" alt="S__13811728" data-base62-sha1="tN7nDd84IgtfqB9TVdQFPu0Ozbd" width="486" height="128"><br>
nvidia-smi:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fca6596bbac1dde77ab21ef940842b7ffbd6299.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8fca6596bbac1dde77ab21ef940842b7ffbd6299" title="S__13811727"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fca6596bbac1dde77ab21ef940842b7ffbd6299_2_690x289.jpeg" alt="S__13811727" data-base62-sha1="kw1UvKsN6WAx99tT4IAnM7HqGil" width="690" height="289" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fca6596bbac1dde77ab21ef940842b7ffbd6299_2_690x289.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fca6596bbac1dde77ab21ef940842b7ffbd6299.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fca6596bbac1dde77ab21ef940842b7ffbd6299.jpeg 2x" data-dominant-color="2A292A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">S__13811727</span><span class="informations">942×395 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Aniruth_Sundararaja1: <p>Check once if your cuda paths are correctly set . if still an issue is there ,try running</p>
<p>sudo nvidia-smi -i 0 -e 1</p>
Elementwise multiplication on an image inside neural network

KVV: <p>I am trying to train a neural network for which the inputs and outputs are both 2D numerical arrays. I want the last layer to be an ‘element-wise multiplying layer’, i.e, If the input for the layer is A<sub>ij</sub>, then I want the output should be W<sub>ij</sub>*A<sub>ij</sub> + B<sub>ij</sub>, where W<sub>ij</sub> and B<sub>ij</sub> are the weight and bias tensors of the last layer.</p>
<p>I wrote down how I am trying to do this below. Seems inefficient and the loss is not going down with each iteration. Is there a clever/efficient way of doing this better ?<br>
At an even more elementary level, will autograd update the gradients dL/dW<sub>ij</sub> and dL/dB<sub>ij</sub>. I think it should. Am I missing something ?</p>
<p>def <strong>init</strong>(self):<br>
self. convs = some convolutions<br>
<strong>self.LastLayerWeights = nn.Conv2d(1,1, Arows, Acols)</strong><br>
<strong>self.LastLayerBias         = nn.Conv2d(1,1, Arows, Acols)</strong></p>
<p>def forward(self,x):<br>
x = Some activations after self.convs<br>
<strong>x = self.LastLayerWeights.weight * x + self.LastLayerBias.bias</strong><br>
return x</p>

Aniruth_Sundararaja1: <p>try using nn.Parameter  : <a href="https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html" class="inline-onebox" rel="noopener nofollow ugc">Parameter — PyTorch 2.3 documentation</a></p>
<p>your code would be like</p>
<pre><code>    self.LastLayerWeights = nn.Parameter(torch.randn(1, 1, Arows, Acols))
    self.LastLayerBias = nn.Parameter(torch.randn(1, 1, Arows, Acols))

    x = self.LastLayerWeights * x + self.LastLayerBias
</code></pre>

KVV: <p>Works much better. Thanks</p>
[FSDP] difference between _composable/fsdp or fsdp

vocaltract: <p>I used torch2.3 and found there is <code>torch.distributed.fsdp</code> and <code>torch.distributed._composable.fsdp</code>.</p>
<p>Because I want to understand source codes of fsdp and try to make some contributions, I think fsdp implementations under these two directories are quite different.</p>
<p>Why there are two directories for fsdp and what is the core reason for this difference?</p>
<p>English is not my native language, sorry for my poor expression.</p>

ptrblck: <p>I believe <code>_composable.fsdp</code> refers to the new <code>FSDP2</code> implementation, but <a class="mention" href="/u/bdhirsh">@bdhirsh</a> or others can correct me.</p>

marksaroufim: <p>Indeed composable FSDP is being referred to as FSDP2. There are some notes that highlight the differences here <a href="https://github.com/pytorch/torchtitan/blob/main/docs/fsdp.md" class="inline-onebox">torchtitan/docs/fsdp.md at main · pytorch/torchtitan · GitHub</a></p>
<p>Personally I found FSDP2 exciting because it works when you have heterogeneous data types <a href="https://x.com/HamelHusain/status/1800315287574847701" class="inline-onebox">x.com</a></p>
What is the status of `functorch`?

StrausMG: <p>The <a href="https://pytorch.org/functorch/stable/" rel="noopener nofollow ugc">official docs</a> say that <code>functorch</code> is deprecated and <code>torch.func</code> should be used instead.</p>
<p>However, <code>functorch</code> has recently received new features like <code>functorch.dim</code> and <code>functorch.einops</code> (as of PyTorch 2.1) that are not a part of <code>torch.func</code>. So I am a bit confused: what is the current relation between <code>functorch</code> and <code>torch.func</code>? If I find features in <code>functorch</code> useful, can I use them or I should wait for them being merged to <code>torch.func</code> ?</p>

Zacharias: <p>This seems like a relevant question! cc <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I don’t see any recent commits to the <code>main</code> branch of <code>pytorch/functorch</code> as seen <a href="https://github.com/pytorch/functorch/commits/main/">here</a> so would stick to the docs and use <code>torch.func</code> in recent PyTorch versions.</p>
FlashAttention Pytorch integration

CDhere: <p>Hi, I’m trying to experiment and make tweaks and potential upgrades to FlashAttention, and wondering where’s the best place to start. Does the Pytorch integration copy-paste/pull from the original FlashAttention repo, or there are implementation changes made along with the integration?</p>
<p>Thanks!</p>

soulitzer: <p>This doesn’t answer your question, but relatedly for anyone interested in making changes to attention but still want to have fast kernels, there’s <a href="https://github.com/pytorch/pytorch/blob/93a33bf3ac0b4c9560b49780eabcad2f76dcf43e/torch/nn/attention/_flex_attention.py#L42" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/nn/attention/_flex_attention.py at 93a33bf3ac0b4c9560b49780eabcad2f76dcf43e · pytorch/pytorch · GitHub</a></p>
DDP implementation in LLM

LeonBinder: <p>Hey Guys,<br>
how can I use DDP to make my two GPUs work together and parallel on answering questions with an LLM like Mistral-7B?</p>

soulitzer: <p>This may be relevant <a href="https://discuss.pytorch.org/t/how-to-inference-under-ddp/103042" class="inline-onebox">How to inference under DDP</a></p>

LeonBinder: <p>I am a little new in this topic but what I want is that I run my model but i dont want the model to be split up between my gpus but to ,work together" parallel to create the answer faster. does this make sense? what is the correct way to achieve this?</p>

LeonBinder: <p>This is my code to simply run the mistral-7B<br>
But how can I make my 2 GPUs (Tesla M40 24GB) act like one large gpu so the output gets faster?</p>
<pre><code class="lang-auto">import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

local_model_path = "/media/data/AI/models/mistral-7B"
# Load the tokenizer and model from the local folder
tokenizer = AutoTokenizer.from_pretrained(local_model_path)
model = AutoModelForCausalLM.from_pretrained(local_model_path, torch_dtype=torch.float16)
# Ensure the model is in evaluation mode
model.eval()

class TransformersLLM:
    def __init__(self, model, tokenizer, device):
        self.model = model.to(device)  # Move model to the specified device
        self.tokenizer = tokenizer
        self.device = device

    def invoke(self, prompt):
        # Tokenize the input prompt and move tensors to the appropriate device
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.device)
        
        # Generate the output using the model
        with torch.no_grad():  # Disable gradient calculations to save memory
            outputs = self.model.generate(input_ids=inputs['input_ids'].to(self.device), max_length=4096)
        
        # Decode the generated output to a string
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

# Initialize the custom LLM class with accelerate
llm = TransformersLLM(model, tokenizer, "cuda")

print(llm.invoke("Who was Marie Curie?"))

</code></pre>

soulitzer: <p>Yes, DDP replicates rather than shards your model. The data is sharded, so that forward/backward can be parallelized.</p>
ONNX export gives different results than python

Tadgh_Kelly: <p>I’m using the <a href="https://github.com/crowsonkb/k-diffusion/tree/21d12c91ad4550e8fcf3308ff9fe7116b3f19a08" rel="noopener nofollow ugc">k-diffusion</a> transformer model, and I’ve found that the ONNX export gives significantly different results to the torch model. I had some warnings about to begin with but now I’ve fixed those. The export seems to work correctly and the <code>onnx.check_model</code> doesn’t raise an error.</p>
<p>Here a part of the pytorch output (right) and the ONNX output (left):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/311ddd271431a27d542b47205e5051e72d8aa8c6.jpeg" alt="image" data-base62-sha1="70vnvjC7A3yujrf4qVUPDCjmC90" width="600" height="300"></p>
<p>Clearly there is some similarity, but there is a larger difference than I would have expected. Both TensorRT and ONNXRuntime are giving the same result, which differs from PyTorch.</p>
Issue with Per-Sample Gradients tutorial

sandy_floren: <p>I’m interested in computing gradients of my model outputs w.r.t. its inputs more efficiently, and I came across this <a href="https://pytorch.org/tutorials/intermediate/per_sample_grads.html" rel="noopener nofollow ugc">tutorial</a>. Unfortunately, when testing out the code <strong>exactly as it is written in the tutorial</strong>, I get an <code>AssertionError</code> when checking that the gradients computed with function transforms are close to the naive computation. This is the code that fails:</p>
<pre><code class="lang-auto">for per_sample_grad, ft_per_sample_grad in zip(per_sample_grads, ft_per_sample_grads.values()):
    assert torch.allclose(per_sample_grad, ft_per_sample_grad, atol=3e-3, rtol=1e-5)
</code></pre>
<p>It seems like an issue to me that code in an official PyTorch tutorial would fail like this.</p>
<p>I checked manually and I get differences between the two gradients as large as 0.0120.</p>
<p>I’m using Python 3.11.6 and CUDA 12.3 on an NVIDIA A30 GPU.</p>

soulitzer: <p>I couldn’t reproduce this on the main branch. What version of PyTorch are you using?</p>

sandy_floren: <pre><code class="lang-auto">Name                      Version            Build
pytorch                    2.2.2         py3.11_cuda12.1_cudnn8.9.2_0  
</code></pre>

soulitzer: <p>Are you able to update to a later version of PyTorch? It also does not fail on Colab (2.3.0)</p>

sandy_floren: <p>Still getting the error with PyTorch 2.3.0.</p>

soulitzer: <p>Hmm maybe an issue specific to your GPU version. cc <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>I would disable TF32 if FP32 thresholds are used as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices">here</a>.</p>

sandy_floren: <p>Thanks, this ended up being the issue.</p>
RuntimeError: get_parameter is not supported on ScriptModules

simosechi: <p>Hi everyone as the title says, I’m wondering if get_parameter is no more supported for ScriptModules.<br>
Here a trivial example:</p>
<pre><code class="lang-auto">import torch

class InputContainer(torch.nn.Module):
    def __init__(self, obj):
        super().__init__()
        for key, value in obj.items():
            setattr(self, key, value)

inputs = {
        "audio" : torch.rand((553172,))
        }
inputs_model = InputContainer(inputs)
inputs_script = torch.jit.script(inputs_model)
inputs_script.save("rmvpe_inputs.pt")
inputs_script.get_parameter("audio")
</code></pre>
<pre><code class="lang-auto">File "c:\Users\simos\miniconda3\envs\rcv\lib\site-packages\torch\jit\_script.py", line 981, in fail
    raise RuntimeError(name + " is not supported on ScriptModules")
RuntimeError: get_parameter is not supported on ScriptModules
</code></pre>
<p>what does it mean? are not docs up to date?</p>
<p>docs ref:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.jit.ScriptModule.html#torch.jit.ScriptModule.get_parameter" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.jit.ScriptModule.html#torch.jit.ScriptModule.get_parameter</a></p>
<p>PyTorch version: 2.2.2+cpu<br>
Python version: 3.10.14</p>

ptrblck: <p><code>TorchScript</code> is deprecated in favor of <code>torch.compile</code>, so you should consider switching to the latter approach for your workloads.<br>
With that being said, directly accessing the attribute via <code>inputs_script.audio</code> seems to work.</p>
Gradient of Threshold function

Lebourdais: <p>The Threshold activation function doesn’t seem to be differentiable (Gradient should be 0 everywhere except on the jump where it’s non differentiable).</p>
<p>How is handled the backpropagation with this type of activation function, is it ignored ?</p>

KFrank: <p>Hi Lebourdias!</p>
<aside class="quote no-group" data-username="Lebourdais" data-post="1" data-topic="205194" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lebourdais/48/67644_2.png" class="avatar"> Lebourdais:</div>
<blockquote>
<p>The Threshold activation function doesn’t seem to be differentiable</p>
</blockquote>
</aside>
<p>Pytorch’s <a href="https://pytorch.org/docs/2.3/generated/torch.nn.Threshold.html" rel="noopener nofollow ugc">Threshold</a> is (usefully) differentiable.  When its input, <code>x</code>, is<br>
greater than the threshold value, its output is <code>x</code> and the gradient is <code>1.0</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>(Gradient should be 0 everywhere except on the jump where it’s non differentiable).</p>
</blockquote>
</aside>
<p>This description describes a <em>step function</em> (the Heaviside function),<br>
rather than pytorch’s <code>Threshold</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>How is handled the backpropagation with this type of activation function, is it ignored ?</p>
</blockquote>
</aside>
<p>For a step function (whose gradient you describe), you can use<br>
pytorch’s <a href="https://pytorch.org/docs/2.3/generated/torch.heaviside.html" rel="noopener nofollow ugc">heaviside() function.</a>  The output of <code>heaviside()</code> carries<br>
<code>grad_fn = &lt;NotImplemented&gt;</code> (if either of its inputs carry<br>
<code>requires_grad = True</code>).  This is pytorch’s way of telling you that<br>
this function isn’t usefully differentiable.</p>
<p>You can also code a step function “by hand:”</p>
<p><code>y = (x &gt; 0.0).float()</code></p>
<p>In this case, the result will not carry <code>requires_grad = True</code>, even if<br>
the input does.</p>
<p>Your main point – that the step function – isn’t <em>usefully</em> differentiable<br>
is quite correct.  Even though the gradient of zero (almost everywhere)<br>
is perfectly sensible and mathematically correct, it’s not useful for<br>
gradient-descent-based optimization because a zero gradient doesn’t<br>
tell you by how much nor in what direction you should adjust your model<br>
weights in order to reduce your loss criterion.</p>
<p>As to why pytorch chooses to have the gradient of <code>heaviside()</code> be<br>
<code>NotImplemented</code> (rather than have <code>requires_grad = False</code>), I don’t<br>
know.</p>
<p>(Depending on your use case, you can use <code>sigmoid()</code> as a “soft” step<br>
function that is usefully differentiable.)</p>
<p>Best.</p>
<p>K. Frank</p>
Two Loss Functions - Negative and Positive

NudnikShpilkis: <p>I have two loss functions that I’m trying to optimize for. One is NLL so the output is almost always negative. The other is L1 Loss so the loss is always positive. They also have different magnitudes. What’s the best way to combine them into one loss? I can add weights like I’ve seen suggested, but that wouldn’t change the sign of the NLL.</p>

ptrblck: <aside class="quote no-group" data-username="NudnikShpilkis" data-post="1" data-topic="205075">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nudnikshpilkis/48/70024_2.png" class="avatar"> NudnikShpilkis:</div>
<blockquote>
<p>One is NLL so the output is almost always negative.</p>
</blockquote>
</aside>
<p>The loss of <code>nn.NLLLoss</code> is bound at <code>[0, Inf]</code>, so positive. Do you mean the log probability output of your model is negative?</p>

NudnikShpilkis: <p>I’m using a custom distribution and loss function, I just meant using negative log-likelihood. My apologies. Is it best to bound my custom nll to [0, Inf]? I can share my code if it helps.</p>

ptrblck: <p>Yes, a “standard” loss function definition would return a zero loss for a perfect prediction and a positive value otherwise.</p>

NudnikShpilkis: <p>Gotcha! Thank you. Assuming I have a custom distribution/loss function what should I do to ensure always returns values between [0, Inf]?</p>
<p>For reference here’s the likelihood equation. r, alpha, a, b are outputs of my model and x, tx and T are data.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f743c54036b158d2a60e6c15ce8b432289e71b5.png" alt="Screenshot 2024-06-24 at 10.36.59 AM" data-base62-sha1="kt3jlpgqCzpMjHZ4VT3TxNggeKV" width="556" height="96"></p>
Can we use pin_memory and pin_memory_device of DataLoader with mps?

mattlbeck: <p>I tried enabling memory pinning in a DataLoader via <code>DataLoader(..., pin_memory=True, pin_memory_device="mps")</code>, but the device is seemingly ignored and it defaults to CUDA.</p>

ptrblck: <p>Based on <a href="https://github.com/pytorch/pytorch/blob/e53d9590287cbf97521f96d055910394f6e9a849/torch/utils/data/dataloader.py#L1047-L1053">this code</a> it seems <code>mps</code> is not a valid <code>pin_memory_device</code> argument.</p>
Num_workers and determinism

Omroth: <p>I’ve noticed that despite comprehensively seeding before starting training:</p>
<pre><code class="lang-auto">seed = 7
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
</code></pre>
<p>My training data is not loaded in the same order.</p>
<p>But it occurs to me that this is “expected” with num_workers &gt; 1, because of race conditions for machine resources between the worker threads.  Is that correct?  Does it cause anyone issues?</p>

soulitzer: <p>You need to pass a generator to DataLoader. From <a href="https://pytorch.org/docs/stable/notes/randomness.html:" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/randomness.html:</a></p>
<pre><code class="lang-auto">def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    numpy.random.seed(worker_seed)
    random.seed(worker_seed)

g = torch.Generator()
g.manual_seed(0)

DataLoader(
    train_dataset,
    batch_size=batch_size,
    num_workers=num_workers,
    worker_init_fn=seed_worker,
    generator=g,
)
</code></pre>
How does optimizer.step() work over a batch and multiple output neurons?

Delve: <p>Hi, beginner with pytorch here, not understanding exactly how backpropagation with loss.backward() and optimizer.step() is working with my code. Can someone help me clarify this? My question is as follows in 2 parts:</p>
<ol>
<li>I have a neural network that I’m training that 3 output neurons. So, when I feed an <em>input</em> into my model, I get a <em>predicted</em> with size 3x1. I then compare that to my <em>actual</em> of size 3x1 to get a loss:</li>
</ol>
<p><em>loss = torch.nn.functional.mse_loss(predicted, actual)</em><br>
<em>self.optimizer.zero_grad()</em><br>
<em>loss.backward()</em><br>
<em>self.optimizer.step()</em></p>
<p><em>loss</em> here is a single number (1x1). Is loss supposed to be a single number here, even with the output layer size being 3? My understanding is that since loss is an mse over the 3 <em>actual - predicted</em> differences (aka summing over them), it contains the information needed to calculate gradients &amp; backpropagate through all 3 output neurons. Please correct me on anything I’m wrong about here.</p>
<ol start="2">
<li>In reality I’m computing batches of 32 at a time, such that my <em>predicted</em> and <em>actual</em> are 32x3 in size. Using the above lines, <em>loss</em> is still a 1x1, with which I update my model at the end of the batch. Is this the correct way to backpropagate when doing mini-batch training like I’m doing? My <em>loss</em> here is a single number that averages over losses in the 3 layers and across the 32 samples in the batch. Or should I be getting 1 loss per sample instead?</li>
</ol>
<p>Sorry if this question doesn’t make a whole lot of sense. Currently very confused <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

soulitzer: <p>This sounds fine to me. The idea is that when you backpropagate through the loss function, the output elements aka inputs to the loss will receive the corresponding gradient proportional to the derivative of their contribution.</p>
Pytorch crashes and tells me to report a bug when running a transformer model

hantan: <p>I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: handle_0 INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":15, please report a bug to PyTorch. 
</code></pre>
<p>I get this error when I try to run a transformer model I implemented following this tutorial: <a href="https://pi-tau.github.io/posts/transformer/#decoder-block" class="inline-onebox" rel="noopener nofollow ugc">An even more annotated Transformer :: pi-tau</a></p>
<p>The error only occurs when I call LogSoftmax or Softmax over the final outputs, not if I skip softmax altogether.</p>
<p>Package versions:<br>
Torch==2.1.0<br>
Cuda==12.1</p>
<p>GPU = RTX 3060ti</p>
<p>When I lower the dimensionality of the model the error does not appear. I suppose this has got to do with that my GPU runs out of memory then?</p>
<p>Does this mean that Softmax and LogSoftmax are very computationally expensive as they seem to be what finally causes the error?</p>
<p>Ps. This is my first post here, let me know if I shold change/add anything. Thanks!</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/hantan">@hantan</a>,</p>
<p>Can you add a minimal reproducible example code that reproduces the error?</p>

HusharBabu: <p>I got the same error, surprisingly I decreased the batch size and the error was resolved.</p>
Convert pyTorch to CoreML

Kamal_T: <p>I converted this sample code to CoreML using coremltools 7 and the output is showing as MultiArray (Float16 1 × 256 × 128 × 128). I am after a MultiArray of 512 x 512 similar to the input Image (Color 512 × 512).</p>
<p>Do I need to add layers to do this ?</p>
<p><a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/neural_style_tutorial.html</a></p>
<p>The model definition shows as:</p>
<p>model=Sequential(<br>
(0): Normalization()<br>
(conv_1): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(style_loss_1): StyleLoss()<br>
(relu_1): ReLU()<br>
(conv_2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(style_loss_2): StyleLoss()<br>
(relu_2): ReLU()<br>
(pool_2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(conv_3): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(style_loss_3): StyleLoss()<br>
(relu_3): ReLU()<br>
(conv_4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(content_loss_4): ContentLoss()<br>
(style_loss_4): StyleLoss()<br>
(relu_4): ReLU()<br>
(pool_4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
<strong>(conv_5): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))</strong><br>
(style_loss_5): StyleLoss()<br>
)</p>
UNET won't even overfit on 8 examples from BraTS dataset

mopi47: <p>I have been trying to train some image segmentation models on the BraTS23 dataset, unfortunately I couldn’t get good results, so I started to doubt my code or the way I calculate the metrics.<br>
then I decided to train a UNET on the 8 first examples (which translates to about 1000 4channels 2d inputs) from this dataset, my goal is to overfit this model, so at least I can say the model can learn, but even with this it won’t overfit, I used the mean IOU as metric and the best I can get is 33 % after 100 epoch.</p>
<p>I have prepared a colab notebook if you would like to take a look at my code:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1zFReBRG6pE8OHZW1mLgSCOTR_pRVxtcO?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/f5cb66a76ad313db68a3becdc32da4e1/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1zFReBRG6pE8OHZW1mLgSCOTR_pRVxtcO?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1zFReBRG6pE8OHZW1mLgSCOTR_pRVxtcO?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colab</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Passing a dictionary to getitem

yonatansmn: <p>Hi,<br>
How do I get my <strong>getitem</strong> function to accept a dictionary, e.g.:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset
class DatasetCustomGetitem(Dataset):
    ....

def __getitem__(self, data: dict)
    ....
</code></pre>
<p>The Dataset is wrapped with torch DataLoader.<br>
I need it only for inference.</p>
<p>Thanks</p>

Omroth: <p>Need more information on your use-case.  How is this being integrated into your training run?</p>

yonatansmn: <p>I edited my question</p>

Omroth: <p>It would help me if I could see how you propose using it in your training loop.  The Dataset class is meant to give everything you need for one training instance.  It doesn’t make sense to me, without more information, why you would want to pass something else to it (that is different for each item, otherwise you would make it a member of your Dataset and pass it in the constructor.) But I expect you do have a good reason, but without knowing it I can’t work out what the best solution is.</p>
<p>Generally, you are not going to be able to easily pass a custom object to Dataset::getItem as it is used within the machinery of the DataLoader class.</p>

yonatansmn: <p>I need to implement it as part of a complex pipeline I have no control on…<br>
When training, at initialisation the Dataset receives a list of folders. <strong>getitem</strong> works “as usual”: receiving an index, depicting the relevant folders, reading files (image, csvs…), does some data processing and outputs it.<br>
At inference, I cannot do any reading; Instead I want to directly get the data (in the form of a dictionary, which always have the same keys, with their values’ being a numpy array.).</p>

Omroth: <p>I’m not particularly sure I can help, but it is possible this will help you:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset
class DatasetCustomGetitem(Dataset):
    def __init__(self, data_dict):
       self.data_dict = data_dict

    def __getitem__(self, idx)
       return self.data_dict[idx]

    def __len__(self):
       return len(self.data_dict.keys())
</code></pre>

yonatansmn: <p>Thanks for the quick response. I thought of this solution, but this will force me to store a lot of data in memory. I wonder if there’s a more elegant solution, e.g. writing a custom data fetcher.</p>

Omroth: <p>Then just store all the elements onto disk, and have <strong>getItem</strong> load the nth one.</p>

Omroth: <p>The Dataset class was designed to solve these problems, I think.</p>

yonatansmn: <p>I’m not allowed to do any reading.<br>
In your first suggestion, would erasing the dict key from self.data_dict can create some error?<br>
For example:</p>
<pre><code class="lang-auto">    def __getitem__(self, idx)
       data= self.data_dict[idx]
       # do stuff on the data
      delete self.data_dict[idx]
      return data
</code></pre>
Pictures of fruits and vegatables

Rita_Morgan: <p>I have bunch of pictures in a folder. My garden of various fruits and vegetables. There are close to 1k pictures I accumulated over the years. I would like to use Pytorch to identify 1) if its a fruit or vegatable 2) the vegatable or fruit.</p>
<p>How can I achieve this?</p>

Aniruth_Sundararaja1: <p>Refer image classification docs</p>
<p><a href="https://medium.com/bitgrit-data-science-publication/building-an-image-classification-model-with-pytorch-from-scratch-f10452073212" class="onebox" target="_blank" rel="noopener nofollow ugc">https://medium.com/bitgrit-data-science-publication/building-an-image-classification-model-with-pytorch-from-scratch-f10452073212</a></p>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></p>
Why does LSTM accept incorect input size

Kai_Hammond: <pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, INPUT_SIZE, HIDDEN_SIZE):
        super().__init__()
        self.lstm = nn.LSTM(INPUT_SIZE, HIDDEN_SIZE, batch_first=True)

    def forward(self, input_data):
        lstm_out, _ = self.lstm(input_data)
        return lstm_out

model = LSTM(4, 1)

input = torch.tensor([[[2, 5, 3, 2, 2, 5], [2, 4, 2, 2, 2, 5]],
                         [[5, 2, 3, 2, 2, 5], [10, 2, 7, 2, 2, 5]]], dtype=torch.float32)
input.shape
# torch.Size([2, 2, 6])

model(input)

#tensor([[[-0.1295],
#         [-0.1073]],
#
#        [[ 0.1431],
#         [ 0.2922]]], grad_fn=&lt;TransposeBackward0&gt;)

</code></pre>
<p>I believe this shouldn’t work because I set the input size as 4, but am inputting inputs of size 6. Why is this not throwing an error? Am I understanding the calculation of the LSTM incorrectly?</p>

Aniruth_Sundararaja1: <p>Hi <a class="mention" href="/u/kai_hammond">@Kai_Hammond</a></p>
<p>Your code works because PyTorch allows variable sequence lengths by default when batch_first=True in the nn.LSTM module .</p>
<p>So LSTM by default processes the incoming input to handle variable size input.</p>
<p>For shorter sequences, it  ignores the remaining time steps.<br>
For longer sequences, it  truncates the excess time steps.</p>
What kind of loss to use with multi-hot encoded multiclass ordinal features?

layman: <p>I have some ordinal classes with a range of 1-5 each and some implicit relationships, and decided to multi-hot encode those such that each level gets an activation like</p>
<pre><code class="lang-auto">{
1: [0, 0, 0, 0]
2: [1, 0, 0, 0]
...
5: [1, 1, 1, 1]
}
</code></pre>
<p>And then concatenate for each class so the resulting vector might look like<br>
<code>[0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1]</code></p>
<p>What is a good loss function to use here? Is <code>MultiLabelMarginLoss</code> sufficient? Not sure if I can get the proper penalizing effect from a one-vs-all approach given multiple bits belong to the same class each.</p>

Aniruth_Sundararaja1: <p>BCE - Binary Cross entropy loss would be better than MultiLabelMarginLoss</p>
A strange warning 'Unresolved attribute reference 'to' for class 'int' '

Leonard0: <p>I am learning how to build a LeNet5 CNN using pytorch. There is warning says <strong>'Unresolved attribute reference ‘to’ for class ‘int’</strong> ’ I dont know how to deal with it. Here is the code with warning at  <code>images = images.to(device) </code></p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = images.to(device)  
        labels = labels.to(device)

        outputs = model(images) 
        loss = cost(outputs, labels)  

        optimizer.zero_grad()  
        loss.backward()  
        optimizer.step()  

        if (i + 1) % 400 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                  .format(epoch + 1, num_epochs, i + 1, total_step, loss.item()))
</code></pre>

Leonard0: <p>/(ㄒoㄒ)/~~ plz help me.</p>

vocaltract: <p>maybe <code>type(labels) == int</code>  or <code>type(images) == int</code>？</p>
Android build and vulkan_wrapper.h

sjmerel2: <p>I’m trying to build libtorch for Android, using the build_pytorch_android.sh script, but am getting errors due to Vulkan:</p>
<pre><code class="lang-auto">  Cannot find source file:

    /Users/steve/Library/Android/sdk/ndk/25.1.8937393/sources/third_party/vulkan/src/common/vulkan_wrapper.h
</code></pre>
<p>And indeed, that file does not exist there.</p>
<p>Can someone point me to some up-to-date instructions on how to build the native library for Android?</p>

sjmerel2: <p>For anyone coming across this with the same question, I had to specifically disable both Vulkan and XNNPACK support to build the native code for Android (-DUSE_VULKAN=OFF -DUSE_XNNPACK=OFF).</p>
<p>I would be interested in any tips on how to build without disabling those features, though, since I’m not sure what the consequences is of disabling them!</p>

sjmerel2: <p>I built with version 21 of the NDK, and I no longer had to turn off those features.</p>

kabyanil: <p>I get the following error when using NDK 21.4.7075529 and 21.0.6113669 -</p>
<pre><code class="lang-auto">pytorch/third_party/XNNPACK/src/xnnpack/math.h:332:13: error: invalid output constraint '=t' in asm
      : [i] "=t" (i)
</code></pre>

kabyanil: <p>Which command did you add the flags to?</p>