akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDAGraph.cpp:145] Warning: Waiting for pending NCCL work

Joeyoung: <p>Hi,</p>
<p>I wrote the code for using CUDA Graph for multi processing (GPUs).<br>
The code works without the error.<br>
However, I am getting the following warning messages.<br>
[rank 0]:[W CUDAGraph.cpp:145] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())<br>
[rank 1]:[W CUDAGraph.cpp:145] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())</p>
<p>Although the code works, the warning messages bother me.<br>
Could you enlighten me about why the warning message happens and how to fix it?</p>
<p>Thanks,<br>
Joe</p>

ptrblck: <p>The watchdog can poll during a CUDAGraph capture via events and crash the capture as described <a href="https://github.com/pytorch/pytorch/pull/104487">here</a>. To avoid this, <a href="https://github.com/pytorch/pytorch/pull/110665">this PR</a> waits for all pending work to finish before starting the graph capture.</p>

Joeyoung: <p>Thanks! I can silence the warning messages by synchronizing before capture.</p>

ptrblck: <p>Yes, this would be one way to make sure the pending work is done before capturing the graph.</p>
How to create your own backpropagation graph

Max_Unhold: <p>Hey, since I am stuck with the error <code># RuntimeError: Function 'AddmmBackward0' returned nan values in its 0th output.</code> and there seems to be no way of getting around it, I would like to set the backward graph by myself.</p>
<p>How would you do that?</p>

ptrblck: <p>You can write a custom <code>autograd.Function</code> defining your own <code>backward</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>
Complicated indexing question

David_Wasserman: <p>X is a tensor of shape (N, D, P) such that for each n and d, X[n, d, :] is either all 0’s, or it contains a single block of consecutive 1’s and the rest are 0’s.</p>
<p>Y is a tensor of shape (N, P) such that each row is sorted increasing.</p>
<p>I want to create a tensor Z of shape (N, D, 2) such that<br>
a. If i is the index of the first 1 in X[n, d, :], then Z[n, d, 0] = Y[n, i]<br>
b. If j is the index of the last 1 in X[n, d, :], then Z[n, d, 1] = Y[n, j]<br>
c. If X[n, d, :] is all 0’s, then I don’t care what Z[n, d, :] is.</p>
<p>Is there an efficient way to create Z?</p>

ptrblck: <p>Could you post an example how these tensors could look like by constructing it manually, please?</p>
In-place Cholesky Decomposition

Kevin96: <p>Hi,</p>
<p>I am using PyTorch’s Cholesky decomposition to compute the Cholesky factor of a very large matrix (16 GB).</p>
<p>I noticed that the peak memory consumption is doubled when calling <a href="https://pytorch.org/docs/stable/generated/torch.linalg.cholesky.html" rel="noopener nofollow ugc">torch.linalg.cholesky</a> and the reason seems to be memory allocation for the returned tensor.</p>
<p>This causes memory overflow on my 24 GB GPU. I am wondering if there is an option in PyTorch to do in-place Cholesky updates? Note that I do not need to compute the backward pass – it is okay to completely override the input matrix. In-place updates should be theoretically possible to save memory.</p>
<p>Thanks</p>

s769: <p>I’m also interested in this and wanted to check if there are any updates.</p>

AlphaBetaGamma96: <p>If you want to alleviate any memory calculations with <code>torch.linalg.cholesky</code>, why not use a <code>torch.no_grad()</code> decorator to remove any gradient calculations? That might help solve the problem without requiring too many changes?</p>

s769: <p>I tried this, but it still gives the same out of memory error. The matrix I have is of size 100000x100000 double precision (~80GB). I can get the factorization to work by directly calling cuSOLVER through the C API. Just wanted to see if there was a more convenient way through pytorch.</p>
Torch.std not working properly?

eleven: <pre><code class="lang-auto">import numpy as np
import torch
a = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
t = torch.tensor(a)
np_std = np.std(a, axis=0)
t_std = torch.std(t, dim=0).numpy()
print(np_std, t_std) #[1.63299316 1.63299316] [2. 2.]
assert np_std == t_std # why aren't these equal, and which one is correct?
</code></pre>

eleven: <p>NumPy and PyTorch use different default definitions for calculating the standard deviation:</p>
<ul>
<li>
<p>NumPy’s <code>np.std</code> by default calculates the <strong>population standard deviation</strong>, dividing by ( N ) (where ( N ) is the number of elements).</p>
</li>
<li>
<p>PyTorch’s <code>torch.std</code> by default calculates the <strong>sample standard deviation</strong>, dividing by ( N - 1 ) (applying Bessel’s correction for an unbiased estimator).</p>
</li>
</ul>
<p>This code works:</p>
<pre><code class="lang-auto">import numpy as np
import torch
a = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
t = torch.tensor(a)
# sample standard deviation (torch default)
np_std = np.std(a, axis=0, ddof=1.0)
t_std = torch.std(t, dim=0, unbiased=True).numpy()
print(np_std, t_std) #[2. 2.] [2. 2.]
assert np.array_equal(np_std,t_std) # pass
# population standard deviation (numpy default)
np_std = np.std(a, axis=0, ddof=0.0)
t_std = torch.std(t, dim=0, unbiased=False).numpy()
print(np_std, t_std) #[1.63299316 1.63299316] [1.63299316 1.63299316]
assert np.array_equal(np_std,t_std) # pass
</code></pre>
Padding sequence data required for CNN based autoencoder?

L_Z: <p>Let me breakdown my question into a few subset questions:</p>
<ol>
<li>Should we pad the sequences to the same length for more accurate results? More accurate reconstructed sequences and low reconstruction error. I have heard how if the sequence are in different length, a reconstructed sequence’s length will be different from the corresponding input sequence, is that so?</li>
<li>I know how inside each batch, it requires the sequences to be the same length. I am not sure why this is the case? My understanding about having batches is to speed up the training process, and if we apply batch normalization function in PyTorch, we need the batch size to be greater than 1, since it normalized among the sequences in the batch (Correct me if I am wrong here). Then can I train a model without applying DataLoader to my dataset? which means that the batch size is 1, then I can train with different length sequence.</li>
<li>Also in order to get a more accurate reconstruction result, should I make sure that all of my time-series sequence to have a fixed starting point? Like a similar starting point. (ie. zero-crossings for sinusoidal)</li>
</ol>
Where can I report the typo in the error message?

songsong0425: <p>Hi, I think it is too minor problem, but I want to notify a typo in the CUDA error message.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c94e9b0a0b334dc8b952ad5f3f5403b4fc93754.png" data-download-href="https://discuss.pytorch.org/uploads/default/6c94e9b0a0b334dc8b952ad5f3f5403b4fc93754" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c94e9b0a0b334dc8b952ad5f3f5403b4fc93754.png" alt="image" data-base62-sha1="fuyBT9hupwnNBTTIjtIluMti7Fa" width="690" height="39" data-dominant-color="2C2C2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1176×67 5.21 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As you can see, I found that <code>capacity</code> was misspelled as <code>capacty</code>.<br>
Can anyone notice it to the corresponding administrator?</p>
<p>Thank you.</p>

ptrblck: <p>This was already fixed a year ago in <a href="https://github.com/pytorch/pytorch/pull/111155">this PR</a>. You might want to update PyTorch to see the fixed error message.</p>
LFW dataset train and test distribution

andergalvan: <p>Hello everyone,</p>
<p>I’m a bit new to PyTorch and I’d like to use the LFW dataset for a face recognition problem. I noticed that if there are images of a person in the training set, there are no images of that person in the test set (looking in each set how many images there are of each person). Am I doing something wrong? This is the code I have used:</p>
<p>train_data = datasets.LFWPeople(root=“data/LFW”, split=“train”, image_set=“original”, transform=ToTensor(), download=True)<br>
train_counter = dict(Counter(train_data.targets))</p>
<p>test_data = datasets.LFWPeople(root=“data/LFW”, split=“test”, image_set=“original”, transform=ToTensor(), download=True)<br>
test_counter = dict(Counter(test_data.targets))</p>
<p>Thank you in advanced!<br>
Ander</p>

melgor: <p>I know that it has been 8 months but here is the answer:<br>
LFW data is designed not for a classification task, it is a face-retrieval task. It means that during test time your task is to build an index of known persons  (from half of the dataset).<br>
And second half of the dataset should be used as a query image.<br>
Then you calculate the distance between a query and all persons in the gallery. You take the most similar image to a query and check if the query matches your query label.</p>
<p>Thanks to that protocol, classes during training and testing can be different, like in LFW.</p>
<p>In my blog-post I wrote little bit more about it, with link to notebooks with code implementations: <a href="https://medium.com/p/42574d13f6a6" class="inline-onebox" rel="noopener nofollow ugc">Foundation Models for Computer Vision | by Bartosz Ludwiczuk | Medium</a></p>
About otuput of torch.matmul with special values like as +inf, -inf, nan

liger99: <p>If you look at example 1 below, it is a 4x4 matmul z = torch.matmul(x, y), and it is a situation where nan should be output by accumulating +inf and inf.<br>
But matrix output (z matrix) are all composed of +/inf.<br>
example 1)<br>
x =  tensor([[ 1.2208e+38,  1.5069e+38,  3.4027e+37,  7.5471e+37],<br>
[ 6.7193e+37, -9.0691e+37, -1.7984e+38,  1.5571e+38],<br>
[-1.9089e+38, -1.7563e+37,  2.3544e+37, -9.8721e+37],<br>
[-9.5343e+37, -5.0244e+37, -7.0133e+37,  2.2247e+37]])<br>
y =  Parameter containing:<br>
tensor([[-9.2652e+36, -5.3796e+37, -7.5069e+37,  6.6170e+37],<br>
[ 6.2397e+37,  1.1426e+38, -1.6466e+38,  2.9774e+38],<br>
[-3.9087e+37, -5.1765e+36,  5.3396e+37, -2.4418e+38],<br>
[-6.0303e+37,  7.8149e+37,  4.1132e+37,  1.4304e+37]],<br>
requires_grad=True)<br>
z =  tensor([[-inf, -inf, -inf, inf],<br>
[-inf, -inf, -inf, inf],<br>
[inf, inf, inf, -inf],<br>
[inf, inf, inf, -inf]], grad_fn=)</p>
<p>If you look at example 2 below, you can see that nan is output only when +/-inf is directly entered as input.<br>
example 2)<br>
x =  tensor([[inf, -inf, inf, -inf],<br>
[-inf, inf, inf, -inf],<br>
[-inf, -inf, -inf, inf],<br>
[inf, -inf, inf, inf]])<br>
y =  Parameter containing:<br>
tensor([[inf, -inf, -inf, inf],<br>
[-inf, -inf, -inf, inf],<br>
[-inf, -inf, -inf, inf],<br>
[-inf, inf, -inf, inf]], requires_grad=True)<br>
z =  tensor([[nan, nan, nan, nan],<br>
[nan, nan, nan, nan],<br>
[nan, inf, nan, nan],<br>
[nan, nan, nan, nan]], grad_fn=)</p>
<p>This result is different from the numerical operations we know.<br>
I would like to hear the opinion of who know about that problem.</p>

ptrblck: <p>Using your tensors I get:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.matmul(x, y)
Out[9]: 
tensor([[nan, nan, nan, nan],
        [nan, nan, nan, nan],
        [nan, nan, nan, -inf],
        [nan, nan, nan, nan]], grad_fn=&lt;MmBackward0&gt;)

torch.matmul(x.cuda(), y.cuda())
Out[10]: 
tensor([[nan, nan, nan, nan],
        [nan, nan, nan, nan],
        [nan, nan, nan, -inf],
        [nan, nan, nan, nan]], device='cuda:0', grad_fn=&lt;MmBackward0&gt;)
</code></pre>
<p>for <code>torch==2.6.0.dev20241004+cu124</code>.</p>

liger99: <p>Thank you for your quick reply.<br>
Maybe I think this problem is related to Pytorch version.</p>
Efficient way to set tensor values using lower triangular indices

Munich: <p>Hello,</p>
<p>I’m looking for an efficient way to properly set batched (size = 2) lower triangular tensors each with 3 x 3 dimensions. Lets call this tensor Sigma [2, 3, 3]. I’ve created parameters Theta [2, 6] which will be used to fill Sigma. Note that 6 parameters are required to fill/set a lower triangular matrix with dim = 3 (e.g. (3*(3+1))/2). Currently my code is:</p>
<pre><code class="lang-auto">Theta = torch.nn.Parameter(torch.Tensor(2, 6))
torch.nn.init.xavier_uniform_(Theta, gain=1.414)
Sigma = torch.nn.Parameter(torch.Tensor(2, 3, 3))
torch.nn.init.zeros_(Sigma)
tril_indices = torch.tril_indices(row=3, col=3, offset=0)
Sigma[0, tril_indices[0], tril_indices[1]] = Theta[0]
Sigma[1, tril_indices[0], tril_indices[1]] = Theta[1]
</code></pre>
<p>The above code works but is there a more efficient way that I replace the last two lines with a single line of code? If my batch size is say 16 I don’t want to have 16 separate lines of code setting the appropriate tensor values.</p>
<p>Ideally, I’d like Sigma[index_tensor] = Theta. But because Theta and Sigma have different shapes I am getting a errors for broadcasting. I’ve also tried thinking about using scatter_ but the function seems like it can’t be used for my use case.</p>
<p>Thanks</p>

kirtanp: <p>Did you find a solution for it?</p>
<p>You could always use a loop to set it but I was wondering if someone found a better solution.</p>

Martin36: <p>The solution I found is the following:</p>
<pre><code class="lang-auto">Theta = torch.rand(2,6)
Sigma = torch.zeros(2,3,3)
tril_indices = torch.tril_indices(3,3)
Sigma[:, tril_indices[0], tril_indices[1]] = Theta
</code></pre>
<p>Hope it helps</p>
When I run without CUDA: Function ‘PowBackward0’ returned nan values in its 0th output

MartensCedric: <p>My code was running fine <em>with</em> CUDA, but now that I run it with <code>device="cpu"</code>, with the flag <code>torch.autograd.set_detect_anomaly(True)</code>, the runtime error is raised:</p>
<pre><code class="lang-plaintext">RuntimeError: Function 'PowBackward0' returned nan values in its 0th output.
</code></pre>
<p>Looking closely at the call stack:</p>
<pre><code class="lang-auto"> File "&lt;ipython-input-468-be9e157834e4&gt;", line 83, in forward
    self.grad_mag = torch.sqrt(self.grad_x**2 + self.grad_y**2)
  File "/usr/local/lib/python3.10/dist-packages/torch/_tensor.py", line 41, in wrapped
    return f(*args, **kwargs)
 (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:111.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
</code></pre>
<p>Indicating that the error is when backwarding through:</p>
<pre data-code-wrap="python"><code class="lang-python">self.grad_mag = torch.sqrt(self.grad_x**2 + self.grad_y**2)
</code></pre>
<p>Firstly, I don’t see why this issue is on CPU but not on CUDA. Secondly, I don’t understand why it would get NaNs from the backward pass</p>
<pre data-code-wrap="python"><code class="lang-python">print("grad_x: ",self.grad_x.isinf().any(), self.grad_x.isnan().any())
print("grad_y: ",self.grad_y.isinf().any(), self.grad_y.isnan().any())
self.grad_mag = torch.sqrt(self.grad_x**2 + self.grad_y**2)
print("grad mag ", self.grad_mag.isinf().any(), self.grad_mag.isnan().any())
</code></pre>
<p>which outputs:</p>
<pre><code class="lang-auto">grad_x:  tensor(False) tensor(False)
grad_y:  tensor(False) tensor(False)
grad mag  tensor(False) tensor(False)
</code></pre>
<p>If it makes any difference, I’m optimizing with LBFGS</p>
Is there a library for automated regression testing for machine learning training code?

heth27: <p>I want to test my deep learning training code in an automated way, to make sure that refactoring does not change anything about the update step of my training. I guess I also need some kind of metric for coverage of my computational graph.</p>
<p>Is there a library that does that for pytorch or jax? I do not want to test the trained model (there are libraries for that) but the training code.</p>
Placing tensors in different GPU in multinode, multiGPU setting?

AKT_TARAFDER: <p>We got 2 nodes, each with 2 GPUs. Trying to run a simple experimental all_reduce from the 4 GPUs. And we want to place tensors to their respective GPUs</p>
<p>code:</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
import os

def main():
    rank = torch.distributed.get_rank()
    world_size = torch.distributed.get_world_size()
    
    print(f"Hello from node {os.getenv('HOSTNAME')} with rank {rank} out of {world_size} processes!")


    tensor = torch.ones(1).to(torch.cuda.current_device())
    
    dist.all_reduce(tensor)
    print(f'local_rank:{rank}, device:{tensor.device}, tensor:{tensor}')
    print(f'current_device:{torch.cuda.current_device()}')

if __name__ == "__main__":
    # Initialize the distributed process group
    try:
        torch.distributed.init_process_group(backend='nccl') 
   
        rank = torch.distributed.get_rank()
       
        print(f'rank:{rank}')
        
        local_rank = int(os.environ['LOCAL_RANK'])
        torch.cuda.set_device(local_rank)
        main()

    finally:

        dist.destroy_process_group()

</code></pre>
<p>for running:<br>
in node 0:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=0 --master_addr="x.y.z.10" --master_port=12346 dist.py --backend nccl
</code></pre>
<p>and in node 1:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=1 --master_addr="x.y.z.10" --master_port=12346 dist.py --backend nccl
</code></pre>
<p>you should replace “x.y.z.10” with master node IP of your machine.</p>
<p>The issue is Node1 has (gpu0, gpu1), and Node2 has (gpu2, gpu3)</p>
<p>(1) Are the GPUs like 0,1,2,3 for each node?<br>
(2) We are setting the device using torch.cuda.set_device(), and trying to place the tensors using torch.cuda.current_device(),<br>
(3) Why all_reduce not working?<br>
(4) can we choose to use process in gpu1 from node0, to shift some portion of tensor to gpu0 of node2<br>
(5) Overall how to place tensors individually in each of the 4 GPUs.</p>
<p>The code does not run when we want to place tensors using torch.cuda() either or use all_reduce.</p>
<p>Printing without them always shows cuda:0 and cuda:1 for both nodes.</p>
Does Executorch support int16x8 quantization?

andresovela: <p>Hi,</p>
<p>I’m currently working on getting audio models to run on HiFi 4 DSPs, coming from TFLite Micro.</p>
<p>TFLM supports <a href="https://ai.google.dev/edge/litert/models/post_training_integer_quant_16x8" rel="noopener nofollow ugc">int16x8 quantization</a> and I’m wondering if Executorch does too.</p>
<p>I just started looking into Executorch, so I’m pretty ignorant about PyTorch stuff at the moment. I read something about Mixed Precision support in PyTorch, but I couldn’t find anything about that in Executorch.</p>

lucylq: <p>Hey <a class="mention" href="/u/andresovela">@andresovela</a>, thanks for taking a look at ExecuTorch!</p>
<p><a class="mention" href="/u/jerryzh168">@jerryzh168</a>, is there support for int16x8 quantization?</p>
<p>ExecuTorch supports various quantization schemes - see <a href="https://pytorch.org/executorch/stable/quantization-overview.html#quantization-overview" class="inline-onebox" rel="noopener nofollow ugc">Quantization Overview — ExecuTorch 0.3 documentation</a></p>

jerryzh168: <p>it’s not right now probably, but it’s possible to express in quantizer API, and also will require xnnpack or some backend to implement an efficient op for it I think</p>

andresovela: <p>I see. Thanks for the response!</p>
Usingng torch on machienes with globally installed cuda runntime

Black_Mesa: <p>Nvidia ships cuda runntime installtion and respective docker containers with installed cuda runntime binaries  in /usr/local/cuda .</p>
<p>The torch pip package under linux declares the exact same runntime libraries as pip packages dependecies with binary wheels from nvidia .</p>
<p>Thus on a preconfigured system with the cuda runntime installing torch from pip downloads a 2gb cuda runntike twice</p>
<p>Is there a a way to install a cuda compatible pip package without pulling the nvidia pip package dependencies ?</p>

ptrblck: <p>No, since your locally installed CUDA toolkit wouldn’t be used at all. If you want to use your locally installed CUDA toolkit you can build PyTorch from source.</p>
Custom way of reading weights of a layer

Izan_C_G: <p>Hello,</p>
<p>I am looking for an efficient way to read the weights of a convolutional layer while modifying them as they are being read.</p>
<p>The idea is to ensure that when performing inference later, the weights are already modified, thereby avoiding any additional time overhead during image inference.</p>
<p>The objective is to accomplish this during runtime, either in parallel with the execution of other layers or at the start of the inference process when processing an entire dataset.</p>
<p>Is this feasible? Would it be possible to implement a reading or custom operator that allows the weights to be read or modified in parallel to the inference process? Would I need to modify the Cuda kernels used by Pytorch?</p>
<p>Thank you very much.</p>
Parallel for in a Hook

Izan_C_G: <p>Hello,<br>
I am using hooks to access the inputs of layers in a pre-trained neural network model like ResNet50. Specifically, I am employing the <code>register_forward_pre_hook</code> method. However, as the input tensor is quite large, and I need to iterate over it and modify it, I would like to know if it is possible to parallelise this process within the hook. This is due to the increased computation time on the GPU, as it remains idle while the hook is executed, making the process less efficient.</p>
<pre><code class="lang-auto">def get_activation_info(name):
    def hook(model, input):
        aux = input[0].clone()
        aux = torch.flatten(aux)
       for i in range(len (aux)):
        // loop to be optimised
    return hook
</code></pre>
<p>Thank you !</p>
Audios wav pytorch

jsanchez: <p>good day your support this error RuntimeError: mat1 dim 1 must match mat2 dim 0, I am working pytorch to classify wav audios,<br>
BATCH_SIZE = 32 ##<span class="hashtag-raw">#128</span> para 4conv<br>
EPOCHS = 10<br>
LEARNING_RATE = 0.001</p>
<p>ANNOTATIONS_FILE = “…/data_aymara/metadata/aymara_sound.csv”<br>
AUDIO_DIR = “…/data_aymara/audio”<br>
SAMPLE_RATE = 22050<br>
NUM_SAMPLES = 22050</p>
<p>from torch import nn<br>
from torchsummary import summary<br>
class CNNNetwork(nn.Module):</p>
<pre><code>def __init__(self):
    super().__init__()
    self.conv1 = nn.Sequential(
        nn.Conv2d(
            in_channels=1,
            out_channels=16,
            kernel_size=3,
            stride=1,
            padding=2
        ),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2)
    )
    self.conv2 = nn.Sequential(
        nn.Conv2d(
            in_channels=16,
            out_channels=32,
            kernel_size=3,
            stride=1,
            padding=2
        ),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2)
    )
  
    self.flatten = nn.Flatten()
    self.linear = nn.Linear(32 * 3 * 4, 3)####2 convoluciones ####3 son las clases de salida, 32 capas ocultas
    self.softmax = nn.Softmax(dim=1)

def forward(self, input_data):
    x = self.conv1(input_data)
    x = self.conv2(x)
    x = self.flatten(x)
    logits = self.linear(x)
    predictions = self.softmax(logits)
    return predictions
</code></pre>

ptrblck: <p>What’s the shape of <code>input_data</code>?</p>

jsanchez: <p>in one folder I have sound files in wav format and in another folder an excel csv containing the name of the wav sound file and the category it is and based on the two trainings.</p>

jsanchez: <pre><code>    solved by printing xshape and setting the value to linear
    print(x.shape)###con esto se sabe que debe ir en la seccion A
    x = self.flatten(x)
    
    x = x.view(x.size(0), -1)
</code></pre>
How to register hook to only part of weights

SKYHOWIE25: <p>Hi</p>
<p>I am new to Python and do not very familiar with the lambda expression. I know I can register a hook to a weight by doing</p>
<p><code>h = model.classifier.weight.register_hook(lambda grad: grad * 0.)</code></p>
<p>But, is there a way to register a hook to only part of the weight. For example, to grad[0].</p>
<p>Thank you.</p>

albanD: <p>Hi,</p>
<p>No you cannot, but you can return apply your hook to just part of the gradient:</p>
<pre><code class="lang-auto">def hook_fn(grad):
    new_grad = grad.clone() # remember that hooks should not modify their argument
    new_grad[0].zero_()
    return new_grad

h = model.classifier.weight.register_hook(hook_fn)
</code></pre>

Izan_C_G: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, I have seen your answers and wonder if there is any faster option of modifying the gradients. I tried to implement your code, but when using Hooks, my GPU is stopped, and the time to do training or de-inference is much slower than without Hooks. Is there any way of paralysing the code assuming that I want to check all elements within the gradient tensor?</p>
Is randn() guaranteed to be finite?

pyscorcher: <p>In theory samples from an N(0, 1) distribution can get arbitrarily large, that is, outside of the range of whatever floating point type you’re using. Is it possible that the output of torch.randn can be Inf or NaN?<br>
If it is guaranteed to be finite, is the range (given the type) known or even well defined?</p>
<p>I couldn’t find anything about that in the documentation:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.randn.html#torch.randn" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.randn.html#torch.randn</a></p>

kalekundert: <p>TLDR: the output of <code>randn()</code> is guaranteed to be finite, and the range is ±5.76810741424560546875.</p>
<hr>
<p>I had this same question.  I’m not really at all familiar with the C++ code underlying PyTorch, but I believe that <code>randn()</code> is implemented <a href="https://github.com/pytorch/pytorch/blob/932ae131fbc74408f0452f910e763e99170452df/aten/src/ATen/core/PhiloxRNGEngine.h#L120-L135" rel="noopener nofollow ugc">here</a>.  The relevant lines of code are:</p>
<pre><code class="lang-auto">float u1 = 1 - uint32_to_uniform_float(output_[0]); // uint32_to_uniform_float returns [0,1), we need (0,1] to avoid passing 0 to log.
float u2 = 1 - uint32_to_uniform_float(output_[1]);
return static_cast&lt;float&gt;(std::sqrt(-2.0 * std::log(u1)) * std::cos(2.0 * M_PI * u2));
</code></pre>
<p>The only way this function could output a non-finite value would be if <code>u1</code> was ≤0 or &gt;1.  As the comment says, the <code>1 - uniform()</code> construction avoids this possibility.</p>
<p>Empirically, I also tested this by generating ≈2<sup>43</sup> (≈10<sup>13</sup>) samples from <code>randn()</code>.  All were finite.  There are only 2<sup>32</sup> single-precision floats, so if it were possible for <code>randn()</code> to generate non-finite values, this test probably would have detected that.</p>
<p>With regard to the range, <code>uint32_to_uniform_float()</code> is implemented <a href="https://github.com/pytorch/pytorch/blob/932ae131fbc74408f0452f910e763e99170452df/aten/src/ATen/core/PhiloxRNGEngine.h#L215-L219" rel="noopener nofollow ugc">here</a>, in the same file as <code>randn()</code>.  The relevant lines of code are:</p>
<pre><code class="lang-auto">constexpr float scale = 4.6566127342e-10;
return static_cast&lt;float&gt;(value &amp; 0x7FFFFFFF) * scale;
</code></pre>
<p>I don’t fully understand this algorithm, but the largest value it can produce (if <code>value == 0xFFFFFFFF</code>) is <code>0.999999940395355224609375</code>.  The smallest is 0.  Looking back to the <code>randn()</code> implementation, we see that the smallest and largest normally-distributed samples are obtained when <code>u1</code> is as close to zero as possible and <code>u2</code> is 0.5 and 1, respectively.  Plugging in the extreme values from the uniform distribution, we get <code>±5.76810741424560546875</code>.  For what it’s worth, I’m a bit surprised that this range isn’t wider.</p>
Is it possible to export a model with torch c++ extension to ONNX?

crazyboy9103: <p>Hello,<br>
I’ve been working on a project that needs custom c++ extension like <a href="https://github.com/pytorch/extension-cpp/tree/master/extension_cpp/csrc" rel="noopener nofollow ugc">this</a>, but it seems that it’s very difficult to convert the model to ONNX and run on ONNX runtime.</p>
<p>By registering the operator using torch.onnx.register_custom_op_symbolic as described <a href="https://github.com/onnx/tutorials/blob/main/PyTorchCustomOperator/README.md" rel="noopener nofollow ugc">here</a>, i was able to export the model to ONNX, which failed on inference, something like <code>"Fatal error: x is not a registered function/op"</code>.</p>
<p>According to the guide, it is required to implement the same kernels in ONNX runtime, which just seems very difficult for me, as i dont know anything about Ort.<br>
Also, i noticed that if i want to run using CUDA and TensorRT execution providers, i have to implement them separately. (or may be not?)</p>
<p>I want to know if there is any workaround to this, as the guide seems to be outdated (no update for 3 years), or is it the only possible way for my purpose? If so, where can i find more advanced examples so i can learn from those?’ I’m still learning, so any basic advice or guidance would be really helpful.</p>
When trying to save my c++ network: serialize.h:43:11: error: no match for ‘operator<<’

SystemLord: <p>Here’s my network:</p>
<pre><code class="lang-auto">struct Net : public torch::nn::Module
{
    Net(int64_t input_n, int64_t middle_n, int64_t output_n)
        : linear_1(register_module("linear_1", torch::nn::Linear(input_n, middle_n))),
          linear_2(register_module("linear_2", torch::nn::Linear(middle_n, middle_n))),
          linear_3(register_module("linear_3", torch::nn::Linear(middle_n, output_n)))
    {
    }

    torch::Tensor forward(torch::Tensor input)
    {
        return linear_3(linear_2(linear_1(input)));
    }

    torch::nn::Linear linear_1;
    torch::nn::Linear linear_2;
    torch::nn::Linear linear_3;
};
</code></pre>
<p>I try to save it with:</p>
<pre><code class="lang-auto">Net network = Net(128, 256, 1);
torch::save(network, save_file);
</code></pre>
<p>But I get this:</p>
<blockquote>
<p>serialize.h:43:11: error: no match for ‘operator&lt;&lt;’ (operand types are ‘torch::serialize::OutputArchive’ and ‘const Net’</p>
</blockquote>
<p>Despite the fact that the docs say that &lt;&lt; should be defined for torch::nn::Module</p>
<p>What am I doing wrong?</p>

ptrblck: <p>Try to save the <code>state_dict</code> or any other <code>std::vector&lt;torch::Tensor&gt;</code> instead of the model directly.</p>

SystemLord: <p>Thank you - how precisely do I access the state_dict from the network in c++?</p>
How to Properly Normalize Weights During Training in PyTorch Without Bypassing Autograd?

BeZeBeast: <p>I’m implementing a neural network in PyTorch and need to normalize the weights of certain layers during the forward pass. Specifically, I want to normalize the weights by their L2 norm for some layers. Here’s a simplified version of my code:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

class MyModel(torch.nn.Module):
    def __init__(self, layers, activation_function):
        super(MyModel, self).__init__()
        self.layers = torch.nn.ModuleList(layers)
        self.act_fun = activation_function

    def forward(self, X):
        output = X
        for i, layer in enumerate(self.layers):
            if i &gt; 0:
                # Normalize the weights
                layer.weight.data = F.normalize(layer.weight, p=2, dim=1)
            if i &lt; len(self.layers) - 1:
                output = self.act_fun(layer(output))
            else:
                output = layer(output)
        return output.squeeze()
</code></pre>
<p>My concerns are:</p>
<ol>
<li>Autograd Compatibility: By directly modifying layer.weight.data, am I bypassing PyTorch’s autograd system? Will this prevent gradients from being computed correctly during backpropagation?</li>
<li>Proper Gradient Updates: Will the weight normalization be accounted for when I call loss.backward(), or do I need to handle this differently to ensure correct gradient computation?</li>
<li>Better Practices: Is there a recommended way to normalize layer weights during training in PyTorch that maintains compatibility with autograd and ensures proper gradient updates?</li>
</ol>
<p>I’ve read that modifying .data directly can cause issues with gradient tracking, but I’m unsure how to implement weight normalization correctly in this context.</p>

ptrblck: <p>You could implement a parametrization as described in the <a href="https://pytorch.org/tutorials/intermediate/parametrizations.html">Parametrization Tutorial</a>.</p>

BeZeBeast: <p>Thanks,</p>
<p>But using a parametrization with a simple forward pass that returns F.normalize(self.weight) wouldn’t that be the same as using a model forward pass including</p>
<p>normalized_weight = F.normalize(layer.weight, p=2, dim=1)<br>
output = F.linear(output, normalized_weight, layer.bias)</p>
<p>?</p>
Using SmoothL1Loss

SystemLord: <p>I’m trying to use SmoothL1Loss in libtorch:</p>
<pre><code class="lang-auto">torch::Tensor network_output = network.forward(training_input);
torch::Tensor loss = torch::nn::SmoothL1Loss(network_output, target);
</code></pre>
<p>But that doesn’t seem to be what SmoothL1Loss is expecting… what am I doing wrong?</p>

SystemLord: <p>Never mind, I got it, it’s torch::smooth_l1_loss</p>
Pytorch Installations

nikita01: <p>Hello everyone,</p>
<p>I’m trying to install PyTorch on my Linux system, but when I verify the installation, the output shows as “false.”</p>

ptrblck: <p>Did you follow <a href="https://pytorch.org/get-started/locally/">these install instructions</a>?</p>

nikita01: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>! issue was resolved</p>
Is there an expected distribution for torch.empty?

atiorh13: <p>I was reviewing the <code>gumbel_softmax</code> implementation [1] and realized that the Gumbel distribution was being sampled via <code>gumbels = -torch.empty_like(logits, memory_format=torch.legacy_contiguous_format).exponential_().log()  # ~Gumbel(0,1)</code>. I couldn’t see how this would generate gumbel = -log(eps - log(uniform(0,1) + eps) so I empirically validated the distributions which checks out.</p>
<p>My question is the following: what is the distribution of torch.empty so that these formula generate the same distribution?</p>
<p>[1] <a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#gumbel_softmax" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#gumbel_softmax</a></p>

googlebot: <p>it is garbage rewritten by an inplace operation (here exponential_() creates samples from Exponential(1) and is not a function of input data)</p>
Forward in chunk

Jean_Kim: <p>Hi,</p>
<p>I have a large dataset which must be fully pass a forward and I cannot take advantage of the mini-batches due to the characteristic of my evaluation metric.<br>
Since the whole dataset does not fit to GPU memory, I first thought that I need to chunk it into pieces to achieve partial outputs of my model which should be concatenated after being transferred to cpu.</p>
<p>Basically this is what has been done for this:</p>
<pre><code class="lang-auto">chunks = DataLoader(data, batch_size)

model = model.to(device)

output = []
for batchidx, data in enumerate(chunks):
  data = data.to(device)
  this_out = model(data)
  output.append(this_out.cpu())

output = torch.cat(output, ..)
</code></pre>
<p>But I found that during the transfer from GPU to CPU, the whole computational graph is copied for each chunk and stays within the GPU memory, so eventually I end up with OOM. I’m not really sure why this also happens even when I moved ‘this_out’ to cpu.<br>
This is somewhat a large overhead when I am supposed to have the same structure for every chunk. Does anyone have a good solution for this kind of situation?</p>

ptrblck: <aside class="quote no-group" data-username="Jean_Kim" data-post="1" data-topic="211586">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jean_kim/48/72865_2.png" class="avatar"> Jean_Kim:</div>
<blockquote>
<p>I’m not really sure why this also happens even when I moved ‘this_out’ to cpu.</p>
</blockquote>
</aside>
<p>This is expected since the <code>to()</code> operation is differentiable if tensors are moved between devices and all intermediate tensors created in the computation graph will stay on their original device. The backward pass will thus move the gradient back to the original device.</p>
<p>If you only want to collect the outputs for some sort of evaluation metric, you could execute the forward pass in a <code>torch.no_grad()</code> context, which will not store the intermediate tensors, and <code>torch.cat/stack</code> the outputs.</p>
Choosing the neural network model based on the outcome of a primary NN model

sriramgs: <p>Hi,</p>
<p>My model’s (ResNet50) weights are frozen except its final Fully Connected layer. I am thinking of this approach: For e.g.</p>
<p>Model 1: FC will be <code>Linear(in_features=2048, out_features=2, bias=True)</code><br>
Model 2: FC will be <code>Linear(in_features=2048, out_features=10, bias=True)</code><br>
Model 3: FC will be <code>Linear(in_features=2048, out_features=5, bias=True)</code></p>
<p>My Model 1 will do binary classification for any provided image. Then, based on its outcome, it will choose Model 2 or Model 3 and completes the task for the same image. In all these cases, only the FC layer is getting changed and learnt. The rest of weights in the architecture are fixed and does not undergo training.</p>
<p>How can I integrate the above points together in pytorch in such a way that if I pass an image to model 1, it follows the flow? If someone has better approach or suggestions for this task, I would be happy to consider. Please share your thoughts.</p>
<p>Thanks in advance!</p>
Question About Non-Static Versions and Manual Modification of Computational Graph

Martin_Pflaum: <p>Hello!</p>
<p>I had a question regarding <code>torch.autograd.Function</code>—is there a non-static version available? I think back in the day this used to be like that. Additionally, is there a way to dynamically modify the computational graph by hand without relying on <code>torch.autograd.Function</code>? I was also wondering if there is a way to know in torch.autograd.Function in the backward function if the flag create_graph and retain graph was set or not in the calling torch.autograd.grad function.</p>
<p>I would really appreciate any guidance or insights!</p>
<p>Best regards,<br>
Martin</p>

albanD: <p>Hey!</p>
<blockquote>
<p>is there a non-static version available</p>
</blockquote>
<p>Not sure what you mean by that? This function (like every other autograd graph in PyTorch) is executed at every forward pass. So from one use to the next, you can do anything you want.</p>
<blockquote>
<p>Additionally, is there a way to dynamically modify the computational graph by hand without relying on <code>torch.autograd.Function</code> ?</p>
</blockquote>
<p>Do you have a concrete example of what you want to do? We’ve fund that autograd.Function (or custom_op) and hooks are enough to get all the behavior we need.</p>
<p>You can see the doc here for needs_input_grad field to know which input require gradients <a href="https://pytorch.org/docs/main/notes/extending.html#extending-torch-autograd" class="inline-onebox">Extending PyTorch — PyTorch main documentation</a></p>
<p>For create_graph, you can just check if <code>torch.is_grad_enabled()</code> or if the inputs require gradients.<br>
For retain_graph, why do you need to know about this one? Since you don’t handle state lifetime there usually, we didn’t expose it.</p>

Martin_Pflaum: <p>Hello,<br>
Thank you for your quick response. What exactly is custom_op? I couldn’t find it online.<br>
By non-static i mean a version where i can modify the behaviour of the backward function outside of the instance. I tried to do that with the ctx provided but seems not to work that well. This would enable me to make the backward function behave differently depending on how often it is runned. I want to differentiate through some numerical scheme analytically because there the solution cannot be safely used directly to calculate the derivatives. Having information of retain_graph is helpful inorder to safe up memory and have more options to implement the problem.<br>
All the best,<br>
Martin</p>

Martin_Pflaum: <p>now that i think about it’s more than enough to know if create_graph is enabled. Thank you for your helpful insights.</p>
How do pixels get mapped to inputs?

matiasV-TYN: <p>Hi,</p>
<p>I’m working with object detection on an embedded device and need to compare performance on the board against off the board (running on a PC). For the integrity of the test, I need to make sure that the pixels are mapped to the inputs the same way on both cases.<br>
How does this happen with PyTorch on a PC so that I can replicate it on the embedded device?</p>
<p>Edit, clarification: I’m referring to how each individual pixel get’s mapped to each input. So for example, does the top left pixel get mapped to the first input, or the last one.</p>
<p>Thanks in advance.</p>

J_Johnson: <p>Here is an example code of a classifier in PyTorch.</p>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></p>
<p>Pretty straight forward. RGB gets separate to 3 channels, while Y and X take on dimensions. Then there is a dimension for batch size(usually the first dim).</p>

matiasV-TYN: <p>Thanks for the reply. Maybe I wasn’t clear enough, but I’m referring to how each individual pixel get’s mapped to each input. So for example, does the top left pixel get mapped to the first input, or the last one.</p>

J_Johnson: <p>The standard conversion from a PIL image to  tensor is via <code>torchvision.transforms.ToTensor</code>.</p>
<p><a href="https://pytorch.org/vision/main/generated/torchvision.transforms.ToTensor.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/generated/torchvision.transforms.ToTensor.html</a></p>
<p>That states:<br>
“Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W)…”</p>
<p>The order of the dimensions may change(i.e. channels first), but the location of pixels does not change.</p>
<p>That’s assuming you aren’t doing any image augmentations.</p>
Issues with Model Training and Overfitting in PyTorch

nolrahirose: <p>Hello everyone,</p>
<p>I’m reaching out for help regarding an issue I’m facing while training my model in PyTorch.</p>
<p><strong>Issue:</strong> I’ve noticed that my model is overfitting significantly on the training data, which is affecting its performance on the validation set.</p>
<p><strong>Details:</strong></p>
<ul>
<li>I’m using PyTorch version [insert version number].</li>
<li>The architecture I’m working with is [insert model architecture, e.g., CNN, RNN].</li>
<li>My training dataset consists of [insert dataset size and type].</li>
<li>I’ve implemented [mention any techniques you’ve tried, e.g., dropout, data augmentation], but the overfitting persists.</li>
</ul>
<p><strong>Questions:</strong></p>
<ol>
<li>What strategies can I employ to reduce overfitting in my model?</li>
<li>Are there specific hyperparameters I should focus on adjusting?</li>
<li>How can I effectively evaluate my model’s performance to ensure it’s generalizing well?</li>
</ol>
<p>I appreciate any insights or advice you can provide!</p>
<p>Thank you for your help!</p>

cortical_iv: <p>What type of task (e.g., classification, object detection)? How much data? How are you augmenting? HOw do you split into training/validation? What are you doing? Do you have code you can share a link to?</p>
Significiant time difference between minor model architecutre change

Louis2099: <p>I’m running ppo lagrangian algorithm from following repo: <a href="https://github.com/akjayant/PPO_Lagrangian_PyTorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - akjayant/PPO_Lagrangian_PyTorch: Implementation of PPO Lagrangian in PyTorch</a></p>
<p>By changing one of the model’s architecture from:</p>
<pre><code class="lang-auto">&gt;  class MLPCritic(nn.Module):
&gt;       def __init__(self, obs_dim, hidden_sizes, activation):
&gt;           super().__init__()
&gt;           self.v_net = mlp([obs_dim] + list(hidden_sizes) + [1], activation)
&gt;       def forward(self, obs):
&gt;           return torch.squeeze(self.v_net(obs), -1) # Critical to ensure v has right shape.
</code></pre>
<p>To:</p>
<pre><code class="lang-auto"> class MLPCritic_affine_Q_wide(nn.Module):
      def __init__(self, obs_dim, ac_dim, hidden_sizes, activation):
          super().__init__()
          self.q_w_net = mlp([obs_dim] + list(hidden_sizes) + [obs_dim], activation)
          self.fuse_layer = mlp([obs_dim + ac_dim] + [1], activation)
      def forward(self, obs, act):
          w = self.q_w_net(obs)
          result = self.fuse_layer(torch.cat((w, act), dim=-1))
          return result
</code></pre>
<p>During a simple test where:<br>
obs_dim = 2<br>
act_dim = 1<br>
The backward updating time increased by 10 times, but there is no significiant increase of parameter size and model deepth.<br>
Why such minor change to the network has a profound impact to the running time?</p>

vmoens: <p>Without further details on the network, device and such it’s a bit hard to tell. Feel free to share more details, like a fully reproducible example!</p>
<p>Ideally, you should run your models under the pytorch profiler to get a good understanding of what’s going on:<br>
<a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html</a></p>
<pre data-code-wrap="python"><code class="lang-python">with profile(activities=[ProfilerActivity.CPU], record_shapes=True) as prof:
    net1(data).backward() # or net2
prof.export_chrome_trace("trace.json")
</code></pre>
<p>then open it in <code>chrome://tracing</code> in your browser.</p>
Gymnasium Single Frame Render with TorchRL

ajberlier: <p>I can do the following with Stable-Baselines3, but unsure how to do it with TorchRL. I simply want a single frame image to be saved off, not a full rollout video. I could not find a solution in the TorchRL docs. Please let me know if I am missing something. Thank you!</p>
<pre><code># initial conditions image
img = env.render()
images = wandb.Image(img, caption=f"Initial Condition State for Seed {env_seed}")
wandb.log({"Initial Conditions": images})
</code></pre>

vmoens: <p>If you have a <code>GymEnv</code> you can do <code>GymEnv.whatever</code> and if <code>whatever</code> is not registered in the <code>GymEnv</code> class you will get it from the base env (ie, your gym env)</p>
<pre data-code-wrap="python"><code class="lang-python">from torchrl.envs import GymEnv
env = GymEnv("Pendulum-v1")
env.reset()
print(env.render())
</code></pre>
<p>Hope that helps!<br>
(if you want the rendered frames just create the env with <code>from_pixels=True, pixels_only=False</code> and if you want just one frame, well call <code>env.step(data)</code> - but that will make a step, not return just one rendered image <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> )</p>