akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to discern the computational graph and the corresponding gradient calculation process after I modifies the value of tensors with ".data"

BaeHann: <p>I am informed that modifying the value of a tensor with .data is dangerous since it can generate wrong gradient when backward() is called. However, I write two programmes(below) only to find that the generated gradient is so weird that it can’t be figured out how the gradient is calculated.</p>
<p>The first program is like this:</p>
<pre><code class="lang-auto">a = torch.tensor([1, 2, 3.], requires_grad=True)
out = a.sigmoid()
c = out.data  
c[0]=1
c[1]=3
c[2]=4
weight = torch.ones(out.size())
d = torch.autograd.grad(out,a,weight,retain_graph=True)[0]
</code></pre>
<p>d is tensor([ 0., -6., -12.])</p>
<p>while the second one is :</p>
<pre><code class="lang-auto">a = torch.tensor([1, 2, 3.], requires_grad=True)
out = a.sigmoid()
c = out.data 
c.zero_()
weight = torch.ones(out.size())
d = torch.autograd.grad(out,a,weight,retain_graph=True)[0]
</code></pre>
<p>d is tensor([0., 0., 0.])</p>
<p>I know the vector to vector derivative is a little complex.However, since the value of <em>a</em> and the computational flow from <em>a</em> to <em>out</em> is identical in these two programmes. So the value of <em>d</em> should nor differ from each other.</p>
<p>I sincerely appreciate all your help and suggestions. The question can also be found in <a href="https://stackoverflow.com/questions/76021517/how-to-discern-the-computational-graph-and-the-corresponding-gradient-calculatio" rel="noopener nofollow ugc">https://stackoverflow.com/questions/76021517/how-to-discern-the-computational-graph-and-the-corresponding-gradient-calculatio</a></p>

bdhirsh: <p>This is a good example of how .data can cause silent correctness issues.</p>
<p>The derivative of sigmoid(x) is “sigmoid(x) * (1-sigmoid(x)”. The autograd engine will probably implement this by taking the tensor that you got from running sigmoid(x) in the forward, and saving directly to be reused in the backward compute.</p>
<p>Since you are manually modifying the output of the sigmoid call in a way that is hidden from autograd (the .data field), autograd will end up using your mutated “out” tensor in the backward, giving you arbitrary gradients. In the second example, since you mutated out.data to set it to zero, then you get “0 * (1-0)”</p>

BaeHann: <p>Thank you for your help!<br>
Please allow me to paraphrase your answer in my own language. for example: z = f(x)，when computing the value of dz/dx, pytorch doesn’t always strictly calculate the formulation of f’(x) and then plug in the actual value of x. Sometimes(such as what pytorch has done for sigmoid()), the pytorch will directly get the value of dz/dx from the value of z.</p>

bdhirsh: <p>Yep! (Depending on the particular operator. not all operators use z in the compute of dz/dx).</p>
Question about SpecAugment

g1777: <p>i want to use the same specAgment parameters as this paper:<br>
" We use<br>
SpecAugment [27, 28] with mask parameter (F = 27), and ten<br>
time masks with maximum time-mask ratio (pS = 0.05), where<br>
the maximum-size of the time mask is set to pS times the length<br>
of the utterance."</p>
<p>here’s my code:</p>
<pre><code class="lang-auto">def feature_augmentation(spec,audio_length):
    p=0.05
    masking = FrequencyMasking(freq_mask_param=27)
    spec = masking(spec)
    spec = masking(spec)
    masking = TimeMasking(time_mask_param=80,p=p*audio_length)
    for _ in range(0,5):
        spec = masking(spec)

    return spec
</code></pre>
<p>could someone please tell me if i did it right ? any help would be highly appreciated!</p>
Estimate theoretical FLOPs of backward pass of a DNN

Vinayaka_Hegde: <p>Hi, as a part of my research, I wanted to estimate the total FLOPs of a DNN theoretically as the total number of multiplication operations performed in the network during forward pass and backward pass.</p>
<p>I have used Lenet5 as my model for simplicity. I’m able to understand the total number of flops for forward pass. The formula for forward pass (assuming <code>bs</code> to be the batch_size of the input) is as follows :</p>
<ol>
<li>Fully connected layer of <code>x</code> input neurons and <code>y</code> output neurons = <code>x</code> * <code>y</code> * <code>bs</code>
</li>
<li>Conv2d layer of input size <code>(a,b,c)</code> and filter size <code>(f1,f2,f3)</code> and conv_output size <code>(x,y,z)</code> = <code>bs</code> * <code>(f1*f2*f3)</code> * <code>num_filters</code> * <code>(x*y)</code>
</li>
</ol>
<p>I’m unable to come up with a similar formula like above for the backward pass.</p>
<p>I also looked at some of the tools like : <a href="https://gist.github.com/soumith/5f81c3d40d41bb9d08041431c656b233" rel="noopener nofollow ugc">Horace’s flop counter, but with flops metric fixed correctly · GitHub</a> to calculate FLOPs empirically. According to the tool, for backward pass, the formula roughly looks like below :</p>
<ol>
<li>Fully connected layer of <code>x</code> input neurons and <code>y</code> output neurons = <code>2</code> * <code>x</code> * <code>y</code> * <code>bs</code>
</li>
<li>Conv2d layer of input size <code>(a,b,c)</code> and filter size <code>(f1,f2,f3)</code> and conv_output size <code>(x,y,z)</code> = Unable to figure out !</li>
</ol>
<p>But, I’m still unable to figure out how many FLOPs is required for a conv2d or a fully connected layer in the backward pass. So, would be very helpful if anyone can help me with a formula like the above to calculate the exact number of FLOPs for backward pass.</p>
<p>Thanks a lot for your time !</p>
Relationship between Params and GFLOPS

cosmosaa: <p>When I look at the available models in torchvision <a href="https://pytorch.org/vision/stable/models.html" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision 0.15 documentation</a> , the relationship between Params and GFLOPS does not seem to be linearly related as I would expect. For example, <code>VGG11_BN_Weights.IMAGENET1K_V1</code> which has 132.9M weights has a GLOPS of 7.61 while <code>EfficientNet_V2_S_Weights.IMAGENET1K_V1 </code> has much fewer weights at 21.5M, while the GLOPS is around the same at 8.37, and consequently they take around the same time per epoch given the same inputs. I would expect since one has about 15% the number of parameters as the other, that it would take about 15% the amount of time per epoch given the same inputs but that is not the case. What is the relationship between number of params and GFLOPS? It looks like GFLOPS is the better measure to use when comparing how iteration time per epoch compares to another model.</p>

J_Johnson: <p>GFLOPS (not to be confused with GFLOPs) describes how many floating point operations a model can perform <em>per second</em>. More GFLOPS implies the model has a faster throughput, all else being equal.</p>
CUDA and CuDNN version for libtorch

Albert_Christianto: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I want to know what libtorch API that we can use to get the CUDA and CuDNN version?<br>
Ps:<br>
Right now I have a project that using both opencv DNN and Libtorch, but I keep getting warning because unmatch cudnn version. I use pre-built libtorch library.</p>
<p>best regards,<br>
Albert Christianto</p>

ptrblck: <p>For cuDNN:</p>
<pre><code class="lang-auto">long cudnn_version = at::detail::getCUDAHooks().versionCuDNN();
</code></pre>
<p>for CUDA runtime:</p>
<pre><code class="lang-auto">int runtimeVersion;
AT_CUDA_CHECK(cudaRuntimeGetVersion(&amp;runtimeVersion));
</code></pre>
<p>for the driver version:</p>
<pre><code class="lang-auto">int version;
AT_CUDA_CHECK(cudaDriverGetVersion(&amp;version));
</code></pre>
<p>In case you want to use checks in the code you could also use e.g.:</p>
<pre><code class="lang-auto">#if defined(CUDA_VERSION) &amp;&amp; CUDA_VERSION &gt;= 11200
</code></pre>

JamesDickens: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> what are your include headers and namespace declarations?</p>

JamesDickens: <pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;ATen/cuda/CUDAContext.h&gt;
using namespace std;

void print_LibtorchVersion() {
    cout &lt;&lt; "PyTorch version: "
        &lt;&lt; TORCH_VERSION_MAJOR &lt;&lt; "."
        &lt;&lt; TORCH_VERSION_MINOR &lt;&lt; "."
        &lt;&lt; TORCH_VERSION_PATCH &lt;&lt; endl;
}

void print_CUDA_cuDNN_info() {
    long cudnn_version = at::detail::getCUDAHooks().versionCuDNN();
    cout &lt;&lt; "The cuDNN version is " &lt;&lt; cudnn_version &lt;&lt;"\n";
    int runtimeVersion;
    AT_CUDA_CHECK(cudaRuntimeGetVersion(&amp;runtimeVersion));
    cout &lt;&lt; "The CUDA runtime version is " &lt;&lt; runtimeVersion &lt;&lt; "\n";
    int version;
    AT_CUDA_CHECK(cudaDriverGetVersion(&amp;version));
    cout &lt;&lt; "The driver version is " &lt;&lt; version &lt;&lt; "\n";
}
</code></pre>
Print the CUDA version and CuDNN version in LibTorch

JamesDickens: <p>Is it possible to print the version of CUDA and cuDNN that a given build of LibTorch was compiled against in C++?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/cuda-and-cudnn-version-for-libtorch/137194/2">This post</a> shows the corresponding calls.</p>

JamesDickens: <pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;ATen/cuda/CUDAContext.h&gt;
using namespace std;

void print_LibtorchVersion() {
    cout &lt;&lt; "PyTorch version: "
        &lt;&lt; TORCH_VERSION_MAJOR &lt;&lt; "."
        &lt;&lt; TORCH_VERSION_MINOR &lt;&lt; "."
        &lt;&lt; TORCH_VERSION_PATCH &lt;&lt; std::endl;
}

void print_CUDA_cuDNN_info() {
    long cudnn_version = at::detail::getCUDAHooks().versionCuDNN();
    cout &lt;&lt; "The cuDNN version is " &lt;&lt; cudnn_version &lt;&lt;"\n";
    int runtimeVersion;
    AT_CUDA_CHECK(cudaRuntimeGetVersion(&amp;runtimeVersion));
    cout &lt;&lt; "The CUDA runtime version is " &lt;&lt; runtimeVersion &lt;&lt; "\n";
    int version;
    AT_CUDA_CHECK(cudaDriverGetVersion(&amp;version));
    cout &lt;&lt; "The driver version is " &lt;&lt; version &lt;&lt; "\n";
}
</code></pre>
What does .pth mean in torch.save?

qbx2: <p>example: <a href="https://github.com/pytorch/examples/blob/0984955bb8525452d1c0e4d14499756eae76755b/imagenet/main.py#L241" rel="nofollow noopener">https://github.com/pytorch/examples/blob/0984955bb8525452d1c0e4d14499756eae76755b/imagenet/main.py#L241</a></p>
<p>What is pth extension?<br>
Just curious, thanks.</p>

Kaixhin: <p>Don’t think it means anything in particular, it just seems to be a convention for files saved using <code>torch.save()</code>. Compare this to <code>.pck</code>, which is commonly used with Python’s built-in <code>pickle.dump()</code>.</p>

dgriff: <p>Personally I tend to use .dat data files to save and load models but as mentioned above .pth path files seem to be most prominent preference for saving models</p>

qbx2: <p>I use .pkl for pickle.dump, fine.<br>
.dat looks good.</p>

Allen_Huang: <p>Hmm, I wonder where this convention came from? Being new to pytorch, it was a bit confusing because .pth files are already used in the context of python as files in site packages that point to other directories on the system.</p>

marksisson: <p>pth file extension represents the P,T, and H in “PyTorcH”</p>
Training scripts

bayesianroad: <p>A general question on approaching /learning PyTorch.<br>
I started learning PyTorch something like 6 months ago. First I started by writing my own scripts based on some examples. However, I pretty fast understood that there is a large number of ways to structure code and a pretty large number of different tricks for training. There is PyTorch Lightning which tries to simplify training, there is fast.ai, there are training scripts that are provided by PyTorch developers themself, there are training scripts from timm.<br>
I would like to hear the opinions of the people who have more experience. Are there some training scripts that are multitasking suitable vision/NLP for standard tasks and/or which are commonly used? I do not want to reinvent the wheel by myself, but would rather use some already field-proven tools. I find timm scrips pretty complicated for me at this moment. fast.ai - easy to use, but hide a lot of important features behind the simple inferface.</p>
Why is Quantization on the GPU actually not supported?

fabian_schutze: <p>Why is the quantization on the GPU actually not supported?</p>
<p>The CPU quantization works really well and the basic quantization algorithms seem to be mature and on the conceptual level not related to any device. I understand that very large model present new challenges for quantization (outlier features) and I am also exclusively thinking of PTQ.</p>
<p>So, out of genuine curiosity: What makes GPU quantization different from CPU quantization? Why is it difficult to implement?</p>

supriyar: <p>we haven’t had a major use case for int8 quantization on GPU, since the speedup from fp16 seems to work for most models at inference. Moreover, for fast int8 inference there is a dependency on using a 3p backend like TensorRT or custom cuda/cudnn int8 kernels from Nvidia.</p>
<p>However, we are starting to look into enabling int8 quantization on GPU using triton in the context of speeding up transformer based models. cc <a class="mention" href="/u/cdhernandez">@cdhernandez</a></p>
Multi-dimension 1D FFT

FrankiePoon: <p>I am trying to convolve several 1D signals via FFT convolution. Does Pytorch offer any ways to avoid a for loop as below to perform a multi-dimension 1D FFT / iFFT, i.e. I would like to have a batch-wise 1D FFT?</p>
<pre><code class="lang-auto">import torch

# 1D convolution (mode = full)
def fftconv1d(s1, s2):
    # extract shape
    nT = len(s1)
    # signal length
    L = 2 * nT - 1
    # compute convolution in fourier space
    sp1 = torch.fft.fft(s1, L)
    sp2 = torch.fft.fft(s2, L)
    conv = torch.real(torch.fft.ifft(sp1 * sp2))
    return conv

A = torch.randn(20, 25)
B = torch.randn(20, 25)
conv = torch.randn(20, 25*2 -1)

for it in range(20):
    conv[it, :] = fftconv1d(A[it, :], B[it, :])


</code></pre>

ptrblck: <p>Your method should work if you fix the <code>nT</code> calculation and allows you to pass the inputs directly:</p>
<pre><code class="lang-python"># 1D convolution (mode = full)
def fftconv1d(s1, s2):
    # extract shape
    nT = s1.size(-1)#len(s1)
    # signal length
    L = 2 * nT - 1
    # compute convolution in fourier space
    sp1 = torch.fft.fft(s1, L)
    sp2 = torch.fft.fft(s2, L)
    conv = torch.real(torch.fft.ifft(sp1 * sp2))
    return conv

A = torch.randn(20, 25)
B = torch.randn(20, 25)
conv = torch.randn(20, 25*2 -1)

for it in range(20):
    conv[it, :] = fftconv1d(A[it, :], B[it, :])

out = fftconv1d(A, B)
print((conv - out).abs().max())
# tensor(6.2585e-07)
</code></pre>

FrankiePoon: <p>I didnt notice that. Great thanks!!!</p>
Is current pytorch 2.0 version stable?

bing: <p>Hi,<br>
I am trying to install current pytorch 2.0. Is it stable, on running the python version, I am getting the below version-</p>
<p><code>2.1.0.dev20230415</code></p>
<p>Is this the stable one,which everyone is talking about?<br>
Thanks.</p>

ptrblck: <p>On the <a href="https://pytorch.org/get-started/locally/">Get Started</a> page you will find the install matrix showing “Stable (2.0.0)” and “Preview (Nightly)”.<br>
You have installed a nightly release based on the version number which contains the date of the build (April 15, 2023, so today).</p>
RTX 4090 vs 2 RTX 3090

Miguel_Campos: <p>Just that, What would your choice be?</p>
<p>Currently, I already have one RTX3090. The options would be</p>
<ol>
<li>Add another 3090</li>
<li>Replace the 3090 by 4090</li>
</ol>
<p>Thanks</p>

JuanFMontesinos: <p>Short story, as long as your computer can power 2 gpus, buy a second 3090.<br>
Why:</p>
<ul>
<li>2 experiments in parallel → double fun</li>
<li>I think 3090 still have nvlink so you can get a huge gpu.</li>
<li>parallel training in 2 gpus (since they are the same it’s balanced)</li>
</ul>
<p>Buying a better one is just going to give u +20% more performance, which is okish. But again<br>
BE SURE YOUR COMPUTER CAN RUN BOTH<br>
this means you have enough RAM, a power-enough power supply, good cpu etc…</p>
Why is pytorch using only the half of cpu cores (SMT enabled)?

qbx2: <p>I’ve searched why, and it seems to be related to simultaneous multithreading (SMT) and OpenMP. <em>OMP_NUM_THREADS</em> is (num of cpu cores) / 2 by default(?). Is this behavior intended in pytorch? I don’t think that this will help increase performance…</p>

tom: <p>PyTorch has intra-operand and inter-operand parallelism. This means for a given op, you’d want not necessarily want to use all threads. If you have an application where you know you don’t need the latter, you can adjust the defaults.</p>
<p>Best regards</p>
<p>Thomas</p>

qbx2: <p>Thank you for answer! So it is an intended behavior. By the way, I don’t understand what intra-operand and inter-operand are. Could you please explain more?</p>

tom: <p>If you have 4 cores and need to do, say, 8 matrix multiplications (with separate data) you could use 4 cores to do each matrix multiplication (intra-op-parallelism). Or you could use a single core for each op and run 4 of them in parallel (inter-op-parallelism).<br>
In training, you also might want to have some cores for the dataloader, for inference, the JIT can parallelize things (I think).<br>
The configuration is documented here, but without much explanation: <a href="https://pytorch.org/docs/stable/torch.html#parallelism" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#parallelism</a></p>
<p>Best regards</p>
<p>Thomas</p>

Zhen_Liu: <p>Hi, thanks for the great discussion!</p>
<p>Is the choice between intra-op and inter-op determined by PyTorch internally? For me, I found that setting  <code>num_threads</code> or <code>num_interop_thread</code> to a larger number does not change the run time (or even slow down the run time), even though more CPU cores were involved in the computing.</p>
How to use nn.Transformer in the context of vision?

Robin_Lobel: <p>I’m quite familiar with CNN but not in combination with Transformers.</p>
<p>I’d like to include a Transformer as the central element of a UNet-like architecture, similar to:<br>
<a href="http://cs230.stanford.edu/projects_fall_2022/reports/59.pdf" rel="noopener nofollow ugc">DARE-Net: Speech Dereverberation And Room Impulse Response Estimation</a><br>
<a href="https://arxiv.org/pdf/2102.04306.pdf" rel="noopener nofollow ugc">TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation</a><br>
<a href="https://arxiv.org/pdf/2211.08553.pdf" rel="noopener nofollow ugc">HYBRID TRANSFORMERS FOR MUSIC SOURCE SEPARATION</a></p>
<p>My understanding is that after a few Convs I should flatten the output of the last encoding Conv, and pass it to a Transformer, then reshape the output of that Transformer back to a NCHW tensor and pass it to decoding Convs.</p>
<p>However reading <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html#torch.nn.Transformer" rel="noopener nofollow ugc">the documentation for nn.Transformer</a> I struggle to understand 2 points :</p>
<ul>
<li>a Transformer takes a SNE tensor. Am I understanding right that S should be a flat version of my 2D dimensions (HW) and E should be the hidden channels (C) ?</li>
<li>why should I provide both a Source and Target tensor to the Transformer, considering I just have an input tensor and expect an output tensor ? What is the meaning of those Source and Target ?</li>
</ul>
<p>Any clarification would be appreciated !</p>
Second order derivatives fail on some (simple) functions

Jesper_Nielsen: <p>I’m sorry, about the poor subject, but I don’t know how to describe this better. I’m trying to compute some second order derivatives.</p>
<p>If I do this:</p>
<pre><code class="lang-auto">import torch as tc
x = tc.tensor(0.0, requires_grad=True)
loss = x**2
g = tc.autograd.grad(loss, x, create_graph=True)[0]
g2 = tc.autograd.grad(g, x)[0]
</code></pre>
<p>It runs fine. Yay!</p>
<p>It also works well with, say, a sine:</p>
<pre><code class="lang-auto">import torch as tc
x = tc.tensor(0.0, requires_grad=True)
loss = tc.sin(x)
g = tc.autograd.grad(loss, x, create_graph=True)[0]
g2 = tc.autograd.grad(g, x)[0]
</code></pre>
<p>Yay!</p>
<p>But for some other functions this fails. For example:</p>
<pre><code class="lang-auto">import torch as tc
x = tc.tensor(0.0, requires_grad=True)
loss = x
g = tc.autograd.grad(loss, x, create_graph=True)[0]
g2 = tc.autograd.grad(g, x)[0]
</code></pre>
<p>Fails with:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>It also doesn’t work for:</p>
<pre><code class="lang-auto">import torch as tc
x = tc.tensor(0.0, requires_grad=True)
loss = x + 1
g = tc.autograd.grad(loss, x, create_graph=True)[0]
g2 = tc.autograd.grad(g, x)[0]
</code></pre>
<p>Any ideas?</p>

srishti-git1110: <p>Hey,</p>
<p>Please follow <a href="https://discuss.pytorch.org/t/second-order-differentiation-doesnt-work-for-x-1/177459/4">this forums post</a> for some discussion on this. You can maybe try putting follow up questions there itself to keep it all at one place <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to print out logs with LOG(INFO) in Python in PyTorch1.10

big_old: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8289454796623b84e3080254876a47796905c4d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e8289454796623b84e3080254876a47796905c4d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8289454796623b84e3080254876a47796905c4d_2_421x500.jpeg" alt="image" data-base62-sha1="x7LIh8ZU0j4HLEK4PwGDYmFM69T" width="421" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8289454796623b84e3080254876a47796905c4d_2_421x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8289454796623b84e3080254876a47796905c4d_2_631x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8289454796623b84e3080254876a47796905c4d_2_842x1000.jpeg 2x" data-dominant-color="252322"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1230×1460 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>TORCH_DISTRIBUTED_DEBUG=INFO</code> should work.</p>
OutOfMemoryError

surf_reta: <p>When running a model, I got the following error message</p>
<blockquote>
<p>OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 3.29 GiB already allocated; 0 bytes free; 3.44 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>How to fix this error, i.e., what should be the right setting for</p>
<blockquote>
<p>max_split_size_mb</p>
</blockquote>
<p>and</p>
<blockquote>
<p>PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>To debug this issue, are there any GPU-related information I can provide?</p>

ptrblck: <p>I would recommend trying to reduce the memory usage by e.g. lowering the batch size given the GPU memory is quite small with 4GB total memory.</p>
Need to use two pytorch_model.bin at the same time, can I renamed it?

BB1: <p>I need to use two different models in Spark and save them in the same folder. How can I deal with having both models with the same name pytorch_model.bin. If I rename one of them, how can I load the model?</p>

ptrblck: <p>Could you describe your use case a bit more and especially how Spark is used with PyTorch?<br>
Also, could you explain what <code>pytorch_model.bin</code> contains and how it’s created as it doesn’t seem to be a “standard” PyTorch <code>state_dict</code>?</p>
`GradScaler`+AMP causes NaN weights on backward pass

almeidaraul: <p><strong>TL;DR:</strong> when I use AMP <code>GradScaler</code> with two different losses (scaling each one separately), after about 100 epochs, training crashes due to NaN weights on backward.</p>
<p>I am trying to train a self-implemented DC-CDN, which uses two losses (Contrastive Depth Loss and Mean Squared Error).</p>
<p>In my implementation I’ve used <code>autocast</code> for both the forward function and the losses’ computation (in particular, if it helps, I use autocast as an annotator for both of these functions, so as to make sure it is never enabled in another moment during training).</p>
<p>I’ve also used <code>GradScaler</code>, initially summing both losses:</p>
<pre data-code-wrap="Python"><code class="lang-plaintext">scaler.scale(loss1+loss2).backward()
scaler.step(opt)
scaler.update()
</code></pre>
<p>However, as I’ve learned in the <a href="https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html#advanced-topics" rel="noopener nofollow ugc">AMP Recipe</a> this fits an advanced use case, so I’ve changed the above code to this:</p>
<pre data-code-wrap="Python"><code class="lang-plaintext"># each loss is scaled separately
scaler.scale(loss1).backward(retain_graph=True)
scaler.scale(loss2).backward()
scaler.step(opt)
scaler.update()
</code></pre>
<p>This was after reading <a href="https://github.com/pytorch/pytorch/issues/40497" rel="noopener nofollow ugc">this GitHub issue’s discussion</a>.</p>
<p>Disabling <code>GradScaler</code> or <code>autocast</code> (just one, or both) has allowed me to finish my experiments without crashing, but it is my understanding that this could lead to future issues (disabling <code>autocast</code> renders longer training times, which is not ideal, and no gradient scaling could correspond to <code>NaN</code> weights in certain datasets).</p>
<p>Also, the fact that the <code>GradScaler</code> step is not avoiding the <code>NaN</code> weights as it is hints at there something being wrong with my implementation.</p>

ptrblck: <aside class="quote no-group" data-username="almeidaraul" data-post="1" data-topic="177542">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ecccb3/48.png" class="avatar"> almeidaraul:</div>
<blockquote>
<p>training crashes due to NaN weights on backward.</p>
</blockquote>
</aside>
<p>The <code>backward</code> pass calculates the gradients and NaNs could be expected if the scaling factor is too high. The weights are not manipulated at all at this point and <code>scaler.step(optimizer)</code> will skip the parameter update if invalid gradients are detected or is this behavior not observed?</p>
Deep Coral Loss for semantic segmentation

Sarmad_GTU: <p>I have two image’ features outputted from the semantic segmentation feature extractor with the following sizes:<br>
image1 feature = [ 8,2048,16,16]<br>
image2 feature = [ 8,2048,16,16]</p>
<p>i want to compute the Deep Coral between these two features based on eq1 in this paper <a href="https://arxiv.org/abs/1607.01719" class="inline-onebox" rel="noopener nofollow ugc">[1607.01719] Deep CORAL: Correlation Alignment for Deep Domain Adaptation</a> which is for classification not semantic segmentation.</p>
<p>Any suggestions?</p>
Stationary time series

Aimen_Malik: <p><a href="https://github.com/Aimen1996/coursera-test/tree/main/site" rel="noopener nofollow ugc">link to repository</a><br>
I have lmdb file , what i am trying is  to analyze the time series data. I used adf test to check stationarity. when i do test on whole series, adf test give result as “stationary”. but when I make several chunks of series and perform adf test on it, some parts it gives series as “stationary” some parts as “non-stationary”. what should i assume what kind of series is it???<br>
I have attached the files to the data (as lmdb file) and code(adf-test) in the above mentioned link.(link to repository)</p>

Aimen_Malik: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> hello sir, please take a look at my query, i have seen you helping around people here,thanks in advance!</p>

ptrblck: <p>I’m not familiar enough with the Augmented Dickey-Fuller Test and my nave understanding would be that data splits could indeed show a trend while the overall data could be seen as “stationary”.<br>
I’m also not familiar with your use case and don’t know how you would use this information in PyTorch.<br>
In any case, maybe <a class="mention" href="/u/kfrank">@KFrank</a> or <a class="mention" href="/u/tom">@tom</a> would have a better idea how to interpret these results as mathematical experts.</p>

Aimen_Malik: <p>Basically data is pressure values on hydraulic supports installed in coal mining, recorded every second .I am using this data for time series forecasting using LSTM, transformer and non-stationary transformer. but results are poor from both transformer models, I believe I need to preprocess the data for it. I have already performed several transformation as shown in the repository.<br>
I would appreciate someone giving me a direction to deal with such kind of data. <a class="mention" href="/u/kfrank">@KFrank</a> <a class="mention" href="/u/tom">@tom</a></p>
Discrepancy in sum of matrix on mps and cpu (or its matrix view and vector view)

zeligism: <p>I faced a peculiar situation in which I want to sum the elements of a matrix W that resides on MPS using <code>W.sum()</code>. I will describe the problem first, and then I will try to describe the exact operations that resulted in this matrix at the end of this post.</p>
<p>The problem is that <code>W.sum()</code> returns a value that is clearly false (clearly in my case). For example,  we have the following:</p>
<ol>
<li>
<code>W.sum() == W.view(-1).sum()</code> returns False.</li>
<li>
<code>W.view(-1).view(*W.shape).sum() == W.view(-1).sum()</code> returns False.</li>
<li>
<code>W.sum() == W.cpu().sum()</code> returns False.</li>
<li>
<code>W.cpu().to('mps').sum() == W.cpu().sum()</code> returns False!</li>
</ol>
<p>In other words, we have:</p>
<ul>
<li>
<code>W.sum() == W.view(-1).view(*W.shape).sum() == W.cpu().to('mps').sum() == wrong_value</code>, and</li>
<li>
<code>W.view(-1).sum() == W.cpu().sum() == W.sum(1).sum() == W.sum(0).sum() == true_value</code>.</li>
</ul>
<p>This behavior extends to many other functions that reduce a matrix to a single value (e.g. <code>norm()</code> and <code>count_nonzero()</code>), so something is clearly off with the <strong>matrix view of W on MPS device</strong>.</p>
<p>The code that produced such a matrix in my case can be summarized as follows: for each param in a model, create W a matrix view of the param, create W_new a new matrix like W, run some operations on W_new, return param_new a param view of W_new, and copy it onto param <code>param.copy_(param_new)</code>. This all runs under <code>torch.no_grad()</code> context.</p>

ptrblck: <p>I assume you are comparing floating point values directly, which might fail due to their limited precision and a potentially different order of operations.<br>
Here is a small example using <code>sum</code> and forcing a change in the order of operations:</p>
<pre><code class="lang-python">x = torch.randn(100, 100)
s1 = x.sum()
s2 = x.sum(0).sum()
print((s1 - s2).abs().max())
# tensor(1.5259e-05)
</code></pre>
<p>How large are the errors you are seeing in your use case?</p>

zeligism: <p>Sorry for the late reply. The summation itself is not the issue here. Honestly, I was counting non-zeros, so floating point might not explain the issue (the summation example still holds). Also, floating points don’t explain the slightly different number of nonzeros I get when I simply move my model to cpu (or do they? I assume cpu case produces the true value). I will try to come up with a reproducible example but it’s very tricky as I do not exactly know what led to this behavior during training. FYI I have a MacBook Pro (16-inch, 2021) with Apple M1 Pro and PyTorch 2.0 installed.</p>
Loading data in list of lists form

AleTL: <p>Hello!</p>
<p>I want my dataloader to return me a list of size 3, in which each element is likewise a list, in the following form:</p>
<pre><code class="lang-auto">[[tensor1, tensor2], [tensor3, tensor4], [tensor5, tensor6, tensor7]]
tensor1.shape = tensor3.shape
tensor2.shape = tensor4.shape
tensor1.shape != tensor2.shape
tensor5.shape != tensor6.shape != tensor7.shape
</code></pre>
<p>As you can see, it’s a bit intricate format but it’s needed this way. I am using PytorchLightning and in the .fit function, I receive the following error: <code>each element in list of batch should be of equal size</code>. If I use <code>colllate_fn=lambda x:x</code> the problem doesn’t appear but the format of my batch changes and I receive a list with all elements in the batch (<code>len(batch) = batch_size</code>).<br>
Nevertheless, what I want is that each element of the list I show above have a dimension <code>batch_size x d</code>.</p>
<p>How can I do it? Cannot this be done with PyTorch?</p>
<p>Thanks a lot!</p>

ptrblck: <p>Your approach seems to work for me:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        pass
    
    def __len__(self):
        return 10
    
    def __getitem__(self, index):
        tensor1 = torch.randn(2)
        tensor2 = torch.randn(3)
        tensor3 = torch.randn(2)
        tensor4 = torch.randn(3)
        tensor5 = torch.randn(4)
        tensor6 = torch.randn(5)
        tensor7 = torch.randn(6)
        return [[tensor1, tensor2], [tensor3, tensor4], [tensor5, tensor6, tensor7]]
    
dataset = MyDataset()
loader = DataLoader(dataset, batch_size=5)

for a, b, c in loader:
    print(a[0].shape) # torch.Size([5, 2])
    print(a[1].shape) # torch.Size([5, 3])
    print(b[0].shape) # torch.Size([5, 2])
    print(b[1].shape) # torch.Size([5, 3])
    print(c[0].shape) # torch.Size([5, 4])
    print(c[1].shape) # torch.Size([5, 5])
    print(c[2].shape) # torch.Size([5, 6])
</code></pre>

AleTL: <p>Hi, thanks for answering!</p>
<p>It doesn’t work when I used PytorchLightning, I receive the mentioned error. Sorry, I should have specified, I thought it was something that would happen no matter what.</p>
Torchviz subtraction node with one input

Jan_Egil_Glaestad: <p>Note: posted on stackoverflow also, but no takers.</p>
<p>Ive used torchviz to visualize a pytorch graph. One of the nodes have one line in and one line out. Node text “subbackward0 … alpha 1”. Am i rigtht to assume it does nothing?</p>
<p>I did not expect only one line into a sub-box, i expected minimum 2.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ef0c48d912ca5d131f0003db0a085adbc91e9f6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4ef0c48d912ca5d131f0003db0a085adbc91e9f6" title="6WJ1H"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4ef0c48d912ca5d131f0003db0a085adbc91e9f6_2_470x500.jpeg" alt="6WJ1H" data-base62-sha1="bgl5hSle9ymPU142UW8HFMyznfM" width="470" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4ef0c48d912ca5d131f0003db0a085adbc91e9f6_2_470x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4ef0c48d912ca5d131f0003db0a085adbc91e9f6_2_705x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4ef0c48d912ca5d131f0003db0a085adbc91e9f6_2_940x1000.jpeg 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">6WJ1H</span><span class="informations">1017×1081 33.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How do I code dataset to extract patches in 3D image

gali: <p>Hi,<br>
I have a question about the extraction of patches in 3D images.<br>
Say I have many 3D images with different resolutions and for each one, I would extract different numbers of fixed size patches(40<em>40</em>40). How should I code the <strong>getitem</strong> so that the dataloader would extract patches one image by one image and the I can get (batch_size, channel, 40, 40, 40) when enumerating dataloader.</p>
<p>Thanks!</p>
RuntimeError:. DefaultCPUAllocator: not enough memory: you tried to allocate 12726195328 bytes

zhaofan249: <p>train_data_one_func = train_data_one_func.float()</p>
<p>There is an error when I run this line. I try to batch it, but I still run out of memory.</p>
<p>Note: My computer has 48G RAM</p>
<pre><code>     train_data_one_func is a [120000,41,616] tensor
</code></pre>
<p>Can someone tell me how to solve this problem</p>

ptrblck: <p>Your method tries to allocate ~12GB and it seems your host RAM of 48GB might not be enough.<br>
You would thus need to reduce the memory usage by e.g. decreasing the batch size of your input.</p>

zhaofan249: <p>Thank you for answering my question.<br>
But when I run the following code, I run out of memory no matter how small batc_size is. Here is the code for batc_size=1000 and the error message</p>
<pre><code class="lang-auto">batch_size = 1000
batches = [train_data_one_func[i:i+batch_size, :] for i in range(0, train_data_one_func.size(0), batch_size)]

for i in range(len(batches)):
       batches[i] = batches[i].float()
</code></pre>
<p>Error on this line    batches[i] = batches[i].float()<br>
RuntimeError: DefaultCPUAllocator: not enough memory: you tried to allocate 101024000 bytes.``</p>

ptrblck: <p>You would need to check how much free RAM your system has, e.g. via monitoring <code>htop</code>, and make sure your data processing doesn’t fill it all up.</p>
DataLoader object not callable when given as function argument

vthieb: <p>Hi everyone!</p>
<p>I recently tried to use a public repo that you can find here:<br>
<a href="https://github.com/manigalati/usad" rel="noopener nofollow ugc">USAD</a></p>
<p>I have an issue when using the <code>training</code> function. I first create my own train and val loaders that I give as arguments:<br>
<code>training(epochs, model, train_loader, val_loader, opt_func=torch.optim.Adam)</code></p>
<p>It raises an error <code>TypeError: 'DataLoader' object is not callable</code>.</p>
<p>The thing is that this error is supposed to be raised by the line:<br>
<code>optimizer1 = opt_func(list(model.encoder.parameters()) + list(model.decoder1.parameters()))</code></p>
<p>Even weirder, when I try to call these lines in a notebook (not with the <code>training</code> function), this error won’t occur.</p>
<p>Does anyone have an idea about what could be happening when calling the function?</p>
<p>Thanks a lot for your help!</p>

ptrblck: <p>You are most likely passing the wrong number of arguments to this function and the <code>DataLoader</code> objects is mapped to the <code>opt_func</code> argument thus raising this error.</p>

vthieb: <p>You are right, this was simply this: the <code>training</code> function was called with an additional argument which was not defined in the version I had.</p>
<p>Thank you very much!</p>
RuntimeError: DataLoader worker exited unexpectedly while using ThreadPoolExecutor in concurrent.futures

ztysdu: <p>Traceback (most recent call last):<br>
File “/home/sduu1/userspace/zhangtongyu/DEEPVO_KITTI/deepvo_direct_on_real2/main.py”, line 194, in <br>
futures = [executor.submit(test_meanwhile,M_deepvo, test_video,device) for test_video in test_video_list]<br>
File “/home/tongyuz/.conda/envs/deepvo_zty/lib/python3.9/concurrent/futures/_base.py”, line 637, in <strong>exit</strong><br>
self.shutdown(wait=True)<br>
File “/home/tongyuz/.conda/envs/deepvo_zty/lib/python3.9/concurrent/futures/thread.py”, line 235, in shutdown<br>
t.join()<br>
File “/home/tongyuz/.conda/envs/deepvo_zty/lib/python3.9/threading.py”, line 1060, in join<br>
self._wait_for_tstate_lock()<br>
File “/home/tongyuz/.conda/envs/deepvo_zty/lib/python3.9/threading.py”, line 1080, in _wait_for_tstate_lock<br>
if lock.acquire(block, timeout):<br>
File “/home/tongyuz/.conda/envs/deepvo_zty/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 15516) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.<br>
Dear all,<br>
I tried to use mutiple thread(3 threads)  when test the model.In most time, it went on perfectly well.However ,sometime it threw this error sometimes.I wonder why this occurs?</p>
Pruning Language Model failure

Goldname: <p>I am testing out pruning on the DialoGPT LLM from Hugging Face</p>
<p>My pruning appears to succeed but the model outputs haven’t changed much. Does anyone know what the problem is here?</p>
<pre><code class="lang-auto">!pip3 install tqdm boto3 requests regex sentencepiece Transformers sacremoses huggingface_hub tokenizers

import torch

import logging

from transformers import pipeline
from transformers import AutoTokenizer
from transformers import AutoModelForCausalLM
import transformers
from PIL import Image
import requests
from transformers import Blip2Processor, Blip2Model
import torch.nn.utils.prune as prune

model = AutoModelForCausalLM.from_pretrained("microsoft/DialoGPT-medium")
module = model.transformer.h[0].attn.c_attn
prune.random_unstructured(module, name="weight", amount=1.0)
tokenizer = AutoTokenizer.from_pretrained("microsoft/DialoGPT-medium")

# Let's chat for 5 lines
for step in range(5):
    # encode the new user input, add the eos_token and return a tensor in Pytorch
    new_user_input_ids = tokenizer.encode(input("&gt;&gt; User:") + tokenizer.eos_token, return_tensors='pt')

    # append the new user input tokens to the chat history
    bot_input_ids = torch.cat([chat_history_ids, new_user_input_ids], dim=-1) if step &gt; 0 else new_user_input_ids

    # generated a response while limiting the total chat history to 1000 tokens, 
    chat_history_ids = model.generate(bot_input_ids, max_length=1000, pad_token_id=tokenizer.eos_token_id)

    # pretty print last ouput tokens from bot
    print("DialoGPT: {}".format(tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True)))
</code></pre>
Using FetchContent_Declare in CMake to pull LibTorch

B4D_0M3N: <p>As the title says, I am trying to use <code>FetchContent_Declare</code> in the <code>CMakeLists.txt</code>, in order to download <code>LibTroch</code> directly, without further “manual” configuration on Windows.<br>
So, in order to do this, I copied the <a href="https://pytorch.org/cppdocs/installing.html" rel="noopener nofollow ugc">example</a> in the documentation and changed the <code>CMakeLists.txt</code> file to the following:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.25 FATAL_ERROR)
project(exampleApp)

# Set the C++ standard
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED True)

include(FetchContent)

FetchContent_Declare(
        Torch
        URL https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-2.0.0%2Bcpu.zip
        # URL https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-debug-2.0.0%2Bcpu.zip
        # OVERRIDE_FIND_PACKAGE
)
FetchContent_MakeAvailable(Torch)
list(APPEND CMAKE_PREFIX_PATH "${torch_SOURCE_DIR}")

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(exampleApp main.cpp)
target_link_libraries(exampleApp "${TORCH_LIBRARIES}")
set_property(TARGET exampleApp PROPERTY CXX_STANDARD 14)

# The following code block is suggested to be used on Windows.
# According to https://github.com/pytorch/pytorch/issues/25457,
# the DLLs need to be copied to avoid memory errors.
if (MSVC)
    file(GLOB TORCH_DLLS "${TORCH_INSTALL_PREFIX}/lib/*.dll")
    add_custom_command(TARGET exampleApp
            POST_BUILD
            COMMAND ${CMAKE_COMMAND} -E copy_if_different
            ${TORCH_DLLS}
            $&lt;TARGET_FILE_DIR:exampleApp&gt;)
endif (MSVC)
</code></pre>
<p><code>cmake</code> appears to run without any issue, but when compiling, I am getting the following errors:</p>
<p>And the following errors when compiling:</p>
<pre><code class="lang-auto">CLionProjects\untitled\cmake-build-release-visual-studio\_deps\torch-src\include\c10/util/safe_numerics.h(26): error C3861: '_addcarry_u64': identifier not found
</code></pre>
<p>Any idea what how to fix this?</p>
<p><strong>PS:</strong> I know about the <code>OVERRIDE_FIND_PACKAGE</code> flag, but couldn’t get <code>cmake</code> to work with it.</p>
<p><strong>Edit:</strong> Just tried this <code>CMakeLists.txt</code> file on linux and it worked perfectly.</p>
Pruning: "cannot assign x object to parameter"

Goldname: <p>Can someone explain what this error message means?</p>
<blockquote>
<p>TypeError: cannot assign ‘transformers.pytorch_utils.Conv1D’ object to parameter ‘c_attn_orig’ (torch.nn.Parameter or None required)</p>
</blockquote>
<p>I am attempting to use the pytorch pruning function to prune some pre-trained models from HuggingFace</p>
<pre><code class="lang-auto">!pip3 install tqdm boto3 requests regex sentencepiece Transformers sacremoses huggingface_hub tokenizers
import torch
import logging
from transformers import pipeline
from transformers import AutoTokenizer
from transformers import AutoModelForCausalLM
import transformers
import torch.nn.utils.prune as prune

model = AutoModelForCausalLM.from_pretrained("microsoft/DialoGPT-medium")
module = model.transformer.h[0].attn
# print(model)
# print(list(module.named_parameters()))
prune.random_unstructured(module, name="c_attn", amount=0.3)
</code></pre>
<p>Why am I unable to prune the conv1d layer? Why doesn’t prune convert it to the “torch.nn.Parameter” type in order to assign the “_orig” name to it?</p>