akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to compile from tarball downloaded from github?

liangstein: <p>It may be a simple question. I downloaded the tarball from github releases page, however the  tarball doesn’t work. How to make it work ?</p>
<p>(eee) [ pytorch-0.4.1]$ python setup.py  install<br>
fatal: not a git repository (or any of the parent directories): .git<br>
running install<br>
running build_deps<br>
Could not find /home/liangstein/test_tarballs/4.1/pytorch-0.4.1/third_party/gloo/CMakeLists.txt<br>
Did you run ‘git submodule update --init’?<br>
(eee) [ pytorch-0.4.1]$ git submodule update --init<br>
fatal: not a git repository (or any of the parent directories): .git<br>
(eee) [ pytorch-0.4.1]$</p>

tom: <p>I don’t think it would work. It probably is best to check out the git branch v0.4.1 and use that.<br>
If you clone and then do<br>
<code>git checkout -b v0.4.1 origin/v0.4.1</code><br>
should do the trick.<br>
Then you can run the <code>git submodule update --init</code> as you suggested.</p>
<p>Best regards</p>
<p>Thomas</p>

liangstein: <p>After I do<br>
<code>git checkout -b v0.4.1 origin/v0.4.1</code><br>
and run<br>
python setup.py install</p>
<p>However “pip list” shows the wrong torch version:<br>
torch      0.5.0a0+a24163a</p>
<p>How does this happen?</p>

tom: <p>That just means its legit <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"> :<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/9926" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/EKami" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/4030626?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/9926" target="_blank" rel="nofollow noopener">Issue: Wrong version number for the 0.4.1 release</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/EKami" target="_blank" rel="nofollow noopener">EKami</a>
	on <a href="https://github.com/pytorch/pytorch/issues/9926" target="_blank" rel="nofollow noopener">2018-07-27</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/soumith" target="_blank" rel="nofollow noopener">soumith</a>
	on <a href="https://github.com/pytorch/pytorch/issues/9926" target="_blank" rel="nofollow noopener">2018-07-27</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">Issue description
The __version__ provided by Pytorch is wrong
See this line</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Best regards</p>
<p>Thomas</p>
Relationship between CMake build and python setup.py

jiapei100: <p>Sorry to bother everyone again, but I really need to make myself understand the building structure of PyTorch.</p>
<ul>
<li>CMake out-of-source buld: alway can ben done successfully, after my manual configuration.</li>
<li>
<code>python setup.py build --cmake</code> tried to build PyTorch independelty, which seems to ignore ALL my out-of-source cmake configuration, even if with <code>--cmake</code> configuration. And, it looks <code>python setup.py build --cmake</code> wouldn’t let me manually configure the cmake configurations.</li>
</ul>
<p>Again, my questions:</p>
<ol>
<li>Is there any possibility that when I tried to do <code>python setup.py build --cmake</code>, it would force to use cmake out-of-source configuration inside subfolder <strong>build</strong> ?</li>
<li>Is there a similar command that use <strong>ccmake</strong>? say: <code>python setup.py build --ccmake</code> ??</li>
<li>I can clearly tell that by using <strong>cmake build</strong>, I already successfully built out <strong>libtorch_python.so</strong>. So, how to use PyTorch in python, without building <strong>PyTorch</strong> by running <code>python setup.py build -cmake</code> and install again? Is it possible? I mean: with <code>libtorch_pytorch.so</code>, can I load torch module from within python script?</li>
</ol>
<p>Thank you very much…</p>

jiapei100: <p>I got it… Problem solved…<br>
It looks I can do</p>
<ol>
<li>CMake build FIRST</li>
<li><code>TORCH_CUDA_ARCH_LIST="8.6" NO_TEST=1 USE_MKLDNN=0 FULL_CAFFE2=1 python setup.py bdist_wheel</code></li>
</ol>
Custom conv2d algorithm c++ extension

71M073J: <p>Hello, I’m implementing <a href="https://arxiv.org/abs/1504.08362" rel="noopener nofollow ugc">perforated convolution</a> for use during training; (essentially, strided convolution with very specific upscaling to restore output size) With python i can (poorly) simulate the algorithm with strided and transpose conv2d, but i want both performance and exact control over what the algorithm is doing, which is even more important during backward pass.</p>
<p>As of right now, i want to focus on cpu-only implementation, as the control over what is happening is more important than raw compute performance.</p>
<p>I have an issue as i have never used the c++ extensions, and don’t exactly know how to go about doing it. The tutorial extension for <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">LLTM</a> seems to work, copying the <a href="https://github.com/pytorch/pytorch/blob/8d08b103be936d78d5d4ed90c0547aeccb8ce166/aten/src/ATen/native/ConvolutionMM2d.cpp" rel="noopener nofollow ugc">ConvolutionMM2d.cpp</a> and binding the forward/backward functions does not.</p>
<p>JIT compiling fails with errors such as:<br>
<code>In function ‘at::Tensor&amp; at::native::thnn_conv2d_out(const at::Tensor&amp;, const at::Tensor&amp;, at::IntArrayRef, const std::optional&lt;at::Tensor&gt;&amp;, at::IntArrayRef, at::IntArrayRef, at::Tensor&amp;)’: /mnt/c/Users/timotej/pytorch/test.cpp:611:26: error: ‘thnn_conv2d_forward_out’ is not a member of ‘at’; did you mean ‘slow_conv3d_forward_out’?   611 |   return std::get&lt;0&gt;(at::thnn_conv2d_forward_out(output, finput, fgrad_input, self, weight, kernel_size, bias, stride, padding));       |                          ^~~~~~~~~~~~~~~~~~~~~~~       |                          slow_conv3d_forward_out</code></p>
<p>or</p>
<p><code> error: cannot convert ‘at::Tensor’ to ‘c10::ScalarType’ in argument passing</code></p>
<p>So, i suppose i am asking the following:</p>
<ul>
<li>am i doing something fundamentally wrong in making this an extension?</li>
<li>general advice for extensions</li>
<li>is the above MM2d file (and then going into unfolding the tensor) the best way of making “custom” convolution operation?</li>
</ul>
How to check if weight of a linear layer is in computational graph

shivam_singh: <p>I have a network which has one linear layer. I made one forward pass then how to check if weight of that linear layer is in computational graph? As in tensor .grad_fn None means the tensor is not in computation graph, but mod.linear.weight.grad_fn is None even after forward pass <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c960f4d6425daf81b4a417cc2c73f56b22fae25a.png" data-download-href="https://discuss.pytorch.org/uploads/default/c960f4d6425daf81b4a417cc2c73f56b22fae25a" title="Screenshot 2024-05-21 at 11.35.37 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c960f4d6425daf81b4a417cc2c73f56b22fae25a_2_579x500.png" alt="Screenshot 2024-05-21 at 11.35.37 AM" data-base62-sha1="sJtHnyDPzaLI5Ig6gDDX5zEHvGG" width="579" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c960f4d6425daf81b4a417cc2c73f56b22fae25a_2_579x500.png, https://discuss.pytorch.org/uploads/default/original/3X/c/9/c960f4d6425daf81b4a417cc2c73f56b22fae25a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/9/c960f4d6425daf81b4a417cc2c73f56b22fae25a.png 2x" data-dominant-color="1C1C1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-05-21 at 11.35.37 AM</span><span class="informations">716×618 30.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/shivam_singh">@shivam_singh</a>,</p>
<p>In order to populate the <code>.grad</code> attribute of a weight, you need to first compute some loss value using your model, i.e.,</p>
<pre><code class="lang-auto">x = torch.randn(4,100) #batch_size=4
y = net(x) #compute output
loss = torch.mean(y) #take mean over all output/batch
loss.backward() #compute gradients (and populate .grad attributes)
</code></pre>
<p>Then see if the gradient is in <code>self.lay1</code>,</p>
<pre><code class="lang-auto">print(mod.lay1.weight.grad)
</code></pre>
<p>The particular <code>grad_fn</code> the Tensor will used is determined in the backward pass of your model and it is initialized as <code>None</code>.</p>

ptrblck: <p>In addition to <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>’s answer: parameters are leaf variables and will thus not get a <code>grad_fn</code> assigned to them since they were explicitly created. Intermediate tensors will set their <code>grad_fn</code> during the computation graph creation.</p>
saved_tensors_hooks auto delete custom attributes

Yang_Wei: <p>I have a custom attribute defined in the saved tensors. But after unpacking, the attribute is deleted by the hook. How can I obtain the attribute?</p>
<pre><code class="lang-auto">import torch

class f(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, w):
        y = x @ w
        ctx.save_for_backward(x, w)
        return y

    def backward(ctx, grad_output):
        x, w = ctx.saved_tensors
        print(hasattr(x, "aaa"))  # the custom attribute is deleted
        return grad_output @ w.t(), x.t() @ grad_output

def pack_hook(tensor):
    tensor.aaa = 1
    return tensor
        
def unpack_hook(tensor):
    print(tensor.aaa)  # the custom attribute still exists before unpack returns
    return tensor

x = torch.randn(100, 200).bfloat16().cuda().requires_grad_()
w = torch.randn(200, 300).bfloat16().cuda().requires_grad_()
with torch.autograd.graph.saved_tensors_hooks(pack_hook, unpack_hook):
    y = f.apply(x, w).sum()
    y.backward()
</code></pre>
<p>The result is:</p>
<pre><code class="lang-auto">1
1
False
</code></pre>
Encountered Errors while Deploying LibTorch on Qt Creator

baozi: <p>The downloaded package is libtorch-win-shared-with-deps-2.3.0+cpu. It loads normally on the local Visual Studio, but when the corresponding lib library is added to the .pro file of Qt, along with the corresponding header files, and after referencing <span class="hashtag-raw">#include</span> “torch/script.h”, compilation errors occur. The error messages are as follows:</p>
<ol>
<li>D:\software\libtorch\include\ATen\core\ivalue_inl.h:1572: error: C2059: syntax error: “”</li>
<li>D:\software\libtorch\include\ATen\core\ivalue_inl.h:1572: error: C2334: unexpected token before ‘{’; skipping apparent function body</li>
<li>D:\software\libtorch\include\ATen\core\ivalue_inl.h:1788: error: C2760: syntax error: unexpected token ‘(’; expected ‘ID expression’ D:\software\libtorch\include\ATen/core/ivalue_inl.h(1788): error C2760: syntax error: unexpected token ‘(’; expected ‘ID expression’ D:\software\libtorch\include\ATen/core/ivalue.h(500): note: This diagnostic occurred in the compiler generated function “c10::intrusive_ptr&lt;T,c10::detail::intrusive_target_default_null_type&gt; c10::IValue::toCustomClass(void) &amp;&amp;”.</li>
</ol>
Efficient custom backward in 2024

arogozhnikov: <p>Hi, I am trying to implement somwhat optimal autograd function for fused layer norm and linear weight. I I can in theory write it manually using <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">provided instructions</a>, but …</p>
<p>But here is the gotcha - that will be wildly inefficient, as fused layer norm is much faster than manual. Same for backward. Fused layernorm + linear is more efficient, but seemingly I can’t use it either.</p>
<p>My dream implementation would be (very pseudocode-ish)</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(x):
    ctx.save_for_backward(x) # store only input for bwd
    y = F.layer_norm(x, None, None)
    out = F.linear(y, W, b) 
    # would be perfect to use fused kernel for the prev two lines if possible
    return out 

def backward(out_grad):
    x = ... # retrieved from ctx
    y = F.layer_norm(x, None, None)
    y_grad, W_grad, b_grad = F.linear.backward(x, W, b, x_grad) # or something similar
    x_grad, _, _ = F.layer_norm.backward(x, None, None, y_grad) # or something similar
    # return gradients
</code></pre>
<p>So my point is – I want to use fused kernels, same as used internally by torch, but I can’t find a way to use those directly.</p>
Unexpected NaN in forward pass when using flash_attn

a719591339: <p>I`m using flash_attn as following:</p>
<pre><code>                q, q_indices, cu_q_lens, q_max_s = unpad_input(q, x_mask)
                kv, _, cu_kv_lens, kv_max_s = unpad_input(kv, source_mask)
                q, kv = q.view(-1, self.nhead, self.head_dim).bfloat16(), kv.view(-1, 2, self.nhead, self.head_dim).bfloat16() # [N_unmasked, 1(2), H, D]
                dropout_p = 0.1 if self.training else 0.0
                message = flash_attn_varlen_kvpacked_func(
                    q, kv, cu_q_lens, cu_kv_lens, q_max_s, kv_max_s, dropout_p, softmax_scale=None)
                 check_nan(message, 'message1_attn_self', {'q': q, 'kv': kv, 'cu_q_lens': cu_q_lens, 'cu_kv_lens': cu_kv_lens, 'q_max_s': q_max_s, 'kv_max_s': kv_max_s, 'x_mask': x_mask, 'source_mask': source_mask}) # ignore this line for now
</code></pre>
<p>after a few steps of training, the message contains NaN, confirmed by the func check_nan:</p>
<pre><code>def check_nan(x, name, save_dict={}):
    if isinstance(x, list) or isinstance(x, tuple):
        for xx in x:
            check_nan(xx)
    if torch.isnan(x).any():
        save_dict[name] = x
        torch.save(save_dict, 'nan_check_tr.pt')
        raise ValueError(f'nan detected in {name}')
</code></pre>
<p>The code is running on a remote cluster, so I can not debug it directly. So I saved the context vars when NaN is detected.</p>
<p>However, when I try to use the saved nan_check_tr.pt to reproduce the process, there is no NaN.<br>
pt = torch.load(‘nan_check_tr.pt’, map_location=‘cuda’)<br>
m = flash_attn_varlen_kvpacked_func(pt[‘q’], pt[‘kv’], pt[‘cu_q_lens’], pt[‘cu_kv_lens’], pt[‘q_max_s’], pt[‘kv_max_s’], softmax_scale=None)<br>
torch.isnan(m).any()</p>
<pre><code>&gt;&gt;&gt; tensor(False, device='cuda:0')
</code></pre>
<p>I`m sure the environment I used is totally the same(the same docker image). I have no idea what is going on. I have tried different learning rate such as 5e-4, 3e-4, 1e-4. A small lr can train more steps without NaN, but it will  be there anyway.</p>
<p>The saved file is here <a href="https://drive.google.com/file/d/1tui4ed9AITCvmBY4MOnMMIJsjxk0Cakz/view?usp=drive_link" rel="noopener nofollow ugc"> <code>nan_check_tr.pt</code></a></p>
<p>Some important environment:</p>
<p>docker built on: nvidia/cuda:11.6.2-cudnn8-devel-ubuntu20.04</p>
<p>torch: 1.13.1</p>
<p>flash-attn: 2.2.2</p>
<p>pytorch-lightning: 1.9.0</p>
<p>GPU: one single A100(debug)/two nodes with 8 A100 each(training)</p>
<p>Any suggestion is welcome! Thx!</p>

tete: <p>Hello, have you figured this bug out? I met a similar problem, and I am looking forward to any suggestions or insight. Thx!</p>
(Torch Geometric) Issues With Batching

FezTheImmigrant: <p>I am trying to use the Data and DataLoader specified by Torch Geometric; however, I’m running into issues with the size of my data. The size of edge_idx varies for each graph in my dataset as seen below:</p>
<p><code>[Data(x=[43, 10], y=[43, 10], edge_idx=[2, 90]), Data(x=[43, 10], y=[43, 10], edge_idx=[2, 88]), Data(x=[43, 10], y=[43, 10], edge_idx=[2, 92])]</code></p>
<p>Given data of this form, I follow the boilerplate template given here <a href="https://pytorch-geometric.readthedocs.io/en/latest/tutorial/create_dataset.html#creating-larger-datasets:" class="inline-onebox" rel="noopener nofollow ugc">Creating Graph Datasets — pytorch_geometric documentation</a></p>
<pre><code class="lang-auto">
from torch_geometric.data import Data
from torch_geometric.loader import DataLoader

data_list = [Data(...), ..., Data(...)]
loader = DataLoader(data_list, batch_size=32)
</code></pre>
<p>I get the following error, however:</p>
<pre><code class="lang-auto">RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 90 but got size 88 for tensor number 7 in the list.
</code></pre>
<p>This is confusing because the link specifies:</p>
<blockquote>
<p>Here, adjacency matrices are stacked in a diagonal fashion (creating a giant graph that holds multiple isolated subgraphs), …</p>
</blockquote>
<p>I know that I should be able to batch with differing edge_idx sizes per graph, I just don’t know how. Any help would be appreciated!</p>
Best place to subscribe for a CUDA function callback in Pytorch

orangemaru: <p>I am analyzing an undocumented CUDA API in PyTorch, and I want this function to react to each kernel launch during inference/training workloads. I can retrieve this function using a callback from <code>cudaGetExportTable</code> outside PyTorch in a simple test:</p>
<p>For example:<br>
static const CUuuid callbackFunid = …;<br>
// Get the function from the table<br>
cuGetExportTable((const void**)&amp;subscribeFuncAddr, &amp;callback_funcs_id);<br>
// ‘subscribe’ is a function defined elsewhere that has the same signature as the callback function<br>
subscribe = (typeof(subscribe))subscribeFuncAddr;<br>
// ‘res’ equals 0 if successful<br>
res = subscribe(&amp;my_hndl, f, NULL);</p>
<p>I first tested this code for some elementwise kernels before their launch (e.g., before <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cuda/CUDALoops.cuh#L192C3-L192C21" rel="noopener nofollow ugc">this line in CUDALoops.cuh</a>). However, when I reran my workload, I encountered CUDA error code 999 when I tried to subscribe to the function callback (from <code>cuGetExportTable</code>). This indicates that I didn’t retrieve the function properly. I assume this may be because the CUDA driver environment has not been initialized yet.</p>
<p>I then tried to insert my code <a href="https://github.com/pytorch/pytorch/blob/main/c10/cuda/CUDAStream.cpp#L173" rel="noopener nofollow ugc">here in CUDAStream.cpp</a>. It seems that I can get the function callback, but when I profile the workload, I get <code>CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED</code>.</p>
<p>Do you know the best place to insert these codes before launching each kernel in PyTorch? I would appreciate any suggestions and comments.</p>
Torch.linalg.svd() backpropagation

Am_b_reas: <p>Hello,<br>
I have a model that calculates the svd of a matrix in the forward and i’m getting this error in the backward, any idea about how to solve it ?</p>
<pre><code class="lang-auto">RuntimeError: svd_backward: The singular vectors in the complex case are specified up to multiplication by e^{i phi}. The specified loss function depends on this phase term, making it ill-defined.

</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/am_b_reas">@Am_b_reas</a>,</p>
<p>The gradient of an SVD decomposition isn’t defined with respect to the phase as it’s degenerate, and therefore its gradient is ill-defined.</p>
<p>There’s some more info here: <a href="https://discuss.pytorch.org/t/function-linalgsvdbackward0-returned-nan-values-in-its-0th-output/190410" class="inline-onebox">Function ‘LinalgSvdBackward0’ returned nan values in its 0th output</a></p>
<p>There’s also more info in the docs for SVD in case of ill-defined operations too: <a href="https://pytorch.org/docs/stable/generated/torch.linalg.svd.html" class="inline-onebox" rel="noopener nofollow ugc">torch.linalg.svd — PyTorch 2.3 documentation</a></p>
One question: why does not FSDP shard all paramters into each rank in average?

jie_yang: <p>When I use FSDP in pytorch, I print each rank’s parameters and calculate parameter’s number, I find different ranks has different param’s number?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/98cdebe6168182a1468485db5c9f0adf8d8951f2.png" alt="image" data-base62-sha1="lNLL8nkWujqfA5Ob2CRkHFssVPQ" width="197" height="95"></p>
<p>some rank has more sharding parameter? why pytorch design it?</p>
Does Pytorch provide MobileNetv3Small (Large) with smaller alpha?

_lovepytorch: <p>I saw that Pytorch method <code>torchvision.models.mobilenet_v3_small</code> provides only pretrained model with alpha=1, without pretrained weights with smaller alpha.<br>
I want to see performance of the model with smaller alpha on edge devices. Although MobileNetv3 is small, but it is still big for some devices.<br>
If you have any information, please comment here. Thank you.</p>
Inference speed of jit model

kent252: <p>I’m a newbie in torch script. I tried to quantize a resnet18 model and use torch.jit.script to compress the model. The model from 45MB is compressed to 0.01MB in Jit form. However, I expect the inference time of the jit model (run on CPU) will be faster than (or maybe equal to)  the original model (run on GPU) but it is just two times faster when they run on the CPU and the jit model (run on CPU) performs slower (around 4 times) than the original one (run on GPU).<br>
I follow this code to quantize and compress to jit</p>
<blockquote>
<p>num_calibration_batches = 32<br>
from torch.ao.quantization import get_default_qconfig_mapping<br>
torch.backends.quantized.engine = “x86”<br>
prepared_model = prepare_fx(model, get_default_qconfig_mapping(“x86”), example_inputs=torch.rand((7,1,10,20)))<br>
prepared_model.to(‘cpu’)</p>
<p>with torch.no_grad(): # calibrate using random data</p>
<blockquote>
<pre><code>data = torch.rand((1,3,32,32))
prepared_model(data)
</code></pre>
</blockquote>
<p><span class="hashtag-raw">#retrain</span> the model to maintain the accuracy<br>
train(prepared_model.to(‘cuda’), trainloader, valloader, 2, device)<br>
prepared_model.to(‘cpu’)<br>
prepared_model.eval()<br>
model_quantized = convert_fx(prepared_model)</p>
<p>model_quantized_scripted = torch.jit.script(model_quantized) # script model<br>
model_quantized_scripted(torch.rand((1,3,32,32)))</p>
</blockquote>
<p>Please let me know, is that normal or what problem I am faced</p>
A mistake resulted in a 1% improvement in the accuracy of my model

light1003: <p>When I use cosine distance to do triple loss,A<em>B/(|A||B|) Unexpectedly written as A</em>B/(|A||A|),It’s the same as (A<em>B/(|A||B|))</em>(|B|/|A|). It looks like a coefficient has been added.I guess it may be beneficial for the SGD optimizer,because doing so increases the accuracy of the model after training by 1% compared to the correct approach.I don’t know if my conjecture is correct. If anyone is interested, could you try using your model and see if it’s the same?</p>

ptrblck: <p>Could you post more details about your experiments, how many runs you’ve executed as well as the mean +/- stddev of the final performance metric?</p>
Report a bug: torch.linalg.lstsq: Argument 6 has illegal value

auyb: <p>RuntimeError: false INTERNAL ASSERT FAILED at “/opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/native/BatchLinearAlgebra.cpp”:1538, please report a bug to PyTorch. torch.linalg.lstsq: Argument 6 has illegal value. Most certainly there is a bug in the implementation calling the backend library.<br>
The RuntimeError appear when run following code:<br>
import torch<br>
x = torch.arange(10, 50, step=1, dtype=torch.float32)<br>
y = torch.exp(-((x-100)/(0.60056120439323*100))**2)<br>
y = 100 * y + 10 * torch.randn(x.shape)<br>
y = torch.log(y)<br>
x = x.unsqueeze(dim=0)<br>
y = y.unsqueeze(dim=0)<br>
x = torch.linalg.vander(x, N = 3).flip(-1)<br>
coefficients = torch.linalg.lstsq(x, y).solution<br>
print(coefficients)</p>
<p>the code works fine if without add noise to data y, that is remove the line “y = 100 * y + 10 * torch.randn(x.shape)”<br>
torch.<strong>version</strong> == 2.0.1</p>

KFrank: <p>Hi auyb!</p>
<aside class="quote no-group quote-modified" data-username="auyb" data-post="1" data-topic="203064" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f04885/48.png" class="avatar"> auyb:</div>
<blockquote>
<p>RuntimeError: false INTERNAL ASSERT FAILED at “/opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/native/BatchLinearAlgebra.cpp”:1538, please report a bug to PyTorch. torch.linalg.lstsq: Argument 6 has illegal value. Most certainly there is a bug in the implementation calling the backend library.<br>
…<br>
torch.<strong>version</strong> == 2.0.1</p>
</blockquote>
</aside>
<p>This looks like a known bug – see this <a href="https://github.com/pytorch/pytorch/issues/125892" rel="noopener nofollow ugc">pytorch github issue.</a></p>
<p>Note, I can reproduce your issue using your code on both pytorch version 2.3.0<br>
and on the current pytorch nightly, 2.4.0.dev20240519, on the cpu.  Your code<br>
runs for me without error (but the results are <code>nan</code>s) on the gpu on version 2.3.0.</p>
<p>Best.</p>
<p>K. Frank</p>
Creating custom grad

Aashiq_Muhamed: <p>I wrote an autograd function to create a custom grad that is of a different shape than the original parameter. However, I get an error like<br>
<code>Function CustomBackward returned an invalid gradient at index 1 - got [1024, 256] but expected shape compatible with [1024, 2048].</code></p>
<p>Since I’m doing this to mainly save memory is there a way I can disable this check? This gradient is not required for other gradient computations (since it is for the the weight) and I want to manually control how I accumulate these gradients.</p>
Different CPU utilization in training and validation under the same conditions

DOUBLE: <p>I train a semantic segmentation model based on the Pytorch framework, using the same Batch_size and Num_worker in both the val and train stages.<br>
Interestingly. During the training process, the CPU utilization of the main process is 100%, but the CPU utilization of several other Python sub-processes (launched by num_woker) is 25%, and the GPU utilization is 95%.<br>
When the val stage is reached, something interesting happens. The CPU utilization corresponding to the main process remains around 100%, but the CPU utilization corresponding to the remaining python sub-processes (launched by num_woker) soars to 70%, while the GPU utilization decreases to 85%.<br>
In fact, I use the same transform rules for the validation set data and the training set data.</p>
<pre><code class="lang-auto">transform_train = Compose(Resize(resize_shape), Rotation(2), 
                          Normalize(mean=mean, std=std))
train_dataset = My_Dataset(Dataset_Path, "train", transform_train)
train_loader = DataLoader(train_dataset, batch_size=8, shuffle=True,
                          collate_fn=train_dataset.collate, num_workers=5, pin_memory=False)

# ------------ val data ------------
transform_val_img = Resize(resize_shape)
transform_val_x = Compose(ToTensor(), Normalize(mean=mean, std=std))
transform_val = Compose(transform_val_img, transform_val_x)  # Resize --&gt; ToTensor --&gt; Normalize
val_dataset = My_Dataset(Dataset_Path, "val", transform_val)
val_loader = DataLoader(val_dataset, batch_size=8, collate_fn=val_dataset.collate, num_workers=5)

</code></pre>
Spikes in Training loss curve

AI_ML: <p>I’ve trained a image segmentation model using BCE loss and Adam optimizer with no weight decay. No dropout is used in the network. In the training loss curve there is spike after some steps.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae42c68f73518ebe888dafbcefa117c8abc3ffb6.png" alt="tensorboard" data-base62-sha1="oRA938BD7hcDjha9VMX3eG5lR66" width="450" height="356"></p>
<p>What are the possible reasons for this?</p>
Sudden spike in validation loss with regularization

AI_ML: <p>I’ve trained a image segmentation model using BCE loss and adam optimizer.<br>
On this run, I just changed the weight_decay value from 0 to 1e-05 &amp; every other things is same as first run. But I see in the validation curve there is a big spike in the middle</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4aa2b5a4b539ae110f3a61851d7236588928143b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4aa2b5a4b539ae110f3a61851d7236588928143b" title="mergepng"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aa2b5a4b539ae110f3a61851d7236588928143b_2_690x142.jpeg" alt="mergepng" data-base62-sha1="aEfW3xqtlbMMGFqUYVEDsu0SoUX" width="690" height="142" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aa2b5a4b539ae110f3a61851d7236588928143b_2_690x142.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aa2b5a4b539ae110f3a61851d7236588928143b_2_1035x213.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aa2b5a4b539ae110f3a61851d7236588928143b_2_1380x284.jpeg 2x" data-dominant-color="FCFCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mergepng</span><span class="informations">2705×557 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
left plot corresponding first run and right for second run(having weight decay)</p>
<p>What could be the reason of this spike?</p>