akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch DDP with AMP make deadlock

username: taehyung
cooked: <p>I am writing training codes containing AMP and DDP.<br>
My command is <code>torchrun --nproc_per_node=4 main.py</code>.<br>
Code stucks in <code>scaler.scale(loss).backward()</code>, after running several steps.<br>
Weirdly, it progresses well about 20 steps, which differ every time I run the codes, but soon stops without any error message.<br>
Does anyone have a similar experience?</p>
<pre><code class="lang-auto">model.train()
optimizer.zero_grad()

scaler = torch.cuda.amp.GradScaler()
local_progress=tqdm(dataloader, total=len(dataloader), disable=(utils.get_rank()!=0))
for i, batch in enumerate(local_progress):
    input_batch = batch['input'].type(torch.FloatTensor)
    target_batch = batch['target'].type(torch.FloatTensor)

    if cfg.IS_CUDA_AVAILABLE:
        input_batch = input_batch.cuda(non_blocking=True)
        target_batch = target_batch.cuda(non_blocking=True)

    with torch.cuda.amp.autocast():
        output_batch = model(input_batch)
        loss = loss_fn(output_batch, target_batch)
    local_progress.set_postfix(loss='{:05.3f}'.format(loss))
    
    # Codes stuck here 
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad()

    train_loss += loss.item() * len(input_batch)
    num_samples += len(input_batch)

train_loss = round(train_loss / num_samples, 6)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ee6e88a1cff1d6781a1dd444ffc12ef5e45c275.png" data-download-href="https://discuss.pytorch.org/uploads/default/6ee6e88a1cff1d6781a1dd444ffc12ef5e45c275" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6ee6e88a1cff1d6781a1dd444ffc12ef5e45c275_2_690x13.png" alt="image" data-base62-sha1="fP5eUXl5e1XD6L3mKCXKzkXfbZb" width="690" height="13" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6ee6e88a1cff1d6781a1dd444ffc12ef5e45c275_2_690x13.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6ee6e88a1cff1d6781a1dd444ffc12ef5e45c275_2_1035x19.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6ee6e88a1cff1d6781a1dd444ffc12ef5e45c275_2_1380x26.png 2x" data-dominant-color="20242A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2512×48 4.18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
CUDA 12.6 and torch.cuda.is_available return false

username: Eleanor_K
cooked: <p>Hi,</p>
<p>I have NVIDIA-SMI 560.35.03, Driver Version: 560.35.03, CUDA 12.6 installed in the server. Python version is 3.9.18. But I tried installing torch version 2.5 + cu124; 2.4 + cu121. It’s all return torch.cuda.is_available() is false. ps: torch.backends.cudnn.enabled is true</p>
<p>Really need help please</p>

username: ptrblck
cooked: <p>Your NVIDIA driver might not be properly installed if PyTorch cannot communicate with the GPU assuming you’ve installed a CDUA-enabled PyTorch binary (you can check it e.g. via <code>torch.version.cuda</code>).</p>

username: Eleanor_K
cooked: <blockquote>
<blockquote>
<blockquote>
<p>torch.version.cuda<br>
‘12.4’<br>
I think this is cuda 12.4 installed?</p>
</blockquote>
</blockquote>
</blockquote>

username: Eleanor_K
cooked: <p>I checked print(“Torch version:”, torch.<strong>version</strong>); print(“CUDA version:”, torch.version.cuda) and print(“CUDA available:”, torch.cuda.is_available()). The third one return False.</p>

username: ptrblck
cooked: <p>In this case your system seems to have trouble communicating with the device which could happen e.g. if you’ve updated the driver without a restart. Recently, another user explained they were seeing this issue after letting their device sleep and needed to reset the GPU or restart their system.</p>

username: wilhelm
cooked: <p>It is always the same …<br>
Everybody needed to fix this problem at least once.<br>
My homemade recipe is:</p>
<ol>
<li>Delete your conda environment completely</li>
<li>Recreate your conda environment (possibly from file)</li>
<li>Install PyTorch using the conda installer (in your case nightly build)</li>
<li>If torch.cuda.is_available() == False; then:</li>
<li>Repeat step 1, 2 and install PyTorch using pip</li>
</ol>

username: Eleanor_K
cooked: <p>Hi, thanks for the tips. I have been repeating it 10 times. Still not working… Does it work if I downgrade the system CUDA to 12.4 rather than 12.6?</p>

username: wilhelm
cooked: <p>As far as I understood pytorch installs its needed cuda version indipentenly. So, in short, there is no need to downgrade.<br>
I messed up my system so many times, that I would not try to downgrade the driver.<br>
But try at least the following: Install only pytorch with the recipe above without <code>torchaudio</code> and <code>torchvision</code> and see what happens.<br>
So:<br>
<code>$pip3 install torch</code><br>
or<br>
<code>$conda install pytorch pytorch-cuda=12.4 -c pytorch -c nvidia</code></p>
Cases where Fully Sharded Data Parallel is not mathematically equivalent to local training

username: NeuralFlux
cooked: <p>In the <a href="https://arxiv.org/abs/2304.11277" rel="noopener nofollow ugc">PyTorch FSDP paper</a> section 7.2.1, they say</p>
<blockquote>
<p>FSDP cannot ensure that it always achieves the same mathematical equivalence as local training, especially with respect to the optimizer computation. This stems from the fact that the optimizer step operates on the sharded parameters, whose data layout is a function of FSDP’s FlatParameter sharding algorithm that does not respect individual parameter boundaries. As a result, any optimizer computation that depends on an original parameter’s unsharded value (e.g. vector norm), its tensor structure (e.g. approximate second-order optimizers), or require global states over all parameters will become invalid.</p>
</blockquote>
<p>I have 2 questions</p>
<ol>
<li>How does <code>FlatParameter</code> sharding algorithm not respect individual parameter boundaries?</li>
<li>What is the implication of the last sentence? What kind of models cannot be trained using FSDP?</li>
</ol>

username: H-Huang
cooked: <p>cc <a class="mention" href="/u/agu">@agu</a> <a class="mention" href="/u/weifengpy">@weifengpy</a> for FSDP questions</p>

username: weifengpy
cooked: <aside class="quote no-group" data-username="NeuralFlux" data-post="1" data-topic="210081">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neuralflux/48/73006_2.png" class="avatar"> NeuralFlux:</div>
<blockquote>
<ul>
<li>How does <code>FlatParameter</code> sharding algorithm not respect individual parameter boundaries?</li>
</ul>
</blockquote>
</aside>
<p>Support we shard param1 = [1.0] and param2 = [2.0, 3.0, 4.0] over two gpus</p>
<ul>
<li>1st rank have flat_param1 = [1.0, 2.0]. This rank has whole param1, but part of param2</li>
<li>2nd rank have flat_param2 = [3.0, 4.0]. This rank has part of param2, but nothing of param1</li>
<li>one rank might only have part of the one param</li>
</ul>
<aside class="quote no-group" data-username="NeuralFlux" data-post="1" data-topic="210081">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neuralflux/48/73006_2.png" class="avatar"> NeuralFlux:</div>
<blockquote>
<ul>
<li>What is the implication of the last sentence? What kind of models cannot be trained using FSDP?</li>
</ul>
</blockquote>
</aside>
<p>It is suggests point-wise optimizers can be used out of the box, including SGD, Adam, AdamW.<br>
If the optimizer has non-pointwise op, say vector norm for grad clipping, we need NCCL all-reduce norms across ranks</p>
Saving and Loading a transformed dataset

username: adamson
cooked: <p>Hey,</p>
<p>I’m working with pytorch and torchaudio in the context of an audio dataset.<br>
As part of my dataset loading and feature extraction pipeline I’d like to apply a few transforms: resampling to a uniform sample rate, normalizing audio so that peaks are at 0dBFS, and extracting various spectral features (e.g. MFCCs).</p>
<p>All this can be defined nicely with Dataset and Data Loaders to my understanding so far.<br>
My question is about how to serialize the results of such transforms so that they can be effectively cached for future use rather than be applied on the fly each time I start over iterating on the dataset, e.g. when developing different types of classifiers / models that use the data…</p>
<p>I assume this is a very common use case across audio/vision and other data pipelines but I could not find immediate answer for this. I can of course roll my own custom code to save things to disk and load back, but seems like would be a natural functionality to have in the dataset interface? Apologies if redundant question…</p>
<p>Best,</p>
<p>Adam</p>
Libtorch with c++20 on MSVC

username: dingari
cooked: <p>I’m trying to integrate the libtorch C++ frontend into a project that relies on some C++20 (std:c++latest) features. Libtorch will not compile out of the box with MSVC. I get errors related to <code>std::result_of</code> and <code>std::result_of_t</code> being used (in C++17.h), which have been deprecated in favor of <code>std::invoke_result</code> and <code>std::invoke_result_t</code> in C++20.</p>
<p>So it looks like I have two options:</p>
<ol>
<li>Trying to dig into the libtorch source code and use <code>std::invoke_result</code> instead, without fully understanding the code (and then fix any subsequent errors I run into).</li>
<li>Downgrade my project to use C++17, which requires replacing a lot of lines (mostly designated initializers) with more error-prone code.</li>
</ol>
<p>Neither one sounds very fun. Will libtorch support C++20 any time soon?</p>

username: stalek
cooked: <p>I plan also to use LibTorch and  C++ 20 (mainly focusing on concepts).<br>
Is it sufficient just to fix <strong>result_of</strong> or it’s more complex task?<br>
Were you able to do it or just gave up?</p>
<p>I don’t see any activity here so I assume nobody in torch team is interested in compiling it<br>
with modern c++  <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: Willem
cooked: <p>I’m having the same issue. Need to downgrade project now to get things running using libtorch.</p>

username: Rinnekioku
cooked: <p>I have the same issue, I tried to compile app with Clang and GCC and everything works fine even with C++20, but with MSVC we need to downgrade standard</p>

username: stalek
cooked: <p>Does anybody know any plans related to this issue?</p>

username: Freeman_LIU
cooked: <p>Please, libtorch people, can you roll out a version for c++20. It’s 2022, and we are stuck with c++14!<br>
Even if with c++17, I got error below:<br>
libtorch/include/ATen/Functions.h:12966:3: error: reference to ‘optional’ is ambiguous<br>
optional device_{};<br>
^<br>
libtorch/include/ATen/Operators.h:89:7: note: candidate found by name lookup is ‘c10::optional’<br>
class optional;<br>
^<br>
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/optional:585:7: note: candidate found by name lookup is ‘std::optional’<br>
class optional</p>

username: kila_suelika
cooked: <p>I have used pytorch with C++17 for one year and it works perfectly.</p>
<p>Did you use “using namespace std;”? Your problem should be gone after remove this line.</p>

username: kila_suelika
cooked: <p>Libtorch 2.0 has been released now. I have test it with <code>c++/latest</code> and it works perfectly.</p>

username: lskook47
cooked: <p>Same here struggling to build libtorch on cpp20 along with opencv cpp… Is there any update so far?</p>
Float16 matmul works while bfloat16 throws CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmStridedBatchedEx

username: Butanium
cooked: <p>mwe:</p>
<pre data-code-wrap="py"><code class="lang-py">a = th.randn(16, 8, 1024, 1024, dtype=th.float16, device="cuda") # switch to bfloat fails
b = th.randn(16, 8, 1024, 256, dtype=th.float16, device="cuda")  # switch to bfloat fails
c = a @ b
print(c.shape)

CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasGemmStridedBatchedEx(handle, opa, opb, (int)m, (int)n, (int)k, (void*)&amp;falpha, a, CUDA_R_16BF, (int)lda, stridea, b, CUDA_R_16BF, (int)ldb, strideb, (void*)&amp;fbeta, c, CUDA_R_16BF, (int)ldc, stridec, (int)num_batches, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)
</code></pre>
<pre data-code-wrap="bash"><code class="lang-bash">nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

 pip show torch
Name: torch
Version: 2.5.0+cu121

nvidia-smi 
Sat Oct 26 14:11:57 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX TITAN X     On  | 00000000:04:00.0 Off |                  N/A |
| 22%   34C    P8              16W / 250W |   7450MiB / 12288MiB |      0%      Default 
</code></pre>

username: ptrblck
cooked: <p>I cannot reproduce the issue in my setup on a 3090 and <code>bfloat16</code> support is also not available on your Maxwell GPU.</p>

username: Butanium
cooked: <p>Thank you! I should have included this in my first message but the reason I’m confused is that with less dimensions it works just well. I guess the support is partial?</p>
<pre data-code-wrap="py"><code class="lang-py">In [5]: a = th.randn(16, 8,  dtype=th.bfloat16, device="cuda")
   ...: b = th.randn(8, 8, dtype=th.bfloat16, device="cuda") 
   ...: c = a @ b 
   ...: print(c.shape) 
   ...:                                                                                                                                                                                                                                                                             
torch.Size([16, 8])
</code></pre>

username: ptrblck
cooked: <p>Yes, maybe fallback kernels are used for specific shapes. However, since the support was added for Ampere and later devices, don’t depend on it for Maxwell or any other generation before Ampere.</p>
Function_schema.h(49,1): error C2351: obsolete C++ constructor initialization syntax

username: Felix1
cooked: <p>Hi,<br>
I got this error</p>
<p><em>Error: function_schema.h(49,1): error C2351: obsolete C++ constructor initialization syntax</em></p>
<p>Version pythoch in c++: <strong>libtorch-win-shared-with-deps-debug-2.5.0+cpu.zip</strong></p>
<p>I am using visual stuido 2020 and compiling with:<br>
ISO C++17 Standard (/std:c++17)</p>
<p>Thanks very much for you help!<br>
Best!</p>

username: Felix1
cooked: <p>I think I have found the souce of the error. There is a conflict with the includes of the library</p>
<p><strong>wxWidgets</strong></p>
<p>// define this one just in case it occurs somewhere (instead of preferred<br>
// wxTRANSLATE) too<br>
<span class="hashtag-raw">#if</span> !defined(WXINTL_NO_GETTEXT_MACRO)<br>
<span class="hashtag-raw">#if</span> !defined(gettext_noop)<br>
<span class="hashtag-raw">#define</span> gettext_noop(str) (str)<br>
<span class="hashtag-raw">#endif</span><br>
<span class="hashtag-raw">#if</span> !defined(N_)<br>
<span class="hashtag-raw">#define</span> <strong>N_</strong>(s)             (s)<br>
<span class="hashtag-raw">#endif</span><br>
<span class="hashtag-raw">#endif</span></p>

username: Felix1
cooked: <p>I have fixed changing:</p>
<p><span class="hashtag-raw">#define</span> <strong>N_</strong> (s) (s)<br>
to<br>
<span class="hashtag-raw">#define</span> <strong>N_2</strong> (s) (s)</p>
<p>And this code in wxWidgets it is about translation language. And I am not using translation…so…</p>
<p>I marked as fixed.<br>
Best<br>
Felix</p>
RuntimeError: strides() called on undefined Tensor

username: zhazl
cooked: <p>Hello everyone! I have met a problem when I’m using torch.jit.save to save torchscript model as the title.  Anyone has some suggestions for me?</p>

username: Ankita_Bhagat
cooked: <p>Hello zhazl, did you find the solution? I am getting the similar error while implementing model with ArcFace.</p>

username: zhuanling_zha
cooked: <p>yes, you can add the input data(default equal to None) with a type Optional<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aadf0a9b9f61c9e17dbc781d35a5d921ddf6eb57.png" alt="image" data-base62-sha1="onB26TpUnyCnS2NUAOjmAApln6v" width="563" height="398"></p>

username: ali_hamdy
cooked: <p>hello</p>
<p>I faced the same issue, but I have functions that has default arguments</p>
<p>is there another cause of this error</p>
About pytorch version

username: DIVYANSH_RENKUNTLAWA
cooked: <p>I have cuda 12.5 in my laptop but on pytorch still 12.4 is the latest how do i use this pytorch on cuda 12.5 and not only that , pytorch’s one library only works below cuda 12.2 versions .</p>

username: ptrblck
cooked: <p>PyTorch binaries ship with their own CUDA runtime dependencies and your locally installed CUDA toolkit won’t be used. I don’t understand the second question.</p>
Compute grad with regard a slice of the input

username: nazareem
cooked: <p>Hi,</p>
<p>Do you know why I get the following error:</p>
<pre><code class="lang-python">n, d = 10, 3
lin1 = nn.Linear(d, d)

# case 1
x = torch.randn(d).requires_grad_()
y = lin1(x)
vec = torch.ones(d)
gr = torch.autograd.grad(y, x, vec, retain_graph=True)[0]
print(gr.shape)  # returns: torch.Size([3])

# case 2
x = torch.randn(n, d).requires_grad_()
y = lin1(x)
vec = torch.ones(d)
gr = torch.autograd.grad(y[0], x[0], vec, retain_graph=True)[0]

# Raises Error:
# RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
</code></pre>
<p>why does taking x[i] remove it from the computation graph, I tried to clone it or to use <code>.narrow</code> but in vain!</p>
<p>y[0] only depends on x[0], so I don’t want to compute the gradient with regard to the full input!<br>
Any help is appreciated!</p>

username: ptrblck
cooked: <p>I guess the issue is caused by indexing <code>x</code> since it would create a new view, which was not used directly in the computation graph.<br>
The same error would be raised if you use <code>gr = torch.autograd.grad(y, x.clone(), vec, retain_graph=True)[0]</code> in the first example.<br>
<a class="mention" href="/u/alband">@albanD</a> is there a more elegant way using some new (beta) features for this type of computation?</p>

username: albanD
cooked: <p>Not sure which beta feature you’re referring to?<br>
But in general, autograd works at the Tensor level, so no you won’t be able to compute the gradient wrt a subset of the Tensor any faster than computing the whole gradient and then taking the subset you care about.<br>
To do so, you will have to do the forward with a Tensor that represent the subset you care about.<br>
Does that make sense?</p>

username: thash
cooked: <p>Hello, <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Is there anything I could do to solve this issue by now?</p>
Torch C++ program failed to compile with weird error

username: Thanh_Bao_Nguyen
cooked: <p>I’m trying to install libtorch c++ on my machine. I followed in official instruction from <a href="https://pytorch.org/cppdocs/installing.html" rel="noopener nofollow ugc">pytorch page</a> and compiled, but I recieved this warning:</p>
<pre><code class="lang-auto">CMake Warning (dev) in CMakeLists.txt:
  No project() command is present.  The top-level CMakeLists.txt file must
  contain a literal, direct call to the project() command.  Add a line of
  code such as

    project(ProjectName)

  near the top of the file, but after cmake_minimum_required().

  CMake is pretending there is a "project(Project)" command on the first
  line.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) in CMakeLists.txt:
  cmake_minimum_required() should be called prior to this top-level project()
  call.  Please see the cmake-commands(7) manual for usage documentation of
  both commands.
This warning is for project developers.  Use -Wno-dev to suppress it.
</code></pre>
<p>I followed the instruction from the docs above, modify the CMakeLists.txt file by setting the CMAKE_PREFIX_PATH so I don’t have to copy paste the libtorch path to the terminal. Here are my files:</p>
<p>CMakeLists.txt</p>
<pre><code class="lang-auto">project(test)
set(CMAKE_PREFIX_PATH /home/bao/Downloads/libtorch-shared-with-deps-2.5.0+cpu/libtorch)
find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(${PROJECT_NAME} example-app.cpp)
target_link_libraries(${PROJECT_NAME} "${TORCH_LIBRARIES}")
set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 14)
</code></pre>
<p>main.cpp</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;

int main()
{
    std::cout &lt;&lt; "this is a test";
}
</code></pre>
<p>My project structure:<br>
cpp/<br>
main.cpp<br>
CMakeLists.txt<br>
./build</p>
<p>This is the full output from the terminal when I cmake … , apart from the warning above</p>
<pre><code class="lang-auto">-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /home/bao/Desktop/cpp/build
</code></pre>
<p>And when I tried to cmake --build . --config Release, nothing happens. Like no warning nor errors nor output, the terminal printed nothing. Would anybody know why this happens, and how to fix it?</p>
Reparametrization Trick (Normal distributions)

username: jaytimbadia
cooked: <p>How torch.normal works underhood?</p>
<p>I am asking this with regard to reparamatrization trick?</p>
<pre><code class="lang-auto">import torch
import numpy as np

N = 1
mu_grads = []
std_grads = []


def reparametrize(mu, std):
    eps = torch.rand_like(std)
    return mu + std * eps

mu = torch.tensor([12.], requires_grad=True) # set mu = 12 and store gradient
std = torch.tensor([42.], requires_grad=True) # set std = 42

for i in range(N):
    mu.grad = None # reset the gradient value
    std.grad = None

    z = reparametrize(mu, std)
    z2 = z ** 2
    z2.backward()

    mu_grads.append(mu.grad.detach().cpu().numpy())
    std_grads.append(std.grad.detach().cpu().numpy())

print(f"Estimated dE[z^2]/dmu={np.mean(mu_grads):.2f}")
print(f"Estimated dE[z^2]/dstd={np.mean(std_grads):.2f}")
</code></pre>
<p>The above code works fine!<br>
But below code does not?</p>
<pre><code class="lang-auto">import torch
import numpy as np

N = 1
mu_grads = []
std_grads = []

mu = torch.tensor([12.], requires_grad=True) # set mu = 12 and store gradient
std = torch.tensor([42.], requires_grad=True) # set std = 42

for i in range(N):
    mu.grad = None # reset the gradient value
    std.grad = None

    z = torch.normal(mu, std) # the random sampling happens here
    # but here also same thing should happen right as reparametrize function.
    # mu + std * torch.randn(1)  &lt;- Is this how torch.normal works?
    z2 = z ** 2
    z2.backward()

    mu_grads.append(mu.grad.detach().cpu().numpy())
    std_grads.append(std.grad.detach().cpu().numpy())

print(f"Estimated dE[z^2]/dmu={np.mean(mu_grads):.2f}")
print(f"Estimated dE[z^2]/dstd={np.mean(std_grads):.2f}")
</code></pre>
<p>mu + std * torch.randn(1)  ← Is this how torch.normal works?</p>
<p>Please help!<br>
Thank you!</p>

username: jaytimbadia
cooked: <p>Can anyone please help?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>If I’m not mistaken, <code>torch.normal</code> will call into <a href="https://github.com/pytorch/pytorch/blob/78a0158540a669893245534b89af55a9de4cb082/aten/src/ATen/native/TensorFactories.cpp#L1046-L1063">this <code>normal</code> function</a> which will use the <code>mean</code> and <code>std</code> inputs as plain scalar types and will thus detach them from the tensors, which is why your first approach would work.</p>

username: jaytimbadia
cooked: <p>Can you please elaborate on why it will detach and on plain scalar types?</p>
<p>Thanks!</p>

username: wilhelm
cooked: <p>I do not know, if this answer your question, but I think you should use <code>rsample</code> from the Normal distribution in pytorch for the reparametrization trick.</p>
<p>Thus:</p>
<pre><code class="lang-auto">normal = torch.distributions.normal.Normal(loc = mu, scale = std)

sample = normal.rsample()
</code></pre>
Run backward without accumulating gradients

username: Nikolay_Blagoev
cooked: <p>Currently I have this code, which runs a batch through the model and stores the result for later use. Before actual loss computation, the gradient with respect to the input needs to be computed:</p>
<pre><code class="lang-auto">x.requires_grad = True
x.retain_grad()
tmp = x
x = ml[i](x, start_p, mask)
x.retain_grad()
x.backward(torch.ones_like(x),retain_graph=True)
intermediate_gradients.append(tmp.grad)
x.grad = None
</code></pre>
<p>However running this backward pass will inadvertently accumulate the gradients of the weights into the .grad parameter of the network weights. Is there a way to avoid this last part? Using no_grad is not the solution, as later when the actual loss is computed, the gradients will need to be accumulated.</p>

username: ptrblck
cooked: <p>You could use the <code>inputs</code> argument in <code>backward</code> which will accumulate the gradients only for these tensors. From the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html#torch.Tensor.backward">docs</a>:</p>
<blockquote>
<p><strong>inputs</strong> (<em>sequence</em> <em>of</em> <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor"><em>Tensor</em></a><em>,</em> <em>optional</em>) – Inputs w.r.t. which the gradient will be accumulated into <code>.grad</code>. All other tensors will be ignored. If not provided, the gradient is accumulated into all the leaf Tensors that were used to compute the <code>tensors</code>.</p>
</blockquote>
Torch-TRT - any reason not to use torch.cuda.empty_cache?

username: s0ulf1re
cooked: <p>I have been reading on a different forum post that using torch.cuda.empty_cache() is not usually recommended. However, I see a lot of GPU memory being released permanently when using it once.<br>
Here is what I am doing:<br>
<span class="hashtag-raw">#loading</span> model<br>
model = torch.load(‘modelName.pt’)<br>
<a class="hashtag-cooked" href="/c/compile/41" data-type="category" data-slug="compile" data-id="41"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>torch.compile</span></a> with batch size 16<br>
batch_size = 16<br>
trt_model = torch_tensorrt.compile(model,<br>
inputs = [torch_tensorrt.Input((batch_size, 3, 224, 224),)],<br>
enabled_precisions= {torch_tensorrt.dtype.half}<br>
)<br>
Without using torch.cuda.empty_cache(), nvidia-smi reports 17954 MiB of GPU memory being used. Any inference on the model keeps the memory usage the same. However, if I use torch.cuda.empty_cache(), the memory usage drops to 2428 MiB. Any inference still shows the usage to be 2428 MiB. Inference times seem to be very similar between both cases.</p>
<p>Is there any reason using torch.cuda.empty_cache() would not be recommended here? Thanks for any help.</p>

username: ptrblck
cooked: <p>Based on your memory usage it seems <code>TorchTRT</code> might use a large workspace (or another intermediate allocation) during its initial execution steps and is not freeing it afterwards.<br>
We generally do not recommend clearing the cache as it will synchronize your device and since PyTorch clears the cache itself after e.g. benchmarking cuDNN algorithms or when running into an OOM.<br>
<a class="mention" href="/u/narendasan">@narendasan</a> would know details about <code>TorchTRT</code>’s execution and where this memory jump might come from.</p>
PyTorch: Removing a one-hot-encoding padding from a batch of sequences of one-hot encodings

username: Moe_Joe
cooked: <p>My inputs to a PyTorch LSTM layer look something like this: [batch_size, sequence_length, one_hot_length]</p>
<p>As an example of the data, it might be something like:</p>
<p>[0 0 1]<br>
[0 1 0]<br>
[0 0 1]<br>
or<br>
[0 1 0]<br>
[1 0 0]</p>
<p>in the case of the second sequence, I am padding it the max sequence length, so it becomes</p>
<p>[0 1 0]<br>
[1 0 0]<br>
[0 0 0]</p>
<p>Let’s say I get my batched input from the dataloader. I can’t go around removing the padding from every sequence, because it would make a jagged tensor. I can’t use PackedSequence and pack_padded_sequence because those functions are not for the one-hot-encoding case. The batched input is passed in directly to the LSTM layer, which itself accounts for the batching, so there’s not really an intermediate place for me to pass in something else. Is there any sort of relatively simply way of handling this? It seems like I just can’t do batched training on this particular problem unless I force the batches to all have the same length. Which I suppose is not disastrous, but as I am pretty new to PyTorch, I wanted to check first if I’m missing some obvious solution.</p>
<p>I’ve looked around for a while and surprisingly (at least to me) nobody else has really asked about how to do this. I figure people must just be forcing batches to have the same length.</p>
How to determine if forward is called by backward checkpointing?

username: Luke20000429
cooked: <p>I am training a model with Pytorch, and I use <a href="https://pytorch.org/docs/stable/checkpoint.html#torch-utils-checkpoint" rel="noopener nofollow ugc">activation checkpointing</a> to save memory. There are some host functions inside the module that I don’t want it to run again during backward when recovering the intermediate state. Is there anyway to know if the module is running under checkpointing recovery or normal forwarding?<br>
To be specific, I added some <code>nvtx.push/pop_range</code> inside a module, but I found the pop after the last kernel is not called during the checkpointing recovery, and results in a very long nvtx range. Thus, either we can solve this bug or just skip the push when running in backward would be helpful.</p>

username: Luke20000429
cooked: <p>Look like the problem is magically resolved if I run it under</p>
<pre><code class="lang-auto">with nvtx.annotate(...):
    module()
</code></pre>
Neural Netwok for classify 2D map position

username: Afonsod3v
cooked: <p>Hello all,</p>
<p>I’m new to Neural Networks and my math is secundary school only.</p>
<p>Here is the problem I want to solve using PyTorch:</p>
<p>Im building a robot to explore my house rooms.<br>
I need a way for the robot to identify its 2d map position.</p>
<p>The robot has 2 sensores: front distance sensor (tof) and compasso sensor - which unfortunately I can’t rely on because indoor I have too much magnetic intererence.<br>
So basically I just have the distance sensor and the robot can rotate but I dont know the rotate angle.</p>
<p>Here is my idea:</p>
<p>I built a 2D map, a grid let say 40x40.<br>
Each grid cell holds 8 distances for each 8 angles (0, 45, 90, …, 315 degrees) to neighborg cells.</p>
<p>The robot Will make 5 distance readings, ex:<br>
60, 75, 60, 75, 90</p>
<p>So, 2D map Will have this data:<br>
Cell 0: 30, 45, 60, 75, 90, 75, 60, 45<br>
Cell 1: etc…</p>
<p>I want to Neural Network to learn the 2D map, and based on robot reading, It will classify which cell is more likely to be.</p>
<p>I have reed other papers like SLAM (Simultaneous Localization And Mapping) but I’m too far to reach this. I’m looking for a Simple PyTorch solutions to ML learn a 2D map based on cell parâmeters.</p>
<p>Any help will be very appreciated.<br>
Marco</p>
Getter version of slice_scatter, e.g. slice_select

username: ilykuleshov
cooked: <p>Hello! I was wondering if there is any getter version of <a href="https://pytorch.org/docs/stable/generated/torch.slice_scatter.html" rel="noopener nofollow ugc">slice_scatter</a>, which would be able to get slices along a specific dimension.<br>
I could write it using brackets and <code>slice(None)</code>, however it would look rather nasty:</p>
<pre><code class="lang-auto">t = torch.randn(5, 6, 4, 5, 8, 3, 10)
dim = 4
crop_start = t[(slice(None),) * dim + (slice(1, None),)]
crop_end = t[(slice(None),) * dim + (slice(0, -2),)]
</code></pre>
<p>I’m looking for something that would be probably named <code>slice_select</code> (by analogy to <code>slice_scatter</code>:</p>
<pre><code class="lang-auto">crop_start = torch.slice_select(t, start=1, dim=dim)
crop_end = torch.slice_select(t, end=-2, dim=dim)
</code></pre>
<p>Is something of the sort present in PyTorch, or possibly planned for future releases?</p>
Are features within an LSTM connected

username: pumplerod
cooked: <p>with an LSTM layer with multiple features, do the features within the LSTM have any effect on one another?  Is there some level of a fully connected layer for the features of each time slice?</p>
<pre><code class="lang-auto">lstm = torch.nn.LSTM( 2, hidden_size=10, num_layers=1, batch_first=True)
features = torch.rand( (1,10,2))

'''
tensor([[[0.3317, 0.1547],
         [0.1294, 0.4251],
         [0.3386, 0.1474],
         [0.9933, 0.0323],
         [0.1047, 0.8910],
         [0.2937, 0.5866],
         [0.7244, 0.8024],
         [0.6509, 0.6176],
         [0.2874, 0.3786],
         [0.1911, 0.8797]]])
'''
</code></pre>
<p>With the above example would feature column 1 have any effect or interaction with feature column 2 for each time step?  Or are they treated separately within the LSTM such that each feature series is independent of one another?</p>

username: J_Johnson
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/5413be03e49308852b3bb18980f008dcbba68673.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5413be03e49308852b3bb18980f008dcbba68673" title="1000022804"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5413be03e49308852b3bb18980f008dcbba68673_2_536x500.jpeg" alt="1000022804" data-base62-sha1="bZMp79u9TBmKaVXW1SrkYuWqXej" width="536" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5413be03e49308852b3bb18980f008dcbba68673_2_536x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5413be03e49308852b3bb18980f008dcbba68673_2_804x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5413be03e49308852b3bb18980f008dcbba68673_2_1072x1000.jpeg 2x" data-dominant-color="F3F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1000022804</span><span class="informations">1240×1155 90.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you see those Xs inside circles? Those are <a href="https://en.m.wikipedia.org/wiki/Matrix_multiplication" rel="noopener nofollow ugc">matrix multiplications</a> (matmul). When you matmul(or matrix-vector product) two matrices(or in this case, a vector and a matrix), all parts of the matrix(or vector) interact. So, yes, the input features interact with one another before the output.</p>
How to add a new SDPA backend in Python?

username: fno2010
cooked: <p>As far as I know, all the current SDPA backends are written in C++ (e.g., <code>flash_attention</code> and <code>mem_efficient_attention</code>) or provided by C++ libraries (e.g., CuDNN). As the SDPA backend selection code is written in C++, it can only call backends via C++ APIs. If I have an SDPA backend completely written in Python, how can I add it to PyTorch?</p>
Slow convolutions in triton

username: AidanGoldfarb
cooked: <p>When using torch.compile() on torch.vision models on the GPU, the triton version is normally (densenet and alexnet are slower) faster than the CUDA version. However, when testing single convolutions (torch conv2d), the triton version is always ~20% slower. Compilation overhead is accounted for, the median of multiple runs is taken. I have replicated convolutional parameters from the source code of the vision models, as well as run an ablation on kernel size, stride, and padding. In and out channels are fixed at 3 and 64 respectively, as is the case in all vision models.</p>
<p>When profiled with the torch profiler, it appears to have great speedups, but wallclock time shows them to be much slower.</p>
<p>Any ideas on why this seems to be the case?</p>

username: nickjeliopoulos
cooked: <p>For 1x1 convolutions, I have consistently observed 2x-3x higher latency than baseline PyTorch. I would also like to know the reason for this specifically.<br>
It seems in general that compiled Triton is not able to handle convolutions very well (multiple sources on Triton Github issues)</p>
Torch::max() without dynamic memory allocation

username: dshin
cooked: <p>If I run the following code…</p>
<pre><code class="lang-auto">    torch::Tensor t = torch::rand({5});
    torch::Tensor m = torch::zeros(0);
    for (int i=0; i&lt;5; ++i) {
      m = torch::max(t);
      std::cout &lt;&lt; m.data_ptr() &lt;&lt; std::endl;
    }
</code></pre>
<p>…I get the following output:</p>
<pre><code class="lang-auto">0x558373cc8e00
0x558373cda3c0
0x558373cda480
0x558373cda4c0
0x558373cda580
</code></pre>
<p>As you can see, the data pointer is constantly changing, indicating that there is some dynamic memory allocation going on. How can I rewrite the line <code>m = torch::max(t);</code> to prevent the data pointer from changing on each loop iteration?</p>
<p>I ultimately just want the max value as a float, so I am fine with a solution that just directly returns a float.</p>
<p>For some other functions, the answer to the analogous question to the one I ask is “<em>use the <code>*_out()</code> version</em>”. For example, there is <code>torch::softmax()</code> and <code>torch::softmax_out()</code>. For <code>torch::max()</code>, however, <code>torch::max_out()</code> does not appear to follow this same idiom. At least, I couldn’t figure out how to make <code>torch::max_out()</code> work, and also could not find any documentation on the arguments/return-value of the function.</p>

username: rbaleksandar
cooked: <p>You can’t from what I can tell from the documentation. This applies to both the Python and C++ API. If you don’t want a tensor (in this case one that contains a single value), you will have to get the max (same applies to min btw) yourself.</p>
Which is faster to load for traning?

username: JohnHerry
cooked: <p>When we try to make some thing about LLM (dataset contains audio or image modality， not only text) from scratch, there are a lot of training data to consume for training,  most of them are little, small feature files. and that may make bottleneck on dataloader specilly on big batch size.<br>
My question is, which is the better feature save format for training loading speed? the numpy  np file? the torch pt file? or the joblib serialized file? which is faster for loading during traning?</p>
How can I write a Convolution like torch.nn.Conv2D using CUDA C

username: Aakira
cooked: <p>Recently I have had some computations that require specialization on the GPU. I can still leverage PyTorch’s existing modules, however, I’ve noticed that performance is severely impacted due to intermediate steps and sequential processing in the implementation, leading to increased memory usage and runtime. I feel everything could be done better if I were to move it to CUDA C and somehow bind it to Python similar to how PyTorch operates.</p>
<p>However, this seems to be a rather uncommon topic, and although I’ve searched online, I haven’t found an answer that satisfies me.</p>
<p>So, to code a kernel like Conv2D in CUDA C and use it as an API in PyTorch, where should I start? What knowledge do I need to acquire? Could you provide me with some accompanying resources?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>Custom extensions can be implemented as described in <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">this tutorial</a>. Before going down this path you might want to check if e.g. <code>torch.compile</code> could speed up your use case.</p>

username: Aakira
cooked: <p>I find many parts of that tutorial to be a non-friendly starting point. For example, take a look at this statement:</p>
<blockquote>
<p>For the “ahead of time” flavor, we build our C++ extension by writing a <code>setup.py</code> script that uses setuptools to compile our C++ code. For the LLTM, it looks as simple as this:</p>
</blockquote>
<pre><code class="lang-auto">from setuptools import setup, Extension
from torch.utils import cpp_extension

setup(name='lltm_cpp',
      ext_modules=[cpp_extension.CppExtension('lltm_cpp', ['lltm.cpp'])],
      cmdclass={'build_ext': cpp_extension.BuildExtension})
</code></pre>
<p>Terms like “ahead of time” or the arguments of <code>setup()</code> and their purpose are not fully explained. This seems to be a prerequisite to read further. Is there another starting point to start with?</p>
Ddp with torch and ignite

username: Abdelkareem_Elkhateb
cooked: <p>What is the best way to use multiple GPU(8) to train a pytorch model on a windows server.<br>
I have 8 rtx 2080TI, i am trying to train  a image denoise model, I tried the nn.DataParallel(net,device_ids=device_ids).cuda()  and it uses on 20% of one gpu from the 8.<br>
I tried  using ignite (found it simpler than lightning ) but it uses a ddp stratgey that supports two backens which are nccl and gloo.<br>
for nccl i get the following :</p>
<pre data-code-wrap="py"><code class="lang-py">{
    "name": "ValueError",
    "message": "Unknown backend 'nccl'. Available backends: ('gloo',)",
    "stack": "---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[22], line 6
      3 spawn_kwargs[\"nproc_per_node\"] = 2
      4 config[\"backend\"] = \"nccl\"
----&gt; 6 with idist.Parallel(backend=config[\"backend\"], **spawn_kwargs) as parallel:
      7     parallel.run(training, config)

File c...conda\\envs\\genai\\Lib\\site-packages\\ignite\\distributed\\launcher.py:225, in Parallel.__init__(self, backend, nproc_per_node, nnodes, node_rank, master_addr, master_port, init_method, **spawn_kwargs)
    223 if backend is not None:
    224     if backend not in idist.available_backends():
--&gt; 225         raise ValueError(f\"Unknown backend '{backend}'. Available backends: {idist.available_backends()}\")
    226 else:
    227     arg_names = [\"nproc_per_node\", \"nnodes\", \"node_rank\", \"master_addr\", \"master_port\"]

ValueError: Unknown backend 'nccl'. Available backends: ('gloo',)"
} 
</code></pre>
<p>and for the gloo i get the following erorr</p>
<pre data-code-wrap="py"><code class="lang-py">    191     ctx = _concrete_contexts[method]
    192 except KeyError:
--&gt; 193     raise ValueError('cannot find context for %r' % method) from None
    194 ctx._check_available()
    195 return ctx

ValueError: cannot find context for 'fork'
</code></pre>

username: Abdelkareem_Elkhateb
cooked: <p>I am following this code <a href="https://pytorch-ignite.ai/tutorials/intermediate/01-cifar10-distributed/" class="inline-onebox" rel="noopener nofollow ugc">Distributed Training with Ignite on CIFAR10 | PyTorch-Ignite</a></p>

username: ptrblck
cooked: <p><code>NCCL</code> is not supported on Windows so you would need to use <code>gloo</code>. Unfortunately, I’m not familiar with <code>gloo</code> and don’t know what might be causing the issue.<br>
However, searching for this error message seems to point to a workaround using:</p>
<pre data-code-wrap="python"><code class="lang-python">multiprocessing.set_start_method("spawn", force=True)
</code></pre>
nn.TransformerEncoderLayer output changes with batch size

username: tamirs
cooked: <p>Observing significantly different outputs from torch’s TransformerEncoderLayer when changing batch size (although I used batch_first=True) - anyone knows why?</p>
<p>Code to minimally reproduce:</p>
<p>#######</p>
<pre><code class="lang-auto">import torch
import numpy as np
import random 
import os

os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":16:8"

def set_random_seeds(seed=0, device='cuda:0'):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)  
    torch.use_deterministic_algorithms(True)  
    
    os.environ['PYTHONHASHSEED'] = str(seed)
    
    if device != 'cpu':
        torch.cuda.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False


set_random_seeds()


l1 = torch.nn.TransformerEncoderLayer(1024, 1, 1024, 0.0, batch_first=True).double().cuda()
l1.eval()


x = torch.rand((128, 1024)).double().cuda()
diff = (l1(x)[:2] - l1(x[:2])).abs().mean()
print(diff)
</code></pre>
<h6><a name="p-457799-h-1" class="anchor" href="#p-457799-h-1"></a></h6>
<p>I get diff outputs of around 0.18 with torch version 2.3.1+cu121</p>

username: ptrblck
cooked: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html#transformerencoderlayer">docs</a>:</p>
<blockquote>
<p><strong>batch_first</strong> (<a href="https://docs.python.org/3/library/functions.html#bool"><em>bool</em></a>) – If <code>True</code>, then the input and output tensors are provided as (batch, seq, feature). Default: <code>False</code> (seq, batch, feature).</p>
</blockquote>
<p>So in your case the missing <code>dim0</code> is interpreted as a single sample and you are slicing in the <code>seq</code> dimension. Adding the missing batch dimension gives the expected small mismatch:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.rand((128, 128, 1024)).double().to(device)
diff = (l1(x)[:2] - l1(x[:2])).abs().mean()
print(diff)
tensor(5.5092e-16, device='cuda:0', dtype=torch.float64,
       grad_fn=&lt;MeanBackward0&gt;)
</code></pre>

username: tamirs
cooked: <p>Yes that clears it, totally missed that.<br>
Thank you!</p>
How to compute Nested Tensors's mean and var?

username: xuliaoqing250
cooked: <p>For torch.Tesnor it is easy to compute mean and var, but I can not find ways to compute mean and var for   Nested Tensors. Nested Tensors support layer_norm operation which include mean and var operation. Is there any ways to compute mean fo Nested Tensors?<br>
Thank for your help!</p>
<pre><code class="lang-auto">x = torch.randn(1, 192)
y = torch.randn(10, 192)
nested = torch.nested.nested_tensor([x, y])
nested.mean(dim=-1)#Don't support
layer_norm(nested)#support
</code></pre>