akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is it possible to speedup the value assignment process in cuda tensor?

username: zizhao.mo
cooked: <p>Hi,</p>
<p>I have a question about the value assignment on PyTorch.<br>
For example, I want to copy values from a cuda tensor ‘a’ to a cuda tensor ‘b’, from a series of locations. A simple solution is:</p>
<p>b[loc_list] = a[loc_list]</p>
<p>where the ‘loc_list’ is a list of locations, e.g., loc_list = [1,3,5,7,9]. However, this assignment seems inefficient, since its overhead can increase with the length of loc_list.</p>
<p>My question is, is it possible to parallelize this assignment process? No limitation on methods and any C++ implementation is OK.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>If you transform <code>loc_list</code> to a tensor and move it to the GPU, you’ll remove the H2D copy (assuming you are reusing the indices a few times and can thus benefit from the reduction of copies).</p>

username: zizhao.mo
cooked: <p>Hi ptrblck,</p>
<p>Thanks for your response. I would like to know whether PyTorch can parallelize the assignment process for multiple rows in variable ‘b’ after I transform the ‘loc_list’ into a tensor?</p>

username: zizhao.mo
cooked: <p>Hi ptrblck,</p>
<p>Sorry to bother you again. This time I try to illustrate my problem more detailed with my example code as follows:</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    device = 'cuda:2'
    torch.cuda.set_device(device)

    stream = torch.cuda.Stream()
    tensor_a = torch.zeros((100, 10000, 10000), device=device, dtype=torch.float16)
    tensor_b = torch.ones((100, 10000, 10000), device=device, dtype=torch.float16)

    a_indices = torch.as_tensor([1, 3, 7, 9, 11, 13, 17, 19, 21, 25], device=device, dtype=torch.int)
    b_indices = torch.as_tensor([0, 1, 2, 3, 4, 5, 6, 7, 9, 11], device=device, dtype=torch.int)
    
    with torch.cuda.stream(stream):
        # tensor_a[:, a_indices, :] = tensor_b[:, b_indices, :]
        tensor_a[:, a_indices, :].copy_(tensor_b[:, b_indices, :], non_blocking=True)
</code></pre>
<p>In the above code, I want to copy some data from tensor_b to tensor_a. However, this process is really time-consuming. My objective is to hide this overhead, but either wrapping them in a stream or using the non-blocking flag does not take effect. I think the reason is that I am fetching a non-contiguous memory.</p>
<p>My question is, is it possible to make this assignment process asynchronous, which will not block the main process and will not block the subsequent operation in CUDA? If Pytorch cannot do this, is there any workaround with libtorch?</p>
<p>Looking forward to your response.</p>
<p>BR</p>
Check if algorithms are in deterministic mode

username: mspanring
cooked: <p>Hi!</p>
<p>Is there a way to check if torch is using deterministic algorithms after I have set <code>torch.use_deterministic_algorithms(True)</code>?</p>
<p>Naively, I would have assumed that at least <code>torch.backends.cudnn.deterministic</code> is set to <code>True</code> after this call to reflect that convolutions are using deterministic algorithms.</p>

username: ptrblck
cooked: <p>It depends on your knowledge about the used algorithms, but you could e.g. profile the code via Nsight Systems and check which kernels are used as some kernels are known to be (non-)deterministic. I’m not aware of any other way to “check” it besides setting the variable and assuming the framework respects this setting.</p>

username: mspanring
cooked: <p>Thank you very much for the confirmation!</p>
<p>I think that it may be valuable in some cases, especially for debugging, to have the means to check if the used modules are set to be deterministic.</p>
<p>I will add it as a feature request.</p>

username: chris.rapson
cooked: <p>I assume you made a feature request and it was implemented, because now there is a way to check. For others who stumble across this thread, the answer is:</p>
<p><code>torch.are_deterministic_algorithms_enabled()</code></p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.are_deterministic_algorithms_enabled.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.are_deterministic_algorithms_enabled.html</a></p>
The backward for _scaled_mm

username: yjguo
cooked: <p>_scaled_mm is defined at <a href="https://github.com/pytorch/pytorch/blob/release/2.5/aten/src/ATen/native/native_functions.yaml#L7024" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/native_functions.yaml at release/2.5 · pytorch/pytorch · GitHub</a>, its out_dtype can be different than the input dtype.  And so I’m curious about the dtype things in the backward functions.  I checked in file <a href="https://github.com/pytorch/pytorch/blob/release/2.5/tools/autograd/derivatives.yaml" class="inline-onebox" rel="noopener nofollow ugc">pytorch/tools/autograd/derivatives.yaml at release/2.5 · pytorch/pytorch · GitHub</a>, and do not see anything about scaled_mm.</p>
<p>so, my question is, how does the autograd know what’s the backward function for _scaled_mm (I know the backward function is also mm, but how are they connected in code)? thanks.</p>

username: ptrblck
cooked: <p><code>_scaled_mm</code> is the raw operation while the Autograd usage would depend on derived <code>dtypes</code> as described <a href="https://dev-discuss.pytorch.org/t/float8-in-pytorch-1-x/1815">here</a>.</p>

username: yjguo
cooked: <p>thanks, a nice introduction on fp8.</p>
Embedded Python can't import torch in a C++ project

username: Himateja_Nallani
cooked: <p>Hi, PyTorch community!<br>
This is my first post, so please bear with me. I have been working with Network Simulator (NS-3), a C++ code base and the project was built using CMake. I am calling “import torch” using embedding Python Interpreter as follows:</p>
<pre><code class="lang-auto">Py_Initialize();
PyGILState_STATE gilState = PyGILState_Ensure();        // Get the GIL
PyRun_SimpleString("import torch");
PyRun_SimpleString("torch.__version__");
PyGILState_Release(gilState);   // Release the GIL
</code></pre>
<p>The above code is part of a file called “tcp-madras.cc”, and a dynamic library called “libinternet.so” gets built using CMake.</p>
<p>When I run the main() executable that includes “libinternet.so” the program throws a <strong>Segmentation Fault</strong>. I did some debugging using gdb and found the following:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/22a79ccd269f479f2e40ca8fff2c77af809bccb9.png" data-download-href="https://discuss.pytorch.org/uploads/default/22a79ccd269f479f2e40ca8fff2c77af809bccb9" title="gdb debug error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22a79ccd269f479f2e40ca8fff2c77af809bccb9_2_690x258.png" alt="gdb debug error" data-base62-sha1="4Wzld9AqiUr19fpvkR3TiKNUnnX" width="690" height="258" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22a79ccd269f479f2e40ca8fff2c77af809bccb9_2_690x258.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22a79ccd269f479f2e40ca8fff2c77af809bccb9_2_1035x387.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22a79ccd269f479f2e40ca8fff2c77af809bccb9_2_1380x516.png 2x" data-dominant-color="161A19"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gdb debug error</span><span class="informations">1778×666 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please note that everything works fine if I call “import torch” using a Python interpreter from the terminal.<br>
I would be happy if someone could point me in the right direction to resolve this issue.</p>
<p><strong>Environment:</strong></p>
<blockquote>
<p><em># PyTorch Version: ‘2.5.0+cu124’</em><br>
<em># Python version:  3.12.3</em><br>
<em># OS: Windows 10</em><br>
<em># WSL version: 2.3.24.0</em><br>
<em># CUDA/cuDNN version: 12.6.77</em><br>
<em># GPU: RTX 4060</em></p>
</blockquote>

username: Himateja_Nallani
cooked: <p>Hi Community,<br>
This issue has been resolved. The problem is that the torch module requires OpenSSL, whereas my project references BoringSSL. This I came to know by checking the OpenSSL version using PyRun_SimpleString(“import ssl; print(ssl.OPENSSL_VERSION)”)</p>
<p>Hence by including the following lines in project’s CMakeLists.txt the issue is solved:<br>
find_package(OpenSSL REQUIRED)<br>
target_link_libraries(myproject OpenSSL::Crypto)</p>
In DataLoader, is it possible to control which CPU workers are assigned which indices?

username: Sam_Lerman
cooked: <p>Let’s say I have <code>5</code> CPUs, a dataset of length <code>100</code>, and batch size of <code>8</code>. Let’s say the <code>DataLoader</code> assigns <code>8</code> indices in <code>[0, 99]</code>. For example: <code>{1, 2, 33, 55, 61, 62, 77, 78}</code>.</p>
<p>Now I want to be able to control which CPU workers are assigned to fetch the corresponding data. I want CPU 1 to fetch indices in <code>[0, 19]</code>, CPU 2 to fetch indices in <code>[20, 39]</code>, CPU 3 to fetch indices in <code>[40, 59]</code>, CPU 4 to fetch indices in <code>[60, 79]</code>, and CPU 5 to fetch indices in <code>[80, 99]</code>.</p>
<p>Consequently, CPU 1 should be assigned to fetch <code>{1, 2}</code>, CPU 2 should be assigned to fetch <code>{33}</code>, CPU 3 should be assigned to fetch <code>{55}</code>, CPU 4 should be assigned to fetch <code>{61, 62, 77, 78}</code>, and CPU 5 should not be assigned to fetch any indices.</p>

username: nivek
cooked: <p>You can use <code>[get_worker_info()](https://pytorch.org/docs/stable/data.html#torch.utils.data.get_worker_info)</code> and <code>worker_init_fn()</code> that is being passed to <code>DataLoader</code> to configure the worker to only read certain fraction of the Dataset. For example, you can modify the <code>Dataset</code> object at each worker or you can make <code>__getitem__</code> to ignore certain indices based on <code>worker_id</code>.</p>

username: Sam_Lerman
cooked: <p>Would those discarded indices still be retrieved by other workers or no?</p>

username: nivek
cooked: <p>Each worker should have an independent sampler that generates all the indices, but you can decide what to do with them at each worker.</p>

username: Sam_Lerman
cooked: <p>Okay, that was the solution I thought of</p>

username: s63jgn
cooked: <p>Hi, it’s a relief I am not the only one facing this issue. Thank you so much for posting this! However, I’ve some trouble understanding how you solved this. Can you really have multiple samplers, one per worker? And how do you achieve every worker getting all the indices of one batch and not just a subset?</p>
<p>Maybe you could even share some minimal code? I’ve detailed my current problem and thoughts in <a href="https://discuss.pytorch.org/t/in-dataloader-with-ddp-and-multiple-workers-can-i-control-which-indices-are-assigned-to-which-ranks-and-workers/213644">this new thread</a></p>
Torch.save() like open(mode='a')

username: ZimoNitrome
cooked: <p>I am running a training script and I want to save the output tensors of my validation set after each epoch.</p>
<p>My script runs for an arbitrary amount of epochs so I would like to append tensors to a file after each epoch.</p>
<p>What is the best way to go about this?</p>
<ul>
<li>I could <code>torch.save()</code> to one new file every epoch, but that will create a lot of files.</li>
<li>I could <code>torch.save()</code> to a single file each epoch, but then I would need to <code>torch.load()</code> that file each epoch to append to the single data structure and re-save it.</li>
<li>I could add to an ever-increasing list inside my script and <code>torch.save()</code> that each epoch, but that would use up more and more memory.</li>
</ul>
<p>Are there better alternatives? Like appending a text representation of the output tensor and append it to a text file?</p>

username: ZimoNitrome
cooked: <p>The following code seems to be quite efficient for my use case:</p>
<pre><code class="lang-python"># Save tensors each output
netG.eval()
with torch.no_grad():
    pred_val = netG(fixed_target)
netG.train()

f_path = Path(f"./progress/batch_outputs.h5")
if not f_path.exists():
    f = tables.open_file(str(f_path), mode='w')
    atom = tables.Float64Atom()
    batches_ea = f.create_earray(f.root, 'batches', atom, shape=(0, *pred_val.shape))
else:
    f = tables.open_file(str(f_path), mode='a')
    f.root.batches.append(pred_val.unsqueeze(0).cpu().numpy())
f.close()
</code></pre>
<p>Adapted from <a href="https://stackoverflow.com/a/30379177/7829241" class="inline-onebox" rel="noopener nofollow ugc">python - save numpy array in append mode - Stack Overflow</a></p>

username: qpwo
cooked: <p>Thank you for sharing your solution!</p>
Memory thrashing: `num_alloc_retries` keeps growing, `reserved_bytes` fluctuates

username: Jerome_Ku
cooked: <p>Training a model that uses <code>Zero3</code> (deepspeed, equivalent to <code>fsdp</code>) along with other forms of parallelism (tensor + sequence).</p>
<p>What I’ve noticed is that as I increase the sequence length of the model the throughput becomes highly unstable:</p>
<ul>
<li>logging <code>cuda.memory_stats</code> and specifically the counters <code>num_device_alloc</code>, <code>num_alloc_retries</code>, <code>num_device_free</code>, I see that these counters grow over time, and that stats such as <code>reserved_bytes.all.current</code> fluctuate.</li>
<li>this is in contrast to the same model but run at a smaller sequence length, where the counts are 0 and the <code>reserved_bytes</code> hits a plateau and stabilizes (and where throughput is stable throughout).</li>
</ul>
<p>I’ve already tried setting <code>expandable_segments</code>.  Are there other things to try given memory thrashing / fragmentation?</p>

username: carolgrey98
cooked: <blockquote>
<p>Training a model that uses <code>Zero3</code> (deepspeed, equivalent to <code>fsdp</code>) along with other forms of parallelism (tensor + sequence).</p>
<p>What I’ve noticed is that as I increase the sequence length of the model the throughput becomes highly unstable: <a href="https://www.nextcare-urgentcare.com" rel="noopener nofollow ugc">nextcare urgent care login</a></p>
<ul>
<li>logging <code>cuda.memory_stats</code> and specifically the counters <code>num_device_alloc</code>, <code>num_alloc_retries</code>, <code>num_device_free</code>, I see that these counters grow over time, and that stats such as <code>reserved_bytes.all.current</code> fluctuate.</li>
<li>this is in contrast to the same model but run at a smaller sequence length, where the counts are 0 and the <code>reserved_bytes</code> hits a plateau and stabilizes (and where throughput is stable throughout).</li>
</ul>
<p>I’ve already tried setting <code>expandable_segments</code>. Are there other things to try given memory thrashing / fragmentation?</p>
</blockquote>
<p>Hello,<br>
Here are few additional steps to consider: enable ZeRO-3’s offloading capabilities to shift optimizer and gradient states to CPU or NVMe memory, which can help reduce GPU memory usage; try reducing the batch size to stabilize memory usage; enable gradient checkpointing, which saves memory at the expense of extra computation; monitor memory usage using tools like cuda.memory_stats to identify unusual patterns; and ensure you are using the latest version of DeepSpeed, as updates often come with performance improvements and bug fixes.<br>
Let me know if this helps!</p>

username: Jerome_Ku
cooked: <p>Thanks - fixed the issue by profiling memory usage and reducing per layer memory requirements slightly.</p>
<p>Options such as updating deepspeed or offloading are not feasible given need for stable training env and throughput requirements.</p>
Understanding FSDP prefetching

username: anonymous_user_83259
cooked: <p>How can I get a better understanding of the prefetching process during FSDP1/2 forward and backward?</p>
<p>I hope to learn the following:</p>
<ul>
<li>The maximum amount of memory overhead prefetching will consume</li>
<li>The point at which a given parameter’s prefetch will be triggered</li>
<li>If FSDP1/2 exposes any hooks for when parameter / gradient fetching is initiated.</li>
</ul>

username: agu
cooked: <p>I should write a more detailed post on this at some point. Let me get back to you.</p>

username: tj.solergibert
cooked: <p>Maybe It’s a bit old, but check this post! <a href="https://dev-discuss.pytorch.org/t/fsdp-cudacachingallocator-an-outsider-newb-perspective/1486" class="inline-onebox" rel="noopener nofollow ugc">FSDP &amp; CUDACachingAllocator: an outsider newb perspective - distributed - PyTorch Developer Mailing List</a></p>

username: anonymous_user_83259
cooked: <p>Currently, I do not know whether FSDP2 will actively prefetch the gradients for the last layer (the first layer to be computed in the backward pass) in a training step.</p>
<p>I have not inspected fsdp2’s source code in detail, and imagine the following "What if"s:</p>
<ul>
<li>FSDP never reshards the gradients of the root FSDP module, but only shards inner wrapped FSDP submodules</li>
<li>FSDP starts to gather the root module’s grads when .backward() occurs, leading to an initial blocking step before any computation can occur</li>
<li>FSDP is aware of the computational graph (even in eager?) and gathers an appropriate FSDPParamGroup ahead of time, after it has tracked the first backward()</li>
</ul>
<p>Are any of the above cases true? What really happens?</p>
Libtorch C++ on macOS m1

username: Lupiya_Mujala
cooked: <p>I have been banging my head against this for the past week but not making any progress. I keep trying to use the library on my mac following <a href="https://pytorch.org/cppdocs/installing.html" rel="noopener nofollow ugc">these instructions</a> but I keep getting the same error:</p>
<pre><code class="lang-auto">
**main.cpp:1:10:** **fatal error:** **'torch/torch.h' file not found**
1 | #include &lt;torch/torch.h&gt;
</code></pre>
<p>I don’t know what I’m missing.</p>
How to handle .detach() in opacus enabled model

username: Rahat_Kibria
cooked: <p>The overall picture of my model is expressed in the following pseudo-code</p>
<p>(SimSiam Pseudocode, PyTorch-like<br>
here: f indicates backbone + projection mlp)</p>
<p>for x in loader: # load a minibatch x with n samples<br>
x1, x2 = aug(x), aug(x) # random augmentation<br>
z1, z2 = f(x1), f(x2) # projections, NxD<br>
L = D(z1, z2) # loss<br>
L.backward() # back-propagate<br>
optimizer.step() <span class="hashtag-raw">#update</span>(f, z1) # SGD update</p>
<p>def D(p, z): # negative cosine similarity<br>
z = z.detach() # stop gradient<br>
p = normalize(p, dim=1) # l2-normalize<br>
z = normalize(z, dim=1) # l2-normalize<br>
return -(p*z).sum(dim=1).mean()</p>
<p>I want to incorporate opacus for performing differential privacy, I have discovered that the .detach() is causing problems with the opacus. I am getting this error: “ValueError: Per sample gradient is not initialized. Not updated in backward pass?”<br>
When I do not use .detach(), the code runs well without the error. But it creates issues with accuracy and performance as in Siamese networks one of the projector outputs should remain out of the gradient flow to avoid collapse.</p>
<p>I have also discussed the problem in the opacus git-hub &gt; issue. Any suggestions will be appreciated. Thanks in advance.</p>
Remove feature but increase accuracy

username: tony_chung
cooked: <p>I am new to PyTorch and have tried different methodl to increase the accuracy of my model (e.g. dropoff, data augmentation, normalization, batch normalization). However, most of the dones’t help much. Instead, when I remove some of the feature in my model (e.g. from 10 features to 2 feastures), the accuracy of the model increase a lot. Wondering if anyone got similar experience when developing / training the model. Appreicate if you can share your view / experience.</p>
Error in F.scaled_dot_product_attention

username: olegkufa
cooked: <p>When i’m using F.scaled_dot_product_attention with SDPBackend.CUDNN_ATTENTION and with float attn mask (some values in mask is -inf) i got a lot of nans in output tensor.If i use SDPBackend.EFFICIENT_ATTENTION or SDPBackend.MATH all is okey, if try SDPBackend.FLASH_ATTENTION  got<br>
RuntimeError: No available kernel. Aborting execution</p>
Dependencies for `csprng` becoming untenable?

username: HarrisonWilde
cooked: <p>Hi, so I am using Opacus and PyTorch in a package I am building, and I would like to be able to ensure some level of long term support for the dependencies of the project. I want to use Opacus fully and so would like to utilise the <code>torchcsprng</code> package, but it is still requiring v1.8.1 of PyTorch which is now pretty old. Is there any plan to continually support <code>torchcsprng</code>? I see it hasn’t received much maintenance in a while, but there is a pretty loud warning from Opacus when you don’t use <code>secure_rng</code>, so I am not sure how best to plan the development of my package with these conflicting issues in mind. I would ideally like to use PyTorch 2.0 and the latest version of Opacus…</p>
<p>Thanks!</p>

username: sm-egan
cooked: <p>Glad you posted about this! I’m running into this issue as well. I’m very interested in comparing the performance of the Philox PRNG to the AES-based CSPRNG implemented in torchcsprng on a DP benchmark, but can’t install the package on any of the platforms accessible to me (M3 Apple Silicon or Google CoLab, mostly running CUDA 12.2) because of incompatibilities with recent versions of CUDA, and there’s definitely no MPS support.</p>
<p>As far as I can tell, no other open source DP/ML library has an option for a cryptographically-secure RNG. I’ve looked primarily at jax-privacy and tensorflow-privacy.</p>
Assigning value in a for loop is very slow

username: kevin2004
cooked: <pre><code class="lang-auto">int main() 
{
    auto in = torch::full({9}, 0.0);
    for (int i = 0; i &lt; 2000000; i++)
    {
        in[0] = 0.1;
    }

    return 0;
}
</code></pre>
<p>The above code takes 3000ms to run. But if I comment <code>in[0] = 0.1;</code>, it takes 0.005ms to run. Is it possible to make it faster?</p>
Quantize a single tensor obtained from a float32 model

username: Boltzmachine
cooked: <p>I have a model consisting of two parts: the first is an encoder of float32, and the second is a quantized LLM.<br>
If I load the LLM in bfloat16, I can do <code>encoder(x).bfloat()</code> and feed it into the LLM. But for LLM in 8bit, I cannot find a corresponding operation to convert the output of the encoder.</p>

username: jerryzh168
cooked: <p>if you want to feed a single int8 tensor, you can probably still do <code>encoder(x).char()</code> I guess? but this is not really quantization, since int8 quantization requires scale/zero_point as well</p>

username: jane3144
cooked: <aside class="quote no-group quote-modified" data-username="Boltzmachine" data-post="1" data-topic="209780" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/boltzmachine/48/32323_2.png" class="avatar"> Boltzmachine:</div>
<blockquote>
<p>I have a model consisting of two parts: the first is an encoder of float32, and the second is a quantized LLM. <a href="https://www.prosperidad-social.com.co" rel="noopener nofollow ugc">Prosperidad Social</a><br>
If I load the LLM in bfloat16, I can do <code>encoder(x).bfloat()</code> and feed it into the LLM. But for LLM in 8bit, I cannot find a corresponding operation to convert the output of the encoder.</p>
</blockquote>
</aside>
<p>Hello,</p>
<p>You can manually quantize the encoder output to 8-bit integers using the same method the LLM expects during inference. Libraries like <strong><code>bitsandbytes</code></strong> or <strong><code>transformers</code></strong> from Hugging Face often handle these operations under the hood. Another approach is fake quantization, where you simulate the 8-bit range without converting the data type: def fake_quantize(tensor, min_val=-1, max_val=1, levels=256):<br>
step = (max_val - min_val) / (levels - 1)<br>
return torch.round((tensor - min_val) / step) * step + min_val<br>
This approach keeps the data as <code>float32</code> but simulates 8-bit precision.</p>
<h4><a name="p-460404-h-4-llm-library-specific-hooks-1" class="anchor" href="#p-460404-h-4-llm-library-specific-hooks-1"></a>4. <strong>LLM Library-Specific Hooks:</strong></h4>
<p>Check if the LLM’s library offers an input transformation or pre-processor API. Many 8-bit models have specific hooks for handling data types, especially in PyTorch, TensorFlow, or ONNX-based models.</p>
<p>Hope that helps !</p>
Any way to have DataParallel and DistributedDataParallel automatically handle buffers?

username: pytorcher
cooked: <p>I have a module that has values instantiated with register_buffer functions.  Is there any way to have these values automatically handled by DataParallel and DistributedDataParallel?  I built a custom DataParallel with a custom gather that will work for them, but am having some more significant issues implementing the same with DDP.  Would be great to just set the buffers up in a way that it’s handled automatically.</p>
<p>As an extra level of complexity some of these values should be summed across the replicas, some should be averaged, and one takes the max value.</p>

username: pytorcher
cooked: <p>Thank you for this detailed response!  Three questions about my initial DataParallel implementation before moving into Distributed.</p>
<p>1 - When I inspect the gather() function outputs appears to be a list of tensors.  The buffers I’m looking to adjust are registered to modules.  This is how I’m currently doing it, does this seem ok?  I’m doing this over every buffer of which there are a handful, so if there is a quicker way to do this that would be great.  Currently my 2GPU test is 10X slower than my 1GPU test.</p>
<pre><code class="lang-auto">Average = self.gather([self.replicas[x].average for x in self.device_ids].mean(0)
self.module.average = Average
</code></pre>
<p>2 - My understanding is that gather is just called once at the end of the DataParallel forward() function.  Some of these buffers are actually modified during the backward function.  Is there a way to have those adjusted automatically?  Currently I’m calling a backwardGather() function after the backward function by hand every batch.  Which is working, but not ideal.</p>
<p>3 - You mentioned that it handles parameters automatically.  Is there a way to to just switch these from buffers to parameters and have them handled? Or it only handles parameters that are part of the autograd graph so there’s not any way to have them be adjusted on the side?</p>
Euclidian Loss (self written)

username: string111
cooked: <p>Hey there, I am trying to implement euclidian loss (from <a href="https://arxiv.org/pdf/1409.1556.pdf" rel="nofollow noopener">VGG paper</a>). It is not really described there well, but what I am assuming is, that it just measures the euclidian distance between two coordinates and returns it as sort of a loss:<br>
So, my deep CNN wants to predict two coordinates (output tensor of size <code>(batch_size, 1, 2)</code>) lets say <code>out = [[0.3, 0.6]]</code> where <code>batch_size = 1</code>, and my <code>ground_truth = [[0.5, 0.7]]</code> the euclidian loss should be <code>loss = math.sqrt((0.3 - 0.5)**2) + math.sqrt((0.6 - 0.7)**2)</code></p>
<p>1st question: Do I understand the concept of euclidian loss right?<br>
2nd question: Does the below implementation looks right to you? (first time I create a own loss function)</p>
<pre><code class="lang-auto">class EuclidianError(nn.Module):
    """Implements euclidian distance as an error"""
    def forward(self, x_pred, x_ground_truth):
        err = torch.zeros(1)[0]

        for i in range(len(x_ground_truth)):    # i is index of batch
            for j in range(len(x_ground_truth[i])):    # j is index of to predict coordinates within one item of a batch
                for k in range(len(x_ground_truth[i][j])):    # k is index of x, y position in coordinates
                    err += math.sqrt((x_ground_truth[i][j][k] - x_pred[i][j][k])**2)

        return err
</code></pre>
<p>Thank you for your help!</p>

username: ptrblck
cooked: <p>I’m not sure why there are 3 nested loops, since your data seems to have only 2 dimensions.<br>
If you want to use it as a criterion without implementing the backward method, you should stick to PyTorch functions. So you should change <code>math.sqrt</code> to <code>torch.sqrt</code>.<br>
Alternatively, you could get rid of the for loops (which can be pretty slow) and use a faster approach:</p>
<pre><code class="lang-python">x = torch.randn(10, 2)
y = torch.randn(10, 2)
loss = torch.sqrt((x - y)**2).sum()
</code></pre>

username: string111
cooked: <p>Figuered it out the hard way myself, you are totally right! Thanks!</p>

username: assafge
cooked: <p><code>torch.linalg.norm(x - y)</code> will produce similar result? it’s basically the same?</p>

username: ptrblck
cooked: <p><code>torch.linalg.norm</code> uses the Frobenius Norm for matrices and the 2-norm for vectors as stated in the <a href="https://pytorch.org/docs/stable/generated/torch.linalg.norm.html">docs</a>. You could compare both approaches for your use case to make sure the results are expected.</p>
Extract ReLU outputs of EfficientNet and Incetption

username: Flock1
cooked: <p>I want to recreate <a href="https://www.sciencedirect.com/science/article/pii/S1077314220300333" rel="noopener nofollow ugc">this</a> paper and I was wondering how do I extract <code>ReLU</code> outputs of EfficientNet and Inception models?</p>
<p>I read about the models and <code>ReLU</code> isn’t an explicit layer but a part of the convolution layer. I want to confirm if just using the CNN output would be appropriate for this purpose.</p>
<p>Moreover, EfficientNet uses <code>SiLU</code> instead of <code>ReLU</code>. So what should I do in that case?</p>

username: diana45peters
cooked: <aside class="quote no-group quote-modified" data-username="Flock1" data-post="1" data-topic="213574" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flock1/48/12220_2.png" class="avatar"> Flock1:</div>
<blockquote>
<p>I want to recreate <a href="https://www.sciencedirect.com/science/article/pii/S1077314220300333" rel="noopener nofollow ugc">this</a> paper and I was wondering how do I extract <code>ReLU</code> outputs of EfficientNet and Inception models?<br>
<a href="https://www.wellstar-mychart.com" rel="noopener nofollow ugc">Official Site</a><br>
I read about the models and <code>ReLU</code> isn’t an explicit layer but a part of the convolution layer. I want to confirm if just using the CNN output would be appropriate for this purpose.</p>
<p>Moreover, EfficientNet uses <code>SiLU</code> instead of <code>ReLU</code>. So what should I do in that case?</p>
</blockquote>
</aside>
<p>Let’s break this down:</p>
<ul>
<li><strong>Extracting ReLU Outputs</strong>: In models like EfficientNet and Inception, ReLU (Rectified Linear Unit) is typically integrated within the convolutional layers. To extract the outputs after ReLU, you can use the output of the convolutional layers directly, as the ReLU activation function is applied within these layers1.</li>
<li><strong>EfficientNet and SiLU</strong>: You’re correct! EfficientNet uses the SiLU (Sigmoid Linear Unit) activation function instead of ReLU1. If you need to extract the outputs after the activation function, you should use the output of the SiLU layer instead.</li>
</ul>
<p>Best regards,<br>
DianaP.</p>

username: Flock1
cooked: <p>In the paper, they mentioned that they extracted the ReLU outputs of the EfficientNet model. Why did they mention that? Is there any specific version that uses ReLU?</p>
<p>And regarding Inception, if I extract the CNN outputs, they would automatically be outputs of ReLU activation?</p>
-inf appears in the attention weight of function: scaled_dot_product_attention

username: mingtan408
cooked: <p>I use the equivalent python code from PyTorch official document to instead nn.functional.scaled_dot_product_attention.</p>
<pre><code class="lang-auto">def scaled_dot_product_attention_manual(self, query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None) -&gt; torch.Tensor:
        L, S = query.size(-2), key.size(-2)
        scale_factor = 1 / math.sqrt(query.size(-1)) if scale is None else scale
        attn_bias = torch.zeros(attn_mask.shape, dtype=query.dtype, device=query.device)
        if is_causal:
            assert attn_mask is None
            temp_mask = torch.ones(L, S, dtype=torch.bool).tril(diagonal=0)
            attn_bias.masked_fill_(temp_mask.logical_not(), float("-inf"))
            attn_bias.to(query.dtype)

        if attn_mask is not None:
            if attn_mask.dtype == torch.bool:
                attn_bias.masked_fill_(attn_mask.logical_not(), float("-inf"))
            else:
                attn_bias += attn_mask
        attn_weight = query @ key.transpose(-2, -1) * scale_factor
            
        attn_weight += attn_bias.to(query.device)
        attn_weight = torch.softmax(attn_weight, dim=-1)
        attn_logits = attn_weight

        attn_weight = torch.dropout(attn_weight, dropout_p, train=True)
        return attn_weight @ value, attn_logits
</code></pre>
<p>But the attn_logits contain nans, and attn_weight contain -inf.</p>
<p>The dtype of these tensor is torch.float16</p>
<p>When I directly apply the nn.functional.scaled_dot_product_attention to on the same q_states and kv_states, the outputs is normal.</p>
<p>Could someone tell me how to modify my own sdpa function?</p>
Dassl enumerate dataloader

username: kendal
cooked: <p>for self.batch_idx, batch in enumerate(self.train_loader_x):<br>
I want to use dassl to train my model, but when I executed to the code above, the whole model was stuck without raising any error.<br>
I tried to set num_workers=0 as many invitation introduced, but the result seems the same.<br>
I submitted my work to slurm with requirements as follows:<br>
<span class="hashtag-raw">#SBATCH</span> -G 2 ### two gpus<br>
<span class="hashtag-raw">#SBATCH</span> -N 1 ##<span class="hashtag-raw">#one</span> node<br>
<span class="hashtag-raw">#SBATCH</span> -n 1 ##<span class="hashtag-raw">#one</span> progress<br>
Could anyone has an impressive depth of knowledge in this field help me?</p>
Non blocking copy from CPU to GPU

username: shira
cooked: <p>Hi,<br>
I need a little help.<br>
I know there is copy_() function</p>
<p>Tensor.copy_(<em>src</em>, <em>non_blocking=False</em> ) → [Tensor]<br>
Copies the elements from <code>src</code> into <code>self</code> tensor and returns <code>self</code></p>
<p>so this is function to copy tensor  to specific allocation in the memory<br>
I want to know if there is such a function to copy the whole module to specific allocation <strong>at once</strong>, (not each tensor a lone) in non blocking way.</p>
<p>function like .to()   - that allows to transfer the whole  module.<br>
but to specific allocation (which this is not exsists in .to() function)</p>
<p>Thanks:)</p>

username: shira
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji only-emoji" alt=":pray:" loading="lazy" width="20" height="20"><br>
Any-one did such a copy?</p>
Converting ‘deform_conv2d’ to coreml mlmodel format

username: mht_1421
cooked: <p>I am trying to export a pytorch model to mlmodel format, I can’t do that due to the error saying that Pytorch: the op deform_conv2d is not implemented. Is there someone who has a solution?</p>

username: Naman_Jaswani1
cooked: <p>Were you able to solve it <a class="mention" href="/u/mht_1421">@mht_1421</a> ?</p>

username: mht_1421
cooked: <p><a class="mention" href="/u/naman_jaswani1">@Naman_Jaswani1</a> nope.</p>
A node type in export IR graph

username: mhs4670go
cooked: <p>I’m working on graph transformation with export IR graph in ExportedProgram after <code>torch.export.export</code>.</p>
<p>When I implemented a pass for the transformation, I had one question about a node type. For example, this constant propagation pass in <a href="https://github.com/pytorch/executorch/blob/651af12b19e9aa109ce974a0d95a1710b397d6c4/exir/passes/constant_prop_pass.py#L147" rel="noopener nofollow ugc">executorch codes</a> replaces redundant torch.fx.Node with torch.Tensor.</p>
<p>But, as <a href="https://pytorch.org/docs/stable/export.ir_spec.html#node" rel="noopener nofollow ugc">a spec</a> says, all the nodes are torch.fx.Node.</p>
<blockquote>
<p>A Node represents a particular computation or operation and is represented in Python using the <a href="https://pytorch.org/docs/stable/fx.html#torch.fx.Node" rel="noopener nofollow ugc"><code>torch.fx.Node</code></a> class.</p>
</blockquote>
<p>Is it okay to use torch.Tensor, too? It wouldn’t violate the spec?</p>

username: mhs4670go
cooked: <p>Ah, I missed some points. The pass internally creates placeholder fx node instead.</p>
Scaled_dot_product_attention higher head num cost much more memory

username: Nova
cooked: <p>Hi, I found Scaled_dot_product_attention cost much more memory when head number is large(&gt;=16). This is my code to reproduce the issue.</p>
<pre><code class="lang-auto">
import torch
length = 10000
dim = 64
head_num1 = 8
head_num2 = 16
batch = 1
shapes = [[batch, head_num1, length, dim//head_num1], [batch, head_num2, length, dim//head_num2]]
for shape in shapes:
    torch.cuda.reset_peak_memory_stats()

    shape2 = [1,1, length, length]
    q = torch.rand(shape, dtype = torch.float16).cuda()
    k= torch.rand(shape, dtype = torch.float16).cuda()
    v = torch.rand(shape, dtype = torch.float16).cuda()

    attn_mask = torch.ones(shape2, dtype = torch.bool).cuda()
    x = torch.nn.functional.scaled_dot_product_attention(
                                q, k, v, attn_mask=attn_mask, dropout_p=0)

    peak_memory = torch.cuda.max_memory_allocated()      
    print(f"head number {shape[1]} case peak memory: {peak_memory / 1e6:.2f} MB")
</code></pre>
<p>environment:</p>
<pre><code class="lang-auto">Python Version: 3.9.19
PyTorch Version: 2.1.0+cu121
CUDA Available: Yes
CUDA Version: 12.1
Current GPU: Tesla V100-SXM2-16GB
CUDDN: 8902
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">head number 8 case peak memory: 405.17 MB
head number 16 case peak memory: 6716.14 MB
</code></pre>
<p>just double the head number need 16x more memory… is it normal?</p>
<p>I tried on another machine</p>
<pre><code class="lang-auto">Python Version: 3.9.19
PyTorch Version: 2.1.0+cu121
CUDA Available: Yes
CUDA Version: 12.1
Current GPU: NVIDIA GeForce GTX 1070
CUDDN: 8902
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">head number 8 case peak memory: 405.17 MB
head number 16 case peak memory: 406.49 MB
</code></pre>
<p>I think this should be right but I failed to achieve this on many other machines. Hope someone help, many thanks.</p>

username: Nova
cooked: <p>this is because dim = 64, head_num2 = 16, 64 // 16 = 4 and 4 is not divisible by 8. Pytorch becomes inefficient in this case.</p>
<p>To avoid this, also need to set dim = 128 as 128 // 16 = 8.</p>
Checking what autodiff is differentiating

username: gideonsimpson
cooked: <p>I’m just getting started with pytorch, but one thing I’m curious about is there a way to (visually) see the autograd tree?  I’m computing some intermediate values inside of <code>with torch.no_grad():</code> blocks, but I want to verify that the code is doing what I think it’s doing when it comes to differentiation.</p>

username: Aknw_Fen
cooked: <p>Did you check here?</p>
<p><a href="https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html</a></p>
<p>For multi steps, it needs the <code>backward(retain_graph=True)</code> or it forgets the intermediate gradients.</p>
<p>If you want the partial derivatives on each arrow, then I haven’t seen it. But you can just print <code>w</code>, <code>w.grad()</code>.</p>
Simple quantisation reproduction - how to convert state dict to int8

username: leopere
cooked: <p>Hello! I am learning more about quantisation and decided to code it. I am interested in a simple scalar quantisation on the model weights only. My approach is the following:</p>
<ul>
<li>get all the tensors related to model parameters (let them be p)</li>
<li>quantise them and save them in <code>int8</code> (let them be p_quantised)</li>
<li>add a pre-forward hook that dequantises the <code>int8</code> to <code>fp32</code> (p_quantised → p)</li>
<li>add a (post-)forward hook that replaces the dequantised-<code>fp32</code> with the previous int8 (p → p_quantised)</li>
</ul>
<p>My main issue is that i am unable to force the model state dict to actually use int8 rather than <code>fp32</code>. So that if i do:</p>
<pre><code class="lang-auto">sd = model.state_dict()
p = sd['some_parameter']  # p is in fp32
p_q = quantise(sd['some_paramter']) # p_q is now in int8
sd['some_paramter'] = p_q
model.load_state_dict(sd)
</code></pre>
<p>then the model has the values of p_q but they are in <code>fp32</code>. Adding .type(float.int8) or similar did not work. Is there a workaround to this behaviour? What is the suggested path to implement this (just for learning)? Do i need to use torch.quantisation library?</p>
<p>Thanks!</p>

username: jerryzh168
cooked: <p>yeah you’ll have change the model.weight from float32 to integer as well for this I think.</p>
<p>not sure what’s your goal but you can also start with using our flows: <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a> and understand how it works</p>
torch.ao.nn.quantizable.modules.activation.MultiheadAttention not loading the pre-trained model weights correctly

username: Vivekanand_Sahu
cooked: <p>I have a pre-trained model weights saved as .pt and I am using the model for quantization. I replaced the torch.nn.MultiheadAttention to torch.ao.nn.quantizable.modules.activation.MultiheadAttention<br>
but the loss I am getting is 10 times more. The rest of the things are kept same.</p>
<p>I imported the library like this</p>
<pre><code class="lang-auto">from torch.ao.nn.quantizable.modules.activation import MultiheadAttention
</code></pre>
<p>Then changed the code like this:</p>
<pre><code class="lang-auto">        #self.attn = nn.MultiheadAttention(
        self.attn = MultiheadAttention(
            embed_dim,
            num_heads,
            dropout=attn_dropout,
            add_bias_kv=add_bias_kv,
        )
</code></pre>
<p>Then importing the weights like this:</p>
<pre><code class="lang-auto">pretrained_dict = torch.load("/part-vol-2/weaver-core/particle_transformer/models/ParT_full.pt")

# Load only the parameters that exist in the model
model_dict = pre_trained_model_quant.state_dict()
model_dict.update(pretrained_dict)
pre_trained_model_quant.load_state_dict(model_dict)
</code></pre>
<p>Is there anything else that needs to be doe for configuration? Why am I getting this error?</p>

username: jerryzh168
cooked: <p>I don’t think we have tested this for training, you can check out how we use this module in <a href="https://github.com/pytorch/pytorch/blob/53f8a5fde268b51260f136d3b06fc1e86e6912a7/test/quantization/core/test_quantized_op.py#L3062" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/quantization/core/test_quantized_op.py at 53f8a5fde268b51260f136d3b06fc1e86e6912a7 · pytorch/pytorch · GitHub</a></p>
QConfig for Resnet50 with weights dtype quint8

username: giommariapilo
cooked: <p>Hello!<br>
I am trying to perform quantization aware training on Resnet50 on imagenet, but differently from the default I want the weights to be unsigned (that is quint8, i think)</p>
<pre><code>model = get_resnet50(pretrained = True, num_classes = 1000)
assert torch.cuda.is_available(), f'GPU not available'

model.to(cpu_device)
# Make a copy of the model for layer fusion
fused_model = deepcopy(model)

model.eval()

fused_model.eval()
device = torch.device("cuda")

 
fused_model = torch.quantization.fuse_modules(fused_model,
                                              [["conv1", "bn1", "relu"]],
                                              inplace=True)
for module_name, module in fused_model.named_children():
    if "layer" in module_name:
        for basic_block_name, basic_block in module.named_children():
            torch.quantization.fuse_modules(
                basic_block, [["conv1", "bn1", "relu1"], ["conv2", "bn2"]],
                inplace=True)
            for sub_block_name, sub_block in basic_block.named_children():
                if sub_block_name == "downsample":
                    torch.quantization.fuse_modules(sub_block,
                                                    [["0", "1"]],
                                                    inplace=True)
quantized_model = QuantizedResNet50(model_fp32=fused_model)
quantization_config = torch.quantization.QConfig(activation=torch.quantization.HistogramObserver.with_args(reduce_range=True), 
                                                 weight=torch.quantization.PerChannelMinMaxObserver.with_args(dtype=torch.quint8, qscheme=torch.per_channel_affine))
quantized_model.qconfig = quantization_config
torch.quantization.prepare_qat(quantized_model, inplace=True)
</code></pre>
<p>this is how I configure, then I launch the training but I get</p>
<p><code>AssertionError: Weight observer must have a dtype of qint8</code></p>
<p>Is there any way to do it? Am I missing something?</p>
<p>Thank you!</p>

username: Firas_Guediri
cooked: <p>in your weight observer arguments try changing dtype=torch.quint8 to dtype=torch.qint8:</p>
<p>quantization_config = torch.quantization.QConfig(activation=torch.quantization.HistogramObserver.with_args(reduce_range=True),<br>
weight=torch.quantization.PerChannelMinMaxObserver.with_args(dtype=<strong>torch.qint8</strong>, qscheme=torch.per_channel_affine))</p>

username: giommariapilo
cooked: <p>Yes but I want the weights to have dtype quint8, is it possible?</p>

username: jerryzh168
cooked: <p>this is not supported right now because backends (fbgemm kernels) expects qint8 for weight</p>

username: giommariapilo
cooked: <p>so it doesn’t even support qint32 for weight quantization?</p>

username: jerryzh168
cooked: <p>yeah that’s correct I think</p>
Load custom trained parameters into quantized model

username: imafan
cooked: <p>Hi guys, I created a custom simulated quantized Conv2d layer and replaced all of the convolutional layers in Ultralytics YOLOv8 with this to train it successfully. How can I incorporate pytorch’s quantized layers by replacing my custom quantized layers with pytorch’s builtin layers, as well as loading the parameters (weight, bias, scale, zero point) from my custom layer to take advantage of the speedup during inference?</p>

username: jerryzh168
cooked: <p>that might be tricky since you’d need to match numerics as well, wondering why you need to implement custom quantized layers if you want to use pytorch’s builtin layers in the end?</p>
Per channel setting for QAT Quantization

username: Kevin_Ryu
cooked: <p>Torch 2.4.1, Ubuntu 22.04 environment.<br>
I created quantizer instances with XNNPACKQuantizer and x86_inductor_quantizer, and used get_symmetric_quantization_config() for XNNPACK and the appropriate config function for x86. I then set both is_qat and is_per_channel to True using the set_global function.</p>
<p>Issue:<br>
After running prepare_qat_pt2e, when I check the returned model, only the weights are still set to per-tensor. I’d like to understand why it isn’t changing to per-channel and how to fix it. Conversely, when is_per_channel is set to False, everything, including the weights, changes to per-tensor.</p>

username: jerryzh168
cooked: <p>can you paste model quantization code and print the results?</p>