akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Correctlly assign weights BinaryClassification BCEWithLogitsLoss

username: Codenamics
cooked: <p>Hey</p>
<p>Can I ask if this is the correct way to assigned weights for unbalanced dataset?</p>
<p>my class weights are<br>
{0: 1.94, 1: 0.67}</p>
<p>criterion = torch.nn.BCEWithLogitsLoss(pos_weight = torch.FloatTensor([1.94 / 0.67]).to(device))</p>
<p>above is correct?</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/bceloss-with-class-weights/196991/4">This post</a> might be helpful.</p>

username: Codenamics
cooked: <p>num_negative / num positives</p>

username: Codenamics
cooked: <p>Patric question:</p>
<p>according to your post it is:</p>
<p>num_neg / num_positive</p>
<p>so if:</p>
<p>0: 10000 1:20000</p>
<p>it will be</p>
<p>10000/20000</p>
<p>but if</p>
<p>0: 20000 1:10000</p>
<p>20000/10000?</p>
<p>is this correct? my understanding</p>

username: ptrblck
cooked: <p>Yes, the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html">docs</a> are also giving the example:</p>
<blockquote>
<p>For example, if a dataset contains 100 positive and 300 negative examples of a single class, then <code>pos_weight</code> for the class should be equal to <code>300/100=3</code>. The loss would act as if the dataset contains <code>3×100=300</code> positive examples.</p>
</blockquote>
Torch.stack and device

username: Oktai15
cooked: <p>I want to stack list of something and convert it to gpu:</p>
<p><code>torch.stack(fatoms, 0).to(device=device)</code></p>
<p>As far as I know, tensor was created on cpu firstly and then would be transferred to specified device. How to put it on gpu straight?</p>

username: ptrblck
cooked: <p>If both <code>tensor</code>s are already on the GPU, the result will also have the save device:</p>
<pre><code class="lang-python">a = torch.randn(10, device='cuda:0')
b = torch.randn(10, device='cuda:0')

c = torch.stack((a, b))
print(c.device)
&gt; device(type='cuda', index=0)
</code></pre>

username: Oktai15
cooked: <p>yes, but if <code>fatoms</code> (look at my example) is list?</p>

username: ptrblck
cooked: <p>If your <code>fatoms</code> list contains CPU <code>tensor</code>s, then your example is the way to go.<br>
You could push the content onto the GPU before calling <code>torch.stack</code>, which is what my example shows.</p>

username: ndvbd
cooked: <p>is there a way to cost-effectively push a list of tensors to gpu?</p>
MPS support on torch 2.3.0

username: EarlyBird
cooked: <p>When I run PyTorch 2.3.0 it tells me that pytorch has no link to MPS<br>
With pytorch 2.2.2 it works.</p>
<p>Is there any way to get 2.3.0 to use MPS?</p>
<p>Regards<br>
Sven</p>
Pytorch with MPI backend

username: dolphin_09
cooked: <p>Hi Everyone,<br>
I am trying to run MPI with Pytorch from Source for distributed runs. I am able to build, compile and instal. But post installation, i am unable to import torch.</p>
<p>I am using OpenMPi and Pytorch latest version.</p>
<p>Let me know if i have to export any variables or if there is anything other information needed from side to proceed further.</p>

username: fduwjj
cooked: <p>Can you post what errors you run into after installing?</p>
Failure to connect dist process to individual GPU

username: AKT_TARAFDER
cooked: <p>I got 2 gpu in 1 node, P100, and V100<br>
I want to assign 1 process to 1 GPU. But this code fails.<br>
It shows problem assigning to different ranks.<br>
Running it without assigning to specific devices works.</p>
<pre><code class="lang-auto">
import os
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from typing import Callable



import os
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from typing import Callable

def init_process(rank: int, size: int, fn: Callable[[int, int], None], backend="nccl"):
    """Initialize the distributed environment."""
    os.environ["MASTER_ADDR"] = "127.0.0.1"
    os.environ["MASTER_PORT"] = "29505"  # Ensure the port is not in use
    os.environ["CUDA_VISIBLE_DEVICES"] = str(rank)  # Assign each process to a specific GPU
    try:
        dist.init_process_group(backend, rank=rank, world_size=size)
        torch.cuda.set_device(rank)  # Set the current device
        fn(rank, size)
    finally:
        dist.destroy_process_group()  # Ensure proper cleanup

def do_all_reduce(rank: int, size: int):
    """Perform an all-reduce operation."""
    group = dist.new_group(list(range(size)))
    tensor = torch.ones(1).cuda(rank)  # Make sure the tensor is on the correct GPU
    dist.all_reduce(tensor, op=dist.ReduceOp.SUM, group=group)
    print(f"[Rank {rank}] All-Reduce result: {tensor[0].item()} on GPU {rank}")

if __name__ == "__main__":
    size = 2  # Number of processes / GPUs
    mp.set_start_method("spawn")
    processes = []

    for rank in range(size):
        if rank &gt;= torch.cuda.device_count():
            raise ValueError(f"Invalid GPU rank: {rank}. Number of GPUs available: {torch.cuda.device_count()}")
        p = mp.Process(target=init_process, args=(rank, size, do_all_reduce, "nccl"))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

</code></pre>

username: fduwjj
cooked: <aside class="quote no-group" data-username="AKT_TARAFDER" data-post="1" data-topic="207791">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/akt_tarafder/48/35960_2.png" class="avatar"> AKT_TARAFDER:</div>
<blockquote>
<p><code>torch.cuda.set_device(rank)  # Set the current device</code></p>
</blockquote>
</aside>
<p>Maybe it’s because you do this:</p>
<pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"] = str(rank)
</code></pre>
<p>Usually we do set visible_devices same across ranks with a list of GPUs like [0,1, …]</p>
<p>Is this local_rank or global rank?</p>
About error module 'torch._inductor' has no attribute 'lowering'

username: luyahan
cooked: <p>AttributeError: module ‘torch._inductor’ has no attribute ‘lowering’</p>
<p>torch/_inductor/decomposition.py:728</p>
<pre><code class="lang-auto">  if (
        torch._inductor.lowering.should_fallback_max_pool2d_with_indices(
            kernel_size, dilation
        )
        or window_size &gt; torch.iinfo(torch.int8).max
    ):
</code></pre>
<p>What i should do?</p>

username: ptrblck
cooked: <p><code>import</code> the needed namespace:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

torch._inductor.lowering.should_fallback_max_pool2d_with_indices
# AttributeError: module 'torch._inductor' has no attribute 'lowering'

import torch._inductor.lowering
torch._inductor.lowering.should_fallback_max_pool2d_with_indices
# &lt;function torch._inductor.lowering.should_fallback_max_pool2d_with_indices(kernel_size, dilation)&gt;
</code></pre>
Hash Issue: ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE

username: bobmane
cooked: <p>Installing Deep-Live-Cam<br>
All goes well until end of install:</p>
<p>Windows PowerShell<br>
…<br>
…)<br>
Downloading ht… 2.2/2.6 GB 163.4 kB/s eta 0:46:10<br>
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.<br>
torch==2.0.1+cu118 from <a href="https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-win_amd64.whl#sha256=f58d75619bc96e4322343c030b893613701caa2d6db8017155da226c14171335" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-win_amd64.whl#sha256=f58d75619bc96e4322343c030b893613701caa2d6db8017155da226c14171335</a> (from -r requirements.txt (line 11)):<br>
Expected sha256 f58d75619bc96e4322343c030b893613701caa2d6db8017155da226c14171335<br>
Got ae8f20e6388b9b8df854b5b1175502cb28b092a7d008a693d75c67b219e57000</p>
<p>(venv) PS C:\Deep-Live-Cam&gt;</p>
<p>–</p>
<p>and in requirements.txt from C:\Deep-Live-Cam it has no hash on line 11:</p>
<p>–extra-index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a></p>
<p>numpy==1.23.5<br>
opencv-python==4.8.1.78<br>
onnx==1.16.0<br>
insightface==0.7.3<br>
psutil==5.9.8<br>
tk==0.1.0<br>
customtkinter==5.2.2<br>
pillow==9.5.0<br>
torch==2.0.1+cu118; sys_platform != ‘darwin’<br>
torch==2.0.1; sys_platform == ‘darwin’<br>
torchvision==0.15.2+cu118; sys_platform != ‘darwin’<br>
torchvision==0.15.2; sys_platform == ‘darwin’<br>
onnxruntime==1.18.0; sys_platform == ‘darwin’ and platform_machine != ‘arm64’<br>
onnxruntime-silicon==1.16.3; sys_platform == ‘darwin’ and platform_machine == ‘arm64’<br>
onnxruntime-gpu==1.18.0; sys_platform != ‘darwin’<br>
tensorflow==2.13.0rc1; sys_platform == ‘darwin’<br>
tensorflow==2.12.1; sys_platform != ‘darwin’<br>
opennsfw2==0.10.2<br>
protobuf==4.23.2<br>
tqdm==4.66.4<br>
gfpgan==1.3.8</p>
<p>so do i add to line 11 torch==2.0.1+cu118; sys_platform != ‘darwin’ --hash=sha256:ae8f20e6388b9b8df854b5b1175502cb28b092a7d008a693d75c67b219e57000</p>
<p>or something else?</p>
Onnx resnet50 export

username: alexander_seen
cooked: <p>I made very simple python script which loads <strong>torchvision ResNet50</strong> model and tries to export to onnx file in two ways (<strong>torch.onnx.export</strong> and <strong>torch.onnx.dynamo_export</strong> )</p>
<pre><code class="lang-auto">import torch
import torch.onnx

import torchvision

torch_model = torchvision.models.detection.fasterrcnn_resnet50_fpn_v2( weights='DEFAULT')
torch_model.eval()
torch_input = torch.randn(1, 3, 32, 32)

is_dynamo_export = False

if (is_dynamo_export):
    onnx_program = torch.onnx.dynamo_export(torch_model, torch_input)
    onnx_program.save("onnx_dynamo_export_ResNET50.onnx")        
else:
    torch.onnx.export(torch_model,               # model being run
                      torch_input,                         # model input (or a tuple for multiple inputs)
                      "onnx_export_ResNET50.onnx",   # where to save the model (can be a file or file-like object)
                      export_params=True,        # store the trained parameter weights inside the model file
                      opset_version=10,          # the ONNX version to export the model to
                      do_constant_folding=True,  # whether to execute constant folding for optimization
                      input_names = ['input'],   # the model's input names
                      output_names = ['output'], # the model's output names
                      dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes
                                    'output' : {0 : 'batch_size'}})  
</code></pre>
<p>The errors were appeared:</p>
<pre><code class="lang-auto">torch\onnx\_internal\exporter.py", line 1439, in dynamo_export
  raise OnnxExporterError(

torch.onnx.OnnxExporterError: Failed to export the model to ONNX. Generating SARIF
</code></pre>
<pre><code class="lang-auto">torch.onnx.errors.SymbolicValueError: Unsupported: ONNX export of Pad in opset 9. The sizes of the padding must be constant. Please try opset version 11. [Caused by the value '535 defined in (%535 : int[] = prim::ListConstruct(%405, %534, %405, %533, %405, %532), scope: torchvision.models.detection.faster_rcnn.FasterRCNN::
</code></pre>
<p>Both methods works good with extremally simple models such as</p>
<pre><code class="lang-auto">class MyModel(nn.Module):

    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = torch.flatten(x, 1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
Next step after NLP specialization

username: Abdulrahman_Ahmed
cooked: <p>I’m about to finish the last course in NLP specialization on Coursera, i want to know the next step in the study to be ready for work?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3f91824d08ed881016abe28ae5dec212703f894.png" data-download-href="https://discuss.pytorch.org/uploads/default/d3f91824d08ed881016abe28ae5dec212703f894" title="Specialization-Logo"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3f91824d08ed881016abe28ae5dec212703f894_2_500x500.png" alt="Specialization-Logo" data-base62-sha1="ufcqyDHaDYuq1bWkWsIFqkmUc5e" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3f91824d08ed881016abe28ae5dec212703f894_2_500x500.png, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3f91824d08ed881016abe28ae5dec212703f894.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3f91824d08ed881016abe28ae5dec212703f894.png 2x" data-dominant-color="6E80B6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Specialization-Logo</span><span class="informations">600×600 63.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: AbdulsalamBande
cooked: <p>Congrats on finishing the specialization! To me, the best next step is to get hands-on experience—build projects, contribute to open source, and keep coding to solidify your skills.</p>
How to dynamically quantize MultiheadAttention using torch.ao.nn.quantizable.modules.activation.MultiheadAttention?

username: Vivekanand_Sahu
cooked: <p>I am using using a transformer model for my project where I am using MultiheadAttention. I want to quantize it. In the <a href="https://github.com/pytorch/pytorch/blob/main/torch/nn/quantizable/modules/activation.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/nn/quantizable/modules/activation.py at main · pytorch/pytorch · GitHub</a> it has been mentioned to use the following</p>
<pre><code class="lang-auto">from torch.ao.nn.quantizable.modules.activation import MultiheadAttention
</code></pre>
<p>This is my code which gives an error:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.ao.nn.quantizable.modules.activation import MultiheadAttention

class DummyModel(nn.Module):
    def __init__(self, embed_dim, num_heads):
        super(DummyModel, self).__init__()
        self.embedding = nn.Embedding(10, embed_dim)
        self.attention = MultiheadAttention(embed_dim, num_heads)
        self.linear = nn.Linear(embed_dim, embed_dim)
        self.output = nn.Linear(embed_dim, 1)
        
    def forward(self, x):
        x = self.embedding(x)
        x, _ = self.attention(x, x, x)
        x = self.linear(x)
        x = torch.relu(x)
        x = self.output(x)
        return x

# Instantiate the model
embed_dim = 16
num_heads = 2
model = DummyModel(embed_dim, num_heads)


from torch.ao.quantization import quantize_dynamic

# Switch the model to evaluation mode
model.eval()

# Apply dynamic quantization
quantized_model = quantize_dynamic(
    model, 
    {nn.Linear, MultiheadAttention}, 
    dtype=torch.qint8
)
</code></pre>
Best practices for adapting models

username: heth27
cooked: <p>What are best practices for adapting models that are already implemented in the library, for example I want to adapt efficientnet by constraining the weights<br>
<code>from torchvision.models import EfficientNet_V2_S_Weights, efficientnet_v2_s</code><br>
Does it make more sense for someone else to be able to understand my code to copy all the necessary modules and indicate my changes, or do something with monkey patching, like Conv2d.forward = constraint_forward</p>
How to perform mixed precsion on single F.linear?

username: weight_theta
cooked: <p>Dear Community,</p>
<p>I was wondering whether I can further reduce memory consumption on a specific operation that I identified that is using a lot of memory. I noticed that 1) the <strong>A</strong> and <strong>B</strong> matrix both as float.32 and 2) the resulting coefficients <strong>c</strong> also float.32, lead to rather large memory usage.</p>
<p>While I can manually cast A and B to float.16 so that the resulting coefficients <strong>c</strong> are also float.16, leading to quite reduced memory constraint, I was wondering whether it is possible to use mixed precision instead of manually casting the tensors to float.16 ?</p>
<p>I am not sure whether this is necessary, since we dont need gradients on <strong>A</strong> and <strong>B</strong> but on the coefficients <strong>c</strong>.</p>
<p>This is my code:</p>
<pre><code class="lang-auto">self.A = torch.from_numpy(GBTA).to(device, dtype=torch.float16).requires_grad_(False)
self.B = torch.from_numpy(GBTB).to(device, dtype=torch.float16).requires_grad_(False)
c_t = F.linear(c_t.half(), self.A[t].half()) + self.B[t].squeeze(-1) * input
</code></pre>
<p>I am not sure how to wrap gad scalar into this, as we already do auto-cast mixed precision and pass the scalar from gradscalar into the training  function:</p>
<pre><code class="lang-auto"> scaler = GradScaler()
            train_loss, train_class_acc, train_noobj_acc, train_obj_acc = (
                trainholov4_enas_vid_bptt(
                    device,
                    train_loader,
                    model,
                    optimizer,
                    scheduler,
                    loss_f,
                    scaled_anchors,
                    scaler,
                    conf_thresh=0.8,
                    mode="ciou",
                    target_batch_size=args.target_batch_size,
                    ngpus=args.ngpus,
                )
            )
</code></pre>
<p>So it may not be needed to add gad scalar and maybe something like this would just work ?</p>
<pre><code class="lang-auto">self.A = torch.from_numpy(GBTA).to(device, dtype=torch.float16).requires_grad_(False)
self.B = torch.from_numpy(GBTB).to(device, dtype=torch.float16).requires_grad_(False)
with autocast():
     c_t = F.linear(c_t.half(), self.A[t].half()) + self.B[t].squeeze(-1) * input
</code></pre>
<p>Please let me know, I would be happy to hear about suggestions or answers.</p>

username: ptrblck
cooked: <p>The casts should not be needed when <code>autocast</code> is used. Do you see a different behavior and memory usage between <code>autocast</code> and the explicit transformations?</p>
Lstm chatbot not learning

username: Faruk_Celik
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85add63534048504e03c5688bd8a292fbc9e8c52.png" alt="Screenshot 2024-08-10 065109" data-base62-sha1="j4zWIBshsl6wHvCcCRgrR9mrQT8" width="604" height="491"><br>
output after 150 epochs:<br>
Question: ‘hello’, Answer: 'i do ’<br>
Question: ‘how was the traffic’, Answer: ‘a set for rules in the underlying entities in work’<br>
Question: ‘what is a bug’, Answer: 'transparent system ’</p>
<pre><code class="lang-auto"># Define LSTM model class
class NCGM1(nn.Module):
    def __init__(self, num_emb, num_layers=1, emb_size=128, hidden_size=128):
        super(NCGM1, self).__init__()
        
        # Embedding layer
        self.embedding = nn.Embedding(num_emb, emb_size)

        # MLP layer for embedding
        self.mlp_emb = nn.Sequential(
            nn.Linear(emb_size, emb_size),
            nn.LayerNorm(emb_size),
            nn.ELU(),
            nn.Linear(emb_size, emb_size)
        )
        
        # LSTM layer
        self.lstm = nn.LSTM(
            input_size=emb_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            batch_first=True,
            dropout=0.25
        )

        # MLP layer for output
        self.mlp_out = nn.Sequential(
            nn.Linear(hidden_size, hidden_size // 2),
            nn.LayerNorm(hidden_size // 2),
            nn.ELU(),
            nn.Dropout(0.5),
            nn.Linear(hidden_size // 2, num_emb)
        )
        
    def forward(self, input_seq, hidden_in, mem_in):
        # Embed input sequence
        input_embs = self.embedding(input_seq.type(torch.LongTensor).to(input_seq.device))
        # Pass through MLP for embedding
        input_embs = self.mlp_emb(input_embs)
                
        # Pass through LSTM layer
        output, (hidden_out, mem_out) = self.lstm(input_embs, (hidden_in, mem_in))
                
        # Pass through MLP for output
        return self.mlp_out(output), hidden_out, mem_out```</code></pre>

username: Codenamics
cooked: <p>Train accuracy, Val accuracy, Val Loss? might help to diagnose</p>

username: Brock_Brown
cooked: <p>Validation loss would help as Codenamics says, but also if your validation loss is still steadily decreasing, your training graph seems to indicate that your model can still learn more, the line hasn’t flattened.</p>

username: Codenamics
cooked: <p>yeah but still it can be overfit. we need val loss</p>
Downloading pytorch

username: johann4y
cooked: <p>Hey sorry I am very new to this sort of stuff. I have a quick question, if I downloaded cuda 12.6 from Invidia can I still download pytorch pip cuda 12.4 or would it no longer work? Thank you</p>

username: ptrblck
cooked: <p>Yes, you can download any PyTorch binary as it ships with its own CUDA dependencies and your locally installed CUDA toolkit will be used if you build PyTorch from source or a custom CUDA extension.</p>
Unable to create Adam Optimizer

username: Abhishek_Shivakumar
cooked: <pre><code class="lang-auto">        std::vector&lt;torch::Tensor&gt; params;
        for (size_t i = 0; i &lt; 3; i++) 
        {
            params.push_back (torch::randn (10));
        }
         
        auto options = torch::optim::AdamOptions (learning_rate);

        // Construct the optimizer without moving param_groups
        torch::optim::Adam optimizer (params, options); // base learning rate
</code></pre>
<p>I am unable to create the optimizer object without hitting a bad access error.</p>