akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use torch.exp() with flatten()

Jusseh: <p>Hi everyone,<br>
I’ve trained a model using CNN like this:</p>
<pre><code class="lang-auto">model = nn.Sequential(
                      nn.Conv2d(in_channels=3,out_channels=16,kernel_size=5,padding_mode = 'zeros'),
                      nn.Flatten(),
                      nn.Linear(3701776,256),
                      nn.ReLU(),
                      nn.Dropout(0.2),
                      nn.Linear(256, 128),
                      nn.ReLU(),
                      nn.Dropout(0.2),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                      nn.Dropout(0.2),
                      nn.Linear(64, 5),
                      nn.LogSoftmax(dim = 1)
                     )
</code></pre>
<p>I can train the model just fine calculating loss / accuracy like this:</p>
<pre><code class="lang-auto">    with torch.no_grad():
      # Set the model to evaluation mode
      model.eval()
      
      # Validation pass
      for images, labels in test_dataloader:
        
        log_ps = model(images)
       
        test_loss += criterion(log_ps, labels)
        
        ps = torch.exp(log_ps)
        top_p, top_class = ps.topk(1, dim = 1)
        equals = top_class == labels.view(*top_class.shape)
        accuracy += torch.mean(equals.type(torch.FloatTensor))
</code></pre>
<p>But I don’t know how to test samples, since when I try to call</p>
<pre><code class="lang-auto">ps = torch.exp(model(img))
</code></pre>
<p>I get an error:</p>
<p>mat1 and mat2 shapes cannot be multiplied (16x231361 and 3701776x256)</p>
<p>Probably since I’m not passing in batches, but I’m not sure how to fix this. Any help would be greatly appreciated</p>

ptrblck: <p>The <code>torch.exp</code> call should not fail as it’s an elementwise operation and thus won’t raise a shape mismatch. Check the input shapes to your model and make sure these are valid.</p>

Jusseh: <p>Thank you for your awnser,</p>
<p>The input shape during training is: torch.Size([64, 3, 485, 485]), 64 being the batch size and torch.exp(model(img)) works fine.</p>
<p>But when I try to test the model afterwards the input shape is: torch.Size([3, 485, 485]), which causes the error. I’m not sure how to fix this, since I don’t want to test a batch?</p>

ptrblck: <p>Add the missing batch dimension explicitly via <code>model(x.unsqueeze(0))</code> and it should work.<br>
The reason for the error is that the <code>nn.Flatten</code> will keep <code>dim0</code> of the intermediate activation and will treat it as the batch dimension. However, when no batch dimension is passed to the model, <code>dim0</code> will represent the channel dimension and the shape mismatch error will be raised.</p>

Jusseh: <p>That worked, thanks a lot!</p>
Can transformer automatically learn the length of sequences?

JimW: <p>I am new to transformers and I need to use transformer to predict a score based on a input sequence of variable length.</p>
<p>Due to the nature of my use case, longer sequnences tend to provide higher score for regression, although it is not guaranteed. However, my pytorch experiment only learns to recognize the sequence patterns which can lead to a sudden increase of the score, but it doesn’t seem to learn the fact that “Longer sequence tends to provide higher scores”. So my question is whether transformer can automatically learn the length of the sequence or I need to explictly feed the length of the sequence as an additional input to the network?</p>
<p>My current implementation is like this:</p>
<blockquote>
<p>import torch.transformer as  transformer<br>
input_seq =  …<br>
score = transformer(input_seq)</p>
</blockquote>
Reading GPU memory from another process

deepfailure: <p>Hello,</p>
<p>I have two processes:</p>
<p>Process A C++ with cublas</p>
<p>Process B with python</p>
<p>The goal is to read data from process A in the GPU and train in process B</p>
<p>How can I start looking into that? My question is from the point of view of process B.</p>

ptrblck: <p>You could use CUDA IPC handles, but since Process A is a pure C++ code you would need to unpack the data manually as described <a href="https://discuss.pytorch.org/t/using-cuda-ipc-memory-handles-in-pytorch/17548/4">here</a>.</p>

zizhao.mo: <p>Hi,</p>
<p>If two processes are both written in Python, can the cuda tensor be shared in PyTorch from one process to another?</p>
<p>BR</p>
How to avoid model not detecting a class in some of the epochs

palguna_gopireddy: <p>I have trained a Vgg16 model. I am able to train the model, while testing the model I found that one class is not getting recognized in some epochs, but in other epochs it is detecting all the classes. (which I came to know by printing the output labels using <code>np.unique(y_predict)</code>)</p>
<p>I printed the classification report and accuracy using <code>sklearn.metrics import classification_report,confusion_matrix</code>, which resulted the following warning only on those epochs which it did not detect one of the classes, other classed it did not result this waring as it detected all the classes. (I know the waring is appearing as it was caused by zero dvision error)<br>
<code>UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use </code>zero_division<code> parameter to control this behavior.</code></p>
<p>I am able to detect all the classes in the dataset using some other model in all the epochs, but Vgg16 is not detecting one of the classes, in some epochs, detecting all the classes in some epcohs.</p>
<p>My request to how to solve this problem, i.e. how to make the model detect all the classes in all the epochs? I there a way like using the softmax at the end of the layer or any other solution?</p>
Class imbalance in pixel classification

YASJAY: <p>Hi,</p>
<p>Dealing with a dataset with pixel-level annotations, say A1, A2, A3 and background pixels.<br>
There is huge class imbalance between A’s and the bg pixels. How is one supposed to tackle this?</p>

ZdsAlpha: <p>For training you will have to convert pixels to labels for cross-entropy loss. You can use bincount to count occurrence of each type of pixel and then use weighted cross-entropy loss.</p>

YASJAY: <p>Thanks, did exactly the same.<br>
Is there a good resource on choosing the perfect weights? I have been using 1/frequency as the weights.</p>

ZdsAlpha: <p>Not sure but I do the same. 1/frequency is good. You can also do something like 1/softmax(frequency). This one will not break if there is a class that does not exist in dataset (ie 1/frequency -&gt; inf) or extremely infrequent. Depends on your requirements. BTW I never tested it, just made it up.</p>

YASJAY: <p>Can we make use of the ignore label argument to the loss function for a class like background pixels?</p>

ZdsAlpha: <p>Use ignore_index argument. That should do the job.</p>

SatyamGaba: <p>if some class is not there in your input images then why would you have labels for it? Also to be double sure you can use ignore_index argument as mentioned in the above reply.</p>

Sharma-Tu: <p>Can you share the network architecture for such a pixel level mutli-class classifier?</p>
Fine-tuning with LoRA using peft

dumpling: <p>Hi, I am a beginner. I want to fine-tune a model using LoRA with the peft package, it’s this model:  <a href="https://github.com/lbcb-sci/RiNALMo" class="inline-onebox" rel="noopener nofollow ugc">GitHub - lbcb-sci/RiNALMo: RiboNucleic Acid (RNA) Language Model</a>. However, since only torch.nn.Linear and Conv1D are supported, I cannot fine-tune it and select the layers I want as the linear layers for, say, query key and value matrices are wrapped in another module. If I show the named modules, layers, they are wrapped basically. Can anyone help to circumvent this? Is there a way to access the linear layers wrapped in a bigger module?</p>

Mayank_Kumar: <p>what do you mean by they are wrapped in another module ?</p>
<p>Like in Huggingface transformers, query matrix is at:</p>
<pre><code class="lang-auto">model.vit.encoder.layer[0].attention.attention.query
</code></pre>
<p>Is this what you need ?</p>

dumpling: <p>Thanks for the response! So if you do the following for this rinalmo model:</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
    if param.requires_grad:
        print(name)
</code></pre>
<p>You will get something like this:</p>
<pre><code class="lang-auto">embedding.weight
transformer.blocks.0.mh_attn.Wqkv.weight
transformer.blocks.0.mh_attn.out_proj.weight
transformer.blocks.0.attn_layer_norm.weight
transformer.blocks.0.attn_layer_norm.bias
transformer.blocks.0.transition.0.beta
transformer.blocks.0.transition.0.linear.weight
transformer.blocks.0.transition.0.linear.bias
transformer.blocks.0.transition.0.linear_gate.weight
transformer.blocks.0.transition.0.linear_gate.bias
transformer.blocks.0.transition.2.weight
transformer.blocks.0.transition.2.bias
transformer.blocks.0.out_layer_norm.weight
transformer.blocks.0.out_layer_norm.bias
transformer.blocks.1.mh_attn.Wqkv.weight
transformer.blocks.1.mh_attn.out_proj.weight
transformer.blocks.1.attn_layer_norm.weight
transformer.blocks.1.attn_layer_norm.bias
transformer.blocks.1.transition.0.beta
transformer.blocks.1.transition.0.linear.weight
transformer.blocks.1.transition.0.linear.bias
transformer.blocks.1.transition.0.linear_gate.weight
transformer.blocks.1.transition.0.linear_gate.bias
transformer.blocks.1.transition.2.weight
transformer.blocks.1.transition.2.bias
transformer.blocks.1.out_layer_norm.weight
</code></pre>
<p>My question is, how can i fine-tune the query, key, and value matrices when they are wrapped in mh_attn? If i inspect the model, the layers i want to fine tune with lora is in “flash_self_attn” but they are not listed there?</p>

Mayank_Kumar: <p>Wow <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20">, weird. Can you try printing all parameters regardless of their requires_grad param ? To see if “flash_self_attn” is even their.</p>
Initialization of distributed environment on Windows using gloo backend

Yikai_Xu: <p>I’m trying to conduct single-node multi-CPU training on a Windows platform, here is my environment  initialization code</p>
<pre><code class="lang-auto">import os
import time
import torch
import torch.distributed as dist
import torch.multiprocessing as mp


def run(rank, size):
    """ Distributed function to be implemented later. """
    print("rank", rank)
    print("size", size)
    pass


def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["USE_LIBUV"] = "0"
    os.environ["MASTER_PORT"] = "12355"
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)


if __name__ == "__main__":
    size = 2
    processes = []
    mp.set_start_method("spawn")
    for rank in range(size):
        p = mp.Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
</code></pre>
<p>which lead the following error：</p>
<p><code>[W907 22:07:29.000000000 socket.cpp:697] [c10d] The client socket has failed to connect to [Yikai-Xu-Office]:12355 (system error: 10049: The requested address is not valid in its context</code></p>
<p>some regular check such as filewall setting and port availability has already been made and is useless. However,it can run successfully on a Mac platform, can sombody solve this for me?</p>
<p>Note: I must on windows and use single-machine multi-CPU mode for special reason, hence gloo seems the only choice for me while NCLL not support CPU and MPI install requires reinstall pytorch from source which is too complicated for me.</p>
<p>Thanks in advance!</p>

Eta_C: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/face_holding_back_tears.png?v=12" title=":face_holding_back_tears:" class="emoji" alt=":face_holding_back_tears:" loading="lazy" width="20" height="20"> do you have any suggestion?</p>

ptrblck: <p>No, unfortunately I have never used the combination of Gloo and Windows and am not familiar with common setup issues. Maybe <a class="mention" href="/u/kwen2501">@kwen2501</a> might know.</p>
Obtaining Toeplitz matrix for convolution

McLawrence: <p>In this <a href="https://github.com/pytorch/pytorch/issues/6124#issuecomment-404950023" rel="nofollow noopener">issue</a> <a class="mention" href="/u/ezyang">@ezyang</a> references an implementation of convolutions that uses the Toeplitz matrix.</p>
<p>I have a <code>state_dict</code> (and also a <code>nn.Module</code> class) from a network and explicitly need these Toeplitz matrices for further calculations but I admittedly have not a strong grasp on the things going on in <code>ATen</code> and how I could use that directly in Python. Is there a way to do this?</p>

Mazhar_Shaikh: <p>Hi McLawrence,<br>
Please refer to the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Unfold" rel="nofollow noopener">nn.Unfold module</a>. This would enable you to generate the toeplitz matrices(column matrix).<br>
Hope this helps!</p>

McLawrence: <p><a class="mention" href="/u/mazhar_shaikh">@Mazhar_Shaikh</a>  Using unfold, I can create a matrix from the input and do a matrix multiplication with the kernel vector.<br>
However, what I need would be a matrix from the kernel, not the input. Which seems not possible with unfold (as the kernel is smaller than the input). Do you know of an alternative?</p>

MassimoWRivolta: <p>I’m not sure if this could be useful. However, since I recently needed a Toeplitz matrix in custom layer and this was the only resource I could find, I prepared a function that returns the Toeplitz matrix for real values using the Unfold module.</p>
<p>Here’s the function:</p>
<pre><code class="lang-auto">    def pytorch_toeplitz(V):
    '''
    It creates the Toeplitz matrix for each row in V.

    INPUT:
    V: torch tensor (batch_size x d)

    OUTPUT:
    T: (batch_size x d x d)

    EXAMPLE:
    V = torch.tensor([[1, 0.5, 0.1], [1.3, 0.9, -0.1]])
    T = pytorch_toeplitz(V)
    print(T.shape)
    print(T[0])
    print(T[1])
    '''
        
    d = V.shape[1]
    A = V.unsqueeze(1).unsqueeze(2)
    A_nofirst_flipped = torch.flip(A[:, :, :, 1:], dims=[3]) 
    A_concat = torch.concatenate([A_nofirst_flipped, A], dim=3) 
    unfold = torch.nn.Unfold(kernel_size=(1, d))
    T = unfold(A_concat)
    T = torch.flip(T, dims=[2])

    return T</code></pre>
How can I release allocated memory inside a non-default CUDA stream?

dem123456789: <p>In the following code, I notice that CUDA memory usage increases each time I create a new stream. Even after calling <code>gc.collect()</code> and <code>torch.cuda.empty_cache()</code>, the memory allocated on the GPU seems to multiply with each iteration.</p>
<p>Is there a way to release the memory allocated by each non-default stream?</p>
<p>Here’s the code for reference:</p>
<pre data-code-wrap="python"><code class="lang-python">import gc
import torch
import torch.nn as nn

def run(model):
    model = model.to('cuda')
    input = torch.rand(10, 100)
    target = torch.rand(10, 100)
    input = input.to('cuda')
    target = target.to('cuda')
    model.train(True)
    with torch.cuda.stream(stream):
        output = model(input)
        loss = torch.nn.functional.mse_loss(output, target)
        loss.backward()
    stream.synchronize()
    print('allocated', torch.cuda.memory_allocated())
    print('reserved', torch.cuda.memory_reserved())

model = nn.Sequential(nn.Linear(100, 100), nn.Linear(100, 100))
for i in range(4):
    print(i)
    stream = torch.cuda.Stream()
    run(model)
    model.train(False)
    model.to('cpu')
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.synchronize()
</code></pre>
<p>And here is the memory allocation printout:</p>
<pre><code class="lang-auto">0
allocated 17219584
reserved 25165824
1
allocated 34258944
reserved 46137344
2
allocated 51298304
reserved 67108864
3
allocated 68337664
reserved 88080384
</code></pre>
<p>Why does the allocated memory keep increasing with each new stream, and how can I properly release it?</p>
Index tensor must have same dimensions as input tensor

Uk_Jo: <p>Help me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>import torch<br>
import random<br>
t = torch.randn((32,100))<br>
a = torch.tensor([random.randint(0,100) for a in range(32)])<br>
torch.gather(t, 1, a)</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “”, line 6, in <br>
RuntimeError: invalid argument 4: Index tensor must have same dimensions as input tensor at C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorEvenMoreMath.cpp:453</p>
</blockquote>

ptrblck: <p>If you want to use <code>a</code> as the index in dim1 for <code>t</code>, this code should work:</p>
<pre><code class="lang-python">t[torch.arange(t.size(0)), a]
</code></pre>

Uk_Jo: <p>torch.gather(t, 1, a.reshape(32, 1))</p>
<p>I had a mistake. It also works!!</p>
Return the derivative of a nn with respect to one of its input as a model

JosueCom: <p>As the title says, is there a way to return the derivative of a nn module with respect to one of its input as a separate nn module where the operations in the nn module are known?</p>
<p>For instance, could the graph generated by grad() be converted to a nn module that doesn’t rely on grad inside the module? I am interested in training a nn model using grad(create_graph=true), then have the structure of its derivative with respect to an input to implement the same math operations with obtained parameters on another program.</p>
nn.Embedding performance issue?

Tianxia_Wang: <p>The <code>nn.functional.embedding</code> function on the CPU shows that the time taken for FP32 operations is approximately double that of FP16 and BF16 operations, which is reasonable due to the differences in per-element size.</p>
<p>However, on CUDA, the time taken for FP32 operations is very similar to that of FP16 and BF16 operations, which seems unreasonable.</p>
<p>I ran this cuda code on colab T4 GPU, and I pasted the output below.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn
import time

device = 'cuda'

vocab_size = 8192
n = 768
token_count = 384 * 1024
loop_times = 1000

for dtype in [torch.bfloat16, torch.float16, torch.float]:
    indices = torch.randint(0, vocab_size, (token_count,), dtype=torch.int, device=device)
    weight = torch.rand((vocab_size, n), dtype=dtype, device=device)

    # warm up
    for i in range(0, 10):
        output = nn.functional.embedding(indices, weight)
        torch.cuda.synchronize()

    start = time.time_ns()
    for i in range(0, loop_times):
        output = nn.functional.embedding(indices, weight)
        torch.cuda.synchronize()
    end = time.time_ns()

    print('elapsed time: %f ms' % ((end - start) / 1e6 / loop_times))
    print(output.shape)
    print(output.dtype)
</code></pre>
<p>Output:</p>
<pre><code class="lang-plaintext">elapsed time: 10.571431 ms
torch.Size([393216, 768])
torch.bfloat16
elapsed time: 10.287250 ms
torch.Size([393216, 768])
torch.float16
elapsed time: 12.261419 ms
torch.Size([393216, 768])
torch.float32
</code></pre>
<p>Change <code>device = 'cuda'</code> to <code>device = 'cpu'</code>. The output is pasted below.</p>
<pre><code class="lang-plaintext">elapsed time: 376.148446 ms
torch.Size([393216, 768])
torch.bfloat16
elapsed time: 355.628159 ms
torch.Size([393216, 768])
torch.float16
elapsed time: 720.934025 ms
torch.Size([393216, 768])
torch.float32
</code></pre>

AbdulsalamBande: <p>GPUs are optimized for matrix operations in linear and convolutional layers, but embeddings rely on memory lookups and don’t benefit from that acceleration. Even if acceleration is applied, FP32 and FP16/BF16 are processed in parallel, which minimizes the performance gap on the GPU.</p>
Libtorch CPP Api for Memory Format Channels Last

Abhyuday98: <h2><a name="p-453864-background-1" class="anchor" href="#p-453864-background-1"></a>Background</h2>
<p>I am have been using the pytorch on Android.<br>
As mentioned in the following article <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html" rel="noopener nofollow ugc">link</a>, the processing is faster on channel_last format, and in a real world use case, I have also observed the same affect, where the inference is faster on the channels last memory format.<br>
Now for both the Android and Python libraries, there is a function to convert the input tensor to channels_last memory format.<br>
Python:</p>
<pre><code class="lang-auto">x = x.to(memory_format=torch.channels_last)
</code></pre>
<p>Android (Kotlin):</p>
<pre><code class="lang-auto">val inputTensor = TensorImageUtils.bitmapToFloat32Tensor(
                bitmap,
                IMAGE_MEAN,
                IMAGE_STD,
                MemoryFormat.CHANNELS_LAST
            )
</code></pre>
<p>However, I now have a requirement of running the model in the Android NDK. That means I need to implement the inference in C++ using the libtorch library. However, I can’t find the function to do the memory format change to channels last.</p>
<p>This is what I am currently doing is the following in libtorch in C++:</p>
<pre><code class="lang-auto">cv::Mat image = imgP.clone();
        auto tensorImage = torch::from_blob(image.data, {1, image.rows, image.cols, 3}, torch::kByte).to(torch::kFloat32).permute({0, 3, 1, 2}).div(255.0);
tensorImage = tensorImage.to(torch::MemoryFormat::ChannelsLast);

</code></pre>
<h2><a name="p-453864-question-is-my-apporach-in-c-correct-2" class="anchor" href="#p-453864-question-is-my-apporach-in-c-correct-2"></a>Question: Is my apporach in C++ correct?</h2>

ptrblck: <p>Yes, transforming the inputs and parameters should work as shown <a href="https://discuss.pytorch.org/t/does-torch-nn-conv2d-support-tensors-for-nhwc-layout-in-libtorch/208193/4">here</a>.</p>
Unet Model focuses on small structures and not the overall structure

Radu_Popovici: <p>Hi, I am working on a Unet model for HnE images of prostate cancer and I am encountering the issue where the model focuses on small structures and not the overall tissue structure. I am using a combination of Tversky and Focal Loss for my loss function with alpha 0.8, beta 0.2 and gamma 3.0, where the importance of Tversky is 0.8 and focal is 0.2. Do you have any thoughts on how I can make the model not focus so heavily on the small structures? The HnE image is 3100x3100 and I have extracted patches of 1024x1024 with a side of 512 which have been resized to 256. Thank you!</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/742a0fd66f93634d37e640b4608a296a7d25cd38.png" alt="Screen Shot 2024-09-07 at 17.47.39 PM" data-base62-sha1="gzDveeKehQhZPGBQicEhfV3N2mk" width="568" height="185"></p>
How to train only a certain number of parameters of a PyTorch tensor

Chandan_Tankala: <p>Hello,</p>
<p>0</p>
<p>I have a 2 X 2 tensor in PyTorch. I want the first row entries of this tensor (matrix) to be fixed, i.e., I do not want the first row entries to be parameters that are trained using gradient descent, and the second row entries be updated by the training process (using gradient descent). I saw online that I can use the register_buffer function in PyTorch to create a gradient_mask but I am not sure about the exact implementation.</p>
<pre><code class="lang-auto">matrix = torch.nn.Parameter(torch.zeros(2, 2))
register_buffer(‘gradient_mask’, ???)
</code></pre>
<p>Also, after I create this gradient_mask I do not know how to apply it to my variable: matrix. Any thoughts? Thanks.</p>

AbdulsalamBande: <p>I believe you can manually store the first row and then restore it after training - I created a simple example</p>
<pre><code class="lang-auto">import torch
import torch.optim as optim
matrix = torch.nn.Parameter(torch.randn(2, 2))
print("Original Parameters",matrix )
gradient_mask = torch.tensor([[0, 0], [1, 1]], dtype=torch.float32)
optimizer = optim.SGD([matrix], lr=0.01)
input_data = torch.randn(2, 1)
output = torch.matmul(matrix, input_data)

#Calculate the loss
loss = output.sum()
optimizer.zero_grad()
loss.backward()

# Apply the gradient mask to zero out gradients for the first row
with torch.no_grad():
    matrix.grad *= gradient_mask

optimizer.step()

# Restore the first row to its initial value
initial_first_row = matrix[0].clone()
with torch.no_grad():
    matrix[0] = initial_first_row
print("Parameters after restoring the first row:\n", matrix)

</code></pre>
Tversky Focal loss

Radu_Popovici: <p>I want to implement a custom loss function of a Unet model for HnE images and I made this so far, though I am not sure if I made any reasoning mistakes. Both my predictions and annotations are of the shape B C H W and my annotations have been one-hot encoded, where there is a 1 in the respective channel.</p>
<pre><code class="lang-auto">class Tversky_Focal_Loss(nn.Module):
    def __init__(self, device, weight=None, alpha=0.85, beta=0.15, gamma=3.0, epsilon=1e-7):
        super(Tversky_Focal_Loss, self).__init__()
        if weight is not None:
            self.weight = weight.to(device=device)
        self.alpha = alpha
        self.beta = beta
        self.gamma = gamma
        self.epsilon = epsilon
        self.device = device
    
    def forward(self, activations, annotations):
        activations = activations.float().to(self.device)
        annotations = annotations.float().to(self.device)

        # Softmax across channels to get class probabilities
        probabilities = torch.softmax(activations, dim=1)  # Shape: [B, C, H, W]
        probabilities = torch.clamp(probabilities, min=self.epsilon, max=1 - self.epsilon)

        # Tversky loss computation
        true_pos = (probabilities * annotations).sum(dim=(2, 3))  # Sum over spatial dimensions (H, W), resulting in [B, C]
        false_neg = ((1 - probabilities) * annotations).sum(dim=(2, 3))  # Sum over spatial dimensions (H, W), resulting in [B, C]
        false_pos = (probabilities * (1 - annotations)).sum(dim=(2, 3))  # Sum over spatial dimensions (H, W), resulting in [B, C]

        # Compute Tversky index per class
        tversky_index = (true_pos + self.epsilon) / (
            true_pos + self.alpha * false_neg + self.beta * false_pos + self.epsilon
        )  # Shape: [B, C]

        # Tversky loss per class, averaged over batch and classes
        tversky_loss = 1 - tversky_index  # Shape: [B, C]
        t_loss = tversky_loss.mean()  # Average over batch and classes

        # Focal loss computation
        pt = torch.where(annotations &gt; 0, probabilities, 1 - probabilities)  # Shape: [B, C, H, W]
        focal_loss = -torch.pow(1 - pt, self.gamma) * torch.log(pt)

        # Apply class weights if provided
        if self.weight is not None:
            focal_loss *= self.weight.view(1, -1, 1, 1)  # Shape: [1, C, 1, 1]

        # Focal loss averaged over spatial dimensions and batch
        f_loss = focal_loss.mean(dim=(2, 3))  # Average over spatial dims (H, W) resulting in [B, C]
        f_loss = f_loss.mean()  # Average over batch and classes

        # Final loss combination
        total_loss = f_loss * 0.3 + t_loss * 0.7
        return total_loss

</code></pre>
2d DCT Transform

lifeblack: <p>Hi!<br>
I want to transform a tensor by 2d DCT transform… How could I do this using PyTorch that the output be as the same as matlab?</p>

Nikronic: <p>Hi,</p>
<p>I think this post can answer your questions:</p>
<aside class="quote quote-modified" data-post="1" data-topic="8177">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/b77776/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/solved-discrete-cosine-transform-implementation-in-pytorch/8177">[solved] Discrete cosine transform implementation in pytorch?</a> 
  </div>
  <blockquote>
    UPDATE: I figured it out. This link explains how to incorporate scipy functions into the computational graph. You can use scipy.fftpack.dct <a href="http://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html" rel="noopener nofollow ugc">http://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html</a> 
Hi everyone, 
So I want to backprop the error gradient through the inverse discrete cosine transform. 
I tried implementing a naive versi 
on of the DCT but it was reallyyyyy slow. 
Does anyone have any suggestions for writing a fast, backpropable implementation of the inverse discrete cos…
  </blockquote>
</aside>

<p>Bests</p>

lifeblack: <p>thanks for your reply @ <a href="https://discuss.pytorch.org/u/Nikronic">Nikronic</a>… but it doesn’t work properly and its outputs are wrong (at least in my problem) …</p>

Nikronic: <p>What is your case? Can you create a toy example using numpy?<br>
I have not worked with the code I referenced but at a glimpse, it seems alright.</p>

d4riush: <p>I developed a package that handles 2D DCT encoding/decoding:</p>
<aside class="onebox githubrepo" data-onebox-src="https://github.com/dariush-bahrami/dct-autoencoder">
  <header class="source">

      <a href="https://github.com/dariush-bahrami/dct-autoencoder" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <div class="github-row" data-github-private-repo="false">
  <img width="690" height="344" src="https://opengraph.githubassets.com/efb873b9bb5aa00ad62b77b316b5dd13/dariush-bahrami/dct-autoencoder" class="thumbnail">

  <h3><a href="https://github.com/dariush-bahrami/dct-autoencoder" target="_blank" rel="noopener nofollow ugc">GitHub - dariush-bahrami/dct-autoencoder: Discrete Cosine Transform in PyTorch</a></h3>

    <p><span class="github-repo-description">Discrete Cosine Transform in PyTorch</span></p>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

georgeer: <p>Hello, I tried to load model which larger than my single gpu, so I seperate four part layer to my 4 gpus, after that ,I tried to send query to the model , and I got error in line</p>
<p>x = embed_tokens(input_ids)</p>
<p>my source code is below</p>
<pre><code class="lang-auto"> import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# 加載 tokenizer 和 Gemma 模型
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path="google/gemma-7b-it", token='hf_xxxxxx')
model = AutoModelForCausalLM.from_pretrained("google/gemma-7b-it")

# 假設模型層在 model.model.layers
layers = list(model.model.layers.children())

# 將模型的層分配到不同的 GPU
part1 = torch.nn.Sequential(*layers[:len(layers)//4]).to('cuda:0')  # 前 1/4 層到 GPU 0
part2 = torch.nn.Sequential(*layers[len(layers)//4:len(layers)//2]).to('cuda:1')  # 第二 1/4 層到 GPU 1
part3 = torch.nn.Sequential(*layers[len(layers)//2:3*len(layers)//4]).to('cuda:2')  # 第三 1/4 層到 GPU 2
part4 = torch.nn.Sequential(*layers[3*len(layers)//4:]).to('cuda:3')  # 最後 1/4 層到 GPU 3

# 處理嵌入層和輸出層
embed_tokens = model.model.embed_tokens.to('cuda:0')  # 嵌入層放到 GPU 0
norm = model.model.norm.to('cuda:3')                 # 標準化層放到 GPU 3
lm_head = model.lm_head.to('cuda:3')                 # 輸出層放到 GPU 3

# 準備輸入數據
input_text = "What are the macronutrients, and what roles do they play in the human body?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to('cuda:0')

# 前向傳播：依次在每個 GPU 上運行模型部分，並在每步之間移動張量
with torch.no_grad():
    # 嵌入層計算在 GPU 0 上
    x = embed_tokens(input_ids)

    # 第一部分層計算在 GPU 0 上
    x = part1(x)

    # 將數據從 GPU 0 移動到 GPU 1
    x = x.to('cuda:1')
    # 第二部分層計算在 GPU 1 上
    x = part2(x)

    # 將數據從 GPU 1 移動到 GPU 2
    x = x.to('cuda:2')
    # 第三部分層計算在 GPU 2 上
    x = part3(x)

    # 將數據從 GPU 2 移動到 GPU 3
    x = x.to('cuda:3')
    # 第四部分層計算在 GPU 3 上
    x = part4(x)

    # 最後的標準化和輸出層在 GPU 3 上
    x = norm(x)
    output = lm_head(x)

# 將輸出轉換為文本
generated_text = tokenizer.decode(output.argmax(dim=-1)[0], skip_special_tokens=True)
print(f"Generated text:\n{generated_text}")

</code></pre>
<p>I would really appreciate some help with this! Thanks in advance.</p>

jinooooo: <p>Hi georgeer, you might have to use the torch.nn.DataParallel, you can find an example here - <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" class="inline-onebox" rel="noopener nofollow ugc">💥 Training Neural Nets on Larger Batches: Practical Tips for 1-GPU, Multi-GPU &amp; Distributed setups | by Thomas Wolf | HuggingFace | Medium</a></p>

ptrblck: <p>Is there a way to execute your code without using tokens?</p>
nn.ModuleList() or Not?

Inigo_Parra: <p>Hi friends!<img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
<p>I am currently creating a model and I am in the process of optimizing it. I am used to using ModuleLists and appending layers, but I am not sure this time. Is this more efficient than creating all the layers one by one? Why? Is the loop affecting my running time significantly?</p>
<pre><code class="lang-auto">        self.conv2d_layers = nn.ModuleList()
        self.batch_norm_layers = nn.ModuleList()
        channels = [in_channels, 64, 128, 256, 512]
        current_size = img_size
        for i in range(len(channels) - 1):
            self.conv2d_layers.append(nn.Conv2d(channels[i], channels[i+1], kernel_size, stride, padding))
            self.batch_norm_layers.append(nn.BatchNorm2d(channels[i+1]))

[MORE CODE HERE]

    def forward(self, x: torch.Tensor):
        for conv, norm in zip(self.conv2d_layers, self.batch_norm_layers):
            x = self.activation(norm(conv(x)))
            x = self.dropout_2d(x)
        x = self.flatten(x)
        x = self.dropout(self.activation(self.fc1(x)))
        x = self.fc2(x)
</code></pre>
<p>Thanks in advance!</p>

ptrblck: <p>I doubt it makes a difference in Python to use the loop or to flatten the execution as I don’t think any optimizations would be applied and you should see approx. the same overhead. The loop itself could add a tiny overhead, but again I don’t think it’s noticeable.</p>
Why does modifying the training code affect processes that have already started training (Windows)?

jack_S: <p>For example, given the following code:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader
from torchvision.datasets import FakeData
from torchvision.transforms import ToTensor


dataset = FakeData(size = 2000, image_size = (3, 32, 32), num_classes = 1000, transform = ToTensor())
dataloader = DataLoader(dataset, batch_size = 10, num_workers = 4)

def train_batch(batch:int, images, labels):
    ...

def train_epoch(epoch:int):
    print(f'training ... epoch {epoch}')
    for batch, (images, labels) in enumerate(dataloader):
        train_batch(batch, images, labels)

def train():
    for epoch in range(0, 300):
        train_epoch(epoch)

def main():
    for _ in range(0, 10):
        if _:               
            ... # comment this line instantly after "training ..." printed
        else:
            ...
        train()

if __name__ == '__main__':
    main()
</code></pre>
<p>I commented the line between <code>if</code> and <code>else</code> instantly after <code>training ...</code> printed, only to find the process stopped because of the indent error. Why did this happen?</p>
Tracking progress during inference

Olek: <p>Hi, new to using pyTorch. Im using libTorch to run various pretrained models in c++.<br>
I was wondering, does Torch have any kind of way to track progress during inference? Maybe its even possible using the profiler or something? Is there something I can do to a model before tracing it that would allow me to get intermediate results? Struggling to find any answers for this kind of question.<br>
Thanks all</p>
Identity loss in CycleGAN with different input/output shape generators

foti: <p>How would the identity loss fit when both generators have different input and output channels, e.g. image colorization? The output and input shape should match for the generator while calculating the loss.</p>
<p>Suppose generator AB goes from 1-channel to 3-channel image, when calculating the identity loss you would need it to output a 1-channel image to compare to the identity image. How do you achieve that?</p>
NVIDIA RTX A4000 with cuda capability sm_86 is not compatible with current Pytorch installation

Abhijeet_Kaithwas: <p>I am having cuda version of 12.4 in my GPU’s.<br>
It says current Pytorch installation supports cuda capabilities sm_37, sm_50, sm_60, sm_70, sm_75.<br>
What should I do now?</p>

ptrblck: <p>You have most likely installed an old PyTorch binary shipping with CUDA 10.2 (you can check it via <code>torch.version.cuda</code>) and would need to install any binary shipping with CUDA&gt;=11. Your locally CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension.</p>

Abhijeet_Kaithwas: <blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
print(torch.version.cuda)<br>
11.8</p>
</blockquote>
</blockquote>
</blockquote>
<p>It is showing 11.8 and not 10.2.</p>

ptrblck: <p>All PyTorch binaries compiled with CUDA 11.8 support up to <code>sm_90</code>, so you might need to double check if multiple PyTorch builds are installed or something else is conflicting. E.g. this old <code>2.0.0+cu118</code> installation shows:</p>
<pre data-code-wrap="python"><code class="lang-python">python -c "import torch; print(torch.__version__); print(torch.version.cuda); print(torch.cuda.get_arch_list())"
2.0.0+cu118
11.8
['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
</code></pre>
<p>which supports the RTX A4000.</p>