akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The hidden states of two reverse sequence on bilstm mismatch

Ming_Liang: <p>Suppose I input two sequence [1,2,3,4,5] and [5,4,3,2,1] in to biltm. (Imagine the number is word index). In my understanding, the forward hidden_states of the first seq should be eqaul to the backward hidden_states of the second seq and vice versa. However the results seem mismatch.<br>
Here is my snippet:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

batch_size = 1
seq_len = 5
hidden_size = 3
embed_dim = 5
embed_size = 10
num_layers = 1
num_directions = 2

embedding = nn.Embedding(embed_size, embed_dim)

a = torch.tensor([[1,2,3,4,5]])
b = torch.tensor([[5,4,3,2,1]])
rnn = nn.LSTM(embed_dim, hidden_size, batch_first=True, bidirectional=True, dropout=0)

def my_rnn(input_seq):
    e = embedding(input_seq)
    o, (h, c) = rnn(e)

    o = o.view(seq_len, batch_size, num_directions, hidden_size)
    h = h.view(num_layers, num_directions, batch_size, hidden_size)
    c = c.view(num_layers, num_directions, batch_size, hidden_size)

    return o, h, c

ao, ah, ac = my_rnn(a)
bo, bh, bc = my_rnn(b)

# a forward hidden state = b backward hidden state
print(ah[0, 0], bh[0, 1])

# b forward hidden state = a backward hidden state
print(bh[0, 0], ah[0, 1])
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">tensor([[-0.3088, -0.1794,  0.3160]], grad_fn=&lt;SelectBackward&gt;) tensor([[-0.0156,  0.0029,  0.1224]], grad_fn=&lt;SelectBackward&gt;)
tensor([[-0.1858,  0.0219, -0.0091]], grad_fn=&lt;SelectBackward&gt;) tensor([[-0.2032, -0.0276,  0.2763]], grad_fn=&lt;SelectBackward&gt;)
</code></pre>

ptrblck: <p><a href="https://discuss.pytorch.org/t/clarification-regarding-the-return-of-nn-gru/47363/2">This post</a> by <a class="mention" href="/u/vdw">@vdw</a> explains the outputs, directions etc. and might be helpful.</p>

vdw: <p>Maybe <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">this post</a> is also relevant.</p>
Multiple head loss implementation

akvilonBrown: <p>Greetings!</p>
<p>I’d like to ask for a suggestion from this venerable community on how I can handle a loss for a mulit-task network?<br>
The data goes forward in parallel in an Unet-like network with two heads performing semantic segmentation with different targets (not a multi-class configuration, the tasks differ).<br>
I was unable to find hints here and on Github.<br>
Please suggest or forward where I can find examples.<br>
Should I use two losses for each respective head and sum them?<br>
But I wonder how exactly it is achieved, is it possible without writing a custom loss function?<br>
Both heads use <strong>torch.nn.CrossEntropyLoss()</strong>, can I add them explicitly or should I write a custom implementation for the loss?<br>
Thank you in advance,  I apologize if I violated any rules with my first question (I have an impression the topics are not categorized well here).</p>

KFrank: <p>Hi Iaroslav!</p>
<aside class="quote no-group" data-username="akvilonBrown" data-post="1" data-topic="119476" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/akvilonbrown/48/34909_2.png" class="avatar"> akvilonBrown:</div>
<blockquote>
<p>Both heads use <strong>torch.nn.CrossEntropyLoss()</strong>, can I add them explicitly or should I write a custom implementation for the loss?</p>
</blockquote>
</aside>
<p>Yes, it is perfectly reasonable to add the two losses together, and<br>
there is no need to write a custom loss function.</p>
<p>So, something like:</p>
<pre><code class="lang-auto">loss_task_1 = torch.nn.CrossEntropyLoss() (pred_head_1, target_task_1)
loss_task_2 = torch.nn.CrossEntropyLoss() (pred_head_2, target_task_2)
total_loss = loss_task_1 + weight_task_2 * loss_task_2
total_loss.backward()
</code></pre>
<p>When you now optimize, you will train head-1 and head-2 (in a sense)<br>
separately to perform well on task-1 and task-2, respectively. The shared<br>
part of the network, upstream of the two heads, will be trained jointly to<br>
perform reasonably well on both tasks, making trade-offs in its trained<br>
weights that might, for example, make the task-1 performance a little<br>
worse if doing so yields are larger task-2 performance gain.</p>
<p>Note the <code>weight_task_2</code> parameter I included in my code fragment.  It<br>
allows you to tune your model’s relative performance on the two tasks.</p>
<p>Best.</p>
<p>K. Frank</p>

akvilonBrown: <p>Thank you, <strong>K. Frank</strong>!</p>
What's the fastest way to prepare a batch from different sized images by padding?

Alexander_Soare: <p>This is my collate function for doing what’s in the description. Note that the input images are different sizes, which is the whole point of padding.</p>
<pre><code class="lang-auto">def collate_fn(batch):
        max_h = max([img.shape[-2] for img in batch])
        max_w = max([img.shape[-1] for img in batch])
        # pad all images in batch
        for i, img in enumerate(batch):
            pad_left, pad_right, pad_top, pad_bottom = 0, 0, 0, 0
            if (diff_w := max_w - img.shape[-1]) &gt; 0:
                pad_left = diff_w//2
                pad_right = diff_w - pad_left
            if (diff_h := max_h - img.shape[-2]) &gt; 0:
                pad_top = diff_h//2
                pad_bottom = diff_h - pad_top
            if any([pad_left, pad_right, pad_top, pad_bottom]):
                batch[i] = torch.nn.functional.pad(img,
                                (pad_left, pad_right, pad_top, pad_bottom))
        return torch.stack(batch, axis=0)
</code></pre>
<p>Wondering if there are any neat tricks or obvious utilities to speed this up.</p>
. torch.nn.functional.conv1d padding like torch.nn.Conv1d

Deeply: <p>I would like to use <code>torch.nn.functional.conv1d</code> to implement filtering.</p>
<p>I noticed that, unlike <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html" rel="noopener nofollow ugc">torch.nn.Conv1d</a>, padding for <code>torch.nn.functional.conv1d</code> does not support <code>'reflect'</code> , <code>'replicate'</code> or <code>'circular'</code> modes (so called <code>padding_mode</code>).</p>
<p>Is there a way around this?</p>

ptrblck: <p>You could use the functional API via <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.pad"><code>F.pad</code></a>, which accepts the <code>padding_mode</code> argument and which is in fact also done <a href="https://github.com/pytorch/pytorch/blob/bac4cfd54d44aa0fbc574e6561b878cb406762cc/torch/nn/modules/conv.py#L290-L293">internally</a> in the <code>nn.Conv1d</code> module.</p>

Deeply: <p>You mean to pad the signal before using <code>torch.nn.functional.conv1d</code>. This would work, thanks. I will still need to remove the padding after filtering as I need the exact indices of each point in the signal. Was too lazy to do it, seems this is the only way for now.</p>
How to compile my C++ project against PyTorch source dir?

lithuak: <p>Hi!</p>
<p>I’ve contributed some small Python fixes/additions to PyTorch and now I’m trying to start with a deeper layer of C++ core.</p>
<p>To do that I’m trying to run my C++ experimental code linked with PyTorch built from sources with debug info enabled. And I got somewhat stuck with getting it built.</p>
<p>My configuration is as following:</p>
<p><strong>At ~/pytorch I have the PyTorch repo</strong> cloned from github and successfully compiled.</p>
<p><strong>At ~/projects/my-torch-project I have my own C++ project</strong> that is going to use the PyTorch library compiled in ~/pytorch.</p>
<p>E.g. my main.cpp right now is just trying to repeat the example from <a href="https://pytorch.org/cppdocs/" rel="noopener nofollow ugc">PyTorch C++ API</a>:</p>
<pre><code class="lang-auto">#include &lt;ATen/ATen.h&gt;

at::Tensor a = at::ones({2, 2}, at::kInt);
at::Tensor b = at::randn({2, 2});
auto c = a + b.to(at::kInt);
</code></pre>
<p><strong>My question is: how exactly can I address this in my CMakeLists.txt so that my main.cpp gets compiled and linked?</strong></p>
<p>I’m aware of the “<a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="noopener nofollow ugc">USING THE PYTORCH C++ FRONTEND</a>” and their <a href="https://github.com/pytorch/examples/blob/master/cpp/dcgan/CMakeLists.txt" rel="noopener nofollow ugc">example</a>, yet it looks like that’s not what I need - the example uses the downloaded official package of PyTorch, while I need to use the source base that I can modify and recompile many times in a row.</p>
<p>Thank you!</p>

ptrblck: <p>I assume you’ve build PyTorch in the <code>~/pytorch</code> folder and installed it via <code>pip</code> or <code>conda</code>?<br>
In that case, could you try to use:</p>
<pre><code class="lang-auto">cmake -DCMAKE_PREFIX_PATH=`python -c 'import torch;print(torch.utils.cmake_prefix_path)'`
</code></pre>
<p>instead of the absolute path to the downloaded <code>libtorch</code>?<br>
The small Python inline script should return the path to your locally built cmake files.</p>
How could I obtain unscaled gradient of a tensor

coincheung: <p>Hi,</p>
<p>I used torch.cuda.amp.scaler to train my model with mixed precision mode. Part of my code is like this:</p>
<pre><code class="lang-python">        eps = 1e-8
        mag = 0.1
        im.requires_grad_(True)
        optim.zero_grad()
        with amp.autocast(enabled=args.use_fp16):
            logits, *logits_aux = net(im)
            loss_pre = criteria_pre(logits, lb)
            loss_aux = [crit(lgt_aux, lb) for crit, lgt_aux in zip(criteria_aux, logits_aux)]
            loss = loss_pre + sum(loss_aux) * 0.5
        scaler.scale(loss).backward()
        #  print(im.grad)
        #  scaler.unscale_(im.grad)
        with torch.no_grad():
            grad = im.grad
            grad = (grad / grad.norm() + eps) * mag
        im.requires_grad_(False)
        im += grad
</code></pre>
<p>I need to compute gradient of input image, but I do not know how to unscale it. I am using no optimizer here so I cannot use <code>scaler.unscale_</code>, so how could I compute the unscaled gradient please ?</p>

ptrblck: <p>You can manually unscale the gradients as shown in the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-penalty">Gradient penalty</a> section of the amp examples:</p>
<pre><code class="lang-python">inv_scale = 1./scaler.get_scale()
grad_params = [p * inv_scale for p in scaled_grad_params]
</code></pre>
Latent Code for Recurrent Encoder

Pranavan_Theivendira: <p>Hi All,</p>
<p>I am building a recurrent AutoEncoder using PyTorch. For the encoder, I am using GRU with 2 layers.<br>
Assume our input data is a univariate time-series data in the shape (B,S,1) where B-batch size and S- sequence length. The encoder(GRU with 2 layers) produces 2 outputs (one is the <strong>actual output and other one is the hidden states</strong>). Can somebody explain <strong>which one should I use as the latent code for the AutoEncoder?</strong></p>
<p>Regards<br>
Pranavan</p>
Possible bug with batch with 1 element

straightupbusta: <p>I ran into a strange problem. I am trying out a GRU with pytorch. My forward function goes like this: def forward(self, ids, lengths, masks, labels): … Everything works fine until it gets a batch containing only 1 element. I inserted a print for debugging at the very beginning of the forward function: print(“FORW”, masks, lengths), and the output is FORW tensor([8]) tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]). I.e. lenghts refers to masks and masks refers to lengths? Any idea why this happens?</p>

ptrblck: <p>Could you post an executable code snippet to reproduce this issue using a single sample?<br>
You can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Using GPU to classify an image with a pretrained model

AnhMinhTran: <p>After training a model using a timm library and get a model, I wrote my own script to classify 1 picture. However, it took 7 seconds to classify 1 224x224 image. I thought I had used GPU to predict but it Ross mentioned that I was still using CPU to calculate gradient descent, thats why it took so long.</p>
<p>Isnt tensor.cuda() the solution?</p>
<p>My code can be found below:</p>
<p>Pastebin: <a href="https://pastebin.com/AtFrmBJ1" class="inline-onebox" rel="noopener nofollow ugc">Classify picture - Pastebin.com</a></p>
<pre><code class="lang-auto">
Full code: 

import timm
import torch
import pandas as pd
import torch.nn.functional as F
from torch.autograd import Variable
from PIL import Image
from torchvision.transforms import transforms
import time
 
model = timm.create_model('efficientnet_v2s', num_classes=7)
checkpoint = torch.load("models/model_best.pth-d043d179.pth")
model.load_state_dict(checkpoint)
model.eval()
 
 
def predict_image(path):
    print("Prediction in progress")
    image = Image.open(path)
 
    transformation = transforms.Compose([
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])
 
    img_tensor = transformation(image).float()
    img_tensor = img_tensor.unsqueeze_(0)
 
    if torch.cuda.is_available():
        img_tensor.cuda()
 
    input = Variable(img_tensor)
    output = model(input)
    index = output.data.numpy().argmax()
    # prob = F.softmax(output, dim=1)
    return index
 
 
if __name__ == "__main__":
    imgpath = "data/2.jpg"
    start = time.time()
    index = predict_image(imgpath)
    df = pd.read_csv("file_index.csv")
    pred = df.iloc[index][1]
    print("Predicted Class ", pred)
    end = time.time()
 
    print("[INFO] Prediction took {:.5f} seconds".format(
        end - start))
</code></pre>

ptrblck: <p>You would have to reassign the tensor when calling the <code>to()</code> operator:</p>
<pre><code class="lang-python">    if torch.cuda.is_available():
        img_tensor = img_tensor.cuda()
</code></pre>
<p>Also, you need to call <code>cuda()</code> or <code>to('cuda')</code> on the model as well.</p>
<p>Note that <code>Variable</code>s are deprecated since PyTorch <code>0.4</code>, so you can use tensors in newer versions.</p>
KPConv custom gather operation creating large memory requirements on backward?

kevindckr: <p>I am working on implementing my own use case for a <a href="https://github.com/HuguesTHOMAS/KPConv-PyTorch" rel="noopener nofollow ugc">KPConv KPFCNN network</a> (<a href="https://arxiv.org/abs/1904.08889" rel="noopener nofollow ugc">Original article from Huges et. al.</a>).</p>
<p>After some debugging I have found that during some backward passes of a KPFCNN model (<a href="https://github.com/HuguesTHOMAS/KPConv-PyTorch/blob/master/models/architectures.py" rel="noopener nofollow ugc">defined here ~line 189</a>) the custom gather operation (<a href="https://github.com/HuguesTHOMAS/KPConv-PyTorch/blob/master/models/blocks.py" rel="noopener nofollow ugc">defined here ~line 35</a>), based on model input size, is always the culprit of an OOM exception.</p>
<p>The <code>gather</code> operation</p>
<pre><code class="lang-auto">def gather(x, idx, method=2):
    """
    implementation of a custom gather operation for faster backwards.
    :param x: input with shape [N, D_1, ... D_d]
    :param idx: indexing with shape [n_1, ..., n_m]
    :param method: Choice of the method
    :return: x[idx] with shape [n_1, ..., n_m, D_1, ... D_d]
    """

    if method == 0:
        return x[idx]
    elif method == 1:
        x = x.unsqueeze(1)
        x = x.expand((-1, idx.shape[-1], -1))
        idx = idx.unsqueeze(2)
        idx = idx.expand((-1, -1, x.shape[-1]))
        return x.gather(0, idx)
    elif method == 2:
        for i, ni in enumerate(idx.size()[1:]):
            x = x.unsqueeze(i+1)
            new_s = list(x.size())
            new_s[i+1] = ni
            x = x.expand(new_s)
        n = len(idx.size())
        for i, di in enumerate(x.size()[n:]):
            idx = idx.unsqueeze(i+n)
            new_s = list(idx.size())
            new_s[i+n] = di
            idx = idx.expand(new_s)
        return x.gather(0, idx)
    else:
        raise ValueError('Unkown method')
</code></pre>
<p>Based on the code comments this is a custom implementation of <a href="https://pytorch.org/docs/stable/generated/torch.gather.html" rel="noopener nofollow ugc"><code>torch.gather</code></a> that provides some execution speed-up. I am attempting to understand/determine if this custom operation is truly the culprit of the OOM I am experiencing, though I am not exactly well versed enough just yet to fully understand the memory requirements of the custom operation during a backward pass.</p>
<p>I am currently attempting to replace the use of this operation with <code>torch.gather</code> to determine if any decrease in memory usage is possible. But until then, or if that does not provide a clear answer, <strong>is it possible that this custom <code>gather</code> operation has a larger, or much larger, memory requirement than <code>torch.gather</code>?</strong></p>

ptrblck: <p>This could be the case and you could profile both methods by running the forward and backward operation inside a small fake model and check the memory usage via <code>torch.cuda.memory_summary()</code>.</p>
TypeError: unsupported operand type(s) for *: 'Tensor' and 'NoneType'

el_youssfi_azeddine: <p>Hello, can some figure out this error?<br>
thanks</p>
<p>TypeError                                 Traceback (most recent call last)<br>
 in </p>
<p>—&gt; 44         qx = k * qx<br>
45         Div = Divergence( x_trg ,  qx )<br>
46             ########################################################</p>
<p>TypeError: unsupported operand type(s) for *: ‘Tensor’ and ‘NoneType’</p>

el_youssfi_azeddine: <p>for qx is defined as: qx = Gradient( x_trg , T )</p>

ptrblck: <p>Based on the error message it seems that <code>qx</code> is <code>None</code>. Could you check, if <code>Gradient</code> is returning a value or if the <code>return</code> statement is either not used or is returning <code>None</code>?</p>

el_youssfi_azeddine: <p>Yeak I chesked, gradient returns None, and I don’t know how to make it returning value.<br>
gradient function is defined with:<br>
torch.autograd.grad (<em>outputs</em> , <em>inputs</em> , <em>grad_outputs=None</em> , <em>retain_graph=None</em> , <em>create_graph=False</em> , <em>only_inputs=True</em> ):</p>

ptrblck: <p><code>torch.autograd.grad</code> should yield the desired gradients, so maybe you are missing to assign them to a return value?</p>

el_youssfi_azeddine: <p>thanks for replying, here is gradient function: <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e35a8a34a6bdf702e20abd80da2ede931a99c3f.png" data-download-href="https://discuss.pytorch.org/uploads/default/7e35a8a34a6bdf702e20abd80da2ede931a99c3f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e35a8a34a6bdf702e20abd80da2ede931a99c3f.png" alt="image" data-base62-sha1="i0v7IXJNnfX14zOG3DNXEjxP5HF" width="690" height="161" data-dominant-color="272829"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">708×166 6.26 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I’m not sure why a <code>None</code> value is returned. Could you post a minimal code snippet to reproduce this issue, i.e. valid gradients are expected while you are seeing <code>None</code>s?</p>

el_youssfi_azeddine: <p>Actually the model is composed of several files ( it’s physics-informed neural network), I send you in message the portion of the code that precedes the calling of gradient.</p>

ptrblck: <p>Could you please post the code here, as you shouldn’t share private code with anyone who is not collaborating on your project?</p>

el_youssfi_azeddine: <p>yeah, I can’t share it here, it’s private code <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

ptrblck: <p>That’s understandable. Could you come up with a minimal “fake” example, which would reproduce the issue you are seeing, but wouldn’t show anything from your architecture etc.?</p>
Is there a way to do categorical distribution sampling on the gpu

lulzor: <p>Hi , i just started looking into pytorch today to see if i could speed up my research project. I was wondering if there was a way to sample N*M unique categorical distributions in parallel through pytorch on a gpu. Below is a sample of my current code</p>
<pre><code class="lang-auto"># initialise
num_sets = 2000
num_dimensions = 800
num_distributions = 50

# create a distribution for ever dimension of every dimension (there will be unique distributions in practice)
sets = [torch.full((num_dimensions, num_distributions), 1 / num_distributions
                   , dtype=torch.float)] * num_sets

#to store sample results
set_sampled = [torch.zeros([num_dimensions], dtype=torch.float)] * num_sets


# %%

# function to sample all dimensions
def sample_set(dimensions, sampled_dimensions):
    i = 0
    for element in dimensions:
        rand_generator = torch.distributions.categorical.Categorical(element)
        sampled_dimensions[i] = rand_generator.sample()
        i += 1


#%%

time1 = time.process_time()
i = 0
for element in sets:
    sample_set(element, set_sampled[i])

time2 = time.process_time()

print(time2-time1)
</code></pre>
<p>this is one of the slowest parts of my code hence it would be great if i could speed it up</p>

jeremysalwen: <p>Yes the regular Categorical distribution supports sampling multiple dimensions simultaneously. See the <code>.expand()</code> and <code>.sample(sample_shape=...)</code> methods.</p>
I got strange ValueError from dataloader

111515: <p>Hello,I got a strange error</p>
<pre><code>&lt;ipython-input-154-3032f9891941&gt; in loss_epoch(model, loss_func, dataset_dl, sanity_check, opt)
      5     len_data=len(dataset_dl.dataset)
      6 
----&gt; 7     for xb,clb,yb in dataset_dl:
      8         # convert to tensor
      9         yb=torch.stack(yb,1)

ValueError: too many values to unpack (expected 3)
</code></pre>
<p>It worked 10 minutes ago, but now doesn’t, I just stuck there.<br>
Yes, it is a classification and regression task, so I need 3 outputs from dl.<br>
My loss:</p>
<pre><code class="lang-auto">def loss_epoch(model,loss_func,dataset_dl,sanity_check=False,opt=None):
    running_loss=0.0
    running_metric=0.0
    accuracy=0.0
    len_data=len(dataset_dl.dataset)
    for xb,clb,yb in dataset_dl:
        # convert to tensor
        yb=torch.stack(yb,1)
        yb=yb.type(torch.float32).to(device)
        
        # get model output
        output1,output2=model(xb.to(device))
        
        # get loss per batch
        loss_b,metric_b=loss_batch(loss_func, output2, yb, opt)

        loss2= nn.CrossEntropyLoss()
        clb=clb.cuda()
        acc=(torch.argmax(output1, dim=1)==clb).sum()/float(len(output1))
        #print("OUTPUT1",output1)
        #print("clb",clb)
        #print("OUTPUT2",output2)
        loss2_out = loss2(output1, clb)
        accuracy+=acc
        # update running loss
        running_loss+=loss_b
        # update running metric
        if metric_b is not None:
            running_metric+=metric_b

    # average loss value
    loss=running_loss/float(len_data)+ loss2_out/float(len_data)
    accuracy=accuracy/float(len_data)
    # average metric value
    metric=running_metric/float(len_data)
    
    return loss, metric,accuracy
</code></pre>
<p>My Dataloader:</p>
<pre><code class="lang-auto">    class New_dataset(Dataset):
        def __init__(self, path2data, transform, trans_params):      
          self.labels = labels_df[["xmin","xmax","ymin","ymax"]].values
    
          
          self.imgName=labels_df["image_name"]
          self.ids=labels_df.index
    
          self.fullPath2img=labels_df["path_to_img"]
          self.transform = transform
          self.trans_params=trans_params    
          self.classid=labels_df['class']
        def __len__(self):
            t
            return len(self.labels)
          
        def __getitem__(self, idx):
          
          image = Image.open(self.fullPath2img[idx])  
          label= self.labels[idx]
          idclass=self.classid[idx]
    
          
          image,label = self.transform(image,label,self.trans_params)
    
          return image,idclass,label
</code></pre>
<p>I need 3 outputs from batch DL to calculate 2 losses. It is strange because it worked while i defining and fixing loss function.<br>
Maybe there is another error?</p>

ptrblck: <p>Could you check, if you have accidentally overridden the <code>dataset_dl</code> with another object, which doesn’t return these three values?<br>
Also check, if the underlying dataset still returns the three values via <code>a, b, c = dataset_dl.dataset[0]</code>.</p>
Create a mask that is larger than the n-th quantile of each 2D tensor in a batch

AerysS: <p>I have a <code>torch.Tensor</code> of shape <code>(2, 2, 2)</code> (can be bigger), where the values are normalized within range <code>[0, 1]</code>.</p>
<p>Now I am given a positive integer <code>K</code>, which tells me that I need to create a mask where for each 2D tensor inside the batch, values are 1 if it is larger than <code>1/k</code> of all the values, and 0 elsewhere. The return mask also has shape <code>(2, 2, 2)</code>.</p>
<p>For example, if I have a batch like this:</p>
<pre><code class="lang-python">tensor([[[1., 3.],
         [2., 4.]],
        [[5., 7.],
         [9., 8.]]])
</code></pre>
<p>and let <code>K=2</code>, it means that I must mask the values where they are greater than 50% of all the values inside each 2D tensor.</p>
<p>In the example, the 0.5 quantile is <code>2.5</code> and <code>7.5</code>, so this is the desired output:</p>
<pre><code class="lang-python">tensor([[[0, 1],
         [0, 1]],
        [[0, 0],
         [1, 1]]])
</code></pre>
<p>I tried:</p>
<pre><code class="lang-python">a = torch.tensor([[[0, 1],
                   [0, 1]],
                  [[0, 0],
                   [1, 1]]])
quantile = torch.tensor([torch.quantile(x, 1/K) for x in a])
torch.where(a &gt; val, 1, 0)
</code></pre>
<p>But this is the result:</p>
<pre><code class="lang-python">tensor([[[0, 0],
         [0, 0]],
        [[1, 0],
         [1, 1]]])
</code></pre>

ptrblck: <p>You could use broadcasting to get the desired result:</p>
<pre><code class="lang-python">x = torch.tensor([[[1., 3.],
                   [2., 4.]],
                  [[5., 7.],
                   [9., 8.]]])

quantile = torch.tensor([torch.quantile(a, 1/2) for a in x])

res = torch.where(x &gt; quantile[:, None, None], 1, 0)
print(res)
&gt; tensor([[[0, 1],
           [0, 1]],

          [[0, 0],
           [1, 1]]])
</code></pre>
Transformer Mask Implementation

anirband: <p>I was going over the example here on Transformer encoder for language modelling: <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Sequence-to-Sequence Modeling with nn.Transformer and TorchText — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>
<p>The following function is used to generate mask:</p>
<p>def generate_square_subsequent_mask(self, sz):<br>
mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)<br>
mask = mask.float().masked_fill(mask == 0, float(’-inf’)).masked_fill(mask == 1, float(0.0))<br>
return mask</p>
<p>Shouldn’t the second line be</p>
<p>mask = mask.float().masked_fill(mask == 0, float(1.0)).masked_fill(mask == 1, float(0.0))</p>
<p>instead of mask = mask.float().masked_fill(mask == 0, float(’-inf’)).masked_fill(mask == 1, float(0.0)) ?</p>
Separated mask_transform and image_transform on Albumentation

NamKolorful: <p>I was trying to implement the original UNet and use some augmentations from Albumentation library.<br>
Since UNet expects an input of 572x572 and output of 388x388, the albumentation just merge the images and the masks into one and perform augmentation. Here is some example taken from their repo:</p>
<p>class OxfordPetDataset(Dataset):<br>
def <strong>init</strong>(self, images_filenames, images_directory, masks_directory, transform=None):<br>
self.images_filenames = images_filenames<br>
self.images_directory = images_directory<br>
self.masks_directory = masks_directory<br>
self.transform = transform</p>
<p>from<br>
<em><a href="https://github.com/albumentations-team/albumentations_examples/blob/master/notebooks/pytorch_semantic_segmentation.ipynb" class="inline-onebox" rel="noopener nofollow ugc">albumentations_examples/pytorch_semantic_segmentation.ipynb at master · albumentations-team/albumentations_examples · GitHub</a></em></p>
<p>Since the transfrom recieve 2 inputs, one is image and the other is mask, when I tried to define a self.mask_transfrom = mask_transform and add to <strong>getitem</strong> some lines like below, it returned an error saying it needed image variable to start working so I guess It’s pointless since I wanted to have separate mask and image augmentation</p>
<pre><code>        augmentations2 = self.mask_transform(mask=mask)
        mask = augmentations2["mask"]
</code></pre>
<p>Could it be possible to separate image and mask for albumentation or perhaps there are other libs?</p>