akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pathlib usage makes attribute disappear

iga: <p>I have a module with some <code>torch.jit.export</code>ed methods and some regular methods. This module has some attributes that I want to use in non-scripted methods. These attributes disappear when I script the module.</p>
<pre><code class="lang-auto">class Foo:
    def __init__(self):
        # Commenting out this line gets rid of the AttributeError
        pathlib.Path("/tmp/tmp").mkdir(parents=True, exist_ok=True)
        self.y = 5

class M(nn.Module):
    def __init__(self, f):
        super(M, self).__init__()
        self.f = f

    @torch.jit.ignore
    def get_f(self):
        return self.f  # &lt;-- Getting an attribute error here

m = torch.jit.script(M(Foo()))
m.get_f()
</code></pre>
<p>I am curious if this is a bug or there is some rationale behind this. It seems like assigning <code>self.f = f</code> in <code>M.__init__</code> makes torch script analyze <code>Foo.__init__</code> and discard the attribute if scripting it fails.</p>
<p>Because I never use this attribute in compiled methods, there should not be a reason to try compiling it and discarding. Maybe there is some annotation I can use to ask torch script to just keep this attribute attached to <code>M</code>? Is there some “good” workaround?</p>

ptrblck: <aside class="quote no-group" data-username="iga" data-post="1" data-topic="134885">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/e19b73/48.png" class="avatar"> iga:</div>
<blockquote>
<p>Because I never use this attribute in compiled methods, there should not be a reason to try compiling it and discarding.</p>
</blockquote>
</aside>
<p>I’m not sure I understand the use case completely, but you are indeed using it in the scripted <code>m</code> module.<br>
If you create an eager module first, the attribute will be there:</p>
<pre><code class="lang-auto">module = M(Foo())
m = torch.jit.script(module)
m.get_f() # error
module.get_f() # works
</code></pre>

iga: <p>Yeah, the attribute is obviously there before scripting. My questions are:</p>
<ul>
<li>Why scripting removes this attribute when it is used only in <code>torch.jit.ignore</code>'ed methods?</li>
<li>Is there a good way to make <code>torch.jit.script</code> not remove this attribute? I can just do <code>m.f = f</code> to assign it again after scripting, but this is fairly ugly when you have many attributes.</li>
</ul>

ptrblck: <p>I’m not sure, if there is a way to ignore attributes, which seems to be your case. I.e. if you initialize <code>Foo</code> as an <code>nn.Module</code> it would also be scripted as a <code>RecursiveScriptModule</code>, but I guess you don’t want that.</p>
Hand crafted model degradres through training

ali1s232: <p>So I was trying different activation functions and I came across something weird. Hand crafted this model and I can pretty much get 0% error when I manually assign values to the weights and biases of my network.<br>
Then I tell network to train and after many epochs (2500), I’m telling network to predict again. I was expecting since the original weights are almost perfect the network should just improve that small remaining error, but turns out the network diverges on training data.<br>
I know my training routine is not a problem, as this specific network with these specific input/output is the only example that I found with this behavior. Even in the same model, if I swap my activation functions from ReLU to mish or SiLU the network can train without an issue.</p>
<pre><code class="lang-auto">test(47,)..Submodel(
  (layers): Sequential(
    (0): Linear(in_features=6, out_features=2, bias=True)
    (1): ReLU()
    (2): Linear(in_features=2, out_features=3, bias=True)
    (3): ReLU()
    (4): Linear(in_features=3, out_features=6, bias=True)
    (output_activation): Mish()
    (output_scale): ScaleAndShift(*tensor([0.0200, 0.0700, 0.0700, 0.0200, 0.0200, 0.0200], dtype=torch.float64)+tensor([-1., -3., -3., -1., -1., -1.], dtype=torch.float64))
  )
)
input
[[[1.00000 1.00000 1.00000 1.00000 1.00000 1.00000]]

 [[0.00000 0.00000 0.00000 0.00000 0.00000 0.00000]]

 [[-1.00000 -1.00000 -1.00000 -1.00000 -1.00000 -1.00000]]]
prediction:
[[2.00000 2.00500 2.00500 2.00000 2.00000 2.00000]
 [1.00000 1.02500 1.02500 1.00000 1.00000 1.00000]
 [0.00000 0.03100 0.03100 0.00000 0.00000 0.00000]]
score
0.9998082146798993
done test       2500(  2) :    0.0005 -&gt;    0.6260: v   11.2685;t  12.9918 51
prediction:
[[1.47839 1.55670 1.55706 1.47860 1.47862 1.47843]
 [1.47839 1.55670 1.55706 1.47860 1.47862 1.47843]
 [0.06020 -0.15821 -0.15911 0.05984 0.05971 0.06023]]
score
0.7444793404118504
</code></pre>
<p>Can anyone guess what’s going on here?</p>
Using dataloader to sample with replacement

blade: <p>I have a dataset defined in the format:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, N):
        self.N = N
        self.x = torch.rand(self.N, 10)
        self.y = torch.randint(0, 3, (self.N,))

    def __len__(self):
        return self.N

    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]
</code></pre>
<p>During the training, I would like to sample batches of <code>m</code> training samples, with replacement; e.g. the first iteration includes data indices <code>[1, 5, 6]</code>, second iteration includes data points <code>[12, 3, 5]</code>, and so on and so forth. So the total number of iterations is an input, rather than <code>N/m</code></p>
<p>Is there a way to use <code>dataloader</code> to handle this? If not, what is there any other method than something in the form of</p>
<pre><code class="lang-auto">for i in range(iter):
    x = np.random.choice(range(N), m, replace=False)
</code></pre>
<p>to implement this?</p>

ptrblck: <aside class="quote no-group" data-username="blade" data-post="1" data-topic="134887">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>Is there a way to use <code>dataloader</code> to handle this?</p>
</blockquote>
</aside>
<p>Yes, you could create a custom sampler, draw the indices using your custom logic, and let the <code>DataLoader</code> pass these indices to the <code>Dataset.__getitem__</code>. You might also want to check the <code>BatchSampler</code> as it can pass a batch of indices to <code>__getitem__</code>.</p>
Why Image segmentation results not picking up small masks. How to improve the model for better detection

Sanjana_Mudduluru: <p>Hello,<br>
I implemented UNET architecture to detect hemorrhages on retinal fundus images. I was able to generate decent results but my model doesn’t seem to pick small masks and I was wondering if you can help to improve the scenario. Here are a few examples. (Left: Input RGB retina image, middle:actual mask, right:predicted mask) I circled some small masks.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/098f651ddb21b84f8ceba2b1be2669ae31666e95.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/098f651ddb21b84f8ceba2b1be2669ae31666e95" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/098f651ddb21b84f8ceba2b1be2669ae31666e95_2_690x223.jpeg" alt="image" data-base62-sha1="1mzwdfBtxIV7XnhXh0k5adwfZE9" width="690" height="223" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/098f651ddb21b84f8ceba2b1be2669ae31666e95_2_690x223.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/0/9/098f651ddb21b84f8ceba2b1be2669ae31666e95.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/9/098f651ddb21b84f8ceba2b1be2669ae31666e95.jpeg 2x" data-dominant-color="2D190A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">795×258 33.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1d017424896f439b4f8a436714d310b0f8ab901.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e1d017424896f439b4f8a436714d310b0f8ab901" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1d017424896f439b4f8a436714d310b0f8ab901_2_690x224.jpeg" alt="image" data-base62-sha1="wdDgkI02PavRyjBWOhEK1BlZRxD" width="690" height="224" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1d017424896f439b4f8a436714d310b0f8ab901_2_690x224.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1d017424896f439b4f8a436714d310b0f8ab901.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1d017424896f439b4f8a436714d310b0f8ab901.jpeg 2x" data-dominant-color="261408"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">795×259 32.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Error when compiling custom cuda kernel

Karthik_Ganesan: <p>I’m trying to write my own CUDA kernel but I’m not able to compile it using CPP extension. I am following the guide <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">here</a>, and I was able to get a C++ version working fine. Full code and error messages below:</p>
<blockquote>
<p><strong>global</strong> void linear_forward_cuda( float * input, float * weight, float * bias, float * output){<br>
const int tid = threadIdx.x;<br>
const int bid = blockIdx.x;<br>
const int m = blockDim.x;</p>
<p><strong>shared</strong> float shared_sum[1024];<br>
shared_sum[tid] = weight[(bid*m) + tid] * input[tid];</p>
<p>for (unsigned int i = m/2; i &gt; 0 ; i = i&gt;&gt;1){<br>
if (tid &lt; i){<br>
shared_sum[tid] += shared_sum[tid+i];<br>
}<br>
__syncthreads();<br>
}</p>
<p>if (tid==0) {<br>
output[bid] = shared_sum[0] + bias[bid];<br>
}<br>
}</p>
</blockquote>
<blockquote>
<p>In function ‘at::Tensor linear_forward(at::Tensor, at::Tensor, at::Tensor)’:<br>
/home/karthik/projects/torch_files/approx-mult/cuda/cuda_layers.cpp:58:26: <strong>error: expected primary-expression before ‘&lt;’ token</strong><br>
58 |     linear_forward_cuda&lt;&lt;&lt;threads,blocks&gt;&gt;&gt;(input, weight_transposed, bias, output);</p>
</blockquote>
<p>I am calling this kernel from another .CPP file. And I am adding them both to the setup.py file as shown in the guide I linked above.</p>
<p>This kernel works just fine when I call it via another .cu file and compile that file using just NVCC. My guess is Ninja is trying to compile this file using gcc instead of using nvcc? Since this is all handled by CPP extension, I’m not sure how to fix this. Any help would be much appreciated.</p>
<p>EDIT: I tried renaming both files to .cu to use the NVCC compiler for both and it seems to work. But I’m not sure if that it the right way to fix this.</p>

Hamster: <aside class="quote no-group" data-username="Karthik_Ganesan" data-post="1" data-topic="134723">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/karthik_ganesan/48/34393_2.png" class="avatar"> Karthik_Ganesan:</div>
<blockquote>
<p>EDIT: I tried renaming both files to .cu to use the NVCC compiler for both and it seems to work. But I’m not sure if that it the right way to fix this.</p>
</blockquote>
</aside>
<p>It is. The CUDA kernel launch syntax is not standard C++, hence it fails when you compile with a regular C++ compiler. If you don’t like that, you could put only CUDA code inside the .cu file, define wrappers inside that same .cu file which will simply launch the kernels, and then export those wrappers so that you could use them (and not the CUDA kernel launch syntax) from your other .cpp files.</p>

Karthik_Ganesan: <p>Thank you. It was just because the instructions I was following on the pytorch website said to have the kernel invocation in the cpp file instead of the cu file. But just having the entire thing in the cu file seems to work just fine.</p>
Simple resnet implementation

safin_salih: <p>Hey there,<br>
I am trying to create a simple resnet, possibly with 2-3 resblocks , but having trouble implementing it, can anyone help me here? Thank you</p>

tumble-weed: <p>hey, just a comment, if you post your code and point out where you are having trouble , you might find it easier to get help on this.</p>

safin_salih: <p>Sure, I will do that. Here is what I have so far.</p>
<p>class toy_resnet(nn.Module):</p>
<pre><code>def __init__(self):
    super(toy_resnet,self).__init__()

    self.conv1 = nn.Sequential(
        Conv2d(3, 8, kernel_size=3, stride=1, padding='same'),
        BatchNorm2d(8),
        ReLU(inplace=True),
        Conv2d(8, 8, kernel_size=3, stride=1, padding='same'),
        BatchNorm2d(8),
        ReLU(inplace=True)
         )

    self.conv2 = nn.Sequential(
        Conv2d(8, 8, kernel_size=3, stride=1, padding='same'),
        BatchNorm2d(8),
        ReLU(inplace=True),
        Conv2d(8, 8, kernel_size=3, stride=1, padding='same'),
        BatchNorm2d(8),
        ReLU(inplace=True)
           )

    self.classifier = nn.Sequential(
        nn.Flatten(),
        nn.Linear(24200,37*10),
        nn.Dropout(0.5)
    )
    self.max_pool = MaxPool2d(kernel_size=3, stride=2)
def forward(self,input):
    # x = self.conv1(input.float().to('cuda'))
    # #print(x.shape)
    # x = self.conv2(x.float().to('cuda'))
    # #print(x.shape)
    # x = self.conv3(x.float().to('cuda'))
    #res = input
    x = self.conv1(input.float())
    x = self.max_pool(x)
    res= x

    #x+=input
    #print(x.shape)
    x = self.conv2(x.float())
    out = x+res
    
    x = self.conv2(out.float())
    res = x+out

    out = self.max_pool(res)
    #print(x.shape)
    #x = self.conv3(x.float())
    #print(x.shape)
    x = self.classifier(out)
    return x</code></pre>

tumble-weed: <p>thanks, so your model is supposed to have 370 output classes with a dropout layer as the last layer ( before softmax presumably) . where are you facing issues here?</p>

safin_salih: <p>Well, I just want to know if i am doing it correctly, and my second concern is that it’s not doing so well with comparison to just a simple network with just  few conv layers and filters.</p>

tumble-weed: <p>i guess from the the looks of it the implementation seems fine to me, maybe someone else can comment. the network “not doing so well” as a description is a very sparse to make a comment, i’ll make a general comment. networks with more parameters are prone to overfitting, where their train error goes down but the validation error does not do as well as compared to a network with less parameters. if this is the case , you can search for regularization as a keyword.<br>
good luck</p>

safin_salih: <p>Thank you for your help.</p>
Is there a mechanism for fusing element-wise kernels in C++/libtorch?

Hamster: <p>Assume I do both model definition/training and inference in C++, so we aren’t in the usual use-case where a model was already jitted &amp; trained in Python/PyTorch. I’m finding myself currently fusing ops by hand with custom CUDA kernels when working with libtorch/C++ because I thought the equivalent of <code>@torch.jit.script</code> might not be possible with pure C++ unless there’s a way to define functions “symbolically” in libtorch. But I’m still wondering whether I might be wrong and automatic fusion exists in libtorch/C++?</p>
<p>Thank you!</p>
CER(character error rate) increases while decrease in validation loss

Himanshu_Chaudhary: <p>I am training an OCR model with resnet as backbone and vanilla Transformer as decoder on the Rimes dataset. I tried the following experiments and got these results:</p>
<p>without Augmentation: best validation loss==&gt; 111, best training loss==&gt;2.77 ,Test CER==&gt; 5.75</p>
<p>with Augmentation: best validation loss==&gt; 71, best training loss==&gt;95 ,Test CER==&gt; 10.2</p>
<p>Loss= KL divergence loss with label smoothing in both cases.</p>
<p>Augmentation function:</p>
<pre><code class="lang-auto">albumentations.Compose([
        albumentations.OneOf(
            [
                albumentations.MotionBlur(p=1, blur_limit=5),
                albumentations.OpticalDistortion(p=1, distort_limit=0.05),
                albumentations.GaussNoise(p=1, var_limit=(10.0, 100.0)),
                albumentations.RandomBrightnessContrast(p=1, brightness_limit=0.2),
                albumentations.Downscale(p=1, scale_min=0.3, scale_max=0.5),
            ],
            p=.5,
        ),
        albumentations.Normalize(),
        albumentations.pytorch.ToTensorV2()

    ])
</code></pre>
<p>I am very confused why validation loss decreased and CER increased. It is happening with Rimes dataset only while on other datasets like IAM and Washington there is no such issue.</p>
Custom activation functions?

ZeroMaxinumXZ: <p>New here so… How do I implement and use an activation function that’s based on another function in Pytorch, like for an example, swish?</p>

albanD: <p>If your new function is differentiable then just write it as a python function. If it has parameters, you can use nn.Module and you will need to implement the init and the forward for your function.</p>
<p>If it is not differentiable, you will have to define the backward operation you want using the informations <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="nofollow noopener">here</a>.</p>

ZeroMaxinumXZ: <p><a class="mention" href="/u/alband">@albanD</a> , It keeps giving me a <code>__main__.Activation is not a Module subclass</code>  error, when trying to implement it as a Python func… No parameters except for x and it is differentiable.</p>

albanD: <p>Could you give a code sample that reproduce this please?</p>

ZeroMaxinumXZ: <p><a class="mention" href="/u/alband">@albanD</a></p>
<p>Code sample:</p>
<pre><code class="lang-auto">from math import e
from torch import nn
import torch
def Swish(x):
    return 1/(1+(math.e**-x)) * x

 #Copied from the GAN Tutorial with changes to the activation function.

ngf =  64
nz = 100
nc = 3

main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            Swish,
            #nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            Swish, #Define x, maybe? 
            #nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            Swish,
            #nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            Swish,
            #nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )
</code></pre>
<p>Does x in the Swish function count as a parameter? New to PyTorch so… I also tried to use nn.Module and implementing init and forward, but I got same error.</p>

MariosOreo: <p>Hello,</p>
<p>I think you should rewrite function <code>Swish</code> as a sub-class of <code>nn.Module</code>, and call <code>Swish()</code> instead of <code>Swish</code> in the Sequential.</p>

ZeroMaxinumXZ: <p><a class="mention" href="/u/mariosoreo">@MariosOreo</a>, that worked, thank you.</p>

Stefan_H: <p><a class="mention" href="/u/zeromaxinumxz">@ZeroMaxinumXZ</a> , maybe a little late but what was your final code? Trying to do an activation function my self …</p>
Equivalent of Numpy's list.append() for differentiable Variables

gelazari: <p>Hi, I was wondering what is the equivalent of:</p>
<pre><code>    var = []
    for i in range(0, num):
        newstuff = #dostuff
        var.append(newstuff)
</code></pre>
<p>I’m trying to use torch.cat but it needs a starting Variable to concatenate against. Is there any other way?</p>

justusschock: <pre><code class="lang-auto">var = [] 
for I in range(0, num):
    new_stuff = # do stuff 
    var.append(new_stuff.unsqueeze(0))
var_tensor = torch.cat(var)
</code></pre>
<p>Should work if <code>new_stuff</code> is already a tensor</p>

Kevin_Xue: <p>Hi, i think the last row shoule be <code>var_tensor = torch.stack(var)</code>.</p>
<p>Example see here: <a href="https://discuss.pytorch.org/t/how-to-turn-a-list-of-tensor-to-tensor/8868/4" class="inline-onebox">How to turn a list of tensor to tensor? - #4 by dmadeka1</a></p>
Hugging Face's RoBERTa config and tokenizer do not have matching vocabulary (?)

biobroo: <p>It appears to me that the Hugging Face (i.e., transformers library) has a mismatched tokenizer and config with respect to vocabulary size.  It appears that the RoBERTa config object lists vocabulary size at 30522 while the tokenizer has a much larger vocab.  In the RoBERTa paper, 50k is what is listed as the correct value, which leads me to believe that the problem I am having is due to the config class.  Additionally the documentation for the Hugging Face RoBERTa config says that it is the same as the config of BERT, which has a smaller vocabulary (~30k).  I have some specific questions:</p>
<p>(1) How can I set up RoBERTa so that it will recognize all of the IDs resulting from its tokenizer?<br>
(2) Less importantly, why is this implementation of RoBERTa set up this way?</p>
<p>I am including a toy example to illustrate my problem and facilitate simple answers:</p>
<pre><code class="lang-auto"># setting up RoBERTa
from transformers import RobertaConfig, RobertaModel, RobertaTokenizer
configuration = RobertaConfig()
roberta = RobertaModel(configuration)
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")

# using RoBERTa with a problematic token
text = 'currency'
tokenized = tokenizer.encode(text, 
                           add_special_tokens = False,    # removes beginning of sentence and end of sentence tokens, just for the purpose of this toy example—this is not responsible for the error.
                           return_tensors = 'pt')
roberta(tokenized)

# The KeyError is returned since the token index is greater than the vocabulary size of the config
print(tokenizer.convert_tokens_to_ids(text))   # token index
print(configuration.vocab_size)    # vocabulary size of config

# Just for comparison, here is an unproblematic token
unproblematic_text = tokenizer.convert_ids_to_tokens(10004)
print(unproblematic_text)
encoded = tokenizer.encode(unproblematic_text, 
                           add_special_tokens = False,
                           return_tensors = 'pt')
roberta_output = roberta(encoded)
</code></pre>
Difference between Normal and MultiVariateNormal distributions in Pytorch

ElLoboLoco: <p>Hi,</p>
<p>I have a bit of misunderstanding regarding the difference between the Normal distribution and the MultiVariateNormal distribution in pytorch. As I understand it, Normal distribution are univariate and so loc and scale should be floats. However, it is possible to define Normal distribution with vector mean and vector covariance which is in this case a multivariate gaussian right ? So in pytorch the following works for instance:</p>
<pre><code class="lang-auto">batch_size = 512
dim = 4
mean = torch.rand((batch_size,dim))
log_var = torch.rand(((batch_size,dim))
d = Normal(mean,torch.exp(0.5*log_var))
</code></pre>
<p>But in this case I define a multivariate normal using the univariate Normal distribution from pytorch right  ? How does pytorch interpret this ?</p>
<p>Please help me clear this misunderstanding</p>

KFrank: <p>Hi Samuel!</p>
<aside class="quote no-group" data-username="ElLoboLoco" data-post="1" data-topic="134853" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elloboloco/48/30094_2.png" class="avatar"> ElLoboLoco:</div>
<blockquote>
<p>Normal distribution are univariate and so loc and scale should be floats. However, it is possible to define Normal distribution with vector mean and vector covariance which is in this case a multivariate gaussian right ?</p>
</blockquote>
</aside>
<p>When you do this, you are defining a special case of a multivariate<br>
gaussian distribution where a sample is a vector, and the elements<br>
of that vector are uncorrelated with one another.</p>
<p>That is, the covariance of the distribution is a diagonal matrix whose<br>
diagonal is the square of the vector of standard deviations you passed<br>
into <code>Normal</code>.</p>
<p>Best.</p>
<p>K. Frank</p>
Classification with multi-dimensional labels

23vil: <p>Hi everyone,<br>
I am struggling with finding out if an idea that I had is possible in pytorch.</p>
<p>My goal is to have an image classifier, that selects the most relevant reference image for a reference based super resolution model.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d863f5b7c96c2cad8c25759e747af81e45a35943.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d863f5b7c96c2cad8c25759e747af81e45a35943" title="modelArchitecture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d863f5b7c96c2cad8c25759e747af81e45a35943_2_606x500.jpeg" alt="modelArchitecture" data-base62-sha1="uShio2CJ5sbyih6LrqAG1QXFD6b" width="606" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d863f5b7c96c2cad8c25759e747af81e45a35943_2_606x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/d/8/d863f5b7c96c2cad8c25759e747af81e45a35943.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/8/d863f5b7c96c2cad8c25759e747af81e45a35943.jpeg 2x" data-dominant-color="F1EEEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">modelArchitecture</span><span class="informations">767×632 65.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(green: forward pass, blue:backward)</p>
<p>So first, an image is sent to a classifier. That classifiers output should select one image from a fixed stack of images. This output is then sent to the super resolution model (SRM).</p>
<p>Is it possible to map a classification output to discrete 3d-classes without loosing the possibility to differntiate the whole thing?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.models as models

class RefSelector(nn.Module):
    def __init__(self):
        super(RefSelector, self).__init__()
        self.args = args
        self.RSel = models.resnet18(pretrained=True)
        for param in self.RSel.parameters():
            param.requires_grad_(False)
        self.RSel.fc = nn.Linear(in_features=512, out_features=10, bias=True)
    
    def forward(self, LR):
        return self.RSel(LR)

RefSelector = RefSelector()
inputImage = torch.rand(3,128,128)
refImages = torch.rand(10,3,128,128)
output = RefSelector(inputImage)
</code></pre>
<p>My first idea was to just use any image classifier e.g. ResNet, but of course if I apply</p>
<pre><code class="lang-auto">ReferenceID = torch.argmax(output, dim=1)
</code></pre>
<p>to that and then select the right reference image with that, there is no way a gradient could be calculated in the backward pass, because the reference image has nothing to do at all with the classifiers output.<br>
Has anyone a vague idea how, or if this could be possible?</p>
Sampling with index in 4-D tensor

thecho7: <p>I have a pair of value - index tensors.</p>
<pre><code>value = torch.rand(2, 128, 10, 10) # (batch, feature_size, x, y)
indices = torch.randint_like(torch.zeros(2, 30, 2), min=0, max=9) # (batch, n_sample, xy)
</code></pre>
<p>The expected shape of output tensor is like below,</p>
<pre><code>res: torch.Size([2, 128, 30]) # (batch, feature_size, n_sample)
</code></pre>
<p>This operation means sampling 30 points among the 10x10 = 100 plane.<br>
I tried to use <strong>index_select</strong>, but it only works with index tensor of 1-D.<br>
I have an option that create a bool tensor having size of (2, 10, 10) and use <strong>masked_select</strong>.<br>
However, it is ineffective as the number of samples increase.</p>
<p>I need your help</p>
<p>Thanks</p>

my3bikaht: <ol>
<li>You’re selecting 30 samples per plane from 128 planes, all samples at the same position per batch. Why not just use slices across planes?</li>
<li>With batch_num==2 you can iterate IMO, and also you can just flatten the 10x10 into 100 if 4 dims is hard to keep in mind.</li>
</ol>

thecho7: <p>I couldn’t get it ‘all samples at the same position per batch’ from the first comment.<br>
But flattening a tensor to easily handle is helpful.</p>
<p>Thank you for your advices</p>

my3bikaht: <p>I mean you sample 30 examples per plane. Each of examples is located at the same coords [xi,yi]. Your script is generating 30 different positions per batch, not per each plane. Thus, if this is not a mistake, you can slice through planes to retrieve 30 stacks of 1x128, since sample[i] is always at [xi,yi]</p>
Is there anything similar to grid_sample for 1D input?

torchlight: <p>/is there any way I can make index selection diff wrt indices?</p>

chener: <p>I have the same question. I think it’s important for tasks like stereo matching.</p>

a53153fb407732ef019a: <p>Here is my implementation of grid sample 1d. And I also implement grid sample 1d with grid_sample in Pytorch by regarding 1d as a special case of 2d. Overall, my implementation is 2~3x faster in forward pass. Hope it helps~</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/luo3300612/grid_sample1d">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/luo3300612/grid_sample1d" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/af7164bb17bb9caeb48f1cf7abfd0db873c92140ffe7ee98978a225842aec62e/luo3300612/grid_sample1d" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/luo3300612/grid_sample1d" target="_blank" rel="noopener nofollow ugc">GitHub - luo3300612/grid_sample1d: pytorch cuda extension of grid_sample1d</a></h3>

  <p>pytorch cuda extension of grid_sample1d. Contribute to luo3300612/grid_sample1d development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Libtorch has problems with expand()

x1290148: <p>I’m trying to convert a piece of code. libtorch doesn’t find the method for it.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.expand.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.expand.html</a><br>
<a href="https://pytorch.org/cppdocs/search.html?q=expand&amp;check_keywords=yes&amp;area=default" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/search.html?q=expand&amp;check_keywords=yes&amp;area=default</a><br>
<code>Tensor.</code> <code>expand</code> (*<em>sizes</em> ) → Tensor</p>
<pre><code class="lang-auto">def distance_matrix(x, y=None, p=2):  # pairwise distance of vectors
    y = x if type(y) == type(None) else y
    n = x.size(0)
    m = y.size(0)
    d = x.size(1)
    x = x.unsqueeze(1).expand(n, m, d)
    y = y.unsqueeze(0).expand(n, m, d)
    dist = torch.pow(x - y, p).sum(2)
    return dist
</code></pre>
<p>I tried, but it didn’t work.</p>
<pre><code class="lang-auto">torch::Tensor distance_matix(torch::Tensor x, torch::Tensor y, int p = 2) {
	auto n = x.size(0);
	auto m = y.size(0);
	auto d = x.size(1);
	x = x.unsqueeze(1).repeat( (n, m, d) );
	y = y.unsqueeze(0).expand((n, m, d));
	torch::Tensor dist = torch::pow(x - y, p).sum(2);
	return dist;
}
</code></pre>
Issue about BPR loss

icmp_no_request: <p>Hi,</p>
<p>I worked on implementing bayesian pairwise (BPR) loss function and have some problems:</p>
<ol>
<li>Is the number of negative item a fixed number for all users?</li>
<li>Is the number of positive item same as the number of negative item?</li>
<li>When I backward the loss, it is almost 0 (like 5e-7, 6e-8), how to deal with it?</li>
</ol>
<p>The code snippet is as following:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

emb_size = 8
num_pos = 6
num_neg = 3

user_embedding = torch.randn(emb_size)
pos_embedding = torch.randn(num_pos, emb_size)
neg_embedding = torch.randn(num_neg, emb_size)

pos_score = (user_embedding * pos_embedding).sum(1)
# pos_score = [num_pos]
neg_score = (user_embedding * neg_embedding).sum(1)
# neg_score = [num_neg]

bpr_loss = nn.LogSigmoid()(pos_score.sum() - neg_score.sum())
bpr_loss = -1 * bpr_loss
</code></pre>
<p>Thanks in advance for help.</p>
How to freeze the vector at the second last layer of shallow model?

hema_rathore: <p>Hi there I am using a shallow NN, having 500 neurons in my hidden layer. I have a question , Can I save the coefficients of this layer after training. I need to use these coefficients for further classification/</p>
<p>Plz guide</p>

ptrblck: <p>If you mean the <code>weight</code> parameter (and <code>bias</code>) as “coefficients”, then yes: you can access them via <code>linear_layer.weight</code> and <code>linear_layer.bias</code> (you might need to change the <code>linear_layer</code> name to the object name) or access the parameters in their <code>state_dict</code> via <code>linear_layer.state_dict()</code>.</p>

hema_rathore: <p>No sir Not weight but the resultant output after activation function, Maybe similar to transfer learning. For example, a pre-trained network is used first to fine-tune the model, then the second last layer output needs to be saved for further classification using SVM  or HMM etc.<br>
I am following this paper</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8085174">
  <header class="source">

      <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8085174" target="_blank" rel="noopener nofollow ugc">ieeexplore.ieee.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://ieeexplore.ieee.org/assets/img/ieee_logo_smedia_200X200.png" class="thumbnail onebox-avatar" width="200" height="200">

<h3><a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8085174" target="_blank" rel="noopener nofollow ugc">Speech Emotion Recognition Using Deep Convolutional Neural Network and...</a></h3>

  <p>Speech emotion recognition is challenging because of the affective gap between the subjective emotions and low-level features. Integrating multilevel feature learning and model training, deep convolutional neural networks (DCNN) has exhibited...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Regards</p>

ptrblck: <p>You can get the forward activations directly in the model’s <code>forward</code> method e.g. via:</p>
<pre><code class="lang-python">def forward(self, x):
    act1 = self.layer1(x)
    act2 = self.layer2(act1)
    ...
    out = self.last_layer(actX)
    return out
</code></pre>
<p>and return it directly as an additional output.<br>
Alternatively, you could also use forward hooks to grab intermediate activations from specified layers as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
<p>If you are only interested in the penultimate activation, you could also replace the last layer with <code>nn.Identity</code> without changing the <code>forward</code> at all.</p>
Output vs target in a segmentation task

Erdi: <p>I am trying to learn image segmentation in Pytorch. I learned that if I have say 100x100 images and 5 classes , then the ouput of the model will be (Batch_Size, 5, 100 ,100). Also mask images are  (Batch_Size,100,100). How do we compare 5-channel output to a mask which does not even have a channel dimension? That would maybe make sense if there was a final argmax in the channel dimension in the architecture so that we can have the predictions for each pixel and then we could compare it with the masks but apparently there is no argmax in any model.</p>
<p>What is the math behind nn.CrossEntropyLoss(output, target) ? Any help would be very appreciated.</p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss"><code>nn.CrossEntropyLoss</code> docs</a> show the used formula and you can see that the logits are indexed by the target, so there is no need to use <code>argmax</code> (which would also break the computation graph if applied in the model output).</p>
Can I use this graph as my final result?

hema_rathore: <p>I have got this graph while training my NN, but I am not getting whether I am doing right or not? Shall I consider the final result or this dip will create problem in analysis. Please guide.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/6038c0d789748dff600af886845e65267bfe6a6e.png" alt="Figure 2021-09-18 002504" data-base62-sha1="dJdqGVQ75BQo8Ig88Lw0302ui0S" width="372" height="279"></p>

arya47: <p>Depends on what the analysis is for, i would consider the plot to be sufficient. As for the dip could be cause of difference reasons:</p>
<ol>
<li>Something returning NaN</li>
<li>Randomness (unlikely but possible)</li>
</ol>

hema_rathore: <p>Thanks ! What do you mean by NaN, I mean where it is returning NaN values?<br>
Can I show this graph in my paper and say My accuracy is 94% which is my final result of classification, ignoring this dip.</p>

arya47: <p>I have never encountered such an error and i’m not the best person this <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> I would probably check with a supervisor about including this in a paper and the best supportive argument</p>

hema_rathore: <p>Hi <a class="mention" href="/u/arya47">@arya47</a> did you ask the supervisor?</p>
What is the purpose of `torch.distributed.launch`?

tlim: <p>In the <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py" rel="noopener nofollow ugc">script</a>, it describes using <code>python -m torch.distributed.launch........</code> to spawn the processes but I see that the Pytorch ImageNet example does not use it and is able to spawn the processes too, so what’s the point of it? I see that a lot of 3rd party open-source training repo also call <code> torch.distributed.launch</code> in their bash script for training. I am not very clear on what it does, seems like it setting the environment variables?</p>
<p>Pytorch imagenet example without the usage of the launch file: <a href="https://github.com/pytorch/examples/tree/master/imagenet" class="inline-onebox" rel="noopener nofollow ugc">examples/imagenet at master · pytorch/examples · GitHub</a></p>
<p>It will be very nice if there’s an example of using the launch file in the pytorch example to see what’s the difference.</p>

sandeep: <p>Can anyone comfirm that what shown here <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#:~:text=DistributedDataParallel%20%28DDP%29%20implements%20data%20parallelism%20at%20the%20module,the%20torch.distributed%20package%20to%20synchronize%20gradients%20and%20buffers" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.10.0+cu111 documentation</a>.</p>
<p>Is actually a different way to do the same we can do using torch.distributed.launch ?</p>

Kiuk_Chung: <p>torch.distributed.launch is a CLI tool that helps you create k copies of your training script (one on each process). And as you correctly pointed out it sets certain env vars that ddp uses to get information about rank, world size and so on. The closest analogy is what mpirun is to mpi applications. torch.distributed.launch compliments ddp by making it easy for you to run your ddp training script.</p>
<p>It is one way to launch ddp scripts but not the only way.</p>

tlim: <p>How does it make it easier? Is there a pytorch example over the ImageNet Pytorch example that shows how <code>torch.distributed.launch</code> could be a better option than just calling the arguments directly like for <code>main.py</code> in imagenet example?</p>
Mapping a template type to a ScalarType

andyljones: <p>I’m looking for a function - for a constexpr - that’ll map a type <code>T</code> to a <code>ScalarType</code>. I imagine it’d be called <code>at::ScalarType&lt;T&gt;()</code> or the like, but grepping through the PyTorch codebase I can’t find anything promising.</p>
<p>Concretely, I want to write something like</p>
<pre><code class="lang-auto">template &lt;typename T&gt; 
at::Tensor empty() {
    return at::empty({1}, at::dtype(at::ScalarType&lt;T&gt;());
}
</code></pre>
<p>so that a call like <code>empty&lt;float&gt;()</code> gives me back a <code>at::kFloat</code> tensor.</p>
<p>I understand this is not usually the way the conversion happens, and that control flow should generally be generic -&gt; specific. I know that I could write this as <code>empty(at::ScalarType)</code> and then use the <code>DISPATCH</code> macros to do the templating. For the particular chunk of code I’m working on though, it’d be useful and it feels like it should exist, I just can’t find it.</p>
<p>(More generally, if anyone has advice on searching for a specific signature in a large codebase, that’d be great too. Best I can find is CppDepend, which is $$$)</p>

andyljones: <p>I tripped over <code>caffe2::TypeMeta::Make&lt;T&gt;()</code> in the PyTorch source, so I’ve ended up using</p>
<pre><code class="lang-auto">template&lt;typename T&gt;
at::ScalarType dtype() { return at::typeMetaToScalarType(caffe2::TypeMeta::Make&lt;T&gt;()); }
</code></pre>

etern: <p>found another way: <code>torch::CppTypeToScalarType&lt;long&gt;()</code></p>
<p>and there is <code>ScalarTypeToCPPType</code> in the same src file: <code>c10/core/ScalarType.h</code></p>
Layernorm backward

Trinayan_Baruah: <p>Why does PyTorch uses three different kernels for backward (four when elementwise affine is True) for LayerNorm backward. NVIDIA Apex seems to use only a single kernel or two when elementwise affine is True. Are there some edge cases Apex does not deal with and PyTorch does ?.</p>
<p>Also how is the scale and bias here (<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/layer_norm_kernel.cu#L355" class="inline-onebox" rel="noopener nofollow ugc">pytorch/layer_norm_kernel.cu at master · pytorch/pytorch · GitHub</a>) different from gamma and beta. Isn’t gamma and beta the scale and bias parameters for layernorm?.</p>

tom: <p>While it could be that PyTorch’s kernels are less optimized, I do think that apex uses three kernels if you want all gradients for input and parameters (<code>cuComputePartGradGammaBeta</code>, <code>cuComputeGradGammaBeta</code>, <code>cuComputeGradInput</code>) so the difference is not quite as stark.</p>
<p>That said, if you have benchmarks that show a PyTorch being slower than Apex, I’m quite certain people will look into it more (PyToch did adopt the LayerNorm forward at some point).</p>
<p>Best regards</p>
<p>Thomas</p>

Trinayan_Baruah: <p>I did a high level mapping. The ComputePartGradGammaBeta and ComputeGradGammaBeta are together equivalent to the PyTorch GammaBetaBackwardKernel basically calculating the gradients for gamma and beta if those params are set to trainable. Despite PyTorch doing it through one kernel, the perf of this gradient computation is actually much worse than Apex since the GammaBetaBackward kernel does not effectively utilize the GPU. Basically low occupancy and memory throughput utilization. I can post some benchmark results but we have seen it in many cases. Other than that what Apex does through cuComputeGradInput maps to a set of three kernels in PyTorch(ComputeInternalGradients,ComputeGradientFusedParamsCUDAKernel and LayerNormBackwardKernel). Its not clear why this part is not fused though the GPU utilization is fairly good for these kernels.</p>
Torchaudio backend

Tzeviya: <p>I’m using the <a href="https://github.com/facebookresearch/WavAugment" rel="noopener nofollow ugc">WavAugment</a> package, which imports <code>torchaudio</code>. I’m receiving the following warning:</p>
<pre><code class="lang-auto">UserWarning: "sox" backend is being deprecated. The default backend will be changed to "sox_io" backend in 0.8.0 and "sox" backend will be removed in 0.9.0. Please migrate to "sox_io" backend. Please refer to https://github.com/pytorch/audio/issues/903 for the detail.
    '"sox" backend is being deprecated. '
</code></pre>
<p>I tried adding the following line as recommended:</p>
<p><code>torchaudio.set_audio_backend("sox_io")</code></p>
<p>But I still get the warning. What am I doing wrong?</p>

nateanl: <p>Hi <a class="mention" href="/u/tzeviya">@Tzeviya</a>, what’s your version of torchaudio? Upgrading to 0.9.0 should resolve this issue.</p>

Tzeviya: <p>Thanks <a class="mention" href="/u/nateanl">@nateanl</a>! Upgrading did indeed solve the problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>