akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Call custom functions by jit in C++

username: afshin67
cooked: <p>I have created the following class and saved <code>scripted_module</code> and loaded in <code>C++</code> API by:</p>
<pre><code class="lang-auto">    torch::jit::script::Module module;
    module = torch::jit::load("scriptmodule.pt");
</code></pre>
<p>Now, the question is how can I call <code>func</code> from C++?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()

    @torch.jit.ignore
    def get_rand(self):
        return torch.randint(0, 2, size=[1])

    def forward(self):
        pass 
    
    @torch.jit.export
    def func(self):
        done = self.get_rand()
        print (done)

scripted_module = torch.jit.script(MyModule())
</code></pre>

username: driazati
cooked: <p>Does this answer help?</p>
<aside class="quote quote-modified" data-post="4" data-topic="60926">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/driazati/48/15393_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/jit-for-tolist/60926/4">Jit for tolist()</a> <a class="badge-category__wrapper " href="/c/jit/13"><span data-category-id="13" data-drop-close="true" class="badge-category " title="A category for TorchScript and the PyTorch JIT compiler"><span class="badge-category__name">jit</span></span></a>
  </div>
  <blockquote>
    For methods other than forward you have to explicitly get the method and run it. For this Module 
class M(nn.Module):
    @torch.jit.export
    def infer(self, x):
        return x + 10

torch.jit.script(M()).save("m.pt")

You can run it in C++ with script::Module::get_method 
int main() {
  auto module = torch::jit::load("m.pt");
  auto result = module.get_method("infer")({torch::ones({2, 2})});
  std::cout &lt;&lt; result &lt;&lt; "\n";
}

We have an open issue to improve our C++ documentation to make thi…
  </blockquote>
</aside>


username: driazati
cooked: <p>Also just FYI <code>@torch.jit.ignore</code> functions cannot be exported, if you do <code>scripted_module.save("out.pt")</code> you’ll get</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "../test.py", line 196, in &lt;module&gt;
    scripted_module.save('s.pt')
  File "/home/pytorch/torch/jit/__init__.py", line 1621, in save
    return self._c.save(*args, **kwargs)
RuntimeError: 
Could not export Python function call 'get_rand'. Remove calls to Python functions before export. Did you forget add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:
  File "../test.py", line 192
    @torch.jit.export
    def func(self):
        done = self.get_rand()
               ~~~~~~~~~~~~~ &lt;--- HERE
        print (done)
</code></pre>
<p>Which is expected behavior (since saved models are expected to run without a Python runtime attached)</p>

username: afshin67
cooked: <p>Thanks, this worked.</p>

username: afshin67
cooked: <p>Yes, I faced a similar example and as you mentioned it makes sense.</p>
Why not removing a register_hook() slows down the training gradually?

username: amirhf
cooked: <p>I had to use a <code>register_hook()</code> on a parameter in a custom loss function to modify gradients when backpropagating. However I did not use <code>hook.remove()</code> at first and I noticed training slows down as the model iterates through batches and updates (backprop updates).<br>
My question: what is the mechanism of a <code>register_hook()</code>? What does it pile up and where does it pile up? How does <code>hook.remove()</code> fixes this and what does it exactly free up?</p>

username: albanD
cooked: <p>If you register the hook on your parameter directly, because the parameter does not change, you should register it only once for the whole training. Otherwise you add a new hook everytime and you will have many hooks.</p>

username: amirhf
cooked: <p>That makes sense. Thanks for the explanation!</p>
Resetting Dataparallel after init and using it with a dictionary

username: pytorcher
cooked: <p>Two questions, first is that I have a network that I would like to train for a little while using dataparallel and then modify the network and continue to train with dataparallel. What is the correct way to reinitialize? Should I remove the dataparallel in some way or just call it again on net.module(which runs but doesn’t appear to work as expected)?<br>
Second question is continued from here: <a href="https://discuss.pytorch.org/t/how-to-transition-to-functions-not-being-allowed-to-have-member-variables/62170/2">How to transition to functions not being allowed to have member variables</a>. Now that I am using a dictionary it appears the tensors in that dictionary are not split across GPUs. What’s the best way to include them in the dataparallel?</p>
Runtime error in multithreaded inference scenario

username: MRK
cooked: <p>Hi,<br>
I am trying to run a pytorch model for inference in a multithreaded scenario where multiple threads try to predict using the shared model.</p>
<p>In a non-concurrent case, inference using the model runs perfectly fine but I start getting the following error occasionally for different threads</p>
<p><strong>RuntimeError: The expanded size of the tensor (100) must match the existing size (3) at non-singleton dimension 1.  Target sizes: [40, 100, 300].  Tensor sizes: [16, 3, 1]</strong></p>
<p>I have experimented a little bit and having a mutual exclusion on the prediction solves the problem but that defeats the purpose of having multiple threads.</p>
<p>I have some questions</p>
<ol>
<li>What is the recommended way to have concurrent threads sharing the same model?</li>
<li>Do I need to change any configuration related settings?</li>
</ol>
Unable to understand how pytorch calculates cross-entropy loss

username: bing
cooked: <p>Hi,<br>
I came across this formula of how pytorch calculates cross-entropy loss-</p>
<pre><code class="lang-auto">loss(x, class) = -log(exp(x[class]) / (\sum_j exp(x[j])))
               = -x[class] + log(\sum_j exp(x[j]))
</code></pre>
<p>Could anyone explain in more mathematical way which expression pytorch is using to calculate it as I have encountered only the following ways-<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5be0ba951c20b60ea715044bb2de4e111ebb8546.png" alt="13%20PM" data-base62-sha1="d6MVByxterxheaqb2i1m4Uq3dLE" width="602" height="122"></p>
<p>where p is the target and q is the predicted</p>
<p>while another representation,I have seen is<br>
<code>Loss(Y_hat,Y) = -Ylog(Y_hat)</code></p>
<p>I mean, I am not able to understand the code which pytorch uses to calculate it.</p>

username: ptrblck
cooked: <p>The <code>exp(x)/sum(exp(x))</code> defines the softmax.<br>
Since <code>p_i</code> is the target, we avoid storing it in a one-hot encoded format (one 1, everything else zeros) and multiplying with it, as it’s wasteful. Instead we directly index the logit via <code>x[class]</code>.</p>
Difference of methods between torch.nn and functional

username: zuoxingdong
cooked: <p>Both torch.nn and functional have methods such as <code>Conv2d, Max Pooling, ReLU</code> etc. However, many public codes writes <code>Conv</code> and <code>Linear</code> layer in a class <code>__init__</code> and call it with <code>ReLU</code> and <code>Pooling</code> in <code>forward()</code>. Is there a good reason for that ?</p>
<p>I am guessing that because <code>Conv</code> and <code>Linear</code> consist of learnable parameters which wrapped within functional module. And then define them in <code>__init__</code> as members for the class. For <code>ReLU, Pooling</code> which do not require learnable parameters just to be called in <code>forward()</code> method. Is it like that ?</p>
<p>So that it makes things easier to call weight values in run-time e.g. <code>net.conv1.data</code></p>

username: smth
cooked: <p>Yes, you are spot on. the difference between <code>torch.nn</code> and <code>torch.nn.functional</code> is a matter of convenience and taste. <code>torch.nn</code> is more convenient for methods which have learnable parameters.</p>

username: stared
cooked: <p>Is there some performance difference for layers without learnable parameters (e.g. <code>nn.ReLU</code> vs <code>F.relu</code>)?</p>
<p>Having in mind that in <code>nn</code> I can set <code>inplace=True</code>.</p>

username: smth
cooked: <p><a class="mention" href="/u/stared">@stared</a> there isn’t any performance difference.</p>

username: An_Tran
cooked: <p>Hi <a class="mention" href="/u/smth">@smth</a>,</p>
<p>How about the difference between torch.nn and torch.autograd.Function? thank you for answering my newbie question.</p>

username: ptrblck
cooked: <p><code>torch.nn</code> is a namespace for a lot of modules as well as the functional API.<br>
<code>torch.autograd.Function</code> can be used to create a new function with a custom forward and backward pass as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>
How install Pytorch in Jetson TX2

username: Miguel_Campos
cooked: <p>Hello family</p>
<p>i am trying to install pytorch in a jetson tx2 machine but i cannot to do it. Somebody have any experience with this kind of installation?</p>

username: ptrblck
cooked: <p>Have a look at <a href="https://github.com/pytorch/pytorch#nvidia-jetson-platforms">these binaries</a>.</p>
Expected object of scalar type Short but got scalar type Float for argument #2 'mat2' in call to _th_mm

username: hamid_khb
cooked: <p>Hi guys. as you can see I already tried to generate the input, h and c in ShortTensor format but I still got the same error. would be nice If you can explain what I’m doing wrong.<br>
here is my Code:</p>
<pre><code class="lang-auto">
&gt; class data(Dataset):
&gt;     def __init__(self, samples=10000, number=30):
&gt;         self.x = torch.from_numpy(np.matrix
&gt;                         (np.random.random_integers(0,9,samples*number).reshape(samples, number)))
&gt;         self.y = torch.from_numpy(np.zeros((samples))).type(torch.ShortTensor)
&gt;         for index, row in enumerate(self.x):
&gt;             self.y[index] = 1 if torch.sum(row) &gt;= 130 else 0
&gt;     
&gt; class LSTM(nn.Module):
&gt;     def __init__(self,i_size, h_size, n_layer, batch_size = 30 ):
&gt;         super().__init__()
&gt;         self.lstm = nn.LSTM(input_size=i_size, hidden_size=h_size, num_layers=n_layer)
&gt;         self.h = torch.randn(n_layer, batch_size, h_size).type(torch.ShortTensor)
&gt;         self.c = torch.randn(n_layer, batch_size, h_size).type(torch.ShortTensor)
&gt;         self.hidden = (self.h, self.c)
&gt;         self.linear = nn.Linear(n_layer, 1)
&gt;     
&gt;     def forward(self,x):
&gt;         out, hidden = self.lstm(x.type(torch.ShortTensor), self.hidden)
&gt;         out = nn.Softmax(self.linear(out.short()))
&gt;         return out
&gt; 
&gt; data_set = data()
&gt; train_data = data_set.x[0:8000, :, None]
&gt; train_label = data_set.y[0:8000]
&gt; test_data = data_set.x[8000:, : , None]
&gt; test_label = data_set.y[8000:]
&gt; #train_data = train_data[:,:,None]
&gt; train_data[0].shape
&gt; 
&gt; input_size = 1
&gt; hidden_size = 30
&gt; layer_num = 200
&gt; model_LSTM = LSTM(input_size, hidden_size, layer_num)
&gt; #model_LSTM.cuda()
&gt; y = model_LSTM(train_data)`Preformatted text`
</code></pre>

username: hamid_khb
cooked: <p>self.h = torch.randn(n_layer, batch_size, h_size).type(torch.ShortTensor)</p>
<p>But I was able to solve the problem. It turned out that it actually didnt need ShortTensor but float32. I think LSTM takes only float32. Everything else didn’t work for me.</p>
Not sufficiently precise base_grid for torch.nn.functional.grid_sample?

username: talesa
cooked: <p>Hey!</p>
<p>I’ve noticed that for sufficiently high resolution of images (e.g. full HD) I cannot get a good enough “identity grid” using <code>torch.float32</code> precision, code below.</p>
<p>Posting here:</p>
<ol>
<li>to give a heads up to others</li>
<li>to double check - am I right this just a numerical precision issue?</li>
</ol>
<p>I saw <a class="mention" href="/u/bnehoran">@bnehoran</a> mention in the <a href="https://github.com/pytorch/pytorch/issues/24870" rel="noopener nofollow ugc">issue on github</a></p>
<blockquote>
<p>Allow option for sampling using only the residual displacement/flow. Eliminates the need to constantly add an identity grid to the flow/displacement field, which is imprecise, slow, and very prone to user error.</p>
</blockquote>
<p>If what I’m seeing is what I think I’m seeing it would be a pretty good motivation to add a function which just takes residual displacement as input.</p>
<p>Code:</p>
<pre><code class="lang-auto">import torch.nn.functional as F

W = 1920
H = 1080
shape = [1, 1, H, W]

dtype = torch.float

x_prev = torch.zeros(shape, dtype=dtype)
x_prev[..., :, 0, 0] = 1.

x_next_expected = x_prev.clone()

base_grid = torch.stack((torch.linspace(-1, 1, W, dtype=dtype).unsqueeze(0).repeat(H, 1),
                         torch.linspace(-1, 1, H, dtype=dtype).unsqueeze(-1).repeat(1, W),), dim=-1)
base_grid = base_grid.unsqueeze(0)

x_next = F.grid_sample(x_prev, base_grid, align_corners=True)

print((x_next_expected - x_next).abs().sum())
print(x_next[..., :2, :2])
print(x_next_expected[..., :2, :2])
print(x_next[..., :2, :2] - x_next_expected[..., :2, :2])

assert torch.allclose(x_next_expected, x_next)
</code></pre>

username: tom
cooked: <p>To me it looks like the expectation is implicitly like <code>align_corners=True</code> but you don’t pass that to <code>grid_sample</code>.<br>
At least I get an error of <code>0.75</code> with your code and <code>2.134e-5</code> when adding <code>align_corners=True</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: talesa
cooked: <p>I meant with <code>align_corners=True</code>, I’ll update the code.<br>
Also, isn’t <code>align_corners=True</code> the default as of v1.3? I know it’s going to default to <code>align_corners=False</code> in v1.4.</p>
<p>Thanks,<br>
Adam</p>

username: tom
cooked: <p>Yeah well, then you have to code up the function that computes without the identity grid.</p>
MultiLabelSoftMarginLoss vs BCEWithLogitsLoss vs KLDivLoss

username: Mehran_Ziadloo
cooked: <p>Sorry if my question is too basic. I’m comparing the three loss functions of <a href="https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss" rel="nofollow noopener">BCEWithLogitsLoss</a>, <a href="https://pytorch.org/docs/stable/nn.html#multilabelsoftmarginloss" rel="nofollow noopener">MultiLabelSoftMarginLoss</a>, and <a href="https://pytorch.org/docs/stable/nn.html#kldivloss" rel="nofollow noopener">KLDivLoss</a>. Here are the formula for each one:</p>
<p>For BCEWithLogitsLoss, we have (assuming the reduction is <code>mean</code>):</p>
<p><a href="https://latex.codecogs.com/png.latex?%5Cbg_white%20l%28x%2C%20y%29%20%3D%20%5Cfrac%7B-1%7D%7BC%7D%20%5Csum_%7Bi%3D1%7D%5E%7BC%7D%20y_i%20*%20log%28%5Csigma%20%28x_i%29%29%20+%20%281-y_i%29%20*%20log%281-%5Csigma%20%28x_i%29%29" rel="nofollow noopener"><img src="https://latex.codecogs.com/png.latex?%5Cbg_white%20l%28x%2C%20y%29%20%3D%20%5Cfrac%7B-1%7D%7BC%7D%20%5Csum_%7Bi%3D1%7D%5E%7BC%7D%20y_i%20*%20log%28%5Csigma%20%28x_i%29%29%20+%20%281-y_i%29%20*%20log%281-%5Csigma%20%28x_i%29%29" alt width="446" height="54"></a></p>
<p>And for MultiLabelSoftMarginLoss, we have:</p>
<p><a href="https://latex.codecogs.com/png.latex?%5Cbg_white%20l%28x%2C%20y%29%20%3D%20%5Cfrac%7B-1%7D%7BC%7D%20%5Csum_%7Bi%3D1%7D%5E%7BC%7D%20y_i%20*%20log%28%5Csigma%20%28x_i%29%29%20+%20%281-y_i%29%20*%20log%28%5Csigma%20%28-x_i%29%29" rel="nofollow noopener"><img src="https://latex.codecogs.com/png.latex?%5Cbg_white%20l%28x%2C%20y%29%20%3D%20%5Cfrac%7B-1%7D%7BC%7D%20%5Csum_%7Bi%3D1%7D%5E%7BC%7D%20y_i%20*%20log%28%5Csigma%20%28x_i%29%29%20+%20%281-y_i%29%20*%20log%28%5Csigma%20%28-x_i%29%29" alt width="429" height="54"></a></p>
<p>And finally, for KLDivLoss we have (with <code>batchmean</code> reduction):</p>
<p><a href="https://latex.codecogs.com/png.latex?%5Cbg_white%20l%28x%2C%20y%29%20%3D%20%5Cfrac%7B-1%7D%7BN%7D%20%5Csum_%7Bi%3D1%7D%5E%7BN%7D%20y_i%20*%20%28log%28y_i%29%20-%20x_i%29" rel="nofollow noopener"><img src="https://latex.codecogs.com/png.latex?%5Cbg_white%20l%28x%2C%20y%29%20%3D%20%5Cfrac%7B-1%7D%7BN%7D%20%5Csum_%7Bi%3D1%7D%5E%7BN%7D%20y_i%20*%20%28log%28y_i%29%20-%20x_i%29" alt width="264" height="54"></a></p>
<p>While I understand the first formula (it’s cross-entropy) and its use case, I’m not familiar with the next two. Could someone please explain to me when should I use any of these three loss functions? I’ve been reading the implementation for <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="nofollow noopener">The Annotated Transformer</a> where in my opinion was a good candidate for BCEWithLogitsLoss but the author had decided to use KLDivLoss. In general, what are the use cases for each of these three loss functions?</p>
Word embedding tutorial code does not exit properly

username: Ori_Tzadok
cooked: <p>Hi, I am writing in regard of the following tutorial:</p>
<p>WORD EMBEDDINGS: ENCODING LEXICAL SEMANTICS<br>
<a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html</a></p>
<p>I copied the code of the first part (the Shakespeare Sonnet cassifier). No changes were made.<br>
When I run the script, the program refuses to exit and freezes after printing the losses at the end, and I don’t know why</p>
<p>Any help?<br>
Thank’s</p>

username: zhangguanheng66
cooked: <p>Please open an issue on <a href="https://github.com/pytorch/tutorials" rel="nofollow noopener">pytorch/tutorials</a></p>
Why parameters added to module are not trained, and how to remove some parameters from network?

username: msqiqi10
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn.parameter import Parameter

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.layer1 = nn.Linear(10, 5)
        self.mask1 = Parameter(torch.ones(5))
        self.layer2 = nn.Linear(5, 5)
        self.mask2 = Parameter(torch.ones(5))
        self.layer3 = nn.Linear(5, 1)
        self.mask3 = Parameter(torch.ones(1))
    
    def forward(self, x):
        x = self.layer1(x)
        x = torch.mul(self.mask1, x)
        x = F.leaky_relu(x)
        x = self.layer2(x)
        x = torch.mul(self.mask2, x)
        x = F.leaky_relu(x)
        x = self.layer3(x)
        x = torch.mul(self.mask3, x)
        x = torch.tanh(x)
        return x

def main():
    n = net()
    print(n)
    out = n(torch.randn(5, 10))
    print(out)
    loss = torch.mean(out - torch.randn(1, 5))
    opt_n = torch.optim.SGD(n.parameters(), lr=0.0001)
    loss.backward()
    opt_n.step()
    
if __name__ == "__main__":
    main()
</code></pre>
<p>Here is my network structure and masks are binary. The trainable masks are supposed to block some neuron connections along with the training process.</p>
<p>It seems that after step, masks are not updated with their grads, why would this happen?</p>
<p>And if I want to train mask and layer123 separately, how am I supposed to remove layer123 and masks accordingly? Or is there a better implementation?</p>
<p>Thank you!</p>

username: msqiqi10
cooked: <p>The masks are actually trained properly. The value of masks remaining the same after opt_n.step() is because the lr of optim is too small. So it’s no longer a problem.</p>
Do I need to zero_grad when I use torch.autograd.grad()?

username: gebrahimi
cooked: <p>Do I need to use optimiser.zero_grad(), before using orch.autograd.grad? or it is just used when we use backward() function?</p>

username: albanD
cooked: <p>Just when you use the <code>.backward()</code> function. The goal of <code>zero_grad</code> is to reset the buffers used by <code>.backward()</code>.</p>