akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Random sampling from exponential distribution C++

username: vdg
cooked: <p>Hi,</p>
<p>I want to sample at random from a tensor. For uniform distribution I use:</p>
<pre><code class="lang-auto">auto randtensor = torch::rand({100,1}); //Some tensor to sample from:

auto indices = torch::randint(0, 100, 10, torch::TensorOptions().dtype(torch::kInt64)); // 10 uniformly distributed random indices

auto random_sample = randtensor.index_select(0,indices); //10 random elements from randtensor, uniformly distributed
</code></pre>
<p>Now, instead of using a uniform distribution, I want to use an exponential distribution. Is there a way to generate random indices from an exponential distriburtion?</p>
<p>Thanks!</p>

username: glaringlee
cooked: <p>to my knowledge, there is direct way. You have to write your own index generator using exponential distribution and then create the tensor using generated indices.</p>
TorchScripted loss function breaks in the backward

username: fermat97
cooked: <p>I have a loss function which is torchscripted using <code>torch.jit.script</code> decorator. Before torchscripting it works fine but after torchscripting it fails in the <code>backward</code> path due to some <code>in-place</code> operation. However, I cannot find any <code>in-place</code> operation in my loss function. Any idea?</p>
Tensor.cuda() vs Tensor.to('cuda')

username: mitulraj
cooked: <p>Hello,</p>
<p>I am new to pytorch and trying to understand it. When I see a codes written in pytorch, to utilize GPU sometimes .cuda() is used while .to(‘cuda’) is used sometimes. I want to know if there is any difference between both methods or both are same?</p>

username: albanD
cooked: <p>Hi,</p>
<p>There are the same.<br>
<code>.cuda()</code> is the old api and <code>.to()</code> is a more recent api that can take any device and/or type and so more general.</p>

username: iacob
cooked: <p>Their syntax varies slightly, but <a href="https://stackoverflow.com/questions/62907815/pytorch-what-is-the-difference-between-using-tensor-cuda-and-tensor-totorch/66568863#66568863" rel="noopener nofollow ugc">they are equivalent</a>:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>.to()</th>
<th>.cuda()</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>to('cuda')</code></td>
<td><code>cuda()</code></td>
</tr>
<tr>
<td><code>to('cuda:1')</code></td>
<td><code>cuda(device=1)</code></td>
</tr>
<tr>
<td><code>to('cpu')</code></td>
<td><code>cpu()</code></td>
</tr>
</tbody>
</table>
</div><p>Note: the first row uses the default GPU (this can be set with <code>torch.cuda.set_device()</code>).</p>
Pytorch Mobile - Drastically different output for the same model between PC and Android

username: yonatankarimish
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I just finished training a model with good results, and serialized it using <code>torch.jit.trace()</code>. However, the model performs very poorly when deployed to my Android phone.</p>
<p>While investigating the cause of the performance difference between devices, I ran both models (pc and android) using a zero-tensor as input. To my surprise, I found out the model outputs are totally different.</p>
<p>This would explain the poor results (since the mobile model essentially outputs nonsense values). But is this considered a bug or am I doing something wrong on my end?</p>

username: yonatankarimish
cooked: <p>for reference, here is the code I have written:</p>
<ol>
<li>After training the model (on my pc):</li>
</ol>
<pre><code class="lang-auto">encoder = encoder.to(cpu).eval()
mock_encoder_input = torch.tensor(np.random.rand(1, 1, 138, 80), dtype=torch.float)
traced_encoder = torch.jit.trace(encoder, mock_encoder_input)
traced_encoder.save(colab_data + "/spectrograms/encoder.pt")
</code></pre>
<ol start="2">
<li>Checking the model outputs on the pc:</li>
</ol>
<pre><code class="lang-auto">encoder(torch.zeros(1, 1, 138, 80, dtype=test_spectrograms.dtype))
</code></pre>
<p>with the first 5 scalars in the output tensor being: <code>-7.5927e-01,  1.1893e-02, -6.2612e-03,  5.3497e-03, -3.3745e-02</code></p>
<ol start="3">
<li>Checking the model outputs on my Android phone:</li>
</ol>
<pre><code class="lang-auto">IValue spectroTensor = IValue.from(Tensor.fromBlob(new float[(int)(dim0 * dim1 * dim2 * dim3)], tensorShape));
IValue[] encoderResults = encoder.forward(spectroTensor)
</code></pre>
<p>with the first 5 scalars in the output tensor being: <code>0.23662394, -0.41967326, 0.21712059, -0.312757, 0.29841155,</code></p>

username: xian_kgx
cooked: <p>Can you verify the output of spectroTensor to see if it is really all zeros?</p>

username: yonatankarimish
cooked: <p>Do you mean the output of encoder.forward(spectroTensor)? Because it isn’t zero and shouldn’t be zero.</p>
<p>If you just mean the value of spectroTensor itself - yes, I have already verified it consists of zeros.</p>

username: yonatankarimish
cooked: <p>Well figured it out, and it had <a href="https://stackoverflow.com/questions/5762547/why-arent-my-asset-files-updated" rel="noopener nofollow ugc">nothing to do with PyTorch</a>…</p>
<p>In short - Android will not update asset files when these are changed. You must manually clear the caches through the phone/emulator settings =&gt; apps menu (sigh…)</p>
<p>The app was running an older model (one of many trials), causing the discrepancies I saw.</p>
Where's the implementation of KL divergence loss function?

username: blackbirdbarber
cooked: <p>Simple question but I was not able to find it due to ridiculous Github search and friends.</p>

username: Unity05
cooked: <p><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py" rel="noopener nofollow ugc">pytorch/functional.py at master · pytorch/pytorch (github.com)</a>, line 2549.</p>

username: blackbirdbarber
cooked: <p>Should be in C++ I guess, else in Python.</p>

username: iacob
cooked: <p>It’s <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L2627" rel="noopener nofollow ugc">line 2627</a>.</p>

username: Unity05
cooked: <p>Oups, RIP my eyes then. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>
How to generate source code of compressed model with mask?

username: marsggbo
cooked: <p>As far as I know, most channel compression methods generate a mask for each module in terms of channels. For example, we have a model defined by</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        self.conv1 = nn.Conv2d(3, 10, 3)
        self.conv2 = nn.Conv2d(10, 20, 3)
        self.fc = nn.Linear(20,10)

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = self.fc(out)
        return out
</code></pre>
<p>Suppose we generate a mask as below</p>
<pre><code class="lang-auto">{
    'conv1': 5,
    'conv2': 12,
}
</code></pre>
<p>The mask means that the output channel size of conv1 and conv2 can be compressed to 5 and 12. Now I know how to generate the mask, but I wonder Is any <strong>beautiful and simple</strong> method to  generate the new code</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        self.conv1 = nn.Conv2d(3, 5, 3)
        self.conv2 = nn.Conv2d(5, 12, 3)
        self.fc = nn.Linear(12, 10)

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = self.fc(out)
        return out
</code></pre>
<p>Besides, if the source code builds modules by functions (e.g., <code>_make_layer</code> in resnet), can we still generate the code based on the mask?</p>
Weight argument of Loss Function

username: Tejan_Mehndiratta
cooked: <p>Suppose I have a training set which consists of 4 classes and the number of samples belonging to the 4 classes is 20, 30, 40, 10 respectively. So should I pass the tensor torch.tensor([20,30,40,10]) / 100. to the weight argument of the loss function?<br>
Or should I calculate the values of the weight argument for each batch on the fly in the training loop?</p>

username: KFrank
cooked: <p>Hi Tejan!</p>
<aside class="quote no-group" data-username="Tejan_Mehndiratta" data-post="1" data-topic="114168" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tejan_mehndiratta/48/29857_2.png" class="avatar"> Tejan_Mehndiratta:</div>
<blockquote>
<p>Suppose I have a training set which consists of 4 classes and the number of samples belonging to the 4 classes is 20, 30, 40, 10 respectively. So should I pass the tensor torch.tensor([20,30,40,10]) / 100. to the weight argument of the loss function?</p>
</blockquote>
</aside>
<p>You have this backwards – you want to weight the less-frequent<br>
classes more heavily in your loss function.  The most common<br>
weighting scheme would be the reciprocal of what you have,<br>
<code>100.0 / torch.tensor ([20.0 ,30.0 ,40.0 ,10.0])</code></p>
<aside class="quote no-group">
<blockquote>
<p>Or should I calculate the values of the weight argument for each batch on the fly in the training loop?</p>
</blockquote>
</aside>
<p>My preference is to calculate the weights using the frequency of<br>
classes in the entire training set and use this single set of weights<br>
for each batch.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Tejan_Mehndiratta
cooked: <p>Hi,</p>
<p>But shouldn’t the sum of the weight vector equal to 1?</p>

username: KFrank
cooked: <p>Hi Tejan!</p>
<aside class="quote no-group" data-username="Tejan_Mehndiratta" data-post="3" data-topic="114168" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tejan_mehndiratta/48/29857_2.png" class="avatar"> Tejan_Mehndiratta:</div>
<blockquote>
<p>But shouldn’t the sum of the weight vector equal to 1?</p>
</blockquote>
</aside>
<p>No.  This kind of use case applies to <a href="https://pytorch.org/docs/1.7.1/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss</a>, which computes<br>
a <em>weighted mean</em> (when using the default <code>reduction = 'mean'</code>).  This<br>
means that <code>CrossEntropyLoss</code> divides by the sum of the weights, so<br>
the sum of the weights drops out of the final loss value.</p>
<p>(It doesn’t hurt to have the weights sum to one; it just doesn’t matter.)</p>
<p>Best.</p>
<p>K. Frank</p>
Copying a model

username: Eyal
cooked: <p>Hello,<br>
I’m working on multiple remote machines, which share a directory (minIO)<br>
In one of them, I have a trained model named epoch10.pt, I’ve moved it into the minIO using the Linux cp command, then copied it yet again to the other machine, now when I try to load the model using:<br>
<code>model = torch.load(hyper_params['model_path'])</code></p>
<p>I’m getting the error:<br>
<code>Exception has occurred: RuntimeError Invalid magic number; corrupt file?</code></p>
<p>Thanks in advance.</p>

username: crcrpar
cooked: <p>Hi,</p>
<p>If the <code>hyper_params['model_path']</code> is the output of <code>torch.save(model, path)</code>, it would difficult to load that file in other directories as explained in the tutorial:</p>
<blockquote>
<p>The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. (quote from <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model" class="inline-onebox" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 2.1.1+cu121 documentation</a>)</p>
</blockquote>
<p>The recommended way is to save/load <code>state_dict</code> (<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-state-dict-recommended" class="inline-onebox" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 2.1.1+cu121 documentation</a>).</p>
Support for SiLU operator in Onnx export

username: Srimanth_Tenneti
cooked: <p>Hey guys,<br>
I  am trying to export a model to onnx and it says the operator is not supported. Could I get help with this issue.</p>
<p>Error : RuntimeError: Exporting the operator silu to ONNX opset version 9 is not supported. Please open a bug to request ONNX export support for the missing operator.</p>
<p>Thanks in advance</p>
Custom dataloader for batches of different lengths

username: mathav95raj
cooked: <p>Data : csv file with two columns: a hindi sentence and it’s english translation of same length. Along the row sentences of varying length</p>
<p>How can I get three unique batches in a single iteration of the dataloader? (batch 1 - hindi sentences, batch 2- different english sentences, batch 3 - different english sentences)</p>
<p>Initially I tried setting batch size as three times what I intended and split it.<br>
In that case all the three sets of sentences will be padded to maximum sentence length but what I really want is sentences to be padded within each batch’s maximum length</p>
Cannot create tensors from image in subprocess

username: HeikoSchuett
cooked: <p>I try to run some parallel subprocesses to do some (cpu-based) post processing of network outputs. However, my attempts fail already at the data loading step. I cannot turn the images into torch tensors within the subprocesses. A minimal failing example is this:</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.multiprocessing as mp
from skimage import io

def show_np():
    img_name = '000000000139.jpg'
    image = io.imread(img_name)
    print(image.shape)


def show_torch():
    img_name = '000000000139.jpg'
    image = io.imread(img_name)
    tensor = torch.tensor(image)
    print(tensor.shape)


show_np()
show_torch()


processes = []
for i in range(3):
    p = mp.Process(target=show_np)
    p.start()
    processes.append(p)

for p in processes:
    p.join()

processes = []
for i in range(3):
    p = mp.Process(target=show_torch)
    p.start()
    processes.append(p)

for p in processes:
    p.join()
</code></pre>
<p>This code prints the size when I run the function with only the numpy array or without subprocesses, but when I try to transom into torch.tensor in the subprocess evaluation just stops.</p>
<p>To run the code you need to put an image with the given name into the current folder of course.</p>
<p>Any help is very welcome!</p>

username: ptrblck
cooked: <p>You might need to use the if-clause guard as given in the last example <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html">here</a>. These docs also mention common issues with deadlocks etc.</p>
Random crop in valid range

username: Sebastian_E
cooked: <p>Some torchvision transforms like <code>RandomRotation</code> or <code>RandomAffine</code> will add pad the image with a constant at some places as they change the size of the image.<br>
If I apply this function first and crop later, it is no problem to get an image that doesn’t have any filled regions in it but only a part of the original image (given suitable input and crop sizes). I could achieve this by simply taking the centre crop every time. But is there a way to do that with <code>RandomCrop</code> and control the random crop position, so that it doesn’t pick something in the padded areas? So for example like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e6d6dde7294290bb495bd739d5ee414de7f94544.jpeg" alt="Screenshot 2021-03-08 192501" data-base62-sha1="wW6aqCUe0BC98wmElelwLUELs4k" width="209" height="208"></p>

username: ptrblck
cooked: <p>You could write a custom <a href="https://github.com/pytorch/vision/blob/afc502bd669fe97001a50d0ce2373d13c011ed44/torchvision/transforms/transforms.py#L505-L611"><code>RandomCrop</code></a> method and change how the min. and max. values are created so that the random crop would be performed in your specified bounds.</p>
How filter tensor' rows by a float value

username: Adriano_Santos
cooked: <p>Hi, guys.</p>
<p>I have a tensor x = (a, b, c) , where a is batch, b are coord values and c is a float. I’d like to filter these values by threshold y. I’m doing this:</p>
<p>conf_mask = (prediction[:,:,4] &gt; threshold_conf).float().unsqueeze(2)<br>
prediction *= conf_mask<br>
prediction = prediction[prediction[:,:,4] &gt; 0]</p>
<p>But the tensor shapes are different (previous (a, b, c) after (b, c)). What am I wronging? In fact, I’d like to keep only the values from x that are over to threshold conf.</p>

username: Eta_C
cooked: <p>It looks like the post process of the detection task…</p>
<pre><code class="lang-python">conf_mask = (prediction[..., 4] &gt; threshod_conf)
predicton = prediction[..., conf_mask]
</code></pre>

username: Adriano_Santos
cooked: <p>Yes, <a class="mention" href="/u/eta_c">@Eta_C</a> . Exactly. But, using your suggestion, I’m still getting an error in the second line (predicton = prediction[…, conf_mask]).</p>
<p>IndexError: The shape of the mask [1, 10647] at index 0 does not match the shape of the indexed tensor [1, 10647, 6] at index 1</p>

username: Eta_C
cooked: <p>Sorry, it should be</p>
<pre><code class="lang-python">prediction = prediction[conf_mask, :]
</code></pre>

username: Adriano_Santos
cooked: <p>Hi, Eta! Tks for your help. But it still doesn’t work… the output is the same.</p>

username: LaverneHouston
cooked: <p>This worked perfectly, thanks<a href="https://www.walgreenslistens.mobi/" rel="noopener nofollow ugc">!</a></p>
Initialising a complex tensor

username: Omroth
cooked: <p>Hi guys.</p>
<p>I’m creating a rotation matrix as a pytorch c++ tensor:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be1cc4000ac50c867757a1274da8df624a38f2e1.png" data-download-href="https://discuss.pytorch.org/uploads/default/be1cc4000ac50c867757a1274da8df624a38f2e1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be1cc4000ac50c867757a1274da8df624a38f2e1.png" alt="image" data-base62-sha1="r7OlFiG3MzFEX8GswWsLFDEHLS9" width="690" height="67" data-dominant-color="626262"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">749×73 3.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Where theta needs to be another tensor, which has requires_grad=true.</p>
<p>What would be the best practice way of initialising this tensor given the theta tensor?  Do I just set each element manually, or is there some more clever way?</p>
<p>Cheers.</p>

username: Omroth
cooked: <p>I’m clearly not using the right method.  As a test I have:</p>
<pre><code class="lang-auto">    torch::Tensor theta = torch::tensor({0.18}, torch::dtype(torch::kFloat32).requires_grad(true));

    torch::Tensor cos_theta = torch::cos(theta);

    torch::Tensor t = torch::eye(4, torch::dtype(torch::kFloat32).requires_grad(false));

    t.index({0,0}) = cos_theta;
</code></pre>
<p>This produces an error:</p>
<pre><code>output with shape [] doesn't match the broadcast shape [1]
</code></pre>
<p>And doesn’t seem like the right way of doing it either.  Eventually I want to set each element of the tensor matrix individually with other tensors, such as:</p>
<pre><code>t.index({0,0}) = cos_theta + (ux * (ux * one_minus_cos_theta));

t.index({0,1}) = (ux * uy * one_minus_cos_theta) - (uz * sin_theta);

t.index({0,2}) = (ux * uz * one_minus_cos_theta) + (uy * sin_theta);

//

t.index({1,0}) = (uy * ux * one_minus_cos_theta) + (uz * sin_theta);

t.index({1,1}) = cos_theta + (uy * uy * one_minus_cos_theta);

t.index({1,2}) = (uy * uz * one_minus_cos_theta) - (ux * sin_theta);

//

t.index({2,0}) = (uz * ux * one_minus_cos_theta) - (uy * sin_theta);

t.index({2,1}) = (uz * uy * one_minus_cos_theta) + (ux * sin_theta);

t.index({2,2}) = cos_theta + (uz * uz * one_minus_cos_theta);
</code></pre>
<p>I think I’m probably missing something conceptually with how to construct a tensor matrix out of individual tensor calculations - can someone assist?</p>

username: Omroth
cooked: <p>Ah, it looks like the problem is with using .index.  if I use t[0] it at least doesn’t crash.</p>
<p>I’m still interested in feedback on whether this is how you would approach the problem though.</p>
How to compare tensors in libtorch

username: everybody
cooked: <p>How might one compare two tensors in Libtorch (C++) to check if they are equal?</p>
<p>I want to see if they are equal for an unit test, and understand in python one might:</p>
<pre><code class="lang-auto">torch.all(tens_a.eq(tens_b))
</code></pre>
<p>However torch::all returns a Tensor.</p>
<p>I simply want to achieve:</p>
<pre><code class="lang-auto">if( !torch::all(producedTensor.eq(expectedTensor)) ){
    // fails test, do something
}
else{
    // hooray, do something else
}
</code></pre>

username: Anders_G
cooked: <p>You can just do:</p>
<pre><code>bool success = producedTensor.equal(expectedTensor);</code></pre>

username: everybody
cooked: <p>This works, and is what I am looking for, however people should be aware that if the tensors are different dtypes then it simply throws an c10::Error exception with no message, warning, or justification.</p>
<p>for example the below will SIGABRT:</p>
<pre><code class="lang-auto">produced:
 11  21  31
 12  22  32
 13  23  33
 10  20  30
 13  23  33
 14  24  34
[ CPUDoubleType{6,3} ]
expected
 11  21  31
 12  22  32
 13  23  33
 10  20  30
 13  23  33
 14  24  34
[ CPULongType{6,3} ]
libc++abi.dylib: terminating with uncaught exception of type c10::Error
</code></pre>
<p>generated by simply running:</p>
<pre><code class="lang-auto">	std::cout &lt;&lt; "produced:\n" &lt;&lt; produced &lt;&lt; "\nexpected\n" &lt;&lt; expected &lt;&lt; std::endl;
	bool same = produced.equal(expected);
</code></pre>

username: tom
cooked: <p>This appears to be an OSX thing.</p>
<p>On Linux/gcc, we get very reasonable error and backtrace:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  Expected object of scalar type double but got scalar type long int for argument 'other'
Exception raised from cpu_equal at ../aten/src/ATen/native/ReduceOps.cpp:1230 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0x55 (0x7fc78e452055 in /usr/local/lib/python3.9/dist-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0x66de24 (0x7fc78eae4e24 in /usr/local/lib/python3.9/dist-packages/torch/lib/libtorch_cpu.so)
...
</code></pre>
<p>Maybe on OSX you need to do your own exception catching and what printing (or have a compilation flag that adds that for you or something, I would not know).</p>
<p>Best regards</p>
<p>Thomas</p>
Recurrent Connections break network

username: milebril
cooked: <p>Hi, I am currently working on extending an existing model with some type of recurrent connections to ensure for temporal stability between frames of a sequence.</p>
<p>When training my existing model I get normal results, as expected.<br>
The network I’m working with exists of different modules, one of which is a U-net like module. I have added my recurrent connections to this part of the network at the encoder side of the U-net.<br>
I have done this by simply supplying the output from the previous frame as extra input to each level of the encoder part.</p>
<p>My forward pass of the ConvChain at each level of encoder part of my U-net:</p>
<pre><code class="lang-auto">    def forward(self, x):
        if type(self.hidden_tensor) == type(None):
            self.hidden_tensor = self.init_hidden(x)

        for idx, m in enumerate(self.children()):
            if idx == 0:
                x = m(th.cat([x, self.hidden_tensor], 1))
            else:
                x = m(x)
                
        self.hidden_tensor = x
        self.hidden_tensor.detach_()
        
        return x
</code></pre>
<p>The problem is that my loss goes crazy as seen in the plot below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fca9b9fef03748105ba088d30ca2a1d14da6ffa5.png" data-download-href="https://discuss.pytorch.org/uploads/default/fca9b9fef03748105ba088d30ca2a1d14da6ffa5" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fca9b9fef03748105ba088d30ca2a1d14da6ffa5_2_689x230.png" alt="loss" data-base62-sha1="A39XwffcglkFzQWaY1TAFdIeOkl" width="689" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fca9b9fef03748105ba088d30ca2a1d14da6ffa5_2_689x230.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fca9b9fef03748105ba088d30ca2a1d14da6ffa5_2_1033x345.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/c/fca9b9fef03748105ba088d30ca2a1d14da6ffa5.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">1344×449 18.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My question now is if this is not the way to go with just adding the recurrent connection as extra input? Or how should I approach this?</p>
<p>Kind regards,<br>
Emil</p>
Normalization is important?

username: Kapil_Rana
cooked: <p>I have seen multiple implementations where they have applied normalization in transforms (mean and std).</p>
<p>My question is in respect of images.</p>
<p>Since ToTensor() already converting each pixel in [0-255], why to normalize.</p>

username: tom
cooked: <p>This is the type question that I loved the PyTorch forum for in 2018. (But a small thing first: ToTensor converts to 0…1)</p>
<p>On the face of it, we might except that it isn’t needed: Typically the first thing that happens to images is a convolution and you could just adjust weights and bias to make it take 0…255 pixel values instead of “zero mean, unit variance” (over the entire dataset). So the neural network could easily represent the transformation.</p>
<p>One part might be tradition, if you look at <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf" rel="noopener nofollow ugc">the AlexNet paper</a>, you see that they center the images. In a way this standardization is an easy approximation to a <a href="https://en.wikipedia.org/wiki/Whitening_transformation" rel="noopener nofollow ugc">Whitening transformation</a> which have a history in statistics. Quite likely you will see a (%-pt sized maybe?) drop in accuracy if you remove it.</p>
<p>That said, one of the crucial issues in deep learning is the mathematical stability of the network and the ability to train the network (with mathematical stability of the gradient being one of the key issues). From <a href="https://arxiv.org/abs/1512.03385" rel="noopener nofollow ugc">famous initialization strategies</a> (and our <a href="https://discuss.pytorch.org/t/calculate-gain-tanh/20854/">chit chat here</a>) to <a href="https://arxiv.org/abs/1502.03167" rel="noopener nofollow ugc">BatchNorm</a> and friends to <a href="https://arxiv.org/abs/2102.06171" rel="noopener nofollow ugc">recent</a> <a href="https://arxiv.org/abs/2101.08692" rel="noopener nofollow ugc">work</a> on normalization free training to adaptive optimizers, standardization (to mean zero / unit std as a reference) of activations, gradients, and weight updates has been one of the great underlying themes in training neural networks. Starting with mean zero / unit std inputs would then be a natural starting point.</p>
<p>That said, I do think that doing the normalization as part of the data preparation is not ideal conceptually, and I would really view it as part of the model instead of as part of the data preprocessing.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Kapil_Rana
cooked: <p>Oh I mistyped that one, yes, [0-1]</p>
PyTorch with python2

username: Pranavk
cooked: <p>Hi Can I still use PyTorch with python 2? I have used it before around 6 months ago but now it seems that PyTorch does not support python2 . Is there any work around?</p>

username: crcrpar
cooked: <p>Yes, about a year ago, PyTorch dropped Python2 support as in <a href="https://pytorch.org/blog/pytorch-1-dot-5-released-with-new-and-updated-apis/#python-2-no-longer-supported" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 1.5 released, new and updated APIs including C++ frontend API parity with Python | PyTorch</a>.</p>
<p>So, PyTorc&lt;1.5 is an alternative.</p>

username: ptrblck
cooked: <p>While <a class="mention" href="/u/crcrpar">@crcrpar</a> is correct that older PyTorch releases might work, I would not recommend to use Python2 anymore, as it’s not maintained anymore, so let it rest in peace <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Pranavk
cooked: <p>Hi <a class="mention" href="/u/crcrpar">@crcrpar</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for the suggestion I tried older version and it is working. I am using ROS(Robot Operating System) so I have to use python2 I tried to use python3 with ros but there it is a bit confusing to use so I have to use python2</p>
How to view prediction output as Segmented Image in MultiClass Semantic Segmentation?

username: Ali_Aqdas
cooked: <p>I trained a UNet using nn.CrossEntropyLoss(). Since the total labels were 32 and batch size was 2, I got output prediction of shape (2,32,388,388). How do I get to see the segmented image from this tensor? Images dont have 32 channels!</p>

username: Jeremy_Tavrisov
cooked: <p>Argmax on the second dimension will get you the most probable class per pixel.</p>

username: Ali_Aqdas
cooked: <p>Thanks, it solves the problem</p>

username: angelica-sjsu
cooked: <p>A question related to this is what it the best way to do remapping of classes to the original rgb color?</p>
Regarding LSTMCell example in documentation

username: Victor_George
cooked: <p>I think there is a type for the LSTMCell example given here at api doc page: <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTMCell.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LSTMCell.html</a><br>
Current snippet is:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; rnn = nn.LSTMCell(10, 20)
&gt;&gt;&gt; input = torch.randn(3, 10)
&gt;&gt;&gt; hx = torch.randn(3, 20)
&gt;&gt;&gt; cx = torch.randn(3, 20)
&gt;&gt;&gt; output = []
&gt;&gt;&gt; for i in range(6):
        hx, cx = rnn(input[i], (hx, cx))
        output.append(hx)
</code></pre>
<p>I think the input rather should be:</p>
<pre><code class="lang-auto">input = torch.randn(6, 3, 10)
</code></pre>
<p>Could someone please confirm these?</p>

username: ptrblck
cooked: <p>Yes, that’s right and it was already fixes in the <a href="https://pytorch.org/docs/master/generated/torch.nn.LSTMCell.html?highlight=lstmcell#torch.nn.LSTMCell">master docs</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How augmentation is applied in torch vision

username: talhaanwarch
cooked: <p>These are the transforms that I am applying. But I am confused whether each transformation is applied on original image or they are applied in sequence. It mean first grayscale is applied then on the gray scaled image resizing is applied and random resizef crop is applied on already resized image. To me it look like they work in sequential way. How can I apply each transformation on the input image individually.</p>
<pre><code class="lang-auto">data_transforms = transforms.Compose([
        transforms.Grayscale(),
        transforms.Resize(224),
        transforms.RandomResizedCrop(224),
        transforms.ToTensor(),
        ])
</code></pre>

username: ptrblck
cooked: <p>Yes, the <code>Compose</code> transformation will apply these transformations sequentially.<br>
If you want to apply these transformation on the same input image individually and create multiple outputs, you could apply them directly:</p>
<pre><code class="lang-python">out1 = transforms.Grayscale()(input)
out2 = transforms.Resize(224)(input)
out3 = ...
</code></pre>
<p>Of course you could also define these transformations in a list and index it etc.</p>
Row major indexing

username: Omroth
cooked: <p>I just want to be sure: tensor.index({0,3}) is the top element of the fourth column of the tensor?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>Yes, the indexing would use the row, then column to grab the value.<br>
As a verification you could print the complete tensor as well as the indexed value and compare them.</p>
Order of dataparallel and optimizer

username: mcorange
cooked: <p>Hi everyone,<br>
I came across a problem and can not figure out which one is right and the reason:</p>
<p>model = nn.DataParallel(model)<br>
optimizer = optim.SGD(model.parameters())<br>
#<span class="hashtag">#training</span> code</p>
<p>optimizer = optim.SGD(model.parameters())<br>
model = nn.DataParallel(model)<br>
#<span class="hashtag">#training</span> code</p>
<p>I guess the first one is logically right, since pytorch is going to optimize the parallelized model’s parameters. However, in my experiment, both worked.<br>
I wonder which one is the right way to set up the training and why.</p>
<p>Thanks a lot!!</p>

username: ptrblck
cooked: <p>Both should work, since the <code>model</code> is pushed to the default device and the optimization will also take place on this device.<br>
<a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">This blogpost</a> by <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> explains it in a nice way.</p>

username: mcorange
cooked: <p>Thanks for your sharing! My understanding is that the parameter updating is done after the gradient is reduced to the GPU-1(the GPU which scatters replica and inputs), and the parameter updating only takes place on GPU-1. Please correct me if wrong.</p>

username: ptrblck
cooked: <p>Yes, that’s also my understanding, which is why I think that both approaches should work.</p>
nn.Linear dimensions

username: 90210
cooked: <p>Hi, I am quite new to PyTorch and am a bit confused about how to do a certain task with nn.Linear.</p>
<p>I have an input tensor of size <code>2x28</code>, and I want the output of the Linear layer to be <code>28 x s</code> where <code>s</code> can be any scalar value.</p>
<p>How can this be achieved? Right now, from my understanding, the input of a <code>2x28</code> tensor with <code>s=3</code> results in the output of a <code>2x3</code> tensor. How would I make it so that the output is <code>28x3</code>?</p>
<p>Here’s my basic code so far:</p>
<pre><code class="lang-auto">import torch.nn as nn

x = torch.rand(2, 28)
s = 3

print(x.shape)

linear= nn.Linear(28, s)
out = linear(x)

print('out is:', out.shape)

</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">torch.Size([2, 28])
out is: torch.Size([2, 3])
</code></pre>
<p>Thank you!</p>

username: Dwight_Foster
cooked: <p>So basically what happens is the first number that is passed to the linear layers is the batch size. The way the linear layer works is it computes the outputs over all batches so the batch number doesn’t change only the input number. If you want something of shape [28, s] you would need to put the 28 first as the batch size.</p>
Avg and max pooling addition?/ Global avg/max pooling implementation?

username: Smth_w
cooked: <p>So i found this piece of code from the implementation of the paper “PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition” (It’s supposed to be a 14-layer CNN)</p>
<pre><code class="lang-auto">x = self.conv_block6(x, pool_size=(1, 1), pool_type='avg') #output of the last conv layer, 
x = F.dropout(x, p=0.2, training=self.training) # Dropout, global pooling is supposed to come after this (according to the model architecture at least)

x = torch.mean(x, dim=3) #first step of Global avg pooling?
(x1, _) = torch.max(x, dim=2) # Global max pooling?
x2 = torch.mean(x, dim=2) #next step of Global avg pooling?
x = x1 + x2 #Combining both results

#adding fc layers and sigmoid output
x = F.dropout(x, p=0.5, training=self.training)
x = F.relu_(self.fc1(x))
embedding = F.dropout(x, p=0.5, training=self.training)
clipwise_output = torch.sigmoid(self.fc_out(x))
</code></pre>
<p>Here they claim that “to combine their advantages, we sum the averaged and maximized vectors”, what I don’t understand is why? why is it better to add these two vectors instead of simply using Global avg pooling?<br>
I’ve also seen people combining normal avg_pooling2d with max_pooling2d like this</p>
<pre><code class="lang-auto">x1 = F.avg_pool2d(x, kernel_size=pool_size)
x2 = F.max_pool2d(x, kernel_size=pool_size)
x = x1 + x2
</code></pre>
<p>same thing I’m not sure why is that the addition operation in particular helps with the results, why not using other operations? or just averaging the result instead of adding them?<br>
Sorry if it is a little bit obvious, but I can’t quite grasp the idea behind this yet</p>
<p>PS.- I just realized that x1 is obtained from calculating the max after the mean of x, I thought that GlobalMaxPooling is supposed to be something like torch.max(x, dim=[2,3])<br>
Is there something else I’m missing here?</p>
<p>Thanks for any help!!</p>
Performing matrix-style operations in place

username: Omroth
cooked: <p>Hi all.</p>
<p>I’m often doing this:</p>
<p><code>torch::Tensor c = torch::matmul(a, b);</code></p>
<p>And I’m aware of a lot of unnecessary allocating, when I would really like to have the result of the matmul stored in b (or a).</p>
<p>Is there an easy way to achieve this?</p>

username: tom
cooked: <p>No and this is inherent to the memory access pattern (every value is multiple times to compute other values).</p>
<p>If you wanted to reduce the memory consumption, you could split it to smaller patches along the rows of <code>a</code> (or the columns of <code>b</code>) and overwrite blocks. In the extreme case you would allocate a buffer for a single row, perform the row-wise operation with <code>matmul_out</code> to that buffer and then copy the row to <code>a</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Omroth
cooked: <p>Thanks Tom.</p>
<p>I don’t mind the memory usage, I’m more concerned about the time of the allocs - does PyTorch use a decent memory manager to re-use memory?</p>

username: tom
cooked: <p>On the GPU it has its own cache, I think on mobile, too. On the CPU it uses posix_memalign which relies on libc to do caching (I think they do in contemporary systems), so I would not worry about it.<br>
You can also reserve the memory once and use matmul_out if you love manual, but very likely it will not be the largest lever you have in optimization of any given model.<br>
There is <a href="https://github.com/pytorch/pytorch/tree/master/torch/csrc/jit/runtime/static" rel="noopener nofollow ugc">ongoing work on a static runtime</a>, which also involves static memory planning.</p>
<p>Best regards</p>
<p>Thomas</p>
Does torch TCP connections between processes reliable?

username: mmsz
cooked: <p>Hi, i am wondering in the case of connection break and message lost, for example the connection from master to worker process, 1. will torch handle reconnect? 2. if the connection got restored somehow, will torch retry to send the lost message and let the job keep running or just rely on TCP reliability and simply failed the job? Thanks!</p>

username: agolynski
cooked: <p>Hi!</p>
<ol>
<li>Currently torch doesn’t handle node failures well.</li>
<li>We rely on TCP reliability for node connectivity, but we also have some robustness mechanisms like rpc retries</li>
</ol>
<p>You can also look into <a href="https://pytorch.org/elastic/0.2.2/index.html" class="inline-onebox" rel="noopener nofollow ugc">TorchElastic — PyTorch/Elastic master documentation</a> to handle node failures.</p>
Returning a dictionary from forward call breaks DDP

username: Rakshit_Kothari
cooked: <p>Hello,<br>
My forward call returns a dictionary, <code>out_dict</code>, as follows:</p>
<pre><code class="lang-auto">out_dict = {'main_predict_op': main_differentiable_op,
            'secondary_predict_op': second_differentiable_op}
</code></pre>
<p>It seems DDP does not like this and throws me the following warning:</p>
<pre><code class="lang-auto">RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
</code></pre>
<p>I have set <code>find_unused_parameters=True</code>. From what I understand, output tensor being encapsulated in a dictionary is not favorable for DDP.</p>
<p>How would I go about this?</p>

username: Rakshit_Kothari
cooked: <p>Responding to anyone who faces this. The fact that a tensor is wrapped in a dictionary is not relevant. DDP requires that all output variables be used in the graph. Calling ‘unused_parameters’ option did pretty much nothing.</p>

username: rvarm1
cooked: <p>Hi,</p>
<p>the <code>find_unused_params</code> option in DDP will work if certain params are not used in forward pass, but then all params used in fwd pass get gradients in the backward pass.</p>
<p>There is another case where all params are used in fwd pass but some don’t get grad in backwards, for example if you have:</p>
<pre><code class="lang-auto">a, b = forward()our 
loss = a.sum()
loss.backward()
</code></pre>
<p>It seems like this is what may be happening in your training, but would need your training loop to verify this.</p>
Opencv -> torch tensor conversion does not work

username: Dennis_Hugle
cooked: <p>Dear Community,<br>
I’m experimenting with the C++ API from pytorch. When loading an Image with opencv, I get an error when calling the modules forward function. Using the same logic but initializing as torch::tensor works perfectly. Can someone tell me what I’m doing wrong?</p>
<p>I think the code explains it best. I posted the code everyone can try below:</p>
<p>This is the torch-only version that does work:</p>
<pre><code class="lang-auto">int test_torch_model_torch_dummy_data() {
	torch::jit::script::Module module = torch::jit::load("/path/to/traced/model.pt");
	std::cout &lt;&lt; "model loaded." &lt;&lt; std::endl;

	at::TensorOptions options(at::ScalarType::Float);
	torch::Tensor tensor_image = torch::zeros({ 1, 3, 224, 224 }, options);
	std::cout &lt;&lt; tensor_image.sizes() &lt;&lt; std::endl;

	std::vector&lt;torch::jit::IValue&gt; inputs;
	inputs.emplace_back(tensor_image);

	std::cout &lt;&lt; "input shape torch" &lt;&lt; tensor_image.sizes() &lt;&lt; std::endl;
	auto ret = module.forward(inputs).toTensor();
	std::cout &lt;&lt; "works using torch dummy data.";
	return 0;
}
</code></pre>
<p>This is the opencv-function that does not work:</p>
<pre><code class="lang-auto">int test_torch_model_cv_dummy_data() {
	torch::jit::script::Module module = torch::jit::load("path/to/traced_model.pt");
	std::cout &lt;&lt; "model loaded." &lt;&lt; std::endl;

	auto bgr = cv::Mat(cv::Size(224, 224), CV_8UC3);
	std::vector&lt;int64_t&gt; sizes = { 1, 224, 224, 3 };
	at::TensorOptions options(at::ScalarType::Float);
	at::Tensor tensor_image = torch::from_blob(bgr.data, at::IntList(sizes), options);
	auto tensor_image_permuted = tensor_image.permute({ 0, 3, 1, 2 });

	std::cout &lt;&lt; "input shape cv data" &lt;&lt; tensor_image_permuted.sizes() &lt;&lt; std::endl;

	std::vector&lt;torch::jit::IValue&gt; inputs;
	inputs.emplace_back(tensor_image);

	module.forward(inputs).toTensor();

	std::cout &lt;&lt; "works using cv dummy data " &lt;&lt; std::endl;
}
</code></pre>
<pre><code class="lang-auto">void main() {
	test_torch_model_torch_dummy_data();
	test_torch_model_cv_dummy_data();
}
</code></pre>
Help applying faster R-CNN to mobile

username: Miafr
cooked: <p>Hi all,<br>
I’m starting out with pytorch, and wish to use detection on Android app,<br>
I understand the new version of torchvision 0.9.0 includes support for Faster R-CNN model on mobile, but I couldn’t figure out how exactly to use it, hopefully like this:<br>
<a href="https://github.com/pytorch/android-demo-app/tree/master/ObjectDetection" rel="noopener nofollow ugc">https://github.com/pytorch/android-demo-app/tree/master/ObjectDetection</a></p>
<p>do i need to trace the model? I’m unsure how to use the fasterrcnn_mobilenet_v3_large_fpn specifically.<br>
Thank you!</p>
How to save weights from training and then test

username: jpj
cooked: <pre><code class="lang-auto">  for i, (x,y) in enumerate(zip(feature_trainloader,label_trainloader), 0):
                optimizer.zero_grad()   
                output = self.forward(x)
                target = y
                loss = self.CrossEntropyLoss(output, target)
                print(loss)
                loss.backward()
                optimizer.step()            
  print('Finished Training')
</code></pre>
<p>I want to now calculate the accuracy by putting feature_testloader through the input layer and calculating accuracy by comparing to label_testloader. How can I do that because when the <em>for loop</em> ends and I call <em>forward</em> again the weights will be reset?</p>

username: Dwight_Foster
cooked: <p>The weights should not be reset. You should just be able to do another for loop for your test loaders. The model weights will be saved in memory.</p>
Show FusionGroup for JIT script

username: Monne
cooked: <p>Hi,<br>
In <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="noopener nofollow ugc">Torchscript sample RNN</a> there was FusionGroup shown when printing graph for LSTMCell, but when i’m try to run torchscript LSTMCell from site and print graph, it’s not show a FusionGroup. Is that normal or there something missing for show FusionGroup like torch._C._jit_set_profiling_executor(False)?</p>

username: googlebot
cooked: <p>It is TensorExprGroup with the new executor, and these should appear after second invocation (for cuda, cpu fusion is off by default).</p>
<p>torch._C._jit_set_profiling_executor(False) may also be a good idea with varying shapes.</p>
Exponential Smoothing for Time Series Forecasting in Pytorch

username: boris
cooked: <p>Does anyone know of a Pytroch implementation of Exponential Smoothing (ES)? Since ES is a local model, each time series in a dataset would require its own set of ES params (alpha, beta, etc.). Just wondering how this would be implemented. Any help would be appreciated.</p>

username: googlebot
cooked: <p>It is a pain to do, as implementations with python loop + gradients are slow. You may need something similar to torchaudio.functional.lfilter (that resorts to a C++ loop and doesn’t support backprop I think).</p>
<aside class="quote no-group" data-username="boris" data-post="1" data-topic="114151">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/74df32/48.png" class="avatar"> boris:</div>
<blockquote>
<p>Since ES is a local model, each time series in a dataset would require its own set of ES params (alpha, beta, etc.).</p>
</blockquote>
</aside>
<p>Well, options are obvious:</p>
<ol>
<li>use multiple fixed (never trained) parameter sets, concatenate outputs as an input for a downstream task (e.g. pass them to a rnn layer)</li>
<li>train “amortized” params conditioned on some time series metadata</li>
<li>train “unamortized” params (nn.Embedding lookups by time series id) - with no straightforward way to do out-of-sample predictions</li>
</ol>