akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why does index_add_ and scatter_add_ induce non-deterministic behavior on the CUDA backend?

username: Siddhesh_Mhatre
cooked: <p>Basic question here but was curious why the order of addition at a particular location in a target tensor may vary the value at the location across different calls on index_add_ or scatter_add_</p>

username: ptrblck
cooked: <p><code>scatter_add_</code> and <code>index_add_</code> both use atomic operations (<code>atomicAdd</code> as seen e.g. <a href="https://github.com/pytorch/pytorch/blob/3be2f7c8e61e50bd5bea5649ff9fc6ef39928571/aten/src/THC/THCTensorScatterGather.cu#L163" rel="nofollow noopener">here</a>) as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html#pytorch" rel="nofollow noopener">Reproducibility docs</a>, which introduces these small variations.</p>
How to reduce CrossEntropyLoss mean

username: Msabih
cooked: <p>I want to weight each pixel to compute my loss function. Now first I calculate cross entropy loss with reduce = False for the images and then multiply by weights and then calculate the mean. If I choose all the weights as 1, I should get a consistent result. But its not the case.</p>
<blockquote>
<p>loss_function = torch.nn.CrossEntropyLoss(weight=weight, reduce=False)<br>
loss_function_reduce = torch.nn.CrossEntropyLoss(weight=weight)</p>
<p>loss_tensor = loss_function(input, target)<br>
weights = torch.ones(loss_tensor.size())<br>
loss_tensor = (loss_tensor * weights).mean()</p>
<p>loss_tensor_reduced = loss_function_reduced(input, target)</p>
</blockquote>
<p>Could someone help here ?</p>

username: JuanFMontesinos
cooked: <p>Hi, I think it’s a matter of digging into source code. If I’m not wrong it performs sample-wise mean. Later it averages. In addition, weights are multiplied batch-wise (I think). How are you setting those parameters?</p>

username: Msabih
cooked: <p>I tried with plain .mean() on all data as well as</p>
<blockquote>
<p>loss = (losses_ts.sum(dim=0)/batch_size).mean()</p>
</blockquote>
<p>where losses_ts is the output of CrossEntropyLoss with reduce=False</p>

username: Msabih
cooked: <p>I figured it out. It works like this<br>
Assuming a output with B x H x W</p>
<blockquote>
<p>loss = (loss_unreduced.sum(dim=(1,2))/weights[target].sum()).sum()</p>
</blockquote>
Weights for Cross Entropy Loss that vary every batch

username: Msabih
cooked: <p>Hi,</p>
<p>I have a dataset on which I do segmentation and classify each pixel in an image as 0 or 1. The classes are imbalanced and there is variance in the imbalance depending upon the image. So for some images the imbalance factor is 30(i.e class 1 pixels are 30 times less than class 0 pixels) and for some images, the imbalance factor is 70.</p>
<p>One solution is to set the weight parameter for Cross Entropy Loss based upon the imbalance calculated over entire dataset, this comes out to be 50.</p>
<p>To other solution is to sort the images with respect it imbalance factor and then calculating the weights of cross entropy loss separately for each mini batch. So basically, we are giving different weights to cross entropy loss for each mini batch.</p>
<p>My question is about the later approach ? Is it a valid approach ?</p>
Backpropagate loss of single batch sample rather than mean loss

username: drb12
cooked: <p>For every image in my dataset, I subdivide it into a patch and run the patches through the model as a “batch”. Then I calculate the loss for each patch using <code>nn.CrossEntropyLoss(reduction='none')</code>, which returns a loss for each of my patches in the batch.  I next select only one of the losses that were returned and backpropagate it. I do this by indexing the tensor and then calling <code>loss.backward()</code> as the example below shows. When I do this the <code>grad_fn</code> parameter changes from <code>&lt;BinaryCrossEntropyBackward&gt;</code> to <code>&lt;SelectBackward&gt;</code>. Does this impact my ability to backpropagate just the selected loss?</p>
<pre><code class="lang-auto">loss_all = criterion(m(patches), target)
</code></pre>
<pre><code class="lang-auto">loss_all

tensor([1.0222, 0.8956, 0.4532], grad_fn=&lt;BinaryCrossEntropyBackward&gt;)
</code></pre>
<pre><code class="lang-auto">loss = loss_all[0]
loss
tensor(1.0222, grad_fn=&lt;SelectBackward&gt;)
</code></pre>
<pre><code class="lang-auto">loss.backward()
</code></pre>
Why pytorch model size got increased while loading by c++? I have used torchscript, to load the model into c++ from pytorch

username: dinesh_mareedu
cooked: <p>Can you please help me to know the reason!!</p>

username: mhubii
cooked: <p>what do you mean by, the size increased?</p>

username: Aeroxander
cooked: <p>When I loaded by Python code together with the .pth models (or at least they should be included automatically) into a .pt model for C++, the model actually got substantially smaller than the .pth model!</p>

username: dinesh_mareedu
cooked: <p>Thanks for the concern, Actual model size is 2.3MB. After the torch script procedure for c++, it became 4MB.Actually the problem is resolved…</p>
<p>Can you plz help me to cross compile the pytorch code for pine64…</p>
Completly freezed submodule

username: 11158
cooked: <p>How can I store submodule but in completly freezed way?</p>
<p>Consider the example:</p>
<pre><code class="lang-auto">class PerceptualLoss(nn.Module):
    def __init__(self, vgg):
        super().__init__()
        self.vgg = vgg
</code></pre>
<p>In this example parameters of vgg will be presented in parameters in PerceptualLoss wich is incorrect e.g. while using with Optimizer. But at the same time methods like .cuda(), .cpu(), .to() and others should track parameters of vgg. Is that achivable?</p>

username: colesbury
cooked: <p>set the <code>requires_grad</code> attribute of <code>vgg</code>'s parameters to <code>False</code>. The parameters will still be presented in the parameters of <code>PerceptualLoss</code>, but they won’t be optimized:</p>
<pre><code class="lang-auto">for param in self.vgg.parameters():
    param.requires_grad = False
</code></pre>
Sparse.mm autograd support

username: mshuaibi
cooked: <p>My NN model contains a sparse.mm() operation and when I try to train I get the following error:<br>
<code>RuntimeError: Expected object of backend CPU but got backend SparseCPU for argument #2 'mat2'</code></p>
<p>I’ve seen various posts related to this but I’m unclear as to what the work around/solution is other than converting it to a dense matrix and performing standard dense operations. The idea is to utilize <code>torch.sparse.mm()</code> for its performance.</p>
<p>Thanks!</p>
How to use tensorboar in pytorch

username: isalirezag
cooked: <p>so im using the pytorch-nighty and wanna see how to use tensorboard in pytorch<br>
I am following steps here:<br>
<a href="https://pytorch.org/docs/stable/tensorboard.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/tensorboard.html</a><br>
everything works okay</p>
<p>when i reach to this line:<br>
<code>tensorboard --logdir=runs</code><br>
I have no clue what should i do? how should i visualize?<br>
if i just have that in my code it will give the error that:<br>
<code>SyntaxError: can't assign to operator</code><br>
can anyone give me a suggestion? it should not be difficult I just cannot figure how to do the visualization</p>

username: ptrblck
cooked: <p>In your Python script, you would have to create a <code>SummaryWriter</code> and log some values/images/etc. using it as described in the code example in the <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="nofollow noopener">docs</a>.</p>
<p>In another terminal you then start TensorBoard using <code>tensorboard --logdir=runs</code> with additional arguments if needed.<br>
This will create the tensorboard process, so that you can inspect the saved logs in your browser (default: <code>localhost:6006</code>).</p>

username: Nikronic
cooked: <p>You can check <a href="https://github.com/lanpa/tensorboardX" rel="nofollow noopener">TensorboadX Documentation</a>. It works perfectly and there are some examples to guide you through this.</p>

username: Filos92
cooked: <p>I’m currently experimenting with the new implementation of Tensorboard in Torch.Utilis and so far it works fine.<br>
The only problem I have is that I have to restart the process every time to update the scalar graphs (maybe also the other Graph-Types, but I did not test them).<br>
I see, that Tensorboard detect new log files, but it do not show the data until i kill and restart the process.<br>
Is there a possability to fix this, right now?</p>
How to use WeightedRandomSampler with a concatenated datset

username: streetmafia
cooked: <p>Hi . i have to two datasets on with cancer rich data and one without cancer in its labels . I am trying to run a segementation algorithm by sampling 75% time from cancer rich dataset . I thought i could use weightedrandomsampler to achive this …but i am stuck ,as most of the examples for the sampler are class label based .Can someone help me with an example code ?</p>
How can I use WeightedRandomSampler with a Concatenated Dataset?

username: gewa24
cooked: <p>Hi all,<br>
Edit: Resolved!<br>
Because the data generation schemes differ for my samples depending on their class, I was going to create two different Dataset classes (one for each class) and then concatenate them to form my dataset (using torch.utils.data.ConcatDataset). However, my two classes have different numbers of samples and I’d like to sample from each relatively equally - I was figuring I could use WeightedRandomSampler to achieve this, but I can’t figure out how this works with a concatenated dataset. I have been looking at the source code/documentation but I’m having trouble figuring out generally how the sampling scheme works with a Dataloader and would love if someone could explain how this works in this (concanated dataset) case.<br>
Thanks for any help</p>

username: streetmafia
cooked: <p>Hi…how to manage to use the weighted sampling when you used concat datasets .Can u please post ur solution ?</p>
LSTM multiclass output shape

username: torch
cooked: <p>I’m building a multiclass classification model using a GRU. I’m struggling to get my head around how to shape the output such that the loss can be calculated.</p>
<p>Here is my code:</p>
<pre><code>class Network(nn.Module):
def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim, weights=None):
    super(Net, self).__init__()
    self.embedding = nn.Embedding(input_dim, embedding_dim)
    if weights is not None:
        self.embedding.weight.data.copy_(weights)

    self.embedding.weight.requires_grad = False
    self.lstm = nn.LSTM(embedding_dim, hidden_dim)
    self.linear = nn.Linear(hidden_dim, output_dim)
    self.sigmoid = nn.Sigmoid()

def forward(self, inp):
    embed_out = self.embedding(inp)
    out, (hidden1, hidden2) = self.lstm(embed_out)
    out = self.sigmoid(self.linear(hidden1.squeeze(0)))
    return out.squeeze(-1)
</code></pre>
<p>The output shape in my batch [342, 51] whereas the shape of the label is [32, 51] which means the loss can’t be calculated. Any pointers would be very helpful - thank you.</p>

username: DanielLuci
cooked: <p>Can you give more details? What parameters are you using for embedding_dim, hidden_dim and output_dim? Also what is the format of your label?</p>

username: gslaller
cooked: <p>The LSTM layer takes the tensor of shape (seq_len, batch, features), so to comply with this, you have to call to the lstm with “self.lstm(embed_out.transpose(0,1))”, unless you inp is in the shape of (seq_len, batch) or you have defined the lstm class with “batch_first=True”. I don’t know why you are working on the hidden1, but usually you take the “out = self.sigmoid(self.linear(out)).reshape(-1, class_num)”, so your out is the shape of (seq_len*batch, class_num).</p>

username: torch
cooked: <p>Sure</p>
<p>input_dim = 3908<br>
embedding_dim = 300<br>
hidden_dim = 100<br>
output_dim = 51</p>
<p>Label format is a matrix of ints which represent the multiple classes.</p>

username: torch
cooked: <p>Thanks this is helpful. I still can’t align the final shape however. The target batch size is 32 with 51 classes. I can’t figure out how to get the output of the model to then match with this matrix. I’ve made the your suggestions I’m getting a [9952, 51] output - are there some additional steps I’m missing in terms of getting the model output to match the target shape?</p>
<p>Thanks for your help!</p>

username: gslaller
cooked: <p>Well you target shape isn’t going to be (32,51), unless the length of the sequences is 1. Perhaps you just want “out = self.sigmoid(self.linear(out[-1,:,:])).reshape(-1, class_num)”, which will provide just the last element of each sequence, thus being of a shape (batch, class_num).</p>

username: torch
cooked: <p>Thank you - this solves the issue!</p>
Is there a way to utilize ATen full lib by calling caffe2.python.core.CreateOperators?

username: zieitokk
cooked: <p>Hi I just wondering if there is a way to utilize ATen full lib by creating like</p>
<pre><code class="lang-auto">op = core.CreateOperator(
    "ATen",
    ["X2", "X1"],
    ["Y"],
    operator="grid_sampler"
)
</code></pre>
<p>I wanted to use grid_sampler function in ATen lib and I have checked the <code>native_functions.yaml</code> in <code>aten/src/ATen/native</code> which has function <code>grid_sampler</code>.</p>
<p>However when I run operators above, it shows an error claimed that</p>
<blockquote>
<p>Attempting to run unknown ATen operator configuration: grid_sampler</p>
</blockquote>
<p>It’s really appreciated that if someone can answer my question!</p>
How to learn a multivariabel guassian distribution?

username: amir-abdi
cooked: <p>I am trying ot learn a MultivariateNormal distribution from a fixed set of samples.<br>
Basically, I am trying to maximize the log_prob over some fixed samples. The simple version of the code will be like this:</p>
<pre><code class="lang-auto">class ParametricMultivariateNormal(nn.Module):
    def __init__(self, s):
        super().__init__()
        self.dist = MultivariateNormal(loc=torch.zeros(s, requires_grad=True),
                                       covariance_matrix=torch.eye(s, requires_grad=True))
    def forward(self, value):
        return self.dist.log_prob(value)
</code></pre>
<p>However, when I backward the  <code>-model(value)</code>, the distribution is not updated. What am I doing wrong?</p>
Backward for negative log likelihood loss of MultivariateNormal (in distributions)

username: svd3
cooked: <p>I was trying to minimize the negative log likelihood of a multivariate normal wrt to the mean 1D and covariance matrix<br>
but it gives me an error</p>
<pre><code class="lang-auto">m = torch.distributions.normal.Normal(mu, C)  
# mu is Variable of size(n, ) 
# C is Variable  of size (n, n)
loss = -m.log_prob(x) # x is a variable of size (n, )
# above line gives an error
</code></pre>
<p>I tried to sample from this distribution but it gives sample with size (n, n) not a (n, ) vector</p>
<pre><code class="lang-auto"> m.sample()
</code></pre>
<p>I’m stuck here.<br>
I want to get gradients wrt to mu and Cov. Any idea what can I do</p>

username: fritzo
cooked: <p><code>Normal</code> is a batched univariate distribution. Your <code>mu</code> is being broadcast up to the same shape as <code>C</code>, producing an <code>n,n</code> batch of univariate normals. If you want a <code>MultivariateNormal</code> distribution, use</p>
<pre><code class="lang-auto">n = 5
mu = torch.zeros(n)
C = torch.eye(n, n)
m = torch.distributions.MultivariateNormal(mu, covariance_matrix=C)
x = m.sample()  # should have shape (n,)
loss = -m.log_prob(x)  # should be a scalar
</code></pre>

username: bhavikngala
cooked: <aside class="quote no-group quote-modified" data-username="svd3" data-post="1" data-topic="15339">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/eb8c5e/48.png" class="avatar"> svd3:</div>
<blockquote>
<p>mal is a batched univariate distribu</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/fritzo">@fritzo</a> is the MultivariateNormal also a batched class now? if i give it a mean tensor of shape batch_size x 2 and covar of shape batch_size x 2 x 2 will it consider the input as batched?</p>

username: fritzo
cooked: <p>Yes, the result will be batched.</p>
Can we build libtorch without MKL?

username: dreamfly
cooked: <p>I was able to build libtorch with MKL (installed by conda) by python setup.py install. However, there is no guarantee that the production environment is provided with MKL. As a result, I was wondering if we can build libtorch without MKL?<br>
I’ve tried by uninstalling MKL (through conda) and build libtorch. The building process looks normal. However, when linking libcaffe2_gpu.so to my executable, there are several ‘undefined reference’ error messages I’ve never seen and the Google also provide nothing useful.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5ef9195471390a0bc7fcdd2d415a149e9f5ba228.png" data-download-href="https://discuss.pytorch.org/uploads/default/5ef9195471390a0bc7fcdd2d415a149e9f5ba228" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5ef9195471390a0bc7fcdd2d415a149e9f5ba228_2_690x230.png" alt="image" data-base62-sha1="dyazBpQbCFJTPthuqxrPVzJ1mQg" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5ef9195471390a0bc7fcdd2d415a149e9f5ba228_2_690x230.png, https://discuss.pytorch.org/uploads/default/original/2X/5/5ef9195471390a0bc7fcdd2d415a149e9f5ba228.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/5ef9195471390a0bc7fcdd2d415a149e9f5ba228.png 2x" data-dominant-color="4F2F45"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">840×280 97.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Apologize for the Chinese characters in the screenshot. ‘未定义的引用’  means ‘undefined reference’</p>

username: mhubii
cooked: <p>I think if you build Pytorch from source and you don’t provide MKL, then Pytorch will rely on Eigen as one of its <a href="https://github.com/pytorch/pytorch/tree/master/third_party" rel="nofollow noopener">submodules</a>. That said, you do not even need a linear algebra library on your system for Pytorch to compile</p>
<p>In the <a href="https://github.com/pytorch/pytorch/blob/master/setup.py" rel="nofollow noopener">setup.py</a> you will find</p>
<pre><code class="lang-python">#   BLAS
#     BLAS to be used by Caffe2. Can be MKL, Eigen, ATLAS, or OpenBLAS. If set
#     then the build will fail if the requested BLAS is not found, otherwise
#     the BLAS will be chosen based on what is found on your system.
</code></pre>
<p>And thanks for translating the characters for us, I appreciate that a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
I have a Question in my code. (MLP with binary classification

username: GoldD
cooked: <h2>Hello.<br>
I have a question in my code.<br>
My code has two input parameter, two hidden layer and one output parameter.<br>
I designed to output 0 or 1.<br>
Is this a problem in my code?<br>
Because, accuracy is very low.<br>
Thank you, reading my code.</h2>
<p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
from torch.nn import functional as F<br>
import numpy as np<br>
import matplotlib.pyplot as plt<br>
from torch.autograd import Variable<br>
torch.manual_seed(1)</p>
<p>epoch = 10000<br>
learning_rate = 0.1<br>
input_dim = 2<br>
hidden1_dim = 5<br>
hidden2_dim = 3<br>
output_dim = 1</p>
<p>data = np.loadtxt(‘Data_Set.csv’, delimiter=’,’)<br>
x_data = data [:,:2]<br>
y_data = data [:,2]</p>
<p>X_data = []<br>
Y_data = []<br>
for i in range(0, len(y_data)):<br>
x = x_data[i]<br>
y = y_data[i]<br>
<span class="hashtag">#print</span>(x, “-&gt;”, y)<br>
X_data.append(x)<br>
Y_data.append(y)</p>
<p>X = torch.FloatTensor(np.array(X_data))<br>
Y = torch.FloatTensor(np.array(Y_data))<br>
fc1 = nn.Linear(input_dim, hidden1_dim)<br>
fc2 = nn.Linear(hidden1_dim, hidden2_dim)<br>
fc3 = nn.Linear(hidden2_dim, output_dim)<br>
relu = nn.ReLU()<br>
sigmoid = nn.Sigmoid()</p>
<p>model = torch.nn.Sequential(fc1, sigmoid, fc2, sigmoid, fc3, sigmoid)<br>
criterion = torch.nn.BCELoss(size_average=True)<br>
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)</p>
<p>for step in range(epoch):<br>
optimizer.zero_grad()<br>
hypothesis = model(X)<br>
cost = criterion(hypothesis, Y)<br>
cost.backward()<br>
optimizer.step()</p>
<pre><code>if step % 100 == 0:
    print(step, cost.data.numpy())
</code></pre>
<p>predicted = (model(X).data &gt;= 0.5).float()<br>
accuracy = (predicted == Y.data).float().mean()<br>
print("\nHypothesis: ", hypothesis.data.numpy(), "\nCorrect: ", predicted.numpy(), "\nAccuracy: ", accuracy*100, “%”)</p>

username: MariosOreo
cooked: <p>Hello,</p>
<p>It is better to use ReLU activation function between hidden layers, and use sigmoid at last to generate probability map.</p>
<pre><code class="lang-auto">model = torch.nn.Sequential(fc1, relu, fc2, relu, fc3, sigmoid)
</code></pre>
<p>PS: you could format your code snippet with three `</p>

username: GoldD
cooked: <p>Thanks for your advice.<br>
And I tried “model = torch.nn.Sequential(fc1, relu, fc2, relu, fc3, sigmoid)”<br>
However, accuracy is still low… <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: Seyed_Majid_khorasha
cooked: <p>Hi, I think your learning rate is too big and it’s better to decrease it to 1e-4.</p>

username: GoldD
cooked: <p>Hello. Thanks for your advice.<br>
but, I decreased learning rate low, my output_prediction is only 0… <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>
'NoneType' object has no attribute 'data' for some variables

username: Yasesf
cooked: <p>I’m trying to get gradients of the loss function with respect to some variables in my code. I’m having trouble understanding why I can access the gradients for some variables and why not for some others.Here is a part of my code:</p>
<pre><code>    pert=I+delta
    pert=torch.clamp(pert,0,1)
    I=Variable(I.to(device), requires_grad = True) 
    delta=Variable(delta.to(device), requires_grad = True) 
    pert=Variable(pert.to(device), requires_grad = True) 
    outputs = net(pert)
    targets=targets.long()  
    loss = criterion(outputs, targets)
    loss.backward()
    grad=pert.grad.data
    graddel=delta.grad.data
    gradI=I.grad.data
</code></pre>
<p>I can get the gradients for grad, but for graddel and gradI I get the error:<br>
‘NoneType’ object has no attribute ‘data’</p>
<p>Any ideas?</p>

username: MariosOreo
cooked: <p>It seems that tensor <code>I</code> and <code>delta</code> are not wrapped into the computational graph, thus their grad is None.<br>
I have a dummy code here to explain it</p>
<pre><code class="lang-auto">a = torch.randn(3,3,)
b = torch.randn(3,3,)
c = a + b
a.requires_grad_()
b.requires_grad_()
c.requires_grad_()
c.sum().backward()

a.grad -&gt; None
b.grad -&gt; None
</code></pre>
<p>But you set <code>requires_grad_()</code> at first, like</p>
<pre><code class="lang-auto">a = torch.randn(3,3, requires_grad=True)
</code></pre>
<p>the grad of <code>pert</code> will be None, and the grad of delta and I will be None. Because here <code>pert</code> is non-leaf tensor, its gradient will not be stored.</p>
[LANGUAGE MODELS] From hidden to logits, 3D and 2D tensor multiplication

username: ac1d
cooked: <p>In language modelling, when using RNNs the output (lets denote it by <strong>h</strong>) of your RNN is almost always a tensor with dimensions:</p>
<p><code>h.shape = [time, batch_size, hidden_size]</code></p>
<p>And from here, a common practice is to use a “decoding” linear layer:</p>
<p><code>decoder = nn.Linear(hidden_size, vocab_size)</code></p>
<p>to obtain logits with dimensions:</p>
<p><code>logits.shape = [time, batch_size, vocab_size]</code></p>
<p>Now, I have seen people doing this in two ways. One:</p>
<p><code>logits = decoder(h)</code></p>
<p>where this boils down to a matrix multiplication of the 3D tensor <strong>h</strong> with the 2D tensor inside the decoder.</p>
<p>Two:</p>
<p><code>logits = decoder(h.view(time * batch_size, hidden_size))</code></p>
<p>i.e. they first transform the 3D to a 2D tensor and then they pass it to the decoder, which boils down to a matrix multiplcation between two 2D tensors. Then based on what format do we want the <strong>logits</strong> to be in, we can reshape with:</p>
<p><code>logits = logits.view(time, batch_size, vocab_size)</code></p>
<p>So, finally my questions:</p>
<ol>
<li>Are the two approaches identical?</li>
<li>If not why?</li>
<li>If yes, is there a best practice on which to use and why?</li>
</ol>

username: gslaller
cooked: <p>The main reason they will collapse the BATCH and SEQ_LENGTH axis to a single axis is because of the CrossEntropyLoss layer, which takes the tensor of shape “ITEMS*NUM_CLASSES”. Other than this they are pretty much the same.</p>

username: ac1d
cooked: <p>That makes sense. Thanks a lot on your fast answer! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to use multiprocessing in PyTorch?

username: Shih-han_Wang
cooked: <p>I’m trying to use PyTorch with complex loss function. In order to accelerate the code, I hope that I can use the PyTorch multiprocessing package.</p>
<p>The first trial, I put 10x1 features into the NN and get 10x4 output.</p>
<p>After that, I want to pass 10x4 parameters into a function to do some calculation. (The calculation will be complex in the future.)</p>
<p>After calculating, the function will return a 10x1 array in total. This array will be set as NN_energy and calculate loss function.</p>
<p>Besides, I also want to know whether is another method to create a backward-able array to store the NN_energy array, instead of using “NN_energy = net(Data_in)[0:10,0]”.</p>
<p>Thanks a lot.</p>
<p>======================================================================</p>
<p>import torch</p>
<p>import numpy as np</p>
<p>from torch.autograd import Variable</p>
<p>from torch import multiprocessing</p>
<pre><code>def func(msg,BOP):
    ans = (BOP[msg][0]+BOP[msg][1]/BOP[msg][2])*BOP[msg][3]
    return ans


class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden_1, n_hidden_2, n_output):
        super(Net, self).__init__()
        self.hidden_1 = torch.nn.Linear(n_feature , n_hidden_1)  # hidden layer
        self.hidden_2 = torch.nn.Linear(n_hidden_1, n_hidden_2)  # hidden layer
        self.predict  = torch.nn.Linear(n_hidden_2, n_output  )  # output layer

    def forward(self, x):
        x = torch.tanh(self.hidden_1(x))      # activation function for hidden layer
        x = torch.tanh(self.hidden_2(x))      # activation function for hidden layer
        x = self.predict(x)                   # linear output
        return x


if name == 'main': # apply_async

    Data_in      = Variable( torch.from_numpy( np.asarray(list(range( 0,10))).reshape(10,1) ).float() )
    Ground_truth = Variable( torch.from_numpy( np.asarray(list(range(20,30))).reshape(10,1) ).float() )

    net          = Net( n_feature=1 , n_hidden_1=15 , n_hidden_2=15 , n_output=4 )     # define the network

    optimizer    = torch.optim.Rprop      ( net.parameters() )

    loss_func    = torch.nn.MSELoss()  # this is for regression mean squared loss

    NN_output    = net(Data_in)

    args = range(0,10)
    pool = multiprocessing.Pool()
    return_data = pool.map( func, zip(args, NN_output) )
    pool.close()
    pool.join()

    NN_energy     = net(Data_in)[0:10,0]

    for i in range(0,10):
        NN_energy[i] = return_data[i]

    loss = torch.sqrt( loss_func( NN_energy , Ground_truth ) )     # must be (1. nn output, 2. target)

    print(loss)
</code></pre>
<p>Error messages are shown as follows:</p>
<p>File “C:\ProgramData\Anaconda3\lib\site-packages\torch\multiprocessing\reductions.py”, line 126, in reduce_tensor raise RuntimeError("Cowardly refusing to serialize non-leaf tensor which requires_grad, "</p>
<p>RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries. If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).</p>
How to assign a default value during indexing

username: blodstone
cooked: <p>Say that I have a tensor as follows:</p>
<pre><code>a = torch.tensor([[1, 2, 3], [1, 2, 3]])
</code></pre>
<p>My problem is, what if I only have a look-up table of:</p>
<pre><code>c = torch.tensor([0, 2, 3])
</code></pre>
<p>In which, for every out-of-index, I would like it to be assigned to index 0, such as <em>c[a]</em> will return</p>
<pre><code>tensor([[2, 3, 0], [2, 3, 0]])
</code></pre>
<p>Does anyone know a function to do this?</p>
<p>Thanks for your help.</p>

username: ptrblck
cooked: <p>Would this work?</p>
<pre><code class="lang-python">a = torch.tensor([[1, 2, 3], [1, 2, 3]])
c = torch.tensor([0, 2, 3])

a[a &gt; (c.size(0) - 1)] = 0
c[a]
</code></pre>

username: blodstone
cooked: <p>Yes, it does! Thanks</p>
Do we need to insert blank label into the target before calling the CTC loss?

username: amankhandelia
cooked: <p>I have this question about <code>nn.CTCLoss</code>, I would be grateful if anybody can answer the same.</p>
<p>Example:</p>
<pre><code class="lang-auto">loss = nn.CTCLoss() 
label = 'aman' 
label_dict = {&lt;blank&gt;:0, 'a':1, 'm':2, 'n':3}
</code></pre>
<p>so should encode the label like this<br>
<code>encoded_label = [1, 2, 1, 3]</code><br>
or like this<br>
<code>encoded_label = [0, 1, 0, 2, 0, 1, 0, 3, 0]</code><br>
Note: These  <code>encoded_label</code>  will be passed as  <code>targets</code>  in the  <code>loss</code>  mentioned above.</p>

username: tom
cooked: <p>You want <code>encoded_label = torch.tensor([1, 2, 1, 3])</code>, otherwise the CTC loss will tip over. The CTCLoss implementation does follow the alignment with the enlarged sequence, but does this “padding” on the fly.</p>
<p>Best regards</p>
<p>Thomas</p>

username: SibtainRazaJamali
cooked: <p>What if my labels have variable length<br>
label_1 = torch.tensor([1, 2, 1, 3])<br>
labels2=torch.tensor([1, 2, 3])<br>
do i need to pad it with zeros?</p>

username: gslaller
cooked: <p>No, read the documentation, it differs depending on the device you are working. You should initiate all you label with equal length, perhaps by padding the them. The class acknowledges the length of each target through the “target_lengths” argument which should contain the length of each target in the batch.</p>
How to calculate Confusion matrix?

username: lavender99
cooked: <p>how can i calculte ROC and confusion matrix for segmentation task</p>
<pre><code class="lang-auto">ef get_loss_train(model, data_train, criterion):
    """
        Calculate loss over train set
    """
    model.eval()
    total_acc = 0
    total_loss = 0
    total_dice=0
    result_dice = []
    for batch, (images, masks) in enumerate(data_train):
        with torch.no_grad():
            images = Variable(images.cuda())
            masks = Variable(masks.cuda())
            outputs = model(images).cuda()
            loss = criterion(outputs, masks)
            preds = torch.argmax(outputs, dim=1).float()
            acc = accuracy_check_for_batch(masks.cpu(), preds.cpu(), images.size()[0])
            total_acc = total_acc + acc
            dice = DiceCoefficient()
            score = dice(preds.cpu(), masks.cpu())
            total_dice= score+total_dice
            
            total_loss = total_loss + loss.cpu().item()
    return total_acc/(batch+1), total_loss/(batch + 1),total_dice/(batch+1)
</code></pre>

username: JuanFMontesinos
cooked: <p>sklearn provides all those metrics</p>

username: lavender99
cooked: <p>but i do not know i should put them inside the training loop</p>
Why pytorch assumes my function is a variable

username: Yasesf
cooked: <p>I’m calling a function in another function and get the error ‘local variable’a’ referenced before assignment whereas ‘a’ is a function not a variable.<br>
Any ideas how to fix this?</p>

username: ptrblck
cooked: <p>Probably the function named <code>a</code> is not visible in the current scope.<br>
Could you post some code to reproduce this issue?</p>

username: Yasesf
cooked: <p>I figured what the problem was. I had a variable with the same name somewhere in the code that I was’n aware of.<br>
Thanks</p>
Collate function in DataLoader

username: lanka
cooked: <p>hi,<br>
code explanation <a href="https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/03-advanced/image_captioning/data_loader.py" rel="nofollow noopener">link</a><br>
can you please explain how to effectivly use collate_fn in above tutorial</p>
<pre><code class="lang-auto">data_loader = torch.utils.data.DataLoader(dataset=coco, 
                                              batch_size=batch_size,
                                              shuffle=shuffle,
                                              num_workers=num_workers,
                                              **collate_fn**=collate_fn)
</code></pre>
Images taken as pairs from the dataset

username: Aybuke
cooked: <p>My train dataset has several video folders and in each folder, I have frames that are taken from that video. I should prepare my dataloader with consecutive frame pairs from the dataset such as (t0, t1), (t1, t2), (t2, t3) etc. I should shuffle the loader so that the network will get random pairs of frames from random videos. Could you please help me ??</p>

username: ptrblck
cooked: <p>Your use case sounds like <a href="https://discuss.pytorch.org/t/preparing-dataset-for-video-analyzing/45125">this recent topic</a>. Could you have a look at the suggestions in this topic and check if you could apply them to your use case?</p>
Why is Adaptive Pooling so slow?

username: XT_Li
cooked: <p>I added one adaptive pooling to my network and the training process becomes obvious 3 times slower, is this expected behavior of Adaptive Pooling ?</p>

username: SimonW
cooked: <p>Which adaptive module are you using and what’s the input size? Also, could you profile your network and post the results please (<a href="http://pytorch.org/docs/master/autograd.html?highlight=profiler#profiler">http://pytorch.org/docs/master/autograd.html?highlight=profiler#profiler</a>)? Thanks!</p>

username: XT_Li
cooked: <p>Hi, Thanks for the reply, the network training is only slow in the few hundreds iterations, after that, it becomes fast as not having the adaptive pooling layer.</p>

username: SimonW
cooked: <p>That’s kind of weird. Perhaps disk I/O slowed things down a bit. Glad to know it works now!</p>
Spectral Normalization for Recurrent Layers

username: Ahmed_m
cooked: <p>Hi,</p>
<p>I am trying to implement a GAN model with a discriminator network containing a GRU layer. I use the utils function <em>spectral_norm</em> which works well with convolutional layers but unfortunately, it is not functional for recurrent layers. I tried to follow the implementation for a GRUCell but that’s not possible for the multilayer  GRU as I cannot access its weight tensor. Any advice on how to fix that?</p>
<p>Best</p>
"Local" Conv2dTranpose with "depthwise"

username: Adria_Ciurana
cooked: <p>Hello,<br>
I am trying to implement this type of layer that is somewhat complicated.</p>
<p>The idea is that unlike the Conv2Transpose, this layer has a different filter for each element of the entry. Additionally, this layer applies the <strong>same filter in all the depth dimensions</strong>, and returns the same number of depth (without mixing them, apply the transpose convolution filter by each h,w element).</p>
<p>That is, if we consider an NxM input. And we want to perform the LocalConv2dTransposeDepthWise with kernel_size K, we have NxM KxKx1 filters that perform the same transposed convolution of each channel of the input.</p>
<p>The truth is that I would not know where to start and if it can be implemented using existing pytorch structures.</p>
<p>In the following gif you can see the operation of the layer with a 2x2 entry with depth = 1 and 3x3 kernels:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/7/72ca4831d2bacf2873e29f9e3134832750728652.gif" alt="com-gif-maker%20(4)" data-base62-sha1="gntOLClI12uJVtjsrRbppcOsVc6" width="690" height="388" class="animated"></p>
<p>If someone has any idea, it would be very helpful</p>
Freeze final layer

username: ashwin.raju93
cooked: <p>I am trying to train layer by layer. In a stage I would like to freeze the final layer alone . When i tried to freeze it I encounter an error saying . How do i solve this problem ? any idea ?</p>
<pre><code class="lang-auto">  RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

</code></pre>

username: kelam_goutam
cooked: <p>I guess you need to set <strong>requires_grad = False</strong> in the final layer.<br>
I refereed this - <a href="https://discuss.pytorch.org/t/runtimeerror-element-0-of-variables-does-not-require-grad-and-does-not-have-a-grad-fn-when-trying-to-implement-gumbel-st-sampler/12499">RuntimeError: element 0 of variables does not require grad and does not have a grad_fn when trying to implement Gumbel-ST sampler</a><br>
Do let me know if it works.</p>

username: ashwin.raju93
cooked: <p>Works for me. Thanks.</p>

username: kelam_goutam
cooked: <p>Happy to learn through you.</p>

username: saurav.prakash16
cooked: <p>Hi,<br>
Is it is possible to freeze a layer partially. I want to freeze those parts of a layer which have not changed more than a pre-defined threshold in the last 10 iterations.</p>
Question about torch.load()

username: suke
cooked: <p>Hello, I want to ask a question about torch.load(), I found in same process, if I call torch.load many times.<br>
then 1st time may cost much time, then it will cost too little time.<br>
e.g.<br>
model1 = torch.load()         3.2s<br>
model2 = torch.load()         0.03s<br>
model3 = torch.load()         0.03s</p>
<p>does any one can explain what does pytorch do in first time. and if I call torch.load() 3 times. what is the memory arrangement, will memory cost 3 times than call torch.load once?<br>
Thank you!</p>

username: alex.veuthey
cooked: <p>What <strong>PATH</strong> arguments do you use for <code>torch.load(PATH)</code> with your respective models? Is this just a test script with three times the same checkpoint or different checkpoints?</p>
<p>In the first case, either python or PyTorch is probably clever enough to not reload the model if it’s the same checkpoint.</p>
<p>In the second case, maybe it’s the first time you call a <code>torch</code> function, and it initializes some things, which are not required in the next calls?</p>

username: suke
cooked: <p>in the first case, even I load same checkpoint, actually the memory is increase, so I think maybe there is new instance create.<br>
in the case, maybe it do initialized jobs, do you know detail info about this initialized process?</p>

username: alex.veuthey
cooked: <p>How much does memory increase? Did you try with 3 separate checkpoint files, with different weights?</p>
<p>In any case, I’m not familiar with the internals of PyTorch, these are mainly exploratory questions so that I can learn more about PyTorch thanks to your situation <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> It just seems logical that there would be some initialization at some point for such a complex framework.</p>

username: suke
cooked: <p>when I load in cpu case, the memory will increase about 2M when I call torch.load() every time, I think that is new instance’s memory.<br>
e.g. 1st time 0-&gt;198.75M<br>
2nd time 198.75M-&gt;202.59M<br>
3rd time 202.59M-&gt;204.91M</p>
About the MaxUnpool partial inverse property

username: fseiffarth
cooked: <p>I have a question regarding the class torch.nn.MaxUnpool2d. In the docs it says that it is a partial inverse of torch.nn.MaxPool2d. Moreover, “MaxPool2d is not fully invertible, since the non-maximal values are lost.” So this fact is clear to me, but I had a problem with the following example:</p>
<p>-1 -1</p>
<p>-2 -3</p>
<p>MaxPool2d with 2x2 kernel size would return -1 but MaxUnpool2d of -1 would be</p>
<p>-1 0</p>
<p>0 0.</p>
<p>Now MaxPool2d of this would return 0, which is somehow inconsistent.</p>
<p>This means that “invertibility” restricts to the cases where pooling blocks are not completely negative. Maybe you can mention this in the docs, as it reads that the only problem is that some values are lost.</p>
<p>Another suggestion would be to change the MaxUnpool2d to a right inverse function of MaxPool2d (maybe somehow more natural), i.e.</p>
<p>MaxPool2d(MaxUnpool2d(y)) = y for all input y and not only y with positive value.</p>
Dummy question: Autograd what does it mean?

username: HG1
cooked: <p>Hi,<br>
May be a ‘for dummies’ question.<br>
I have seen many blogs, youtube videos on how the autograd works.<br>
But I haven’t found what it actually means.</p>
<p>As far as I understand the gradient is a kind of derivative.<br>
But what does it mean when I do the following?<br>
<strong>x = Variable(torch.tensor([2.]), requires_grad=True)</strong><br>
<strong>print(x)</strong><br>
<strong>print(x.grad)</strong><br>
<strong>print(x.grad_fn)</strong><br>
<strong>&gt; tensor([2.], requires_grad=True)</strong><br>
<strong>&gt; None</strong><br>
<strong>&gt; None</strong><br>
<strong>y = x + torch.randn(1)</strong><br>
<strong>print(y)</strong><br>
<strong>print(y.grad)</strong><br>
<strong>y.backward()</strong><br>
<strong>print(x)</strong><br>
<strong>print(x.grad)</strong></p>
<p><strong>&gt; tensor([3.5410], grad_fn=)</strong><br>
<strong>&gt; None</strong><br>
<strong>&gt; tensor([2.], requires_grad=True)</strong><br>
<strong>&gt; tensor([1.])</strong></p>
<p>When y.backward() is executed once x.grad = 1<br>
After doing it another time it becomes 2.<br>
What do these values mean?<br>
This does not look to me like a derivative at all.</p>
<p>Regards  Hans</p>

username: JuanFMontesinos
cooked: <p>have a look at this video</p>
<div class="lazyYT" data-youtube-id="MswxJw-8PvE" data-youtube-title="PyTorch Autograd Explained - In-depth Tutorial" data-width="480" data-height="270" data-parameters="feature=oembed&amp;wmode=opaque"></div>

username: HG1
cooked: <p>Hi,<br>
That was one of the video’s that I had already seen.<br>
It explains how it works very well.<br>
But does not what it means.</p>
<p>Why is my gradient 1 when I do a backward operation on y:<br>
<strong>y = x + torch.randn(1</strong><br>
Where<br>
<strong>x = Variable(torch.tensor([2.]), requires_grad=True)</strong></p>
<p>Regards Hans-Peter</p>

username: MariosOreo
cooked: <p>Because the <code>.backward()</code> performs accumulation on each variable with <code>requires_grad=True</code>. That why we call <code>optimizer.zero_grad()</code> before each <code>.backward()</code> in training phase.</p>
<p>Thus, it is clearly to know, x.grad=1 after the first <code>.backward()</code> and x.grad=2 in the second time.</p>
Runtime Error : Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 4, 224, 224] to have 3 channels, but got 4 channels instead

username: Silva_Rubanza
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c1590fb0e64ef59920b583ebc73ae62a87d60f52.png" data-download-href="https://discuss.pytorch.org/uploads/default/c1590fb0e64ef59920b583ebc73ae62a87d60f52" title="27%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c1590fb0e64ef59920b583ebc73ae62a87d60f52_2_690x194.png" alt="27%20PM" data-base62-sha1="rAqXOrPlXsyfKLRB72jS3Dspzou" width="690" height="194" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c1590fb0e64ef59920b583ebc73ae62a87d60f52_2_690x194.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c1590fb0e64ef59920b583ebc73ae62a87d60f52_2_1035x291.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c1590fb0e64ef59920b583ebc73ae62a87d60f52_2_1380x388.png 2x" data-dominant-color="F2F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">27%20PM</span><span class="informations">2376×670 73.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is my model  , i used transfer learning , densenet121 , but when i tried to deploy with flask i keep getting the above error after i upload the picture.<br>
Any help would be highly appreciated.</p>

username: MariosOreo
cooked: <p>Hi,</p>
<p>It seems that [64, 3, 7, 7] is the first conv layer in the network. It receives the input with channel_in=3, but in your use case, the shape of input is [1, 4, 224, 224].</p>
<p>You could check the shape of your input, or having a modification on your network to fit the input size.</p>
How to build after making code changes in cpp files in ATen/native

username: iArunava
cooked: <p>I have successfully build pytorch. And the changes in python files are reflected instantly as I have build using</p>
<pre><code class="lang-auto">NO_CUDA=1 python3 setup.py develop
</code></pre>
<p>But, I am making changes in the <code>cpp</code> files and I am trying to build  it using this</p>
<pre><code class="lang-auto">python3 setup.py clean
NO_CUDA=1 python3 setup.py develop
</code></pre>
<p>and it failes with this error</p>
<pre><code class="lang-auto">subprocess.calledprocesserror: command '['ninja', 'install']' returned non-zero exit status 1.
</code></pre>
<p>Also, its taking about 20mins to reach to this point, any way of doing this faster? I just have to build the native files.</p>

username: LeviViana
cooked: <p>You should use <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#use-ccache" rel="nofollow noopener">ccache</a>, it will change your life!</p>
<p>You won’t need to do <code>python3 setup.py clean</code> before rebuilding, build PyTorch again using ccache, then make your changes, then build it again straight away.</p>

username: iArunava
cooked: <p>I am on Ubuntu, and I am planning to install <code>ccache</code> using this</p>
<pre><code class="lang-auto"># install ccache
sudo apt install ccache

# update symlinks and create/re-create nvcc link
sudo /usr/sbin/update-ccache-symlinks
sudo ln -s /usr/bin/ccache /usr/lib/ccache/nvcc

# config: cache dir is ~/.ccache, conf file ~/.ccache/ccache.conf

# max size of cache
ccache -M 25Gi  # -M 0 for unlimited
# unlimited number of files
ccache -F 0

# deploy (and add to ~/.bashrc for later)
export PATH="/usr/lib/ccache:$PATH"
</code></pre>
<p>After intallation, I should just run</p>
<pre><code class="lang-auto">python3 setup.py develop
</code></pre>
<p>to build, right?</p>

username: LeviViana
cooked: <p>I’d recommend you to run precisely the script in the ccache link above, I always do this without thinking and it works like a charm.</p>
<p>Specifically, in the terminal, I go to the PyTorch folder, I copy &amp; paste all the code between <code>then</code> and <code>fi</code> in the ccache link, then I run <code>export PATH=~/ccache/lib:$PATH</code>. It works a 100% of times, in several machines, I never spent any second trying to understand why <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:">!</p>
<p>BTW, whenever you close the terminal, you should run <code>export PATH=~/ccache/lib:$PATH</code> again for ccache to work.</p>