akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Split a convolution on two GPUs and run out of memory while training

EEstq: <p>Hi, I am new to pytorch. And I have a problem. In a deep learning network,  I split a convolution into two GPU operations, but GPU memory consumption increases while training, and then it reports an error  “RuntimeError: CUDA error: out of memory”</p>
<p>I tried to use del loss and torch.cuda.empty_cache() but it never works…Could you please help me?</p>
<p>This is the code snippet related to the conv part</p>
<p>class Conv2dBlock_Multi(nn.Module):<br>
def <strong>init</strong>(self, input_dim ,output_dim, kernel_size, stride,<br>
padding=0, norm=‘none’, activation=‘relu’, pad_type=‘zero’):<br>
super(Conv2dBlock_Multi, self).<strong>init</strong>()<br>
self.conv = Conv2d_Multi(input_dim, output_dim, kernel_size, stride, bias=True)</p>
<pre><code>def forward(self, x):
    x = self.conv(self.pad(x))
    return x
</code></pre>
<p>class Conv2d_Multi(nn.Module):<br>
def <strong>init</strong>(self,input_dim, output_dim, kernel_size, stride, bias=True):<br>
super(Conv2d_Multi, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(input_dim//2, output_dim//2, kernel_size, stride, bias=True)<br>
self.conv2 = nn.Conv2d(input_dim - input_dim//2,output_dim - output_dim//2, kernel_size, stride, bias=True)</p>
<pre><code>def forward(self, x):
    x1, x2 = x.split([x.size()[1]//2, x.size()[1]-x.size()[1]//2], dim=1)
    self.conv1 = nn.DataParallel(self.conv1, device_ids=[0]).to('cuda:0')
    self.conv2 = nn.DataParallel(self.conv2, device_ids=[1]).to('cuda:1')
    x1_out = self.conv1(x1)
    x2_out = self.conv2(x2)
    x2_out = x2_out.to('cuda:0')
    x = torch.cat([x1_out,x2_out],dim = 1)

    return x</code></pre>

Baileyswu: <p>Maybe you should check whether there is free GPU for you.</p>

EEstq: <p>Thank you for your reply, my GPU storage space is 16000M, but still can’t run. If I operate it on one GPU, I only need 6000M of storage space. Could it be that nn.DataParallel has an error in backpropagation?</p>

SimonW: <p>at every iteration you are wrapping your convs with another <code>DataParallel</code>, so you are doing <code>2^num_iter</code> splits…</p>

EEstq: <p>Thanks for all the reply<br>
I solve this problem finally…<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/603bbb4512dae0778ab5bf2e64643f99c19cad3f.png" data-download-href="https://discuss.pytorch.org/uploads/default/603bbb4512dae0778ab5bf2e64643f99c19cad3f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/603bbb4512dae0778ab5bf2e64643f99c19cad3f.png" alt="image" data-base62-sha1="dJjOiRT7rbuE5Mdr3jSIjit3ift" width="690" height="299" data-dominant-color="F8F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1110×482 18.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Is the printed loss supposed to include the weight decay term?

Enumaris: <p>I built a deep neural net, let’s call it <code>net()</code> which I train like so:</p>
<pre><code>    net = net()
    loss = nn.BCEWithLogitsLoss()
    net.cuda()
    optimizer = optim.Adamax(net.parameters(),lr=0.001,weight_decay=1.0)
    for epoch in range(epochs):
        i = 0
        running_cost = 0
        for minibatch in minibatches:
            X,Y = minibatch
            out = net(X)
            cost = loss(out,Y)
            cost.backward()
            optimizer.step()

            running_cost += cost.data[0]
            if(i%1000==0):
                print(running_cost/1000)
                running_cost = 0
</code></pre>
<p>I find that no matter what I set the <code>weight_decay</code> parameter to, the <code>running_cost</code> will not change. Is that expected behavior? I would have thought that the cost should be calculating the total cost including the weight_decay portion so that if <code>weight_decay</code> is large, the cost should be correspondingly large as well.</p>

tengerye: <p>The <code>running_cost</code> does not include the <code>weight_decay</code>.</p>
How to change a function to be used in cuda? RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

isalirezag: <p>I am using the tutorial on training a classifier for cifar10<br>
So in this tutorial and other places we can do <code>net = net.to(device)</code> to use the it in cuda version</p>
<p>Lets say i have:</p>
<pre><code class="lang-auto">
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5,padding= 5/2)
        self.pool = nn.MaxPool2d(2, 2)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5,padding= 5/2)
        
        self.fc1 = nn.Linear(16 * 32 * 32, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def FunctionIdesigned(self, a,b):
        blabblab
        return hhh
    
    def FunctionIdesigned2(self, aa,bb):
        Here I am using FunctionIdesigned also
        blablabblab
        
        return hh
    def forward(self, x):
        y = FunctionIdesigned2(x,GaussianKernel(3,0.5))

        
        x = self.pool(F.relu(self.conv1(x)))
        x = F.relu(self.conv1(x))
        x = self.pool(F.relu(self.conv2(x)))
        x = F.relu(self.conv2(x))

        

        x = x.view(-1, 16 * 32 * 32)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        
        return x


net = Net()
net = net.to(device)
</code></pre>
<p>when i wanna use the net function it still gives me error that FunctionIdesigned2 is not in cuda.<br>
is there any way to use to(device) to fix it or i should use .cuda manually inside the function?</p>
<p>Error:<br>
<code>RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same</code></p>

joe_hoeller: <p><a class="mention" href="/u/isalirezag">@isalirezag</a> Have you been able to fix this? I am trying to get my Net to GPU.</p>

Diego: <p>Could you post your training loop so we can take a look?. Its likely that the bug is there.</p>
LSTM input in nn.LSTM

Pradumna_Suryawanshi: <p>I have a  multivariate time series with length 48 and at each time step I have 42  variables so for each timestep I get a matix (42 * 30) where 30 is number of times that variable changed  during one time step.<br>
and using “ALL THOSE” 42 variables at 48 time steps (x=48 * 42*30)–(y=1/0) they correspond to a y (it either takes a 1 or a 0).</p>
<p>Now I have 4000 such x and corresponding 4000 such y’s.<br>
I am not able model the input to the Lstm.<br>
I went through many tutorials and online forums but couldnt find anthing.<br>
Can anyone please help me with this.</p>
Curriculum learning

Rodolphe_Lampe: <p>Curriculum learning is the process of giving easy tasks to your neural network before harder examples.</p>
<p>I have a dataset composed of samples having a known discrete difficulty (0 to 10 for example), I’d like my dataloader to give samples of difficulty i for the epoch 100<em>i to 100</em>(i+1).<br>
How can I do that ?</p>

JuanFMontesinos: <p>You should code it yourself. Dataloader is flexible and you can modify its behaviour at any time of the training stage.</p>
Can't speed up depthwise conv with cudnn7102

chenchr: <p>Hello.<br>
I test the depthwise convolution in environment with and without cudnn 7102 by setting<br>
<code>torch.backends.cudnn.enabled=False</code></p>
<p>However, for depthwise convolution, the running time is still the same, while for common convolution with groups is 1, the speed up by cudnn is visible.<br>
It seems that cudnn over 7.0.0 has support groups convolution, but how to benefit from that ?<br>
Any suggestion ?<br>
Thanks!</p>

13952522076: <p>Hi, looks like still no ways to speed up group convolution operations. I have tested even cuDNN 7.6.3 and PyTorch 1.3, the improvement is negligible. All the solutions provided in github are useless. If you found something helpful, please email me, thanks.</p>
Back propagation multiple times before optimizer.step()

shirui-japina: <p>What will happen if back propagation multiple times <strong>before</strong> optimizer.step()?</p>
<p>Coincidentally, I got <code>RuntimeError</code> like this:<br>
<code>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</code></p>
<p>Which means I can back propagation multiple times <strong>before</strong> optimizer.step(). But what will happen if I do that?</p>
<p>Under what circumstances should I use like this?</p>

tom: <p>This is useful in specific cases, e.g. when using <code>torch.autograd.grad</code> (with <code>create_graph=True</code>) to incorporate derivatives in your loss function.<br>
In general, <code>backward</code> accumulates gradients (that’s why you do <code>zero_grad</code> to clear them). Backpropagating the same graph just for the sake of it would be unusual though. It usually is more efficient to only backpropagate once per forward unless you actually have one of those cases where you need the gradient in your loss function or similar.</p>
<p>Best regards</p>
<p>Thomas</p>
Code for Loss Function without C++ ATen implementation

PyExtreme: <p>In which directory are codes there whose C++ ATen implementation is not there?</p>
<p>I am creating C++ Frontend of CrossEntropyLoss and since there is no C++ ATen implementation of it in the native directory, where can I find the implementation of this loss?</p>
<p>Thank you</p>

yf225: <p><code>CrossEntropyLoss</code> is a composition of <code>F::nll_loss</code> and <code>F::log_softmax</code>: <a href="https://github.com/pytorch/pytorch/blob/15be189f0de4addf4f68d18022500f67617ab05d/torch/nn/functional.py#L2012" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/15be189f0de4addf4f68d18022500f67617ab05d/torch/nn/functional.py#L2012</a>, and that’s why it doesn’t have a C++ ATen implementation <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> (I also responded in the C++ torch::nn parity issue: <a href="https://github.com/pytorch/pytorch/issues/25883#issuecomment-544267626" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/25883#issuecomment-544267626</a>)</p>
Backpropogation for WordEmbedding

samane_barzanuni: <p>Hi, I need the backpropgation for my WordEmbedding layer. The code is as below:</p>
<p>class WordEmbedding:<br>
def <strong>init</strong>(self, vocab_size, embedding_dim, init_wt=0.01, weight=None):<br>
self.vocab_size = vocab_size<br>
self.embedding_dim = embedding_dim<br>
self.bias = torch.zeros(0)<br>
if weight is not None:<br>
self.weight = weight<br>
else:<br>
self.weight = torch.randn(vocab_size, embedding_dim) * init_wt<br>
self.grad_bias = torch.zeros(0)<br>
self.grad_weight = torch.zeros(vocab_size, embedding_dim)<br>
self.is_trainable = True</p>
<pre><code>def get_indicator_matrix(self, indices):
    batch_size = indices.size(0)
    self.indicator_matrix = torch.zeros(batch_size, self.vocab_size)
    for i in range(batch_size):
        self.indicator_matrix[i, indices[i]] = 1.0

def forward(self, input):
    self.input = input
    self.batch_size = self.input.size(0)
    output = torch.zeros(self.input.size(0), self.input.size(1) * self.embedding_dim)
    for i in range(self.input.size(1)):
        self.get_indicator_matrix(self.input[:, i].long())
        output[:, i * self.embedding_dim:(i + 1) * self.embedding_dim] = \
            torch.mm(self.indicator_matrix, self.weight)
    return output

def backward(self, grad_output):
    output = 
    return output

def zero_grad(self):
    self.grad_weight.fill_(0.0)</code></pre>
RuntimeError: shape '[-1, 576]' is invalid for input of size 12800

ayaz-amin: <pre><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

train_data = datasets.MNIST('', train=True, transform=transforms.Compose(
    [transforms.ToTensor()]), download=True)

test_data = datasets.MNIST('', train=False, transform=transforms.Compose(
    [transforms.ToTensor()]), download=True)

train_loader = DataLoader(train_data, batch_size=32)
test_loader = DataLoader(test_data, batch_size=32)

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.conv = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        self.fc = nn.Linear(16*6*6, 128)
        self.output = nn.Linear(128, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = x.view(-1, 16*6*6)
        x = F.relu(self.fc(x))
        return F.softmax(self.output(x), dim=1)

net = Network()
optimizer = optim.Adam(net.parameters(), lr=0.001)
loss_function = nn.CrossEntropyLoss()

for epoch in range(3):
    running_loss = 0.0
    for i, (X, y) in enumerate(train_loader):
        optimizer.zero_grad()
        y_pred = net(X)
        loss = loss_function(y_pred, y)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0
</code></pre>

mailcorahul: <p>The tensor shape after <code>x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))</code> will be batch_size x 16 x 5 x 5.<br>
Hence while reshaping <code>x = x.view(-1, 16*6*6)</code>, the error is thrown.</p>
<p>Change <code>self.fc = nn.Linear(16*6*6, 128)</code> to <code>self.fc = nn.Linear(16*5*5, 128)</code><br>
and <code>x = x.view(-1, 16*6*6)</code> to <code>x = x.view(-1, 16*5*5)</code></p>

ayaz-amin: <p>Thanks you for the solution! However, I want to know how to get the output shape of each layer. How do you find the output shape of each layer?</p>

mailcorahul: <p>You can simply calculate a convolution layer’s spatial dims using the standard formula<br>
<code>(W - F + 2P)/S + 1</code> where W can be width/height, F is filter size, P is the padding and S is the stride.<br>
This helps you in choosing the appropriate dimensions for the following <code>nn.Linear</code> layer.</p>
<p>You can also get the output shape by simply printing the <code>torch.Tensor.size()</code> inside the <code>forward</code> method.</p>

ayaz-amin: <p>I think I got it now. Thanks!</p>
Custom untraditional simulation implementations in pytorch - any hope for performance?

benelot: <p>Hey there!</p>
<p>I am currently implementing a biologically-plausible neural network simulation which essentially numerically solves an ODE under the hood. I tried to get it to a similar speed as training a pytorch model, but is seems very hard to do. I recently started to profile my implementations using bottleneck (<a href="https://pytorch.org/docs/stable/bottleneck.html" rel="nofollow noopener">https://pytorch.org/docs/stable/bottleneck.html</a>) and found out that even a single matrix multiplication I perform takes longer than the whole set of operations to update an NN which one batch[1]. The difference is about 13ms for the matrix multiplication vs. 2.5 ms for the forward-backward pass in classical pytorch code (Quick test, I can go more into that). Is there any hope to implement fast gpu-based simulations in pytorch that do not require its classical setup? Another question would possibly be: Are the pytorch higher-order operations (optimizer.step(), loss.backward() etc.) implemented in pytorch lower-order operations such as mm, bmm etc.?</p>
<p>[1] I mean something simple as this<br>
images = images.to(device)<br>
labels = labels.to(device)<br>
output = model(images)<br>
optimizer.zero_grad()<br>
loss = loss_fn(output, labels)<br>
loss.backward()<br>
optimizer.step()</p>

tom: <p>NNs with ODEs have been done successfully, e.g. <a href="https://github.com/rtqichen/torchdiffeq/" rel="nofollow noopener">https://github.com/rtqichen/torchdiffeq/</a><br>
In general, there is nothing to keep you from using the lower-level functions in PyTorch or mixing them with others.<br>
The optimizers are impemented using them, see the code in <a href="https://github.com/pytorch/pytorch/tree/master/torch/optim/" rel="nofollow noopener">torch/optim/</a>.<br>
When benchmarking, one has to be careful about the asynchronous nature of things. If your results are too surprising, chances are that there is room for refinement of your measurement methodology.</p>
<p>Best regards</p>
<p>Thomas</p>

benelot: <p>I realized that possibly my comparison was a bit unfair. I was comparing my recurrent neural network with a feedforward neural network, so some of the computational complexity was vastly different (apart from the ode vs. no-dynamics comparison that is a bit unfair). I will compare again to a regular RNN in pytorch (non-LSTM). Am I correct that the nn.RNN is a regular vanilla RNN? In the description it says it is an Elman-RNN, which is not exactly true since it is without context neurons.</p>
CUDA vs. DataParallel: Why the difference?

wasiahmad: <p>I have a simple neural network model and I apply either <code>cuda()</code> or <code>DataParallel()</code> on the model like following.</p>
<pre><code>model = torch.nn.DataParallel(model).cuda()
</code></pre>
<p>OR,</p>
<pre><code>model = model.cuda()
</code></pre>
<p>When I don’t use DataParallel, rather simply transform my model to <code>cuda()</code>, I need to explicitly convert the batch inputs to <code>cuda()</code> and then give it to the model, otherwise it returns error. [<code>torch.index_select received an invalid combination of arguments - got (torch.cuda.FloatTensor, int, torch.LongTensor)</code>]</p>
<p>But with DataParallel, the code works fine. Rest of the other things are same. Why this happens? Why when I use DataParallel, I don’t need to transform the batch inputs explicitly to <code>cuda()</code>?</p>

smth: <p>DataParallel allows CPU inputs, as it’s first step is to transfer inputs to appropriate GPUs. that’s simply the reason why.</p>

achaiah: <p><a class="mention" href="/u/smth">@smth</a>  Sorry to revive an old thread but I’m looking for a bit of clarification. Are you saying that if we pass our model to  DataParallel then we don’t need to explicitly call <code>.cuda()</code> on our Variables (e.g. <code>inputs.cuda(cuda_device)</code>)?  I was under the impression that the intent of DataParallel was to be used when you were utilizing multiple GPUs. Am I misunderstanding?</p>
<p>Thanks</p>

smth: <p><a class="mention" href="/u/achaiah">@achaiah</a> if you use DataParallel, and the input is on CPU, then the input chunks are parallely copied over to each GPU…</p>
Format for custom Anchor shapes

clck10: <p>Hello all,</p>
<p>I have a very silly question that I cannot seem to find an answer for anywhere else.</p>
<p>I am trying to apply a pre-trained object detection network to a new dataset.<br>
The image dimensions are much smaller than typical images used for object detection.<br>
The objects themselves also have different shapes than typical objects.</p>
<p>Because of this, I defined some custom anchors based on the Anchor Generator class: <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/rpn.py" rel="nofollow noopener">rpn.py</a></p>
<p><strong>The question</strong>: Is “sizes” in AnchorGenerator relative to the input image, or to the feature map after the backbone?</p>
<p>To make things more confusing, I can see the images are also resized to be quite large before feeding them to the backbone. So do the anchor sizes have to be for the original image, the resized/larger image, or the feature maps from the resized image?</p>
<p>I realize my setup is in the painful corner of transfer learning (small dataset, very different from original) so I could be hitting a different wall here. Just want to make sure the anchors are solid.</p>
<p>Thank you very much for your time!</p>
How to convert a variable from tensorflow to pytotch?

DrunkAlex: <p>Hello world!!</p>
<p>I have a question. I have this command in tf <code>tf.tile(tf.expand_dims(inputs, 0), [K, 1, 1])</code> but I need the output in pytorch type. How can I write this expression in torch way or how to convert the result to torch?</p>
<p>Thanks.</p>

InnovArul: <p>the equivalent pytorch expression would be:</p>
<pre><code class="lang-auto">inputs.unsqueeze(dim=0).repeat(K, 1, 1)
</code></pre>
<p>As an alternative way, If you want to convert the tensorflow result to pytorch, depending on tensorflow version, you can either call <code>.eval()</code> (&lt; tf 2.0) or <code>.numpy()</code> (&gt;=tf 2.0) on the tensorflow result.</p>
How to customize model.train() and model.eval() on Pytorch

User1: <p><code>model.train()</code> and <code>model.eval()</code> are flags that tell the model that you are training the model and testing mode respectively. This will make the model behave accordingly to techniques such us <strong>dropout</strong> that have different procedures in train and testing mode.</p>
<pre><code class="lang-auto">class NeuralNet(torch.nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(NeuralNet, self).__init__()
        # Code..

    def forward(self, x):
        out = some_func(x)
         ## Code..
        return out
</code></pre>
<p>Suppose I’ve written a different function <code>some_func()</code> that I want to be executed only on training mode. How can I modify the functions <code>model.train()</code> and <code>model.eval()</code> to do that ?</p>

mailcorahul: <p><code>model.train()</code> and <code>model.eval()</code> sets the calling <code>nn.Module</code>'s and its children’s modules <code>training</code> attribute to <code>True</code> and <code>False</code> respectively.<br>
If you need to invoke functions based on training or testing mode, you can simply use the network’s <code>training</code> attribute.</p>
<pre><code class="lang-auto">def forward(self, x):
    if self.training:
        out = self.train_module(x)
    else:
        out = self.eval_module(x)
    return out
</code></pre>

User1: <p>Mmm, interesting. I tried that, but it’s not clear for me how to run the code accordingly. Thank you for your answer.</p>
How to apply Group Normalization to PyTorch FasterRCNN training?

kkpalczewski: <p>I am following this implementation of RCNN (I will use FasterRCNN instead of MaskRCNN) <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a>.</p>
<p>I want to use Group Normalization for the input layer in my DataSet class. I don’t want to use custom pipeline and create my DataSet and DataLoader since PyTorch training pipeline is very convenient to use.</p>
<p>Is it possible to use GroupNormalization with standard Dataloader and Dataset from torch.utils.data?</p>
<p>There is one (I think) available DataLoader: -&gt; <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader</a> and couple DataSets: -&gt; Dataset: <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset</a> -&gt; IterableDataset: <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset</a> …</p>
<p>Thanks!</p>
Undefined symbol error on importing C++ extension

rahuldey91: <p>I built a custom C++ extension following the guidelines shown here <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>. However, when I import it, I get the following error:</p>
<p><code>ImportError: /usr/local/miniconda/lib/python3.6/site-packages/zbuffertri-0.0.0-py3.6-linux-x86_64.egg/zbuffertri.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC1ENS_14SourceLocationERKSs</code></p>
<p>I followed the discussion <a href="https://discuss.pytorch.org/t/undefined-symbol-when-import-lltm-cpp-extension/32627">here</a> which says both PyTorch and the C++ extensions should be built using -<code>D_GLIBCXX_USE_CXX11_ABI=1</code> which is already the case. I get <code>torch._C._GLIBCXX_USE_CXX11_ABI = True</code> and my compile output does show its using <code>D_GLIBCXX_USE_CXX11_ABI=1</code> while compiling the C++ extension too.</p>
<p>My setup.py file looks like:</p>
<pre><code class="lang-auto">from setuptools import setup
import torch
from torch.utils.cpp_extension import BuildExtension, CppExtension, CUDAExtension

setup(
    name="zbuffertri",
    ext_modules=[
        CUDAExtension(
            "zbuffertri",
            ["zbuffertri.cpp", "zbuffertri_implementation.cu"],
        )
    ],
    cmdclass={"build_ext": BuildExtension},
)
</code></pre>
<p>My setup is the following:<br>
OS: Ubuntu 18.04<br>
PyTorch: v1.1.0<br>
GCC: v4.9<br>
CUDA: v9.0.176<br>
CuDNN: v7.0</p>
<p>Any help would be much appreciated!</p>

hasakii: <p>You should use GCC &gt;= 4.9 to build PyTorch and C++ extensions.</p>

rahuldey91: <p>I am using GCC 4.9 to build the C++ extensions. I didn’t build PyTorch from source but downloaded the binary PyTorch 1.0.0 from conda. Do I absolutely need to build PyTorch from source in order to be able to build C++ extensions? Nowhere in the tutorial did it say that.</p>

hasakii: <p>Maybe you could try to install from the conda <code>pytorch</code> channel?</p>
<p><code>conda install pytorch=1.0.0 cuda90 -c pytorch</code></p>
RuntimeError: cuda runtime error (78) : a PTX JIT compilation failed at /pytorch/torch/csrc/generic/serialization.cpp:137

SeanChenxy: <p>Hi everyone,<br>
I have an error, the information is as follows</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/pytorch/torch/csrc/generic/serialization.cpp line=137 error=78 : a PTX JIT compilation failed
Traceback (most recent call last):
  File "test_video.py", line 221, in &lt;module&gt;
    main()
  File "test_video.py", line 110, in main
    net.load_state_dict(torch.load(trained_model))
  File "/home/sean/anaconda3/envs/pt40/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
    return _load(f, map_location, pickle_module)
  File "/home/sean/anaconda3/envs/pt40/lib/python3.6/site-packages/torch/serialization.py", line 476, in _load
    deserialized_objects[key]._set_from_file(f, offset, f_is_real_file)
RuntimeError: cuda runtime error (78) : a PTX JIT compilation failed at /pytorch/torch/csrc/generic/serialization.cpp:137
</code></pre>
<p>Environment information are:</p>
<ol>
<li>Pytorch0.40</li>
<li>cuda-8.0</li>
<li>Nvidia Titan X, Nvidia Titan Xp</li>
</ol>
<p>Could you help me solve it. Thank you in advance.</p>
I tried .backward(retain_graph=True) twice

Hwarang_Kim: <p>I tried .backward(retain_graph=True) twice</p>
<p>import torch<br>
x = torch.ones(2,2,requires_grad=True)<br>
y=x+2<br>
z=y * y * 3<br>
out=z.mean()<br>
out.backward(retain_graph=True)<br>
print(x.grad)<br>
out.backward(retain_graph=True)<br>
print(x.grad)</p>
<p>The first result of print(x.grad) is tensor([[4.5000, 4.5000], [4.5000, 4.5000]])<br>
and i wondered what if out.backward is performed twice, so i did.<br>
then second print is tensor([[9., 9.], [9., 9.]])</p>
<p>I can`t understand how this result came out<br>
can you please explain ?</p>

paganpasta: <p>Your each call of out.backward() does the following operation<br>
<code>x.grad += dz/dx</code><br>
In short, you accumulate gradients.</p>

Hwarang_Kim: <p>Thank you for replying!<br>
very clear!!</p>
THe problem of incomplete last batch without dropping the last batch

Mohamed_Ragab: <p>I trained my recurrent model with a specific batch size but during the evaluation on the test set, I had the problem of the incomplete batch. For instance my bs=10 but I have 259 samples. Note that dropping the last is not viable to me as it is time-series data.  Any suggestion for a solution without affecting the performance? Or is there a way to vary the batch and the keep the model working?</p>

paganpasta: <p>Not sure what you mean by variable batch size. If you are iterating over a batch loader it will automatically create a batch size of 9 for the last iteration which you can use.</p>

Mohamed_Ragab: <p>Yes it create batch of 9 but  I got an error of dimensions as the trained model accept batch of 10</p>

paganpasta: <p>Can you show a code snippet which you are using to do the evaluation, and also your model class ?</p>

Mohamed_Ragab: <p>def evaluate(model, test_dl, criterion):</p>
<pre><code>model.eval()
epoch_loss = 0
epoch_score =0
with torch.no_grad():

     for inputs, labels in test_dl:
        src = inputs.to(device)
        trg = inputs.to(device)
        labels=labels.to(device)
        rec_inputs,trg_inputs,pred,attn_weights = model(src,0) #turn off teacher forcing
        #denormalize predictions 
        pred=pred*denormalization
        rul_loss = criterion(pred.squeeze(), labels)
        score = scoring_func(pred.squeeze()-labels)
    
        epoch_loss += rul_loss.item()
        epoch_score += score
return epoch_loss / len(test_dl),epoch_score</code></pre>

Mohamed_Ragab: <p>Currently, I have worked around the problem by padding the incomplete batch</p>
Is the scatter_add totally the same as tf version?

Gray: <p>Actually I want to implement the CenterLoss in pytorch, the code in tf is something like</p>
<pre><code class="lang-python">centers_update_op = tf.scatter_sub(centers, labels, diff)
</code></pre>
<p>which is</p>
<pre><code class="lang-auto">labels -&gt; [batch_size, 1]
centers-&gt;[class_nums, emb_size]
diff-&gt;[batch_size, emb_size]
for idx, label in enumerate(labels):
    centers[label,:] += diff[idx]
</code></pre>
<p>I try to use the scatter_add api in pytorch</p>
<pre><code class="lang-auto">centers.scatter_add_(0, labels, diff)
</code></pre>
<p>however, it only affect the first element of the emb_size. I have to use the for loop to implement it. I think it would be rather slow. Is there any api can use to do that? Thanks!</p>

Gray: <p>I figure it out by using <code>index_add_</code>, codes like this</p>
<pre><code class="lang-auto">centers.index_add_(0, labels.flatten(), diff)
</code></pre>
How do RNNs deals with one-hot encoding?

jrmiranda: <p>Hello, I’m starting in NLP and have a simple question.</p>
<p>Let’s say I have a NLP model with a vocabulary of 1000 words and I want to work with sequences of 5 steps. This means that the seq_len=5 and input_size=1000. But in many tutorials and articles all I see is that the input consists of only integers(and not one-hot vectors). So why the input size must be 1000 and not just 1 since is only one integer per word? Are we dealing with only the index of the vectors? Thanks!</p>

vdw: <p>You should look up nn.Embedding. Word embeddings are the de-facto method to vectorize words for most network models and particularly RNNs.</p>
<p>An embedding is a matrix of shape (vocab_size, embed_dim). They also make the use easier since they only use word indexes as input instead of one-hot vectors. Note that multiplying a matrix with a one-hot vector simply selects the corresponding row/column anyway.</p>
How to solve the spikes of loss at some epoch?

John1231983: <p>During training, I found some spikes of loss at epoch  as follows<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad6a42c1c422c79eaf3051650feae8aab822f6a7.png" alt="image" data-base62-sha1="oK6gqNoYtg5h2ZRhLUKxAR5lvrp" width="212" height="107"></p>
<p>Based on the loss of previous loss, if the loss different between current and previous too big, I will not update the gradient. Do we have any function in pytorch do check implement it? Currently, I used as</p>
<pre><code class="lang-auto">
current_loss = cross_entropy(input, target)
if current_loss- prev_loss&gt; threshold):
   current_loss.backward()
   prev_loss = current_loss
</code></pre>

Tony-Y: <p><a href="http://www.cs.toronto.edu/~rgrosse/courses/csc321_2017/readings/L15%20Exploding%20and%20Vanishing%20Gradients.pdf" rel="nofollow noopener">Exploding and Vanishing Gradients</a></p>
<p>If you suffer from exploding gradients, I recommend using <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.clip_grad_norm_" rel="nofollow noopener">clip_grad_norm_</a>.</p>

John1231983: <p><a class="mention" href="/u/tony-y">@Tony-Y</a>: Thanks but how to know the low and up bound of grad to be input for the function</p>

Tony-Y: <aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1211.5063.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1211.5063.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1211.5063.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1211.5063.pdf" target="_blank" rel="noopener nofollow ugc">1211.5063.pdf</a></h3>

  <p class="filesize">610.95 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>One good heuristic for setting this threshold is to look at statistics on the average norm over a sufficiently large number of updates. In our experiments we have noticed that for a given task and model size, training is not very sensitive to this hyper-parameter and the algorithm behaves well even for rather small thresholds.</p>
</blockquote>
Implementing an LSTM from scratch leads to different results

Lewis: <p>Hello,<br>
I am implementing an LSTM from scratch and then comparing it with the PyTorch LSTM, however, the results I get when using the PyTorch LSTM are better than my LSTM implementation. May I know what is wrong in the code below? Or what can be adjusted to get the correct result?</p>
<pre><code class="lang-auto">class LSTMCell(nn.Module):

    def __init__(self, input_size, hidden_size):
        super(LSTMCell, self).__init__()
        self.hidden_size = hidden_size
        self.input_size = input_size
        self.x2h = nn.Linear(input_size, 4 * hidden_size)
        self.h2h = nn.Linear(hidden_size, 4 * hidden_size)
        self.tanh = nn.Tanh()
        self.init_parameters()
    
    def init_parameters(self):
        std = 1.0 / math.sqrt(self.hidden_size)
        for p in self.parameters():
            p.data.uniform_(-std, std)

    def forward(self, inp, states):
        """
        inp shape: (batch_size, input_size)
        each of states shape: (batch_size, hidden_size)
        """
        
        ht, ct = states
        gates = self.x2h(inp) + self.h2h(ht)    # (batch_size, 4 * hidden_size)

        in_gate, forget_gate, new_memory, out_gate = gates.chunk(4, 1)

        in_gate = torch.sigmoid(in_gate)
        forget_gate = torch.sigmoid(forget_gate)
        out_gate = torch.sigmoid(out_gate)
        new_memory = self.tanh(new_memory)
        c_new = (forget_gate * ct) + (in_gate * new_memory)
        h_new = out_gate * self.tanh(c_new)

        return (h_new, c_new)
</code></pre>
<p>Thanks!</p>

Lewis: <p><a class="mention" href="/u/fawaz_sammani">@Fawaz_Sammani</a> can you help please? I believe you do understand in this since your answer to my last post is very related. Hope to hear from you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Fawaz_Sammani: <p>It seems correct to me. I’ve used the same code in one of my projects before (just different variables) and I got very close results with the PyTorch LSTM. May you please post your results and code when using the PyTorch LSTM?</p>
Does torchvision.utils.save_image do compression on save images?

XW324: <p>is save_image lossless?</p>

ptrblck: <p><code>save_image</code> uses the <code>PIL.Image.save</code> method as seen <a href="https://github.com/pytorch/vision/blob/d194082c067d8635370e6fce7345aa03cbb32e86/torchvision/utils.py#L109" rel="nofollow noopener">here</a>, which supports <a href="https://pillow.readthedocs.io/en/5.1.x/handbook/image-file-formats.html" rel="nofollow noopener">these</a> image formats.</p>
Field.build_vocab

Tank: <p>Currently doing the following:</p>
<pre><code>def tokenize_text(text):
    return [letter for letter in text]

def tokenize_ints(int_lst):
    return int(int_lst[0])

text_Field = Field(tokenize = tokenize_text,
               eos_token = '&lt;eos&gt;')

index_Field = Field(tokenize= tokenize_ints)

train_Datafields = [("Index", None),("SRC", text_Field), ("TRG", text_Field)]

train_data = TabularDataset(
        path = "train.csv",
        format = "csv",
        skip_header = True,
        fields = train_Datafields)

text_Field.build_vocab(train_data)
</code></pre>
<p>How do I use that vocab to transform dataset into one hot encoded vectors so that I can pass that into a net?</p>
Solve leaf variable in Custom Embedding Pytorch C++

DeepakSaini119: <p>I have written a custom embedding layer</p>
<blockquote>
<p>struct CustomEmbeddingImpl : torch::nn::Module<br>
{<br>
CustomEmbeddingImpl(int vocab_size, int embedding_size)<br>
{<br>
embedding = register_module(“embedding”, torch::nn::Embedding(vocab_size + 1, embedding_size));<br>
}</p>
<pre><code>torch::Tensor forward(torch::Tensor indices, torch::Tensor weights)
{
    assert((indices.sizes()[0] == weights.sizes()[0]) &amp;&amp; (indices.sizes()[1] == weights.sizes()[1]));

    torch::Tensor x = torch::transpose(embedding-&gt;forward(indices.to(torch::kInt64)), 1, 2);
    torch::Tensor y = torch::unsqueeze(weights, 2);

    torch::Tensor z = torch::matmul(x, y).squeeze(2);
    return z;
}

void initialize()  // init randomly
{
    torch::Tensor&amp; weight = embedding-&gt;named_parameters()["weight"];
    torch::nn::init::normal_(weight, 0, 1);
    for(int j = 0; j &lt; weight.sizes()[1]; j++)
        weight[0][j] = 0.0;
}

void initialize(vector&lt;vector&lt;float&gt; &gt;&amp; pretrained_embs)  // init with pretrained
{
    torch::Tensor weight = embedding-&gt;named_parameters()["weight"];

    assert((pretrained_embs.size() == weight.sizes()[0] - 1) &amp;&amp; (pretrained_embs[0].size() == weight.sizes()[1]));

    for(int i = 0; i &lt; pretrained_embs.size(); i++)
    {
        for(int j = 0; j &lt; weight.sizes()[1]; j++)
            embedding-&gt;named_parameters()["weight"][i + 1][j] = pretrained_embs[i][j];
    }
    for(int j = 0; j &lt; weight.sizes()[1]; j++)
        embedding-&gt;named_parameters()["weight"][0][j] = 0.0;
}
torch::nn::Embedding embedding{nullptr};
</code></pre>
<p>};<br>
TORCH_MODULE(CustomEmbedding);</p>
</blockquote>
<p>Since, I am modifying the embedding layer weight Parameter, when I try to employ autograd here, I get an error that the leaf variable has been moved to the graph interior. How do I solve this issue, while being able to do both kind of initialization(and in both cases, initializing the 0(=padding_idx) with 0s). Thanks.</p>

albanD: <p>Whenever you modify the weights for initialization, you want to make sure to disable the autograd while you’re doing these modifications. You can do so with <code>torch::NoGradGuard guard;</code>.</p>

DeepakSaini119: <p>Thanks a lot <a class="mention" href="/u/alband">@albanD</a>.</p>
Turn off batch-norm but leave dropout on

Ismail_Elezi: <p>Hi all,</p>
<p>I want to play a bit with <a href="https://arxiv.org/pdf/1506.02158.pdf" rel="nofollow noopener">Monte Carlo dropout</a>. However I am using ResNets (yes, I know that they don’t use dropout but added it). The problem is that ResNets also use batch normalization.</p>
<p>Now during the inference stage, I need to do multiple forward steps for each image and average results, while keeping dropout in train mode. At the same time, because batch norm behaves differently during train and eval mode, I need to turn it off. Simply doing:</p>
<pre><code>net = net.eval()
</code></pre>
<p>obviously doesn’t work and sets both dropout and batch norm in eval mode.</p>
<p>Any solutions (I guess it is something relatively straightforward)?</p>

vabh: <p>This should work:</p>
<pre><code class="lang-python">for m in model.modules():
  if isinstance(m, nn.BatchNorm2d):
    m.eval()
</code></pre>

Ismail_Elezi: <p>Worked like a charm! Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

nabsabs: <p>hi <a class="mention" href="/u/ismail_elezi">@Ismail_Elezi</a>, this doesn’t actually turn off batch norm right? I am looking for a way to train with batch_norm but on inference, I just want to use a batch of one and so I want to turn batchnorm off completely. Do you have a suggestion? <a class="mention" href="/u/vabh">@vabh</a></p>

vabh: <p>Hello,<br>
Could you explain what you mean by ‘turn off’ batch norm?</p>
<p>If you set BN to <code>eval</code> mode, it is possible to use a batch size of 1 during inference. This is because during testing the stored <code>running mean</code> and <code>running std</code> are used. These values are not calculated from the batch.</p>

nabsabs: <p>If you set BN to eval mode it will use the running mean and running var parameters to normalize each batch - right? But on inference, if I want to use a batch of one, then isn’t it going to normalize activations based on these batch statistics which might change my classifier’s results? If my understanding is wrong, please let me know <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Memory leaks when a model is transferred from gpu to cpu and save

Tan_Dat_Nguyen: <pre><code class="lang-python">def save_model:
        net = self.my_net.module
        dummy_input = torch.randn(1,3,112,112).cpu()
        net.eval()
        net.cpu() # switch to cpu to save the model to avoid some issues on Pytorch 1.1.0
        script_module = torch.jit.trace(net,dummy_input)
        script_module.save('my_model.pt')
        net.cuda() # switch to cuda to continue training on GPU
        net.train()
</code></pre>
<p>The memory usage keeps increasing after each call to save_model, in other words, memory is being leaked.</p>
<p>May anyone help to me to understand this issue? Thank you.</p>

albanD: <p>Memory on the cpu or gpu side?<br>
How do you measure this?<br>
Could you provide a small script to reproduce this?</p>
Does the generated grad-graph RETAIN inside a for-loop?

Deeply: <p>Does the generated graph retain inside a for-loop, even after the model output is overwritten?</p>
<p><strong>Example</strong>:</p>
<pre><code class="lang-auto">for epoch in range(100):
    for img in dataloader:
         optimizer.zero_grad(); loss=0
         for i in range(5):   
             out_img = M[i](img)   
             loss += criterion_L1(out_img, target[i])
         loss.backward()
         optimizer.step()

</code></pre>
<p>Above, I am using 5 models, each applied to the input image <code>img</code>. So, my simple understanding is although <code>out_img</code> is overwritten each time, backpropagation will apply to all the <code>M</code> models; that is the graph is retained.</p>
<p>Please correct me if I am wrong.</p>

albanD: <p>There is not a single grad graph in pytorch. Whatever you do as computation will be the graph.<br>
So here gradients will flow back to each model in M indeed.</p>
How do I reverse classes

Jordan_Howell: <p>When preparing my data for a convolutional neural network,I pull from two folders representing the classes.  The classes come out with reversed labels.  I use the command:</p>
<blockquote>
<p>class_names = full_data.classes</p>
</blockquote>
<p>to get the classes.  But the classes come out [0, 1] because the names of the folders when i would like it to be [1, 0].  Is there a way to reverse the labels?</p>

albanD: <p><code>class_names = - (class_names - 1)</code> will do that.</p>
How to create a network with multiple forward routines

tueboesen: <p>I have a bit of a special situation that I hope someone can help with.</p>
<p>I wish to create a neural network y=f(x), where f is the network, x is input and y is output. However the network should be made in such a way that it can effectively be split in two parts.<br>
lets say f(<em>) = g(h(</em>)). The idea is that h is the feature extracting part of the network, while g is the classifying part of the network.</p>
<p>I now wish to be able to call either g, h, or f on my data. However I can’t quite figure out how to do this, since each neural network seems to be able to only have 1 forward routine.</p>
<p>I guess one way to achieve this would be to write two separate neural networks, g and h. I take it the autograd wouldn’t have any problem with several neural networks like that. Would there be any performance hits in this approach or is there another more elegant approach?</p>

albanD: <p>Hi,</p>
<p>I would do the following:</p>
<pre><code class="lang-auto">class H:
  def __init__(self, *args):
    # init H

  def forward(self, inp):
    # forward of H
    return out

class G:
  def __init__(self, *args):
    # init G

  def forward(self, inp):
    # forward of G
    return out

class F:
  def __init__(self, *args):
    # init F
    self.g = G()
    self.h = H()

  def forward(self, inp):
    # forward of F
    return self.g(self.h(inp))

model = F()

# forward the whole model
model(inp)

# foward only g
model.g(inp)
# get only g's parameters
model.g.paremeters()

# foward only h
model.h(inp)

</code></pre>

tueboesen: <p>I will try to do something similar to this, unfortunately I don’t think it will work quite as elegantly, since I want to offload the model to cuda in nn.dataparallel, at which point I don’t believe I can access any of the subparameters or functions of the model.</p>
U-Net Implementation for 4D vectors

iamneerav: <p>I am using U-Net with Conv3D and ConvTranspose3D over vectors of size (3,100,100,100). My input size is (32,3,100,100,100) and target size is the same as well. I am having trouble comparing predictions and the output. I can’t understand how to reshape them properly to compare. The code I am using is as follows:</p>
<p><strong>Format</strong>             :   [N ,  C,  D  , W  , H ]<br>
<strong>Input_shape</strong>     :  [32, 3, 100, 100, 100]<br>
<strong>Label_shape</strong>   :  [32, 3, 100, 100, 100]<br>
<strong>Output_shape</strong> :  [32, 3, 100, 100, 100]<br>
<strong>Optimizer</strong>        :   Adam</p>
<p><strong>Loss and Train</strong>:</p>
<pre><code>def dice_loss(pred, target, smooth = 1.):
    pred            = pred.contiguous()
    target          = target.contiguous()    
    intersection = (pred * target).sum(dim=2).sum(dim=2)
    loss             = (1 - ((2. * intersection + smooth) / (pred.sum(dim=2).sum(dim=2) + 
    target.sum(dim=2).sum(dim=2) + smooth)))
    return loss.mean()


def train():
    unet.train(True)
    torch.set_grad_enabled(True)
    running_loss        = 0.0
    running_corrects = 0
    num_samples      = 0

    for data in train_dataloader:
        inputs, labels  = data
        inputs             = inputs.cuda(cuda3)
        labels             = labels.cuda(cuda3)
        optimizer.zero_grad()
        outputs = unet (inputs)
        _,preds = torch.max(outputs, 1)
        loss       = dice_loss(outputs, labels)
        loss.backward()
        optimizer.step()
        num_samples      += inputs.size(0)
        running_loss        += loss.item() * inputs.size(0)
        running_corrects  += torch.sum(preds == labels.long().data)
    epoch_loss  = running_loss / num_samples
    epochs_acc = running_corrects / num_samples
    return epoch_loss, epochs_acc</code></pre>