akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
A question about the output of LSTMCell

username: alan_ayu
cooked: <p>Maybe it is a stupid question but I still don’t understand, why the output of LSTMCell only consists of hx and cx? Should I add another feed-forward layer to compute o(t) based on hx(t-1) and x(t) ?<br>
Thank you.</p>

username: tom
cooked: <p>hx is what is the output. You can try that in nn.LSTM and compare the hx to the last output.</p>
<p>Besteht regards</p>
<p>Thomas</p>

username: alan_ayu
cooked: <p>But the computation of o(t), c(t) and h(t) are different,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3ed6332ff24bce842282b36159e831907968565f.png" alt="image" data-base62-sha1="8XSwvXUREcNwtav8Kxx0ZXUcc0L" width="293" height="71"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/ff607249a191cb3c00164c0bcea47e713e2e38f8.png" alt="image" data-base62-sha1="Araod5frl1tMLeh0yGvKDvGDs12" width="253" height="37"><br>
Is it ok to regard h(t) directly as o(t) ?</p>

username: tom
cooked: <p>Well, h already has o applied. Keep in mind that  is the output gate, not the output.</p>
How to save a modified model

username: bb417759235
cooked: <p>Is there any good way to save the network structure after pruning so that it can be loaded directly？</p>

username: ptrblck
cooked: <p>How did you prune the model? If you’ve removed certain nodes, the <code>state_dict</code> should contain all necessary parameters.</p>

username: bb417759235
cooked: <p>I extracted the parameters via model.state_dict, and removed the channel according to a certain threshold.Give an example below：<br>
conv1 = self.conv1(x)<br>
conv2 = self.conv2(conv1 )</p>
<p>conv1 param  (output_channel1,input_channel1,kernel_x,kernel_y)<br>
conv2 param  (output_channel2,output_channel1,kernel_x,kernel_y)</p>
<p>remove 20% param in output_channel1 and after removed<br>
conv1 param  (output_channel1 x 80%,input_channel1,kernel_x,kernel_y)<br>
conv2 param  (output_channel2,output_channel1 x 80%,kernel_x,kernel_y)</p>
<p>I purned the model channel , but if I want to use this purned parameter, I have to modify the model.py.Obviously this is very troublesome.I think that loading the purned parameters first, reading the corresponding number of channels, and passing the number of channels per layer when generating the network can solve this problem.But I feel that this kind of practice is also more troublesome. I would like to ask you what better way, let me save the structure and parameters directly after purning the parameters.</p>

username: bb417759235
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, do you know how to solve this problem</p>

username: ptrblck
cooked: <p>I’m not sure if there is a better way than to prune the model and save the <code>state_dict</code> with the new model definition.<br>
If you would like to keep the same model definition, you could work with masks to prune the parameters, but I guess that’s not really what you want.</p>

username: bb417759235
cooked: <p>Thank you  for your answer</p>
[SOLVED] What is the best database to store torch.tensor?

username: andreiliphd
cooked: <p>Hello!</p>
<p>I can use torch.save to save any variable but I want to store it in a database.<br>
What is the best solution to this problem?</p>
<p>Andrei</p>

username: andreiliphd
cooked: <p>I solved it. MongoDB can handle Binary data. Mysql can do BLOB.<br>
Just picke.dumps() or torch.save(buffer).<br>
This how it looks:</p>
<pre><code class="lang-auto">class TrainingData(mongoengine.Document):
    date_time = mongoengine.DateTimeField(required=True)
    epoch = mongoengine.IntField()
    train_loss = mongoengine.FloatField()
    test_loss = mongoengine.FloatField()
    train_accuracy = mongoengine.FloatField()
    test_accuracy = mongoengine.FloatField()
    gradients = mongoengine.BinaryField()
</code></pre>
Dataloader : shuffle and sampler

username: Jindong
cooked: <p>Hi, every one,</p>
<p>I am using the sampler for loading the data with train_sampler and test_sampler, but with this method, I have to set the shuffle as False, is there some other way that i can use the train_sampler or test_sampler for data loading in the mean time I can keep the shuffle as True?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You would have to implement shuffling into your sampler.<br>
Have a look at the implementation of <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/sampler.py#L40">RandomSampler</a>.</p>

username: f10w
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> If I use <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html#WeightedRandomSampler" rel="nofollow noopener">WeightedRandomSampler</a>, does it mean that my data will be shuffled? Sorry it’s still not clear to me.</p>

username: ptrblck
cooked: <p>The samples will be drawn from your dataset using the provided weights, so you won’t get a specific order. In that sense your data will be shuffled.</p>

username: f10w
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I get it now. Thank you!</p>

username: vishalthengane
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I am trying to apply sampler for two unbalanced datasets in single data loader(I merged two datasets). I have created an issue <a href="https://discuss.pytorch.org/t/how-to-apply-sampler-to-two-dataset-in-single-dataloader/49778">here</a>.</p>
<p>Thanks!</p>
Triplet Loss on MNIST

username: dickovsky
cooked: <p>Hi everyone, I’m trying to us AlexNet and triplet loss to deal with MNIST set. I think there’s something wrong with how I choose the positive and negative in the triplet. I try to find the furthest positive and closest negative. The average loss of the triplet sticks at 1, which is the margin of the triplet. I tried to adjust the learning rate from 0.01 to 0.000001. However, it doesn’t work.</p>
<pre><code class="lang-auto">model = AlexNet(1).to(device)
criterion = TripletLoss(margin=1)
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

for epoch in range(10):
    
    running_loss = 0
  
    for i, (data, label) in enumerate(train_loader):

        a_input = Variable(data).to(device).float()
        a_output = model(a_input)

        p_index, n_index = [], []

        for index, anchor in enumerate(a_output):

            max_dist, min_dist = 0, 100000
            max_index, min_index = 0, 0
            for index_, anchor_ in enumerate(a_output):
                if index == index_:
                    continue
                a0 = torch.Tensor.cpu(anchor).data.numpy()
                a1 = torch.Tensor.cpu(anchor_).data.numpy()
                dist = np.linalg.norm(a0-a1)
                if max_dist &lt; dist and label[index] == label[index_]:
                    max_dist, max_index = dist, index_
                if min_dist &gt; dist and label[index] != label[index_]:
                    min_dist, min_index = dist, index_
            p_index.append(max_index)
            n_index.append(min_index)

        p_output = torch.stack([a_output[i] for i in p_index])
        n_output = torch.stack([a_output[i] for i in n_index])

        optimizer.zero_grad()

        loss = criterion(a_output, p_output, n_output)
        loss.backward()

        optimizer.step()
        
        running_loss += loss.data
        
        if i % 100 == 99:
            print(f'epoch {epoch+1}, batch {i+1}, loss: {running_loss}')
            running_loss = 0
            
print('Finished')
</code></pre>
<p>The output loss is always a little bit above 100.</p>
<p>Anyone knows what’s happening?</p>
How does bath_first work in GRU/LSTM

username: Tejus
cooked: <p>So I created a GRU for a sentiment classification task.</p>
<pre><code class="lang-auto">class GRU(nn.Module):
    def __init__(self, vocab_size, emb_size,hid_size, out_size, n_layers,bi,drop,pad_tok):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size,emb_size,padding_idx=pad_tok)
        self.gru = nn.GRU(emb_size,hid_size,num_layers = n_layers , bidirectional = bi,dropout=drop,batch_first=True)
        self.out = nn.Linear(hid_size*2 ,out_size)
        self.dropout = nn.Dropout(drop)
    def forward(self, x,text_lengths):
        print("Input shape:",x.shape)
        if(0 in text_lengths):
            text_lengths[len(text_lengths)-1]=1        
        embedd = self.dropout(self.embedding(x))
        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedd, text_lengths)
        out,hidden =self.gru(packed_embedded)
        print("Hidden shape:",hidden.shape)        
        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))
        return self.out(hidden.squeeze(0))
</code></pre>
<p>So in both instances when I changed batch_first to True and False the model works and trains fine.But it shouldn’t as by setting it to true the model expects the first dimension to be the batch size which isn’t in my case (it’s sequence length).The funny thing is the model seems to perform a little bit better when I set it to True , which again I think it shouldn’t . Can someone explain what’s happening behind this?</p>

username: vdw
cooked: <p>PyTorch and other frameworks only complain if the dimensions don’t match expected values. Apart from that, anything goes and there’s always the chance that the network learns at least something. It just doesn’t mean that it’s something meaningful. Here just some food for thoughts and steps I would try:</p>
<ul>
<li>When you say the network trains fine, what does that mean. Only the training loss going down is not a sufficient metric. How about the test loss/accuracy.</li>
<li>
<code>pack_padded_sequence</code> also has <code>batch_first</code> as optional parameter. Did you try various combinations of <code>True/False</code> together with oyr definition of <code>GRU</code>. Intuitively, both should have the same value values for <code>batch_first</code>.</li>
<li>
<code>pack_padded_sequence</code> also has the optional parameter <code>enforce_sorted</code> which is by default <code>False</code>. In this case, the method expects the input sorted w.r.t. the the lengths of the sequences. I cannot see in your code where you sort your input.</li>
</ul>
Where there's no .train() method in this Pytorch official tutorial?

username: John_Deterious
cooked: <p>So, I just learned Pytorch and they say that you must put the NN in train mode by .train() method, then while inferencing .eval() mode. I’m reading this tutorial and there’s no .train() at all. Why is that?</p>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py</a></p>

username: ptrblck
cooked: <p>After initializing a model (or in fact an <code>nn.Module</code>), it will be in training mode by default, so you don’t need to call it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Softmax regression from scratch: NaN loss

username: lorenzo_fabbri
cooked: <p>I’m trying to implement Softmax regression from scratch, but I have a few problems.</p>
<p>The notebook can be visualized at the following <a href="https://nbviewer.jupyter.org/urls/gitlab.com/lorenzo.fabbri92sm/dive-into-deep-learning/raw/master/notebooks/softmax_regression_scratch.ipynb" rel="nofollow noopener">link</a>, or downloaded directly <a href="https://gitlab.com/lorenzo.fabbri92sm/dive-into-deep-learning/blob/master/notebooks/softmax_regression_scratch.ipynb" rel="nofollow noopener">here</a>.</p>
<p>I checked the individual functions and compared the results with the ones PyTorch provides, and they seem correct (i.e., they provide the same values). The problem is that when I train the model, after a few batches the loss becomes NaN. If I replace my implementations of cross-entropy and Softmax, then it seems to be working. So perhaps they are not so correct. It seems that the matrix of weights, <code>W</code>, is NaN already after the first batch.</p>

username: KFrank
cooked: <p>Hi Lorenzo!</p>
<aside class="quote no-group quote-modified" data-username="lorenzo_fabbri" data-post="1" data-topic="49861" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lorenzo_fabbri/48/7696_2.png" class="avatar"> lorenzo_fabbri:</div>
<blockquote>
<p>I’m trying to implement Softmax regression from scratch, but I have a few problems.<br>
…<br>
I checked the individual functions and compared the results with the ones PyTorch provides, and they seem correct (i.e., they provide the same values). The problem is that when I train the model, after a few batches the loss becomes NaN.<br>
…</p>
</blockquote>
</aside>
<p>I didn’t look at your code, but if you wrote your softmax and<br>
cross-entropy functions as two separate functions you are<br>
probably tripping over the following problem.</p>
<p>Softmax contains <code>exp()</code> and cross-entropy contains <code>log()</code>,<br>
so this can happen:</p>
<p>large number → <code>exp()</code> → overflow NaN → <code>log()</code> → still NaN</p>
<p>even though, mathematically (i.e., without overflow),<br>
<code>log (exp (large number)) = large number</code> (no NaN).</p>
<p>Pytorch’s <code>CrossEntropyLoss</code> (for example) uses standard<br>
techniques to combine together, in effect, the <code>log()</code> and the<br>
<code>exp()</code> to avoid the overflow.</p>
<p>If need to write your own loss function, you will need to use<br>
these same techniques to avoid the NaNs.  If you’re just doing<br>
this for learning (Good for you!), you have learned that this is<br>
one good reason to use pytorch’s built-in <code>CrossEntropyLoss</code><br>
function.</p>
<p>Good luck!</p>
<p>K. Frank</p>
Remove part of loss function leads to out of memory Error

username: friskit
cooked: <p>Hi</p>
<p>The loss function of my model has more than one sub-loss which are combined with several factors.<br>
e.g.</p>
<p>loss = factor_1 * loss_1 + factor_2 * loss_2 + factor_3 * loss_3</p>
<p>sometimes for debugging my model, I need to use only one sub-loss like:<br>
loss = factor_3 * loss_3</p>
<p>but this will leads to an OOM error.</p>
<p>I have to use: loss = 0 * loss_1 + 0 * loss_2 + factor_3 * loss_3</p>
<p>is there any more elegant method to deal with it?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Are you storing the losses somewhere?<br>
It’s strange, that using <code>0 * tensor</code> works, while omitting it throws an OOM error.<br>
Could you post a reproducible code snippet so that we could have a look?</p>
GPU vs CPU multiple inferences

username: mezon
cooked: <p>Hello,</p>
<p>I have a question regarding running inference on a model loaded in memory<br>
I initialize the model, run <code>.benchmark</code> (if GPU) and <code>.eval()</code> then do inference 2 times.<br>
Time: GPU_1 = 5sec<br>
Time: GPU_2 = 0.05sec</p>
<p>Time: CPU_1 = 4sec<br>
Time: CPU_2 = 4sec</p>
<p>Now in tensorflow using <code>tf.session</code> the CPU_2 is decreased by a large margin as well.<br>
Is there’s anything I can do to “cache” an inference model in the CPU like GPU does?</p>
<p>Thanks in advance,<br>
John</p>

username: ptrblck
cooked: <p>If you are timing GPU operations, note that CUDA calls are asynchronous, so that you have to add synchronization points before starting and stopping the timer:</p>
<pre><code class="lang-python">torch.cuda.synchronize()
t0 = time.time()
# your operations
torch.cuda.synchronize()
t1 = time.time()
</code></pre>
<p>Most likely you are measuring the CUDA initialization time etc. in <code>GPU_1</code>.</p>
How could I use torch.gels() with batch matrix?

username: xylon
cooked: <p>How could I use <a href="https://pytorch.org/docs/stable/torch.html#torch.gels" rel="nofollow noopener">torch.gels()</a> like <a href="https://pytorch.org/docs/stable/torch.html#torch.solve" rel="nofollow noopener">torch.solve()</a> with A=(batch, m, n) and B=(batch, m, k) ?</p>

username: tom
cooked: <p>I’m afraid you cannot, currently. <a href="https://github.com/pytorch/pytorch/issues/7500" rel="nofollow noopener">Issue #7500</a> has status on the batch operations in master.</p>
<p>Best regards</p>
<p>Thomas</p>

username: xylon
cooked: <p>I got it.</p>
<p>Thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Module 'torch' has no attribute 'cholesky_inverse'

username: Raphikowski
cooked: <p>Dear all,</p>
<p>I try to use the module cholesky_inverse but get the following error message:<br>
AttributeError: module ‘torch’ has no attribute ‘cholesky_inverse’<br>
In the documentation of PyTorch the function is however listed so it should be available.<br>
Has anyone experienced a similar error already or are there any suggestions for possible solutions?</p>
<p>I use version 1.0.1.post2</p>
<p>Best regards,<br>
Raph</p>

username: Raphikowski
cooked: <p>Can anyone help me with this issue?</p>

username: ptrblck
cooked: <p>This method was introduced in PyTorch <code>1.1.0</code>.<br>
Have a look <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">here</a> to find the install instruction.</p>
ValueError: Expected input batch_size (4) to match target batch_size (64)

username: shahaf
cooked: <p>Currently I’m having this probelm (using simple neural network) , please let me know what am I missing.</p>
<blockquote>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as f<br>
from torch.autograd import Variable<br>
import numpy as np<br>
import torchvision.datasets as dset<br>
import torchvision.transforms as transforms<br>
import torchvision<br>
import livelossplot<br>
import matplotlib.pyplot as plt<br>
from fastai.vision import *</p>
<p>class mymodel(nn.Module):<br>
“”" Architecture of the Generator, uses res-blocks “”"</p>
<pre><code>def __init__(self,pretrained=False):
    super(mymodel, self).__init__()
    self.body=nn.Sequential(
        nn.Conv2d(3, 4, kernel_size=5, stride=4*252),
        nn.ReLU(),
    )
    
    self.fc=nn.Linear(256,4)


def forward(self, x):
    x=self.body(x)
    x=x.view(-1,1)
    x.squeeze_(1)
    x=self.fc(x)
    x.unsqueeze_(-1)
    print(x.shape)
    
    return x
</code></pre>
<p>np.random.seed(42)<br>
data = ImageDataBunch.from_folder(path, train=“.”, valid_pct=0.2,<br>
ds_tfms=get_transforms(), size=256, num_workers=4).normalize(imagenet_stats)</p>
<p>learn = Learner(data, mymodel(),metrics=accuracy)<br>
learn.fit_one_cycle(1)</p>
</blockquote>
Help with the implement own autograd.Function

username: ElleryL
cooked: <p>I have a issue with implement my own <code>autograd.Function</code></p>
<p>Consider the following example:</p>
<pre><code class="lang-auto">class Exp(torch.autograd.Function):

    @staticmethod
    def forward(ctx, i):
        result = i.exp()
        ctx.save_for_backward(result)
        print(result) # print1 here for debug
        return result

    @staticmethod
    def backward(ctx, grad_output):
        result, = ctx.saved_tensors
        return grad_output * result
</code></pre>
<p>Now; I have found the following behaviours</p>
<pre><code class="lang-auto">from torch.autograd import Variable
x = Variable(torch.Tensor([3.]),requires_grad=True)
e.apply(x)

tensor([20.0855]) # here comes from the print1
Out[18]:
tensor([20.0855], grad_fn=&lt;ExpBackward&gt;) # here is the return value
</code></pre>
<p>What I have found is that in <code>forward()</code> we seems loosing the track of gradient information. The print shows that <code>result</code> is not <code>grad_fn</code>. However, the actually returned value is.</p>
<p>Can anyone help me to reasoning about it ?</p>
<p>In my true implementation, due to the above issue, I seems to loss some variable (that is during the <code>backward()</code>, some variable has zero gradient and hence no update )</p>
RuntimeError: reduce failed to synchronize: an illegal memory access was encountered

username: daodao
cooked: <p>When I run the pix2pix GAN which implemented by <a href="https://github.com/eriklindernoren/PyTorch-GAN" rel="nofollow noopener">eriklindernoren </a> in the Pytorch version 0.4.1, I got the RuntimeError as:</p>
<pre><code class="lang-auto">Exception ignored in: &lt;bound method _DataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._DataLoaderIter object at 0x7f0f8f5490b8&gt;&gt;
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 399, in __del__
    self._shutdown_workers()
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 378, in _shutdown_workers
    self.worker_result_queue.get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 345, in get
    return ForkingPickler.loads(res)
  File "/usr/local/lib/python3.5/dist-packages/torch/multiprocessing/reductions.py", line 151, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/lib/python3.5/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/lib/python3.5/multiprocessing/reduction.py", line 181, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/lib/python3.5/multiprocessing/reduction.py", line 152, in recvfds
    msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "pix2pix.py", line 141, in &lt;module&gt;
    loss_GAN = criterion_GAN(pred_fake, valid)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py", line 421, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py", line 1716, in mse_loss
    return _pointwise_loss(lambda a, b: (a - b) ** 2, torch._C._nn.mse_loss, input, target, reduction)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py", line 1674, in _pointwise_loss
    return lambd_optimized(input, target, reduction)
RuntimeError: reduce failed to synchronize: an illegal memory access was encountered
</code></pre>
<p>Why encounter the error? Anyone can help me?</p>

username: richard
cooked: <p>Try running your script with CUDA_LAUNCH_BLOCKING=1. That should give a more accurate description of the error</p>

username: NightRainXiaoxiang
cooked: <p>Have your solved this problem? I met the same issue.</p>

username: Hui_Tang
cooked: <p>i met the same problem, I found that it is because my tensors are not on the same gpu and after i set them to be same gpu id, the error is gone</p>

username: ecdrid
cooked: <p>Try something like this</p>
<p><code>CUDA_ViSIBLE_DEVICES=1 python *.py</code></p>
Copy a chunk of one tensor into another one

username: afshin67
cooked: <p>I need to copy a row of one tensor into some part of another tensor, form which the begin and end indexes are available. Withing C++ we can use something like</p>
<pre><code class="lang-auto">  int myints[]={10,20,30,40,50,60,70};
  std::vector&lt;int&gt; myvector (18);

  std::copy ( myints, myints+3, myvector.begin()+4 );
</code></pre>
<p>I can copy three values from <code>myints</code> into <code>myvector</code>, starting at the fourth index.<br>
I was wondering if there is a similar API in <code>torch</code>?</p>

username: LeviViana
cooked: <p>For CPU Tensors, I think that using the <code>data&lt;scalar_t&gt;()</code>(for contiguous tensors, otherwise you should use the <code>accessor&lt;scalar_t, dim&gt;</code> API) method along with <code>std::memcpy</code> can give near to optimal results.</p>

username: afshin67
cooked: <p>I do not want to move the data out of torch tensors. Basically, I am looking for torch APIs to perform this operation directly using the tensors.</p>

username: LeviViana
cooked: <p>I’m not sure what you mean by ‘move data out of tensors’. You won’t be moving that data out of torch tensors, the data and accessor APIs just allow you to access the data from tensors.</p>

username: tom
cooked: <p><code>my_target.select(0, 1).copy_(my_source.slice(1, 0, 10)</code> would be the equivalent of<br>
<code>my_target[1] = my_source[:, :10]</code> in Python (works if the dimensions these sub-tensors are compatible).<br>
Typical methods for selecting are <code>select</code> and <code>slice</code>, possibly <code>narrow</code>. You can chain those.</p>
<p>Best regards</p>
<p>Thomas</p>
Reshaping windows into image

username: ks7g2h3
cooked: <p>I’m building a model that does the following: (i) breaks an image up into small square windows, (ii) applies a network to each window, transforming each into an output window of the same size, and (iii) reassembles the output windows into an output image of the same size as the input image. I’m wondering if there’s an efficient way to do the last step without loops.</p>
<p>The stack of windows output by the network has size (batch * num_windows x num_channels x window_side x window_side) and the output image should have size (batch x num_channels x image_side x image_side).</p>
<p>The naive approach of using just a singe .view() doesn’t work here, but I imagine that there is a combination of transposes and views that will do the job. Any tips?</p>

username: ptrblck
cooked: <p>Do the small windows overlap? If so, you would have to think about a method to unite them (summing, mean, etc.).<br>
If not, you could try the following:</p>
<pre><code class="lang-python">channels = 1
h, w = 12, 12
image = torch.randn(1, channels, h, w)

kh, kw = 3, 3 # kernel size
dh, dw = 3, 3 # stride

patches = image.unfold(2, kh, dh).unfold(3, kw, dw)
patches = patches.contiguous().view(-1, channels, kh, kw)

conv = nn.Conv2d(channels, channels, 1, bias=False)
conv.weight.data.fill_(1.)
output = conv(patches)

tmp = output.view(1, 16, kh*kw).permute(0, 2, 1)
im = F.fold(tmp, (h, w), (kh, kw), 1, 0, (dh, dw))
</code></pre>
<p>Note that you would have to build PyTorch from source, since <code>F.fold</code> is not yet available in the binary release.<br>
Let me know, if this works for you.</p>

username: ks7g2h3
cooked: <p>Thank you for the pointers! This code does the trick!</p>

username: isalirezag
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="19805">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>image = torch.randn(1, channels, h, w)</p>
</blockquote>
</aside>
<p>if it possible to do the same thing for a tensor with shape of:</p>
<pre><code class="lang-auto">image = torch.randn(1, channels,D, h, w)
</code></pre>
<p>where D is depth</p>

username: ptrblck
cooked: <p>Yes, that should also be possible.<br>
If you would like to create the “patches” in D, h, and w, you would need to <code>unfold</code> in the additional D dimension.</p>
My First C++ Pytorch Program | Link Error

username: ashuezy
cooked: <p>I am using codeblocks.</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.

#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
    return -1;
  }

  // Deserialize the ScriptModule from a file using torch::jit::load().
  std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load(argv[1]);

  assert(module != nullptr);
  std::cout &lt;&lt; "ok\n";
}
</code></pre>
<p>I have add libtorch.so in linker path.</p>
<pre><code class="lang-auto">Getting the following output
main.cpp:(.text.startup+0x50): undefined reference to `torch::jit::load(std::string const&amp;, c10::optional&lt;c10::Device&gt;, std::unordered_map&lt;std::string, std::string, std::hash&lt;std::string&gt;, std::equal_to&lt;std::string&gt;, std::allocator&lt;std::pair&lt;std::string const, std::string&gt; &gt; &gt;&amp;)'
collect2: error: ld returned 1 exit status
</code></pre>

username: tom
cooked: <p>I think <code>torch/torch.h</code> is the canonical include for libtorch.<br>
You didn’t show your command line, do you have <code>-ltorch</code> in it? You might also need caffe2 and/or c10 depending on the version you are on.</p>
<p>Best regards</p>
<p>Thomas</p>

username: heethesh
cooked: <p>Try setting this flag <code>"-D_GLIBCXX_USE_CXX11_ABI=0"</code> with CMake.</p>
<p>If you are passing a relative path to the libtorch in CMake, make sure you have the <code>$PWD</code> in the path:</p>
<pre><code class="lang-auto">cmake -DCMAKE_PREFIX_PATH=$PWD/../relative/path/to/libtorch ..
</code></pre>
Different RNN results for the same sequences

username: erickrf
cooked: <p>I found that under some circumstances, an RNN gives slightly different results for instances of the same sequence in a batch of many different sequences.</p>
<p>It is difficult to isolate the problem, and changing small details like the random seed or the constants in the input may make it disappear. With this rather cabalistic set of inputs, though, I get unexpectedly different outputs for the sequence <code>[5]</code></p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

indices = torch.tensor([[64, 35, 35],
        [13, 24,  6],
        [37,  5,  7],
        [11, 44, 10],
        [ 5, 11, 23],
        [19,  4, 36],
        [45, 33, 42],
        [ 5, 11, 23],
        [15, 10,  5],
        [ 7,  4,  8],
        [19,  9,  8],
        [46, 10,  0],
        [12,  4,  0],
        [15, 10,  0],
        [ 6,  7,  0],
        [12,  4,  0],
        [ 6,  7,  0],
        [ 4, 10,  0],
        [ 7,  4,  0],
        [12,  4,  0],
        [25, 26,  0],
        [ 6,  7,  0],
        [12,  4,  0],
        [ 4, 10,  0],
        [ 3,  0,  0],
        [ 3,  0,  0],
        [28,  0,  0],
        [ 5,  0,  0],
        [23,  0,  0],
        [ 3,  0,  0],
        [62,  0,  0],
        [16,  0,  0],
        [16,  0,  0],
        [16,  0,  0],
        [ 3,  0,  0],
        [28,  0,  0],
        [ 5,  0,  0],
        [ 4,  0,  0],
        [16,  0,  0],
        [ 6,  0,  0],
        [16,  0,  0],
        [ 3,  0,  0],
        [28,  0,  0],
        [ 5,  0,  0],
        [16,  0,  0],
        [ 6,  0,  0],
        [16,  0,  0],
        [16,  0,  0],
        [ 3,  0,  0],
        [ 5,  0,  0]])
lengths = torch.sum(indices != 0, 1)

torch.set_printoptions(precision=10)
torch.manual_seed(42)

embeddings = nn.Embedding(66, 5, padding_idx=0)
lstm = nn.LSTM(5, 50, batch_first=True)

embedded = embeddings(indices)
packed = pack_padded_sequence(embedded, lengths, batch_first=True)
hidden, _ = lstm(packed)
padded, _ = pad_packed_sequence(hidden, batch_first=True)

only_5 = (indices[:, 0] == 5) &amp; (lengths == 1)

print('Output for sequences consisting only of [5]')
print(padded[only_5, 0, :3])
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Output for sequences consisting only of [5]
tensor([[-0.0135216247,  0.0015346913,  0.0631567389],
        [-0.0135216247,  0.0015346913,  0.0631567389],
        [-0.0135216247,  0.0015346913,  0.0631567389],
        [-0.0135216229,  0.0015346938,  0.0631567389]],
       grad_fn=&lt;IndexBackward&gt;)
</code></pre>
<p>Is there anyway to fix that?</p>
CUDA Error after calling torch.cuda.is_available()

username: petrell9
cooked: <p>Hi all,</p>
<p>I experiencing a strange error when trying to generalize my training loop for different devices. I have something to see if cuda is available and to do the device setup, however after I call <code>torch.cuda.is_available()</code> moving tensors to the GPU generates a cuda error. I did some testing and found this doesn’t happen if I try to make a cuda tensor before calling <code>torch.cuda.is_available()</code>. Here’s an example of what I mean:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.randn(1,1).to("cuda")
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\Users\Mike\Anaconda3\lib\site-packages\torch\cuda\__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA error: unknown error
</code></pre>
<p>Restarting the python shell</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.randn(1,1).to("cuda")
tensor([[-0.2594]], device='cuda:0')
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.randn(1,1).to("cuda")
tensor([[0.4193]], device='cuda:0')
</code></pre>
<p>I restarted my computer and updated the Nvidia drivers but still encounter this error. I’m using:<br>
Win10<br>
Python 3.7.3<br>
PyTorch 1.1.0<br>
Cuda 10.0<br>
GeForce driver 430.86</p>
<p>Would anyone have an idea of what could be the cause of this issue? Please let me know if I left any relevant details out and thanks for your time. Cheers</p>

username: alex.veuthey
cooked: <p>Do you have any errors when you do <code>nvidia-smi</code> and <code>nvcc -V</code> ? Did you install PyTorch with the correct version of CUDA ?</p>

username: petrell9
cooked: <p>Thanks for the response, it suddenly started working as intended. It’s strange because I didn’t even restart between then and now or install anything new. Unfortunate if someone else has a similar issue but  them’s the breaks I guess. Cheers</p>
How to create an index tensor given a tensor shape

username: dean
cooked: <p>I want to create a function that given a tensor shape, returns its equivalent index tensor. Basically, I want to replicate a 2-D index mask into the same format of a given tensor.shape.</p>
<p>tensor = torch.randn(1,2,3,3)<br>
index_mask = [[0,1,2], [3,4,5], [6,7,8]]<br>
foo(index_mask, tensor) -&gt; 4D_index_max</p>

username: LeviViana
cooked: <p>Sorry but I’m not sure I understood the expected output. Could you give an example ?</p>

username: dean
cooked: <p>I solved it using torch.arange and expand!<br>
Sorry for my poor explanation of the problem, and thanks for the interest <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
C++ convert signed float to tensor?

username: jinfagang
cooked: <p>I have only one last question here:</p>
<p>I have a cv::Mat which is CV_32S format contains signed float with negative values. (there is no other way to store negative values in opencv)</p>
<p>When I convert this Mat to tensor:</p>
<pre><code class="lang-auto">at::Tensor tensor_image = torch::from_blob(in_img.data, {1, in_img.rows, in_img.cols, 3}, torch::kByte);
  tensor_image = tensor_image.permute({0, 3, 1, 2}).to(torch::kFloat32).to(device_type);
  cout &lt;&lt; "tensor_image: " &lt;&lt; tensor_image &lt;&lt; endl;
</code></pre>
<p>The result is totally wrong!! All negative values are gone!! This is totally different in python input model tensor values…</p>
<p>Anybody knows how to convert a cv::Mat with CV_32S type into a tensor and does not effect values store in it???</p>

username: tom
cooked: <p>I think “signed float” is the source of your trouble, OpenCV’s types are <strong>U</strong>nsigned ints, <strong>S</strong>igned ints and <strong>F</strong>loats of various precision.<br>
So using <code>torch::kLong</code> for the <code>from_blob</code> would seem to be the natural way. You’d run into trouble for <code>CV_8S</code> because PyTorch doesn’t have them, but here, just avoid the <code>kByte</code> that corresponds to <code>CV_8U</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: jinfagang
cooked: <p>thanks!! it worked!!!</p>
Https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb

username: vivek_ananthan
cooked: <p><a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb</a></p>
<p>Hi I am trying to understand the NN with pytorch.<br>
I have doubts in gradient calculations…</p>
<p>import torch.optim as optim</p>
<h1>create your optimizer</h1>
<p>optimizer = optim.SGD(net.parameters(), lr=0.01)</p>
<pre><code class="lang-auto"># in your training loop:
optimizer.zero_grad()   # zero the gradient buffers
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()    # Does the update
</code></pre>
<p>From the about code, I understood loss.backward() calculates the gradients.<br>
I am not sure, how these info shared with <code>optimizer</code> to update the gradient.</p>
<p>Can anyone explain this…</p>
<p>Thanks in advance !</p>

username: ptrblck
cooked: <p>When you create an instance of the optimizer, you pass the parameters to it:</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
</code></pre>
<p>The optimizer holds references to these parameters and updates them when calling <code>optimizer.step()</code> using the <code>.grad</code> attribute.</p>
Calculating loss per sample vs calculating loss per dataset

username: etekiller
cooked: <p>I have a dataset of around 4k images. I am iterating over that dataset approximately 1000 times (epochs). I am wondering, what is more efficient and leads to better generalization - calculating loss and doing 1 backpropagation per image or averaging out all losses from one epoch and then backpropagating?</p>

username: el_samou_samou
cooked: <p>Hi,</p>
<p>averaging backprop over the whole dataset and then update corresponds to gradient descent while updating after backpropapgation over an image (or more generaly over a batch of images) corresponds to stochastic gradient descent (SGD). I believe so far SGD is considered to generalize better. I advice you to look for the latest results on that topic.</p>
<p>An additional thing if you choose to use SGD is to increase the size of the batch (it obviously depends on your input size but usually batch are larger especially if you have normalization layers aka batch norm). This is a common practice.</p>
<p>Hope this help!</p>

username: etekiller
cooked: <p>Thank you very much for your answer! It helped a lot :).</p>
How do I multiply every row of a tensor with every row of the other tensor?

username: Zheng_Chen
cooked: <p>Suppose I have a tensor X of dimension <code>a x a</code>, and another tensor Y of the same dimension. How do I multiply every row of X with every row of Y and get an $a x a$ tensor? Thanks!</p>

username: LeviViana
cooked: <p>What do you want is just a matrix multiplication with the transpose.</p>
<pre><code class="lang-python">import torch
x = torch.Tensor([[1, 2], [3, 4]])
y = torch.Tensor([[1, 2], [3, 4]]) + 1
x.matmul(y.transpose(0, 1))
</code></pre>
About torchvision.models

username: Cverlpeng
cooked: <p>hi,<br>
I use pytorch to create model ,this model has dilation &gt;2 layer ?<br>
thanks!!</p>
<pre><code class="lang-python">resnet101 = torchvision.models.resnet101(pretrained=True)
</code></pre>

username: ptrblck
cooked: <p>All conv layers should have a dilation of <code>(1, 1)</code>:</p>
<pre><code class="lang-python">resnet101 = models.resnet101(pretrained=False)

for name, module in resnet101.named_modules():
    if isinstance(module, nn.Conv2d):
        print(name, module.dilation)
</code></pre>

username: Cverlpeng
cooked: <p>thanks!<br>
If I want to set a dilation convolution of more than 2 in the last 2 blocks, and I want them to have a step size of 1, how can I change it?</p>

username: ptrblck
cooked: <p>You could change the dilation by simply assigning a new value to the module:</p>
<pre><code class="lang-python">resnet101.layer4[2].conv1.dilation = (2, 2)
</code></pre>
<p>However, since the outputs might be used on a residual connection etc. you would have to make sure to provide the desired spatial size of the new output.</p>
Finetuning a model on multiple GPU in C++

username: jcl
cooked: <p>In brief, my question is:<br>
How to resize a given (registered) module and reinitialize it with random values ?</p>
<p>My process:<br>
I have a TORCH_MODULE(Net)  c++ class with a NetImpl deriving from torch::nn::Cloneable and a net_imagenet.pt archive (1000 classes)<br>
I want to finetune this model on a different number of classes N.</p>
<pre><code class="lang-auto">torch::nn::Net net(N);   // create a Net with N classes last_linear.sizes()={nbFeatures, N}, last_linear.bias.sizes() = {N}
torch::load(net, "net_imagenet.pt");   // load pretrained parameters (1000 classes), last_linear.sizes()={nbFeatures, N}, last_linear.bias.sizes() = {1000}
</code></pre>
<p>At this stage, the last linear module is of size {nbFeatures, N}, but its bias and weights are of size 1000.<br>
As the last_linear module is registered in Net::reset() (to be cloneable), I can’t find a way to replace it with a new nn::Linear as you do in Python.<br>
I’ve tried various hack to reset the last_linear, but found none which works on multiple gpu:</p>
<ul>
<li>changing last_linear as in python does no good, the original version is still used</li>
<li>renaming last_linear in register_module (ie “last_linear” to “last_linear_ft”) layer before loading the weights fails at torch::load() with a c10 error No such serialized submodule: ‘last_linear_ft’</li>
<li>registering the new version of last_linear as “last_linear_ft” after loading the weigths is ugly (memory consumption) but works in single gpu mode. Although clone() fails : the number of module changed outside of the constructor…</li>
</ul>
<p>I definitely need a reregister_module() or something like that…</p>
<p>Does anyone has successfully finetuned a model with the C++ API ?</p>

username: tom
cooked: <p>Indeed, <code>replace_module</code> seems to be missing, I’ll see if I can put up a PR and link it here.</p>
<p>Best regards</p>
<p>Thomas</p>

username: jcl
cooked: <p>Thanks for you answer !</p>

username: tom
cooked: <p>So I filed <a href="https://github.com/pytorch/pytorch/pull/22546" rel="nofollow noopener">#22546</a>, we’ll see if it is generally liked.</p>
<p>Best regards</p>
<p>Thomas</p>
Using data.parallel with a CNN

username: harizas
cooked: <p>Hi<br>
Im very new to deeplearning and am  trying to understand how to use data parallel with my semantic segmentation training work.<br>
My data loader gives me a batch of 16 files, of both the input image and the ground truth image. I understand I need to transfer the files into the GPU and then train the model.  The following code looks right to me, and when i run it I can see 3 GPUS of 16GB being used. soon after though it bombs, saying out of memory. Is the code below right?<br>
One thing Im confused about is whether when training the model, the entire batch can be sent in at one time, or if we need to send each image one by one.We used to send each image one by one using a custom dataloader and that worked but was very slow. Hence this attempt of speeding training using data Parallel.</p>
<p>These are 3 16GB GPUs.</p>
<pre><code> for epoch in range(0, num_epochs):
        optimizer = get_optimizer(trainable_model, epoch) # optimizer for current epoch
        total_train_loss = 0

        for i_batch, sample_batched in tqdm(enumerate(training_generator)):
            optimizer.zero_grad()
            rgb,mask = sample_batched
            var_rgb = Variable(rgb.float())
            print (var_rgb.shape) &lt; -- this is (16,3,363,400)
            var_rgb = var_rgb.cuda()

            var_mask = Variable(mask.float())
            var_mask = var_mask.cuda()

            output = trainable_model(var_rgb)
            loss = ((criterion(output, var_mask.long()) / (settings.opt['batch_size'])) 
            total_train_loss += loss.data[0]
            loss.backward()

        del var_mask,var_rgb
        optimizer.step()</code></pre>

username: InnovArul
cooked: <p>Can you reduce batch size and check? Basically,  the resolution of images could be large,  such that each gpu is unable to handle ~5 (16/3) images at once.</p>

username: harizas
cooked: <p>Hi Innovarul<br>
think your right, setting a batch size of 6 and am able to run the images through without running out of memory.</p>
Batch size in actual testing

username: nickhir
cooked: <p>Hello,</p>
<p>I created a CNN, trained it on my dataset and saved the model with the best validation accuracy.<br>
Now I want to actually use it, i.e. I want it to classify images for me.<br>
I am wondering which batch size I should use, because I read somewhere that I should use a batch size of 1. I am using 1 GPU for the training</p>
<p>Any help is appreciated!</p>

username: el_samou_samou
cooked: <p>Hi,</p>
<p>Now that you want to use your model (training is over), you should use it in eval mode (call <code>model.eval()</code> before processing your new images). In that case the batch size has no impact so you should set it as large as possible (depending on the memory available on your GPU) to make the process as fast as possible.</p>
<p>Hope this help.</p>

username: nickhir
cooked: <p>That helped alot, because I totally forgot about my batchnorm.</p>
<p>Thank you very much!!</p>
Convert model to Onnx format

username: cemicel
cooked: <p>Hello there,</p>
<p>I’m trying to convert a CRNN model which consists of both Conv and LSTM layers. Everything works fine. However, when I’m using onnx.checker.check_model(model) I’m getting a warning:</p>
<p><em>Warning: ConstantFill was a removed  experimental ops. In the future, we may directly reject this operator. Please update your model as soon as possible.</em><br>
PyTorch saves the model in ai.onnx v9 version. When I try to convert it to version of 10, I’m getting:</p>
<p>Assertion false failed: No Adapter For Current Version $9 for MaxPool</p>
<p>Everything would be okay, but when I try to run the onnx model on ML dotnet, I’m getting RunTimeExeption due to this ConstantFill.</p>
<p>Question <span class="hashtag">#one:</span><br>
What is ConstantFill?</p>
<p>Question <span class="hashtag">#two:</span><br>
Is there any solution?</p>
<p>I’m using:<br>
PyTorch 1.0.1.post2<br>
Onnx 1.5</p>
<p>PS: I know that it is possible to use LSTM layers in Onnx.</p>

username: cemicel
cooked: <p>PyTorch 1.1.0 post2 solves the issue.</p>
Torchvision error while saving images in image segmentation

username: K_awanish
cooked: <p>I am working on image segmentation, on Cityscape dataset, and whenever i uses more than 7 classes torchvision doesn’t save correct coloured image. however when I tried to save it using numpy and PIL it was saved as expected. can someone explain what am I missing here.<br>
Thanks in advance.</p>

username: ptrblck
cooked: <p>Could you post the code you’ve used to save the output using <code>torchvision</code> and PIL?</p>

username: K_awanish
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/46e1fdb021e18f13b57f5000bd9f733482f79a3f.png" alt="Screenshot%20from%202019-07-05%2012-46-50" data-base62-sha1="a73BANXVx27TVOD7MpvIYEvwxAj" width="605" height="338"> <img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9647a6ff2e1b43acaee7ec3c6ca2fd823f84af09.png" alt="Screenshot%20from%202019-07-05%2012-47-01" data-base62-sha1="lrr8mRhBO8KL1MNE2p0rty47c9z" width="538" height="145"></p>

username: K_awanish
cooked: <p>and i am using the standard method provided by torchvision to save images here :<br>
torchvision.utils.save_image(target_rgb, filename_target)<br>
torchvision.utils.save_image(y_threshed, filename_output)<br>
torchvision.utils.save_image(image, filename_input)</p>
<p>and using PIL :<br>
def saving_func(batch_tensor, path):<br>
for i in range(batch_tensor.shape[0]):	<br>
b = np.asarray(batch_tensor.cpu().numpy())<br>
a = b[i,:,:,:]<br>
a =np.swapaxes(a,0,2)<br>
a =np.swapaxes(a,1,0)<br>
a = Image.fromarray(a.astype(‘uint8’))<br>
a.save(path + str("_") + str(i) + str(".jpg"))</p>